Cheat Engine

[Aviar³]

If I remember correctly, you stated before you are a systems administrator. What's that like, and what would you consider necessary knowledge? So far I have down Perl, Bash, PowerShell, C, operating system design, deployment and administration of servers (LDAP, patching on the fly, maintaining drivers, etc). I was just curious what a working gentlemen sees day to day, as theory tends to be worlds apart from the actual practice.

[AhMunRa]

Sounds like you are all set for Linux administration. If you can navigate and do bsh csh and sh fu then you can handle anything Windows Server and the like have to offer. Day to day is mostly user support, with the odd excel spreadsheet parse or converting old Lotus 1-2-3 formulas to work in excel. My documents won't print, etc. Driver maintenance is pretty much a never. You never know what legacy app will break due to a driver update. Windows patches yes, constantly. Troubleshooting why a company wide software won't work on a new computer system, read software conflict with the bloatware on it. Configuring computers for deployment. Probably 90% of what I do, any home user who has broken their install of Windows, or built their own rigs could do. It's the odd problems I live for. A website that's DNS won't resolve in bind yet every other site resolves just fine. Policing email, reading peoples email. Sending them memo's on why it's not ok to have video's sent to their inboxes. Explaining to them why Facebook and the like are not allowed through the network. I'd say the main thing to focus on is the what I call the bullshit factor. When remoting into a computer to fix it while I have the user on the phone, if it's something they did or something they can handle, try to educate them a bit far less likely to call you for a minor "my Outlook didn't close issue" in the future.

Every few weeks or so I do get to pen test the network. I'd suggest learning about network monitoring, from what you described that you are comfortable with might want to take a peek at tcpdump for network monitoring. There is really too much too list. My most fave thing has to be doing the network security checks though. Nothing like when auditors come in and I start arpspoofing them and monitoring their activity.

The best times are when stuff breaks. It can be a hassle but there's nothing like having users not able to use the web due to a proxy service having been blocked by a firewall, which the firewall I don't monitor. Spending a few hours trying to work it out and finding out it's the firewall.

[Aviar³]

I meant down on a list ;-; I would never really learn PowerShell obviously, since bash is much superior (actually, I don't like working on systems that I can't see the source for). I have to learn Perl and managing the server. I've never had to handle deploying things like NIS, NIS+, LDAP, managing work terminals, patching on the fly without reboot, or maintaining a variety of hardware devices and ensuring their drivers are 'compatible' with current configurations. So really, I need the whole administration part, but I don't quite know how one can learn to manage LDAP and such from a single computer. I've used simple things like WireShark to monitor HTTP\header requests, but I've never had a real solid objective, so I don't know exactly why people use the tools other than to see packets, their info and directions (obviously that is the purpose, but I don't know how to look for anomalies). Other questions, do you manage hardware configurations? I assume you also manage firewall policies and IP tables (I read monitor as actively watch), correct? Do you have to manage setting up and guarding services such as Apache, MySQL, etc? Do you set up and administrate SSH, FTP, SSL for HTTPs, etc?

[AhMunRa]

I have configured our bind server to route all requests to MySpace, Facebook, MyLife and the like back to a custom internal page that just gives them a server not available message.

I maintain mostly the *nix side of our network when not taking support calls. We do run MySQL but it's for the IT internal WIKI I setup for our newbies. Our php and apache are not external. I setup and maintain a monitor on ssh, and I monitor the proxy logs to make sure no one is hitting sites they shouldn't. I have squid setup to use ACL's to block content I don't want on our wire, like streaming audio or video. We do have remote access from external sources but only through ssh tunnelling and ssh has 3 part authentication in place, so it can't be brute forced. I did implement a IPTABLES/IPCHAINS firewall on the nix server to auto block repeated attempts to login. At home I have a computer setup with 2 nic as my gateway/router. It uses IPTABLES to route traffic. i can monitor my home network in this way, and also I can use that ability to block unwanted traffic/sites or allow things easier than if I had an actual router. Plus with 4 people using my home network, I can configure the computer to give my packets higher network priority meaning less game lag no matter the network load, read poor mans game router. This has cut down on alot of unneeded traffic. I did get to participate in a system wide update of the mail system from IMAP, and Pine to Exchange. We finally got rid of NT 4.0 authentication and upgraded to ADUC. That was fun, at the time I had never lain eyes on Exchange or ADUC. Neither took long to pick up.

I do get to write custom templates for ADUC to enforce GPO's banning certain things or enforcing settings on workstations. Everything from setting the default homepage to disabling USB ports.

As for the Facebook and myspace. I work for a finance company. After the company in FLA got sued trying to collect via myspace we made it a policy to block them to mitigate likelihood of lawsuit from eager collectors looking to make their goals for the month.

As for administering at home, you can setup VM's and practice. You can design your network across a few system, and configure proxies, setup secure ssh, configure DNS, setup Pine and the like or what have you. This is great practice.

I'd suggest

Perl or Python for log parsing, nothing is worse than staring at logs. Not too long ago I used python to allow me to remote into a computer and log in to a blocked page on the users system to allow the content. This enabled me to not have to log into the server and I could complete the task in 2 steps vs 4, saves about 2 - 3 minutes per shot. It adds up though.
MySQL, PHP also for log parsing, you can take what's parsed and drop it into a db for future reference or tracking, you can also make pretty pages with graphs for managers so they can see what's going on.
DNS obviously.
SAMBA for sharing between Windows and Nix
SSH
Reconfiguring hardware changes without a reboot, say you need to reset your nic ip, #emacs /etc/network/interfaces (make your changes)
#/etc/init.d/networking restart
Most firewall solutions these days are handled by a network appliance. I still run IPTABLES on the box to handle a legacy app on the network that would not function without it. We can not safely allow this app without opening a gaping hole in the network without it.
Adding/removing users to the system.
Creating groups.
Managing permissions.
Monitoring for rootkits. Even though there are only 2 people in the company that can log into that server, I watch it like a hawk.
Monitoring for process usage. ( get used to using things like gawk and sed ) Get used to piping them together to drill down to what you want.

The best advice is don't be afraid to break something. Better to do it on a non-critical system.

I do all my testing on a test server that is setup just for that. It's a mirror of the production server so anything that is getting evaled down to a package update or install is done there first.

[Aviar³]

I'll post more after I finish reading what you wrote, wanted to point out I mixed up nix and NIS, as I had only read about NIS yesterday after reading LDAP. I ended up doing NIS on my virtualized Solaris install, though now I don't know why. I'll have to figure out how it works.

----

Alright, finished reading. I remember hearing about Squid in passing, I'll look it up. I've never managed serious security policies, firewalls, and such mostly since I'm not the one in control for the local router, and Linux kernel panics if trying to install it on my system (Fedora\RedHat\OpenSuse distributions mostly). I'll look into some of the things you mentioned and see what I can pick up. I probably will gear just to trying to read up at some point on improved security on Linux, deploying BIND\Squid, and managing Solaris. I really don't like Windows, so I don't plan on reading up on how to manage a windows based server. Final question, what tasks have you had to automate using scripts?

---

Read the new things.

[AhMunRa]

Mostly log parsing, but I have managed to use Python to do some system level stuff via buttons on webpages. Had to give the python process access to read etc to modify my squid rulesets to allow or block sites. I'm using it now to parse my auth logs looking for brute force attempts on my server and then automatically adding them to my IPTABLES ruleset for blocking.

Oh one other thing. I would suggest looking at Mono for Apache. Allows you to run .asp pages via apache and executes cs code as well, very cool.

[Aviar³]

I've heard of Mono, my personal take on .asp is the same as Windows, but I guess a systems administrator doesn't have the luxury of choosing the workstations as well as the users. Rootkit monitoring sounds complex, but maybe its because I'm basing it off the book I've read on writing one (Shellcoders Handbook). Are there any Solaris systems deployed, or is that not a common configuration? When you went in to work, where there scripts already left by the previous systems administrator, or does each person have to bring their own toolbox? Any particular things you have to worry about policy wise (in the sense of are there any critical decisions software\deploy wise that you can't make, or any delays from upstream). I ask since I would think that the business policies matter as much as the hardware to a systems administrator, usually theres legacy apps like you mentioned that make the job harder, or relying on getting an a-ok from higher up for software.

[AhMunRa]

There's always office politics involved. Usually from a hardware or software side, I've proven myself to be competent in deciding, I do run it by the manager for cursory approval, which is usually a "sounds cool". The IT manager here is not linux savvy. A good fork bomb burried in a cronjob would be easy for me to find, would take some time, but I could find it, he on the other hand would have to start with a fresh install. Which would not work, as all the DNS and SSH configs would be lost due to the Linux server having a raid. He is better adept at the Windows Server administration than I am. That's where our dynamic draws off the others abilities though. So we do work well together. In fact we hang out away from work quite a bit.

Depending on what industry you go to work for will dictate the business rules. Being in finance we have alot of compliance laws we have to maintain.

As for the Solaris we have none deployed and I haven't messed too much with that distro, but Linux distros are fairly similar across the board.

I have used Linux knowledge to tweak network devices as well. We have LaCie NAS devices deployed for network storage for our satelite offices, these things use slimmed down Linux kernel to do their stuff. I cross compiled an ARM distro of Debian with some packages I wanted to include on these device and some stuff I wanted removed. It went very well. I have ssh ability to these devices and they all have gcc so I can compile new packages directly on the boxes themselves.

When I got here, there were scripts in place, many are no longer in use, a lot of them I have revisited and revised much of the code to bring it up to date.

[Yazu™]

>mfw I don't understand a thing you guys are saying
>mfw it's still interesting to read

[InternetIsSeriousBusiness]

[AhMunRa]

Also something you won't really find on google. Windows 7 when registered to a domain does not like to connect to Samba shares that require credentials but samba is not configured to use ADUC to authenticate. This was done as a protection measure. If my password is compromised which it actually was once during an audit, they still cannot access my network shares which contain critical materials.

As a side note, my domain credentials were compromised during the audit from a remote computer I had logged into 2 years prior as myself. After this lesson, no one was allowed to log into remote computer as themselves. It did take them 2 days to crack my password though, as opposed to the other admins password they recovered and cracked in 8 seconds. We called him Rodeo after that. He hated it.

It's that reason though that now every 3rd day I change my domain password.

[Aviar³]

[AhMunRa]

RAID most certainly. All our systems are redundant. Drive failure for a system running 24/7 is high, we have lost 2 in the last year from 2 different servers and these are server drives not your off the shelf stuff.

So far I'm on my 10th year or so of administration. I lost count some time ago. I pretty much fell into my position with little schooling.

Yeah the IT manager and myself have separate ways of dealing with that one. He has a batch file that he uses that's run at startup. I just hit the share and login to the samba share like its a domain, instead of:
username
password

It's
servername\username
sambapassword

Works a treat.

Not so much compiler/tools, but you will have to know how to compile a package without using something like YAST2 or APT, personally I don't see what is difficult with
#./configure && make install ( I like to build things from source )

Though sometimes you will need to know how to specify certain options. Programming not so much, but a grasp of it helps alot. Both in understanding how the machine processes information, stores it, and the like. Reversing more so I have found. When you have something that segfaults you should be able to resolve it without reinstalling it. All of my linux boxes are using X only none have gnome or kde install they are all headless systems, so most of the administration is command line from an ssh session.

All the hardware configuration is done by us, either myself the manager or what ever other admin we have at the time, currently it's the 2 of us and one full time programmer. So we have 100% trust from the machine. First thing that is done is the machine gets wiped and a fresh install done by us of the os. This ensures we don't have anything lurking on the machine. We use an outside supplier that does configure them for us, but it takes 1 disgruntaled employee with a virus or rootkit to screw us over.

[Aviar³]

Thanks for all the tips and details, I feel like I have a better idea of what is asked of a systems administrator now.

As a bonus, I recommend that if anyone complains about something:

"If you wish to file a complaint, please send an e-mail to your own work e-mail, as I read every mail in your inbox."

I might also bug you time to time about small details if I start looking into BIND\Squid, or such material. Probably mostly practice implementations.

[AhMunRa]

Fire away if you have questions.