Bad bad week, kinda…

From REALLY BAD to AWESOME J

REALLY BAD : We had a little exchange issue this week, which was not recovered from so well. That's fixed going forward.

SOMEWHAT BAD : On top of that my biggest project is starting to be realized, which is a datacenter move. That's going okay, but we have some interesting swapping to do. I need to move my f5s this week, and ship a whole bunch of networking gear. I have a Netapp, and lots of servers ready to be installed.

BAD: The issue is that my new contract isn't done yet, which is another thing I have to do. I also just realized we aren't going to do managed backups due to the $5k a month they want to charge us, so now I'm ordering a library as well.

BAD : Found out we opened a German office without any IT signoff, no firewall, connectivity issues, and the staff there speaks no English. Trying to get my hands on that, but it's a bad situation. Need to order some gear.

BAD : We have 4 web conferencing tools in use here, we waste a LOT of money. I want to get everyone on 1 or 2 web conferencing tools. Another annoying project to deal with.

GOOD : Just wrapped up signing a new and much better Akamai contract, which is pretty cool. Also got a LOT of purchases pushed through. Probably about 10 purchases this week for the company.

GOOD : Redoing and reassessing all of our backups due to the exchange issue in all of our offices.

KICKASS : Installed a new helpdesk app that I love that we are evaluating – www.cerb4.com its awesome. I hope to get off the crappy ACE system we are using which is like pulling teeth out of a rabid Rottweiler.

Changes at my Company

I'm not running the group as of 2 weeks ago. That means all operations, enterprise, production, networks, systems, etc etc. I have a good crew of guys, and looking to better augment some of our staff. Happy to have been given the opportunity, and looking forward to making an even larger impact than I have up until this point. Lots of things to get in order which are eating my time aside from all the project management, purchasing, and other engineering I also have to do. Its been very crazy. I'll be posting more on some of the highlights of this week as well.

Monitoring, Reporting, and other stuff :)

Over the last week, I was sick a couple days. I used some of that time to build a cool management dashboard based on data coming from various tools:

1. Coradiant – E2E, Host, Status Codes, Sessions, good versus errored
   
2. Google Analytics – Pageviews
   
3. ACE – Ticket metrics
   

We are also evaluating the Dejaclick product from Alertsite to enhance the basic remote reporting we are using. The tool is very slick, and the pricing is pretty reasonable for what you get. We are also going to be looking at Coradiant Truesight Edge next week to get more data from our Akamaized traffic (which most of it is).

Other than that just ordering a bunch of RAM for some boxes we are turning into ESXi machines, and also ordering some new Dell switches to run 10Gig Ethernet to the new Netapps we are putting in place.

Domain fun

I have been busy moving about 400 domains to godaddy. It's been a lot of fun (NOT) consolidating across our current 14 registrars. Once this is done we can start to do web forwarding and migrating onto our new DNS infrastructure. The new one is fully split, where we have a proper external, internal, and update system running. Good design, and should serve us much better.

Enterprise Fun

I am rewiring a bunch of servers, and reconfiguring a large fileserver on Friday. We need to repartition a 8TB volume into smaller slices. I'm going to use the knoppix CD and some of the tools on that distro to move and recut the NTFS partition so that we can run VSS.

Progressing on the Exchange 2007 migration, everything seems to be 100% now, all we have to do is get some of the mailboxes small enough so we can move them over properly. We are shooting for under 500MB, too bad some of our bad users have 7G mailboxes L

Pain.

Outages

Our site has been having a lot of issues, so we offloading the image serving onto its own 2 boxes. We also added another 15 spindles on the production DB server, which seems to be helping a lot. We are seeing higher traffic volumes on our Coradiant reports, and we are also seeing lower latency. Good work by the whole team to fix the environment.

Licensing

Working on some licensing for Redhat as well for our new environment. I don't really like dealing with Redhat, but in order to have a good supported OS we have no choice. They are already using it in some places. We were deciding if we were going to switch to CentOS, but figured it was smarter to have the fallback on a vendor.

Vmware and Firewalls

We've been pushing more esxi here, and we are going to run our remote offices off a single esxi host running a domain controller, exchange server, and vpn server. We did the same setup for both Shanghai and Geneva offices. We are going to ship them with the new firewalls which should be a good setup, and easy to manage.

 

Tons of progress this week – VM, Exchange, Firewalls, Security, Storage/NAS

Had a very productive, and busy week. Balancing out server crashes taking down the production site, and the build-out project we are working on the team was very busy. I built over a dozen VMs, and 3 ESXi boxes.

I also worked on some of the config to finish out the main exchange 2007 implementation. We are waiting for ESXi boxes to ship to the remote offices, which will house a mailbox server, a domain controller, and possible a client access server.

One of my colleagues (who knows the setup of the WAN well) is working on the configuration and testing of our new firewall infrastructure. We are putting in all Sonicwall NSA series appliances. We have a 2400 for Shanghai, and 3500's for Atlanta and Geneva. The production environment will be off a pair of HA 4500 series boxes. The features, ease of setup, and price were excellent. We also have good resources from the reseller and the vendor if we need them.

We are also running a pilot (which has been going quite slow) of the Q1labs qradar product. I have used this product in previous companies, and its been a great tool for security, network analysis, and troubleshooting. The problem is running the pilot with these huge projects that need to be done over the next few months isn't really feasible. I hope to invest more time in testing it, but my priorities are dictating that not to be the case.

I also got rid of a couple older boxes this week by doing P2V using VMware converter starter edition. Now the enterprise ESXi boxes are pretty much full, so I need more disk space in order to do anymore work. That should be happening as we get the NAS implemented.

We wrapped up the Bluearc v. Netapp stuff, we are going with a couple of Netapps. The solution looks very good, and we should have it up in a couple weeks. I'm looking forward to building out some Exchange, MSSQL, and ESXi clusters using it!

Talk to you all soon, have a nice weekend. Loving the weather in Georgia J

Updates on some projects – NAS/Firewalls

Senior management has given us a date of December to have our current production hosting here in Atlanta (colo, downtown) moved to Geneva (hosting facility we have there). In order to do this we'll need to virtualize a lot, and build out a much better network and storage infrastructure. This will also help us in the future as we grow the business as well.

We pretty much decided at this point that Bluearc is a better solution for us. It was very close between them and Netapp, but it was a matter of performance over more advanced software. We are looking to wrap up the deal soon and get the purchase completed.

We have also decided to go with a sonicwall solution at 5 locations. The product looks very good, and will enable us to have a proper VPN mesh between all sites. We will also be replacing some of our web and spam standalone filters with the new NSA UTM device.

Making some plans to move our Sharepoint and Spam filter from our colo to the corporate office.

Not much else going on at the moment…. That's the update.