Archive for the ‘ Technology ’ Category

Time flies when you work 10 – 12 hours a day

I seriously can’t believe it’s already mid September. I have been working a lot of crazy long days (more than usual) lately. It’s not that I’m required to do so, I just find myself still working at 6 or 7 PM most days. It’s a bug I have. I know that me working late doesn’t make the work go away, but for some reason I continue to do it.

Saturday, had coffee, did a little shopping, then came home and watched football on and off all day. Went out for chinese, then came home and forced myself to sit through the VT vs. LSU game. Sad, sad game. The only good that came of it was Glennon getting the boot, and Tryod getting to step in. There is hope VT fans!

On Sunday, we went out to Alexandria to see Tony & Dawn’s new house, then have dinner in Old Town at Austin Grill. Food was great, so was the company. We walked around Old Town a bit, and caught up. It’s been way too long since we’ve hung out. Definitely gonna try to make it a more regular thing.

Monday evening I decided to take off early (read: on time), went to Bungalow Billiards with Tony & Ivan for $1 Coronitas & pool. We had a good time, then came back to my place to watch the first half of the Monday Night Football game.

Still working on that stupid Debian build. After I got things working last week, I find out that now people want to have some input on how I deploy this. I’m like WTF? I asked the day that I heard about this deal, to get requirements. Not one fucking person spoke up with anything productive. 2 weeks later they want to talk about “standards”. So I have to start from scratch. On top of all this, I can’t even start builds until I get this massive re cabling project done in one of our Data Centers. I finally got the approval on the contracting quote, but of course the contractors are slammed this week. Might have to do a Saturday babysitting session. Once I get all the cabling done, I have to reconfigure & migrate 14 enclosure switches over to our other network. Fun stuff.

Friday night, is migration night for I’m really not looking forward to it, but I’m ready to get it out of the way. Especially since I’m paying for both of these servers right now, which really isn’t cheap.

I love Red Hat

Yesterday afternoon I found out we signed yet another customer wanting a non-standard OS. These guys want Fedora. While it pissed me off that they once again dropped this on my plate, without consulting me first, I wasn’t too worried, as I knew I could make Fedora work with my current kickstart server easily. How easy? Within an hour of being told this customer was signed, I had the iso’s downloaded, extracted to my kickstart server, kickstart config built, and a working deployment to one of my blades! The only thing that didn’t work right off was the Proliant Support Pack, which is to be expected, since I think HP hard codes OS versions in the installer. It was a nice break from the Debian nonsense and a small victory of sorts for me.

As for Debian, I finally had a successful install on a blade this morning. I got the Proliant Support Pack .deb’s from HP installed, and I even got the Altiris agent installed. Believe it or not, the server actually checked into altiris, and I can manage it. This is huge! So I made an image of the server from altiris, and deployed it to another blade. Unfortunately there’s issues there that I need to figure out. But I’m pretty happy I finally got an successful install.

On a side note, while looking into the Fedaora PSP install failure, I figured out something that I’ve been looking for since April.

More reason that Debian sucks

So apparently I Debian sucks so much that it can’t recognize a SAS controller. I tried 3 different installers based on “etch”, none of which could properly “see” a hardware raid. Keeps giving me a drive error. So I run a Red Hat install, and of course it detects and installs perfectly. Do I really need to make driver disks to make this shit work? Seriously. This is 2007 (almost 2008). Driver disks are so 1990’s.

Fucking Debian!

Have I mentioned how much I hate Debian?

So we landed a fairly big customer. Said customer is all about Debian. It went down kind of like this

Sales: “Will you guys support 40 Debian servers?”
Us: “No!”
Sales: “Okay, we’ll need those by the end of September”

So of course not only am I the guy responsible for creating all Linux deployments, I’m one of the only people in the company that know anything about it. I’ve never been a fan of Debian. I’ve tried to like it, and it just never works out. Talk all the shit you want about Red Hat based distros, but there’s a reason they rule the corporate Linux marketplace.

So anyway, now I have to figure out a way to integrate Debian into my kickstart server, or create another deployment server using their lame ass FAI. And did I mention that said deployments will be happening on blades that HP completely does not support Debian on? Should be fun trying to find modules for this shit, let alone managing these servers after the install.

This is the good part…

For all the suck that my job provides, I have to try to remember that I have it pretty good overall. It’s about 11:00am, I’m sitting at panera (aka pantera), eating a bagel, drinking coffee, and working. The better part of my work week is usually spent at home or in coffee shops. As long as I have wifi, and a cell phone connection, I can work from anywhere in the world. I go into my office maybe 2 or 3 times on a good month. I think there was a stretch a couple months back that I went in once in 2 months. I spend a bit of time in the Data Centers, but not anything regular. It’s a luxury that few people I know get to have, and I am thankful to be in the position I am in.

Yesterday turned out to be a 16 hour work day (minus a couple hours for lunch and dinner). I got everything finished that I needed to get done, so in the end it’s worth it. And being that I’m the one who dishes out work, those who failed me, are going to have some rather unpleasant tasks to deal with today.

I’m trying to plan a “business trip” out to California, so I can spend a couple of days at the corporate office, up in Santa Clara, then head down to LA where my part time boss is, then down to San Diego where my director is. Since I have family and friends in LA & San Diego, I figure I can get some time to hang out with people there, since it’s so hard to take a vacation there. I’d like to bring Steph, but she’s not keen on the idea of a “working” vacation. So maybe I’ll take 2 trips out there.

Anyhow, time to get back to doing work.

Responsibility Part Deux

I started my day as I always do around 8:00am. Sometimes it’s a little earlier, sometimes a bit later. Today, I went to the coffee shop to take care of some stuff remotely for about an hour, then I headed to one of my Data Centers to take care of some stuff before my team meeting. During my meeting, I made our priorities very clear. Told my team what we needed to get done today, and that I’d be back to complete stuff later in the afternoon. Then I went to our other Data Center to take care of some stuff there. So at about 4:30 I come back to the DC, expecting to configure 8 switches and go home. I get here, and find that the one thing that I told them absolutely had to be done today, didn’t get done; cable up a blade enclosure stack. Thus, I couldn’t configure the switches that live in these enclosures.

So lucky me, got to start cabling shit up at 4:30PM. If anyone has ever dealt with blades, you know that they are stupid simple to deal with after the enclosure is setup, cuz there’s no cables or anything to deal with. However, setting up a new stack of 4 enclosures is a pretty big ordeal. Each enclosure has 2 switches, each switch has a redundant connection to the patch, then duplicate that to the network cores. in all, you end up with 68 cables that have to be run and labeled. Time consuming to say the least. I finally got everything cabled up at 8:00PM, then took a break to meet Steph for dinner. Now I’m back in the DC configuring my switches. I should be done with this in about half an hour. Then I have to be at the other DC for a maintenance window at Midnight.

It’s times like this that I wish I could be irresponsible, and pass the buck.


So I was searching for new analytics (besides google), and I came across Clicky. Anyone using it? My site is still far too low traffic to generate any useful stats, but the interface looks awesome. Just curious what other people’s experience with them is.

Server migration time again

Over the past month or so, I have been working on a flurry of updates, reconfigurations, migrations of services, and entire servers for my personal sites, and my hosting company. The personal stuff is always fairly painless. If something breaks, it’s not the end of the world. We lose a little ad revenue, and a few people complain that they can’t reach Stvlive, or QuizMeme. I’m a bit of a masochist in this regard, because I always seem to do 8 million projects at once when I should really be focusing on one. In all, I have done, or am in the process of doing 2 complete server migrations (move from one server to another in a completely different Data Center, new IP space, etc), migration of secondary mail & DNS services for my personal sites to a 3rd party, and massive amounts of hardening across all of my sites/servers.

Most of the personal stuff has gone well, and we even revamped some really old stuff on some of the sites, which makes me happy from an InfoSec stand point. I’m mostly content with how things are running on all of those servers, now it’s just the ongoing issues of cleaning things up that have been around since the late 90’s to make things more secure.

My hosting migration however, makes me lose sleep. I have done this a few times over the years, and it usually goes “okay”, but never without some screwup that keeps me up for hours fixing. The problem isn’t from a lack of planning, or skill. I have been doing this a long time, and I am very knowledgeable about these things. Where the problems occur is usually in the little configuration changes (read: hacks) that have been made on the servers over the years, and have been forgotten about. This server has essentially been upgraded and migrated over and over again since 2001. It’s gone through 3 different FULL RedHat releases starting with RedHat 7.1 (going on a 4th now). I can’t even count the number of Plesk versions, I’m an old school customer, so this server (in it’s original form at my house 7 years ago) started at PSA 2.5, and is getting ready to be Plesk 8.2. So as anyone with any admin experience can imagine, the number of “hacks” that would have been put into place over the years to add support for some unsupported feature, or fix a bug. The irony is, that it’s the old “hacks” that were meant to fix something in the past, that break something on the new.

At this point I have most of the behind the scenes work completed. New server is up and running, and has been (mostly) configured. DNS is going to be the biggest hurdle. The current setup is not so good. All DNS is served from the same server. This isn’t really a big deal, because it’s pretty much an all-in-one server, so if DNS goes down, chances are everything else is down too, so the DNS doesn’t really get you anywhere. However, in a migration situation, having a secondary somewhere else is extremely useful because it isn’t going to change. So when the madness happens when I change my name servers at the registry level, propagation isn’t that big of a deal, because the secondary server is still churning out results. So I want to get this server added for all the domains prior to the move.

I host over 200 domains, and unfortunately they were not all registered through me. That means that the owners of all of those domains need to log into their registrar account, and make modifications to their name servers. This is a very simple task for someone with only a little technical knowledge. All of the registrars have documentation on how to do it. The problem is, getting the domain owners to ACTUALLY MAKE THE CHANGE! I am willing to bet most of my customers don’t even know what a registrar is, let alone which one their domain is registered at. Which is going to equal me doing a shit ton of whois lookups for people to point them in the right direction. And in more than a few cases, I’ll probably just have to obtain their login info, and make the change for them. I am actually having some new flash demos made up right now to show people how to login to the various registrars, and make this change, so hopefully that will help a bit. The plus side is, most registrars don’t require an IP address for name servers, so when I actually re-IP my name servers, there shouldn’t need to be any changes on the end user side.

I’m rambling, and I’m sure this is way more information than most people who read my blog care about, but that’s what a blog is for right?

Gotta love responsibility

We had a customer who wanted some basic stuff done during a 3:00AM maintenance window today. Since my boss told me to take Friday off, I assigned this maintenance to one of my guys. I sat down with him yesterday, to make sure that he knew what he was doing, and showed him some docs I made for the issue. So I felt good about him doing it, but told him to call me if there were any problems.

3:08AM – phone rings, I explain some stuff to him
3:18AM – phone rings again, I walk him through something else
3:32AM – phone rings again, I explain something else
4:34AM – phone rings again, he tells me the jobs all failed. I’m pretty sure they are fine, but tired of trying to explain what to look at so I tell him I’m getting online to look at it myself.

So now I’m up. I guess it’s a good thing I’m not working today after all. Though I’m sure as soon as I try to sleep today, someone is gonna call me.

First real post in 2 years

So what’s been happening in my life?

We bought a new house in Ashburn in March. We moved in a few months before that, but were waiting to sell our townhouse. The house is awesome! It’s on about a quarter of an acre, in a great neo-traditional neighborhood. We’ve done a ton to the house already, and it looks great!


Shortly after we moved here, India our Rotti girl lost it completely. After several attempts to reverse her behavior, we had to let her go. Shortly after, we began looking for a new puppy. We wanted to find a dog that would be completely submissive to China, as well as just being an all around awesome dog! So we picked this little guy from a breeder.

Benny The Golden

A little over a year ago, I quit my job at MITRE, and went to work for a “startup” called OpSource. I was hired as the Sr. Linux Engineer for Data Center Operations. These days, I end up doing more management, than engineering, but it’s still a good gig. I mostly work from home, or any other random place with wifi. I’m starting to work in InfoSec now on top of my normal workload, with the intentions of switching over full time as soon as they can afford to replace me.

I could ramble on about the past 2 years forever, but I think I’ll take it one post at a time for now. Commenting is now open, but you must register.