Category: Linux

I love Red Hat

Posted on by DizkoDan

Yesterday afternoon I found out we signed yet another customer wanting a non-standard OS. These guys want Fedora. While it pissed me off that they once again dropped this on my plate, without consulting me first, I wasn’t too worried, as I knew I could make Fedora work with my current kickstart server easily. How easy? Within an hour of being told this customer was signed, I had the iso’s downloaded, extracted to my kickstart server, kickstart config built, and a working deployment to one of my blades! The only thing that didn’t work right off was the Proliant Support Pack, which is to be expected, since I think HP hard codes OS versions in the installer. It was a nice break from the Debian nonsense and a small victory of sorts for me.

As for Debian, I finally had a successful install on a blade this morning. I got the Proliant Support Pack .deb’s from HP installed, and I even got the Altiris agent installed. Believe it or not, the server actually checked into altiris, and I can manage it. This is huge! So I made an image of the server from altiris, and deployed it to another blade. Unfortunately there’s issues there that I need to figure out. But I’m pretty happy I finally got an successful install.

On a side note, while looking into the Fedaora PSP install failure, I figured out something that I’ve been looking for since April.

More reason that Debian sucks

Posted on by DizkoDan

So apparently I Debian sucks so much that it can’t recognize a SAS controller. I tried 3 different installers based on “etch”, none of which could properly “see” a hardware raid. Keeps giving me a drive error. So I run a Red Hat install, and of course it detects and installs perfectly. Do I really need to make driver disks to make this shit work? Seriously. This is 2007 (almost 2008). Driver disks are so 1990’s.

Fucking Debian!

Have I mentioned how much I hate Debian?

Posted on by DizkoDan

So we landed a fairly big customer. Said customer is all about Debian. It went down kind of like this

Sales: “Will you guys support 40 Debian servers?”
Us: “No!”
Sales: “Okay, we’ll need those by the end of September”

So of course not only am I the guy responsible for creating all Linux deployments, I’m one of the only people in the company that know anything about it. I’ve never been a fan of Debian. I’ve tried to like it, and it just never works out. Talk all the shit you want about Red Hat based distros, but there’s a reason they rule the corporate Linux marketplace.

So anyway, now I have to figure out a way to integrate Debian into my kickstart server, or create another deployment server using their lame ass FAI. And did I mention that said deployments will be happening on blades that HP completely does not support Debian on? Should be fun trying to find modules for this shit, let alone managing these servers after the install.

Server migration time again

Posted on by DizkoDan

Over the past month or so, I have been working on a flurry of updates, reconfigurations, migrations of services, and entire servers for my personal sites, and my hosting company. The personal stuff is always fairly painless. If something breaks, it’s not the end of the world. We lose a little ad revenue, and a few people complain that they can’t reach Stvlive, or QuizMeme. I’m a bit of a masochist in this regard, because I always seem to do 8 million projects at once when I should really be focusing on one. In all, I have done, or am in the process of doing 2 complete server migrations (move from one server to another in a completely different Data Center, new IP space, etc), migration of secondary mail & DNS services for my personal sites to a 3rd party, and massive amounts of hardening across all of my sites/servers.

Most of the personal stuff has gone well, and we even revamped some really old stuff on some of the sites, which makes me happy from an InfoSec stand point. I’m mostly content with how things are running on all of those servers, now it’s just the ongoing issues of cleaning things up that have been around since the late 90’s to make things more secure.

My hosting migration however, makes me lose sleep. I have done this a few times over the years, and it usually goes “okay”, but never without some screwup that keeps me up for hours fixing. The problem isn’t from a lack of planning, or skill. I have been doing this a long time, and I am very knowledgeable about these things. Where the problems occur is usually in the little configuration changes (read: hacks) that have been made on the servers over the years, and have been forgotten about. This server has essentially been upgraded and migrated over and over again since 2001. It’s gone through 3 different FULL RedHat releases starting with RedHat 7.1 (going on a 4th now). I can’t even count the number of Plesk versions, I’m an old school customer, so this server (in it’s original form at my house 7 years ago) started at PSA 2.5, and is getting ready to be Plesk 8.2. So as anyone with any admin experience can imagine, the number of “hacks” that would have been put into place over the years to add support for some unsupported feature, or fix a bug. The irony is, that it’s the old “hacks” that were meant to fix something in the past, that break something on the new.

At this point I have most of the behind the scenes work completed. New server is up and running, and has been (mostly) configured. DNS is going to be the biggest hurdle. The current setup is not so good. All DNS is served from the same server. This isn’t really a big deal, because it’s pretty much an all-in-one server, so if DNS goes down, chances are everything else is down too, so the DNS doesn’t really get you anywhere. However, in a migration situation, having a secondary somewhere else is extremely useful because it isn’t going to change. So when the madness happens when I change my name servers at the registry level, propagation isn’t that big of a deal, because the secondary server is still churning out results. So I want to get this server added for all the domains prior to the move.

I host over 200 domains, and unfortunately they were not all registered through me. That means that the owners of all of those domains need to log into their registrar account, and make modifications to their name servers. This is a very simple task for someone with only a little technical knowledge. All of the registrars have documentation on how to do it. The problem is, getting the domain owners to ACTUALLY MAKE THE CHANGE! I am willing to bet most of my customers don’t even know what a registrar is, let alone which one their domain is registered at. Which is going to equal me doing a shit ton of whois lookups for people to point them in the right direction. And in more than a few cases, I’ll probably just have to obtain their login info, and make the change for them. I am actually having some new flash demos made up right now to show people how to login to the various registrars, and make this change, so hopefully that will help a bit. The plus side is, most registrars don’t require an IP address for name servers, so when I actually re-IP my name servers, there shouldn’t need to be any changes on the end user side.

I’m rambling, and I’m sure this is way more information than most people who read my blog care about, but that’s what a blog is for right?

Gotta love responsibility

Posted on by DizkoDan

We had a customer who wanted some basic stuff done during a 3:00AM maintenance window today. Since my boss told me to take Friday off, I assigned this maintenance to one of my guys. I sat down with him yesterday, to make sure that he knew what he was doing, and showed him some docs I made for the issue. So I felt good about him doing it, but told him to call me if there were any problems.

3:08AM – phone rings, I explain some stuff to him
3:18AM – phone rings again, I walk him through something else
3:32AM – phone rings again, I explain something else
4:34AM – phone rings again, he tells me the jobs all failed. I’m pretty sure they are fine, but tired of trying to explain what to look at so I tell him I’m getting online to look at it myself.

So now I’m up. I guess it’s a good thing I’m not working today after all. Though I’m sure as soon as I try to sleep today, someone is gonna call me.

First real post in 2 years

Posted on by DizkoDan

So what’s been happening in my life?

We bought a new house in Ashburn in March. We moved in a few months before that, but were waiting to sell our townhouse. The house is awesome! It’s on about a quarter of an acre, in a great neo-traditional neighborhood. We’ve done a ton to the house already, and it looks great!

House

Shortly after we moved here, India our Rotti girl lost it completely. After several attempts to reverse her behavior, we had to let her go. Shortly after, we began looking for a new puppy. We wanted to find a dog that would be completely submissive to China, as well as just being an all around awesome dog! So we picked this little guy from a breeder.

Benny The Golden

A little over a year ago, I quit my job at MITRE, and went to work for a “startup” called OpSource. I was hired as the Sr. Linux Engineer for Data Center Operations. These days, I end up doing more management, than engineering, but it’s still a good gig. I mostly work from home, or any other random place with wifi. I’m starting to work in InfoSec now on top of my normal workload, with the intentions of switching over full time as soon as they can afford to replace me.

I could ramble on about the past 2 years forever, but I think I’ll take it one post at a time for now. Commenting is now open, but you must register.

been a while…

Posted on by DizkoDan

So I haven’t posted in a while. Been hella busy at work lately. I’ve been in Boston since Sunday for Linuxworld. It was cool, but I don’t think I’d even go for the whole week again. A couple of days would be good. Unfortunately, it seems I know too much, cuz most of the conferences/seminars I sat in on were old news. I guess that’s not really a bad thing though, is it?

I forgot how great Boston is. I love this city. The architecture is fantastic. The people are mostly cool. I wouldn’t mind having a house up here, though I wouldn’t wanna live here in the winter. It’s pretty fuckin cold. As usual, I walked entirely too much this week. The only time I’ve used any other form of transportation I’ve used was the cab I took to the hotel from the train station. I walked about 10 miles the other day around the city, after walking around the expo all day. Today I walked another 5 or so to/from the aquarium, and all around it. It was fun, but I wish Steph was here to see it. It’s too bad, cuz there’s no reason she couldn’t have come, since she’s hardly working this week with the office being remodled.

Other than the trip, not much has been goin on. I’m still waiting on my body kit for my car. They called me today to tell me that infiniti hasn’t even started shipping the parts out. That’s what I get for getting a car barely off the assembly line right?

I decided that we needed a real vacation, so I booked us a trip to Cancun next month for our anniversary. We’ll be staying in the same hotel we stayed at for our honeymoon. It will be nice to just relax for a week. Been a long time since we took a real vacation.

How you all doin?

Posted on by DizkoDan

On the car front, I decided to go with the Infiniti. However, finding the exact one I want, on the east coast is a much bigger challenge than I anticipated. Apparently, Infiniti made the decision to send mostly AWD 05 G35 sedans to the notheast, because thats where the market demand is right now. And I would have to agree to an extent. I would get an AWD, if I didn’t just get the outback for Steph. That was one of the main reasons that we got that car. However, I don’t want to spend the extra money for an AWD car, that’s also gonna get even worse gas mileage than the already less than steller mileage on the RWD G35.

So I went to the dealer near me on Monday to try and get a deal setup for my car. It turns out that all they have is 5 05’s in stock, all AWD, and they wont be getting more until March! I try to get them to order me the car I want, or have it transferred. They try to sell me on an 04. I’m like screw you hippies! I’m out. So I start calling all of the infiniti dealers within a 150 mile radius to try and find the exact car I’m looking for. The dealer in Alexandria tells me that they have located the car in Miami, and can try and get it transferred for a fee. I’m like HELL YEAH! So I waited around there yesterday for a couple of hours waiting for the GSM in Miami to call the GSM in Alexandria to confirm that they will transfer the car. After several attempts to get ahold of him, the GSM at Alexandria tells me to just go home, and he’ll call me as soon as he hears from him today.

So keep your fingers crossed. If this falls through, the only other location they found that had the color/specs I want is in TX, which would cost a lot more to transfer. IF it does work out, I’ll go in today and give em a deposit and the car will go out on a flatbed today/tomorrow and should be here within 24 hours. The waiting is killing me!


I’ve been having a ton of problems with my servers lately. It’s getting to the point where I am constantly proactively monitoring them, in addition to my automated monitoring which pages me when theres a problem. People can’t write secure code to save their lives, so I’m in a battle trying to keep the script kiddiez off my hosting server. I keep trying to deploy a secure kernel patch that should help mitigate the problems, but I keep running into issues where certain necessary services wont start up properly. I have a maintenance window scheduled for tomorrow night to do some upgrades, so one way or another, I’m gonna make this shit work right. Otherwise, I’m gonna have to go through all of my customers code to find their shitty PHP upload script that is allowing the malicious code to get in.

It’s shit like this that makes me want to sell off my business.

Ramble on…

Posted on by DizkoDan

Been a while since I updated this thing, so I thought I’d say hi. Not a whole lot has been happening as of late. Work has been cool. I am basically the resident linux ninja now, which is both good and bad. Obviously it’s good because I am needed. It’s also bad, because I am needed. Luckily we don’t deal with production equipment much, so most of our machines are not critical. The only times I have pressure to get stuff done ASAP, is when someone is trying to do a demo (which has kept me here 12 hours one day, and 15 another day). I don’t really mind though, cuz I enjoy what I do.

Steph & I went to Philly last weekend as a mini vacation. It was nice to spend more than a few hours there for once, and actually get to see the city. As usual, we walked A LOT. We have a problem using transportation on trips, and always end up walking miles and miles everyday. We took a tour on The Big Bus which was cool. It takes you all over town to the major attractions, with a guide who tells you about everything along the way. You can get on and off wherever you want throughout the day. It made it a lot easier to visit all the places we wanted to go in a single day. We stopped at the art museum, and the zoo. Both were cool, but overpriced in my opinion. I guess I’m just spoiled from living in the DC area. We stayed at an awesome hotel called sofitel. They are really big in Europe. There service was fantastic, and the hotel was beautiful.

Other than that, time is just flying by at an unbelievable pace. We finally bought some bikes, and have been trying to get out as much as possible. As soon as I get a rack for the car, we’ll start hitting the trails in the fall. I have been trying to make time to hang out with friends, but it’s been really hard to get together. I have zero interest in going out to clubs these days, which seems to be the same old meeting place. I think I need to have a party or something, to get everyone together.

Posted on by DizkoDan

I’ve been working at my new job for a week now, and It’s been pretty cool. I have finally gotten most of my shit setup, and I’m just waiting on a couple of little things to get done. I’ve spent most of this week setting up a PXE kickstart server for our labs. One of my ongoing projects is upgrading (reinstalling) an asston of legacy RH servers, as well as migrating others. So this was the first thing I wanted to get done, as it will save me lots of man hours. People pretty much leave me alone and let me work, so It’s a nice change. I have some summer hire sitting in my office right now, but hopefully she’ll be leaving soon, and I’ll have the entire office to myself. I don’t like having to turn lights on, and keep my music low for other people. At least she’s nice, and doesn’t bother me.

I’ve been working about 7-4 all week, and It’s been working well for me. I’d like to go in a little earlier, but so far it hasn’t really happened. It’s been nice to actually see Steph everyday, and having a full weekend together is awesome. We’re gonna buy some bikes soon, so we can go riding a lot in the fall. And if all goes as planned, we’re gonna take a weekend trip down to OBX around the end of the month or early next month.

I’m starting to ramble my random thoughts again, so I think it’s time to end this. How is everyone else?