Responsibility Part Deux

I started my day as I always do around 8:00am. Sometimes it’s a little earlier, sometimes a bit later. Today, I went to the coffee shop to take care of some stuff remotely for about an hour, then I headed to one of my Data Centers to take care of some stuff before my team meeting. During my meeting, I made our priorities very clear. Told my team what we needed to get done today, and that I’d be back to complete stuff later in the afternoon. Then I went to our other Data Center to take care of some stuff there. So at about 4:30 I come back to the DC, expecting to configure 8 switches and go home. I get here, and find that the one thing that I told them absolutely had to be done today, didn’t get done; cable up a blade enclosure stack. Thus, I couldn’t configure the switches that live in these enclosures.

So lucky me, got to start cabling shit up at 4:30PM. If anyone has ever dealt with blades, you know that they are stupid simple to deal with after the enclosure is setup, cuz there’s no cables or anything to deal with. However, setting up a new stack of 4 enclosures is a pretty big ordeal. Each enclosure has 2 switches, each switch has a redundant connection to the patch, then duplicate that to the network cores. in all, you end up with 68 cables that have to be run and labeled. Time consuming to say the least. I finally got everything cabled up at 8:00PM, then took a break to meet Steph for dinner. Now I’m back in the DC configuring my switches. I should be done with this in about half an hour. Then I have to be at the other DC for a maintenance window at Midnight.

It’s times like this that I wish I could be irresponsible, and pass the buck.

On the car front, I decided to go with the Infiniti. However, finding the exact one I want, on the east coast is a much bigger challenge than I anticipated. Apparently, Infiniti made the decision to send mostly AWD 05 G35 sedans to the notheast, because thats where the market demand is right now. And I would have to agree to an extent. I would get an AWD, if I didn’t just get the outback for Steph. That was one of the main reasons that we got that car. However, I don’t want to spend the extra money for an AWD car, that’s also gonna get even worse gas mileage than the already less than steller mileage on the RWD G35.

So I went to the dealer near me on Monday to try and get a deal setup for my car. It turns out that all they have is 5 05’s in stock, all AWD, and they wont be getting more until March! I try to get them to order me the car I want, or have it transferred. They try to sell me on an 04. I’m like screw you hippies! I’m out. So I start calling all of the infiniti dealers within a 150 mile radius to try and find the exact car I’m looking for. The dealer in Alexandria tells me that they have located the car in Miami, and can try and get it transferred for a fee. I’m like HELL YEAH! So I waited around there yesterday for a couple of hours waiting for the GSM in Miami to call the GSM in Alexandria to confirm that they will transfer the car. After several attempts to get ahold of him, the GSM at Alexandria tells me to just go home, and he’ll call me as soon as he hears from him today.

So keep your fingers crossed. If this falls through, the only other location they found that had the color/specs I want is in TX, which would cost a lot more to transfer. IF it does work out, I’ll go in today and give em a deposit and the car will go out on a flatbed today/tomorrow and should be here within 24 hours. The waiting is killing me!


I’ve been having a ton of problems with my servers lately. It’s getting to the point where I am constantly proactively monitoring them, in addition to my automated monitoring which pages me when theres a problem. People can’t write secure code to save their lives, so I’m in a battle trying to keep the script kiddiez off my hosting server. I keep trying to deploy a secure kernel patch that should help mitigate the problems, but I keep running into issues where certain necessary services wont start up properly. I have a maintenance window scheduled for tomorrow night to do some upgrades, so one way or another, I’m gonna make this shit work right. Otherwise, I’m gonna have to go through all of my customers code to find their shitty PHP upload script that is allowing the malicious code to get in.

It’s shit like this that makes me want to sell off my business.

Pork chop sandwiches!

So I’m still at work fighting a stubborn hozed harddrive. I’m hella tired, and a bit delerious. These movies have been floating around for some time, but I just cant stop watching this one. The audio is not really work safe, so make sure you put the headphones on if you that sort of thing gets you in trouble. The video is safe.

When the grass is cut the snakes will show

Note to self.. Mail server is cursed.

Just a few weeks ago, I had to completely rebuild my mail server due to a hozed HD. Things were running smooth, and I had managed to get all kinds of shit working that I had been meaning to for a while. Fast forward to Monday night.

Got to work at about 10:30pm and tried to ssh into my mail server. Immediately it returned a connection terminated. So I had Steph go look at the console, and sure enough, it had a kernel panic. She rebooted it, and it did it again. Rebooted again, seemed to be okay. I start investigating the panics, and come to the conclusion that it’s probably bad RAM or the swap partition iz fux0red. As I’m looking around, I get some dma errors (a sign of a dying HD). So when I got home in the morning, I brought the server down to test the RAM. All tested fine. The HD however, told me to go take a flying fuck!

So I spent several hours attempting to clone the drive, but it was too far gone, and kept crashing on me. Finally at about 5:00pm I gave up and went to sleep. I woke up at about 3:00am and started a clean install. Luckily I was able to mount the old HD, and copy the maildir’s, and important configs, so everything seems to be in order now.

Covad got dispatched out again today. Even after I tried to explain to the dispatcher that my DSL was up, just not at the right speed. So when the d00d showed up this morning, I was like go away, and fix my shit at the DSLAM! With any luck this will be closed out today.

Aside from that, I’m working on a migration process to move serve-you.net to a new datacenter. Meanwhile Steph seems to be dying from ebola or some shit.

zzZzzZzzZzzZzzZzzZzz

Is the night over yet? We have been so busy tonight, that I’ve hardly had a moment to think. I got left with about 25 servers tonight. My team is awesome, so we have knocked out most of them, but I’m sure we’ll still be busy till the moment we leave. Fucking 1st shift here is a waste of space. They don’t get shit done, end up turning over all their servers to 2nd shift, who then get slammed since the majority of new servers pop in the late afternoon. Since they end up with all of 1st shift’s mess on top of their own bullshit, we end up with half of their servers. So it’s a vicious circle. If I leave anything for 1st shift, there’s a pretty good chance that it will still be here for 2nd shift. Hopefully things will get better in a couple weeks when we are making some “adjustments” to the shifts. However, I get screwed twice out of the deal, cuz I lose my best employee, and gain the worst of everyone in this organization. My only hope is that he either quits because he wont want to work for me, or he continues being a fuckup on my shift, so I can promptly boot his ass the fuck out of here.

Okay, nuff bitching. I have more work to do.

Another one bites the dust…

It would appear that I have toasted yet another hard drive. And again, it is on my main machine. I am in the process of trying to salvage the data from the old drive, but I fear that a lot of data got corrupted by fsck when it couldn’t read the blocks. With any luck, it will be okay. I just fucking rebuilt this machine less than two monmths ago!

RE: Server shit.. and general life

Well I finally got around to doing some final tests on the server parts. As I suspected it is infact the mobo. This makes me happy actually. Well, not as happy as I would have been if it were just the CPU, but a lot happier than if the problems were in the power supply, or some other dumb shit in the case, which would be a bitch to return. Since the only reason I got this mobo in the first place was to support old RAM, I think I will just toss it, and get a new one that will support DDR, and I’ll just get new RAM. Eventually. All seems well with the HD in my dev machine, so I think I’ll leave it there for a while. I’m not working on any new projects, so the box wouldn’t be doin anything anyway.

As Steph noted in her journal the other day, we got to spend about 7 hours at the emergency room on Saturday night/Sunday morning. The years of neglecting my teeth are finally catching up to me (well, have been catching up for some time, just finally got to a boiling point). I haven’t been to the dentist in about 10 years, and only went every few years as a kid. I never took good care of my teeth as a kid and through my teens. By the time I got into good habits brushing and such, the damage was well underway.

All of my wisdom teeth have been slowly decaying away. I’ve put off going to the dentist over and over, for a variety of reasons (usually related to money). I got used to the constant pain, and it rarely bothered me, so by the time I had insurance again, and could afford to go, I just kind of forgot about it. Well, the pain hit me hard in the past week. Tylenol was doing nothing for it. Finally Saturday night, I just couldn’t take it anymore so I went to the ER for some pain killers. I have an appointment for a consultation with an oral surgeon on Tuesday. I’m sure when they take X-Rays, and take a look in my mouth, they will want to get the fuckerz extracted ASAP. So with any luck, I will have four less teeth within the next couple weeks.

Other than that, work sucks. I have my resume out to several places right now, and hopefully something will pan out soon. I dream of the day when I am making decent money again, and can actually pay my fucking bills. We have had a lot of good people supporting us through the past year and a half (which have been complete shite). If it wasn’t for them, we’d be living in a ghetto ass apartment again, with none of our animals, and definitely none of our “luxuries”.

Enough bitching.

Server Woes

Well as most of you know my webserver which hosts our personal domains, and some friends decided to take a shit last week. It actually started a couple weeks ago when strange noises began to make their way out of this machine. This is usually one of two things, harddrive, or cpu fan. So I chose the weekend to pull the server from the rack and run it opn to verify which it was, and order a new one on Monday. I pull the machine and sure enough, it’s the CPU Fan. I took a big sigh here, cuz a good CPU Fan is only $20 as opposed to about $100 for a new HD. I had an extra fan lying around, so I decided to swap it out in case this one completely died before my replacement would arrive, then I’d be fucked with a fried CPU (see the irony here?). So I shut the bitch down, swap out the fan, plug her back in, hit the power, and NOTHING. The fucker wont post at all.

After doing the normal diagnostics, I came to the conclusion that it could be one of three things. The CPU, mobo, or power supply. This is some rather old hardware, so I decided why not replace it all, instead of taking chances. So I order a new case w/power supply, CPU, mobo, & fan. Everything arrived yesterday, I load the new case up, power on, and NOTHING. Same problem appears… this essentially new machine wont post. After testing several ways, I come to the conclusion that this is a lost cause, and one of these things is dead again. So I decided to temporarily put stvlive on my development server, until I can get to the bottom of these problems.

So last night I stayed up way too fuckin late manually setting up stv on this other machine. By morning, I had everything working, when I realized that there were a lot of environmental things that I would have to rebuild on this temp server to make stv completely happy. Like the fact that this machine had no mail server setup, and the server stv lives on has a pretty complex setup that relays to my real mail server, So it hits me that I wasted several hours when I should have just pulled the HD and swapped it into this box. So I make the decision to do that this evening.

After talking to a few people about it today, the only conclusion I can come up with is that maybe the RAM was bad, and was frying out the mobo. Since this was the only thing that hadn’t changed, this seems to make sense. However, I came home this evening, and swapped out the drives and booted up my test server with the drive from the webserver, and all was good. So I decided to try putting some of the aforementioned RAM in it, and see what happens. I put one stick in, no problems. Put another in, again no problems. So this shoots down my theory completely.

The plus side is, the test server is now up and running as the webserver with all sites. And since the RAM is working, it’s loaded up, so the difference in speed should be minimal except when it’s doing heavy duty processing since it’s only a duron compared to the athlon in the webserver. My thoughts now are that I got a bum CPU, and it’s just a strange coincidence. I’m gonna send the CPU back to exchange it, and hopefully this saga will end. If not, I’ll probably just curl up in the fetal postition and cry like a baby for a while. But the webserver should be in good enough shape to keep running on my dev server for the time being.

Downtime!!!

I am about to start upgrading the hardware in the firewall. stvlive.com, half-asleep.com, and dollar25.org (including the stream) will all be down during this time. If all goes well (it never does), this should take no more than 30 minutes. If the shit hits the fan, it could be a few hours. I’ll post an update once we’re back.

I have had an absolutely shitty day. I have been getting called and paged all damn day. Once again I had to tell NT Techops how to do THEIR fucking job. It gets old. So I finally start to cool off, and relax since the fucktards have not called me in a few hours, so I try to get some work done. I got the new password authentication system up and running for stv. I put some more ram in the WWW server, things are looking better. So then I try to swap some ram out in the FTP server, this is where my night turns back to shit. The module is being a bitch, and wont seat in the slot. So I notice the module is actually seated, it’s the release tab that is fux0red. I bring the box back up, and get no love. Then, smell smoke, and see smoke. The slot is fried, along with the ram. This doesn’t bother me too much, because the machine came back up with that slot empty, and I really don’t give a shit about that box since nothing really important happens on it. Now I decide to setup a new program that I have been using on another box. I am going from term to term checking that permissions and such match. My laziness finally caught up with me… I hit the up arrow a couple times to reuse a command as usual. Instead of this: la ../cgi-bin/ (la is an alias for ‘ls -la’ on my machines)… I accidentally hit enter on: rm -R ../cgi-bin/. Bye-Bye serve-you scripts! You just can’t even imagine how dumb that makes me feel. I mean, I know this shit happens to the best of admins at one time or another, but it still suckz! So of course, I have back-ups of most of the shit, but I had done a shitload of customizations to part of a script, that had not been backed up yet. Oh well,.. life goes on. I can probably do it better the second time around anyhow. So now I’m calm again, and hopefully these fuckz will let me get a good night sleep.