Category: InfoSec

Migrations, Painting, and other fun stuff!

Posted on by DizkoDan

It’s been a busy week. I spent most of last week working on getting setup for that big Debian customer. Not only do I have to get a deployment setup for their 40 servers, but I also had to get the Data Center ready for this customer. We previously had about 15 of our blade enclosures on a different network. So I had to recable, and reconfigure all of the switches before I can build any servers over there. Got that all done. Got a working Debian image setup, and even got NIC bonding working on it, in addition to getting some of our standard stuff configured.

In the evenings Weds, Thurs, & Fri, I pressure washed the fence to prep it for painting. We weren’t planning on painting it ourselves, but after I got some quotes, there was no way I was paying. So we went to the ho depot, bought 10 gallons of fence paint, and started painting on Saturday.

After the problems last week, I rescheduled the serve-you.net migration for Saturday night. This time it went a lot better. I ran into a mail problem, and spent 2 hours trying to figure it out. Then all of the sudden I look back at something I thought I checked, and realized permissions were wrong on a file. The first thing I looked at when the problem occurred! After that, it was just random little issues. I finally got to sleep at about 7:00am.

Sunday, I woke up around noon, and got back to the painting. Steph had some dog event in DC, so she didn’t get home until like 6:00, but she helped me finish up the section I was working on. It’s a very time consuming process, so we only got a little bit done this weekend. If anyone wants to help, there will be cold beer and paint for all this weekend!

Server migration time again

Posted on by DizkoDan

Over the past month or so, I have been working on a flurry of updates, reconfigurations, migrations of services, and entire servers for my personal sites, and my hosting company. The personal stuff is always fairly painless. If something breaks, it’s not the end of the world. We lose a little ad revenue, and a few people complain that they can’t reach Stvlive, or QuizMeme. I’m a bit of a masochist in this regard, because I always seem to do 8 million projects at once when I should really be focusing on one. In all, I have done, or am in the process of doing 2 complete server migrations (move from one server to another in a completely different Data Center, new IP space, etc), migration of secondary mail & DNS services for my personal sites to a 3rd party, and massive amounts of hardening across all of my sites/servers.

Most of the personal stuff has gone well, and we even revamped some really old stuff on some of the sites, which makes me happy from an InfoSec stand point. I’m mostly content with how things are running on all of those servers, now it’s just the ongoing issues of cleaning things up that have been around since the late 90’s to make things more secure.

My hosting migration however, makes me lose sleep. I have done this a few times over the years, and it usually goes “okay”, but never without some screwup that keeps me up for hours fixing. The problem isn’t from a lack of planning, or skill. I have been doing this a long time, and I am very knowledgeable about these things. Where the problems occur is usually in the little configuration changes (read: hacks) that have been made on the servers over the years, and have been forgotten about. This server has essentially been upgraded and migrated over and over again since 2001. It’s gone through 3 different FULL RedHat releases starting with RedHat 7.1 (going on a 4th now). I can’t even count the number of Plesk versions, I’m an old school customer, so this server (in it’s original form at my house 7 years ago) started at PSA 2.5, and is getting ready to be Plesk 8.2. So as anyone with any admin experience can imagine, the number of “hacks” that would have been put into place over the years to add support for some unsupported feature, or fix a bug. The irony is, that it’s the old “hacks” that were meant to fix something in the past, that break something on the new.

At this point I have most of the behind the scenes work completed. New server is up and running, and has been (mostly) configured. DNS is going to be the biggest hurdle. The current setup is not so good. All DNS is served from the same server. This isn’t really a big deal, because it’s pretty much an all-in-one server, so if DNS goes down, chances are everything else is down too, so the DNS doesn’t really get you anywhere. However, in a migration situation, having a secondary somewhere else is extremely useful because it isn’t going to change. So when the madness happens when I change my name servers at the registry level, propagation isn’t that big of a deal, because the secondary server is still churning out results. So I want to get this server added for all the domains prior to the move.

I host over 200 domains, and unfortunately they were not all registered through me. That means that the owners of all of those domains need to log into their registrar account, and make modifications to their name servers. This is a very simple task for someone with only a little technical knowledge. All of the registrars have documentation on how to do it. The problem is, getting the domain owners to ACTUALLY MAKE THE CHANGE! I am willing to bet most of my customers don’t even know what a registrar is, let alone which one their domain is registered at. Which is going to equal me doing a shit ton of whois lookups for people to point them in the right direction. And in more than a few cases, I’ll probably just have to obtain their login info, and make the change for them. I am actually having some new flash demos made up right now to show people how to login to the various registrars, and make this change, so hopefully that will help a bit. The plus side is, most registrars don’t require an IP address for name servers, so when I actually re-IP my name servers, there shouldn’t need to be any changes on the end user side.

I’m rambling, and I’m sure this is way more information than most people who read my blog care about, but that’s what a blog is for right?

First real post in 2 years

Posted on by DizkoDan

So what’s been happening in my life?

We bought a new house in Ashburn in March. We moved in a few months before that, but were waiting to sell our townhouse. The house is awesome! It’s on about a quarter of an acre, in a great neo-traditional neighborhood. We’ve done a ton to the house already, and it looks great!

House

Shortly after we moved here, India our Rotti girl lost it completely. After several attempts to reverse her behavior, we had to let her go. Shortly after, we began looking for a new puppy. We wanted to find a dog that would be completely submissive to China, as well as just being an all around awesome dog! So we picked this little guy from a breeder.

Benny The Golden

A little over a year ago, I quit my job at MITRE, and went to work for a “startup” called OpSource. I was hired as the Sr. Linux Engineer for Data Center Operations. These days, I end up doing more management, than engineering, but it’s still a good gig. I mostly work from home, or any other random place with wifi. I’m starting to work in InfoSec now on top of my normal workload, with the intentions of switching over full time as soon as they can afford to replace me.

I could ramble on about the past 2 years forever, but I think I’ll take it one post at a time for now. Commenting is now open, but you must register.

Posted on by DizkoDan

On the car front, I decided to go with the Infiniti. However, finding the exact one I want, on the east coast is a much bigger challenge than I anticipated. Apparently, Infiniti made the decision to send mostly AWD 05 G35 sedans to the notheast, because thats where the market demand is right now. And I would have to agree to an extent. I would get an AWD, if I didn’t just get the outback for Steph. That was one of the main reasons that we got that car. However, I don’t want to spend the extra money for an AWD car, that’s also gonna get even worse gas mileage than the already less than steller mileage on the RWD G35.

So I went to the dealer near me on Monday to try and get a deal setup for my car. It turns out that all they have is 5 05’s in stock, all AWD, and they wont be getting more until March! I try to get them to order me the car I want, or have it transferred. They try to sell me on an 04. I’m like screw you hippies! I’m out. So I start calling all of the infiniti dealers within a 150 mile radius to try and find the exact car I’m looking for. The dealer in Alexandria tells me that they have located the car in Miami, and can try and get it transferred for a fee. I’m like HELL YEAH! So I waited around there yesterday for a couple of hours waiting for the GSM in Miami to call the GSM in Alexandria to confirm that they will transfer the car. After several attempts to get ahold of him, the GSM at Alexandria tells me to just go home, and he’ll call me as soon as he hears from him today.

So keep your fingers crossed. If this falls through, the only other location they found that had the color/specs I want is in TX, which would cost a lot more to transfer. IF it does work out, I’ll go in today and give em a deposit and the car will go out on a flatbed today/tomorrow and should be here within 24 hours. The waiting is killing me!


I’ve been having a ton of problems with my servers lately. It’s getting to the point where I am constantly proactively monitoring them, in addition to my automated monitoring which pages me when theres a problem. People can’t write secure code to save their lives, so I’m in a battle trying to keep the script kiddiez off my hosting server. I keep trying to deploy a secure kernel patch that should help mitigate the problems, but I keep running into issues where certain necessary services wont start up properly. I have a maintenance window scheduled for tomorrow night to do some upgrades, so one way or another, I’m gonna make this shit work right. Otherwise, I’m gonna have to go through all of my customers code to find their shitty PHP upload script that is allowing the malicious code to get in.

It’s shit like this that makes me want to sell off my business.

All telcos are ass fuckz!

Posted on by DizkoDan

So last night was brutal. Sales got a bug up their asses cuz it was end of the month and they hadn’t sold shit all month, so they made up for it in 12 hours. I had 30 servers waiting for me when I got to work last night. By the time I left at almost 10 this morning, there were only 4 left. My team kickz ass!

So I came home, ate a bagel, and started working on some shit before I go to bed, and all of the sudden the interweb went *p00f*! I knew immediately that the cock bitez at BTN shut off my T1 early! Fuckerz! I was not fully prepared to switch over to the new network yet, so I had to make some quick changes to the new firewall, and got the important machines back online. I still have a shitload of rulez to add to the firewall, but I’m too fuckin tired, and there is really nothing important here anymore except my monitoring machine, a name server (which I now have a secondary offsite), and the mail server.

So now I’m gonna crash.

Downtime!!!

Posted on by DizkoDan

I am about to start upgrading the hardware in the firewall. stvlive.com, half-asleep.com, and dollar25.org (including the stream) will all be down during this time. If all goes well (it never does), this should take no more than 30 minutes. If the shit hits the fan, it could be a few hours. I’ll post an update once we’re back.

Posted on by DizkoDan

Well,.. all the machines are behind the firewall, and so far I only have a couple minor problems to be worked out. It was much less painful than I expected. Anyhow, It’s late. I must get some sleep.

Posted on by DizkoDan

Well,.. I made it into work in 35 minutes door to door. This weekend was good, just went by way too fast (dont they always). Saturday night, Steph, Rudy, and I went out to thai lemon grass for dinner. Food was great, I miss that place. Afterwards, Rudy and I got started on the new firewall config. I stayed up till about 4am workin on it, then finally forced myself to go to bed. Woke up around 11 yesterday, and got right back on it. I put my laptop behind the firewall, and had no problems, so I decided to put Steph’s powerbook behind to test. Her powerbook is connected to the net via an airport. For those of you not familiar with what an airport actually does, it’s basicly a router for a wireless network. Unfortunately because of this, it must have it’s own arp table that you can’t get to, because it was being a bitch everytime I’d make a change. I spent the entire day fighting with her stupid mac hardware, which would just randomly stop communicating with the router. Finally around 9pm, a friend gave me the idea of putting it on a separate hub off of the switch, to see if it would clean up. That did the trick, it was workin great, even figured out a way to get Steph’s irc program through the firewall. So tonight, I will put the servers behind and pray that all is good. Today we start the d00gie obedience classes. It’s an 8 week class that meets every monday. It should be fun. All for now.