Work is really starting to get to me again.  It happens periodically…  A bunch of stuff breaks at once, nobody else can fix it, and it all winds up in my lap.  Suddenly I find myself without enough hours in the day to get everything done.

I’ve got our in-house Exchange Server issues, which I’m going to have to simply abandon today.  I can’t keep working on in-house stuff because it doesn’t earn us any money.  I’m just going to have to call it “good enough” and move on to our clients - which I hate to do.

But in addition to the Exchange issues we have other problems in-house, which these recent Exchange issues have hilighted.  Our Domain is not happy.  There’s constant replication errors.  There’s far more machines showing up in ADUC than we actually have…  DNS is just a mess of old entries and aliases…  We’ve got a ton of users and mailboxes that may or may not actually be in use anymore…  And now I’m wondering what other issues, like the failure to move our Public Folders, may be lurking in my future.

We’ve got a client who is replacing an ancient Linksys router with a new Cisco.

We’ve got a client whose Cisco is malfunctioning and needs to be fixed.

We’ve got another client whose Cisco is acting odd, and may need to be fixed.

Another client has been working on setting up a pile of VPNs between several branch offices and a few remote users, and it just is not working the way it should.  Contant problems, connections dropping, slow connection speed…  I suspect their hardware isn’t up to the task and will need replacement.

We did a server install last week, but were unable to complete all the software setup on-site, so I need to finish that up remotely.

We’ve got a new backup system (tape drive & software) coming in for a local medical office, which will have to be installed after hours some night.

We’ve got a server reload scheduled for this weekend…  We can’t touch their server until 4:30 Friday afternoon.  Which means I’m going to spend most of the weekend reloading it.

And we’ve already got another server reload scheduled for next weekend…  Another place where we can’t touch their server until Friday afternoon, and I’ll wind up spending most of the weekend reloading it.

So I’m looking at a good two weeks of frantically trying to keep my head above water, without even a break on the weekends.  Fun!

Our Exchange 2003 server has been having issues for a while now.  It hasn’t worked right in a year or more.  The drives are filling up and we’re rapidly running out of room.  There are some software oddities that have accrued over the years, and Exchange itself doesn’t work as well as it should.  The remote agent for our backup software has trouble grabbing data.  It falls over periodically for absolutely no reason.  You get the idea…

I’ve been pushing to replace this server for almost a year now.  About six months back I finally got approval to buy some new hardware and load it up with Server 2003 x64 and Exchange 2007 - which I did.  And I was supposed to go ahead and migrate all the data over to the new server shortly after it was built…  But that’s where things fell apart.

They weren’t willing to let me take down our existing Exchange server long enough to migrate the data to the new one.  Nobody could part with their email or calendar for the time required.  So the migration got delayed…  And delayed…  And then put on the back burner…  Somewhere along the line our new server was shut down and shoved into a corner.

Recently our Exchange 2003 server has been having even more issues than normal.  It is falling over on a daily basis.  People can’t send or receive mail.  The public folders are unavailable half the time.  Backups are failing more often than not.  And now it is suddenly important that we get a working Exchange server up and running.

Which means that in addition to all the other work that was scheduled for today, I’m trying to get an Exchange 2007 server up and running.

And because this server has been offline for over three months, replication between it and the rest of the domain has been shut down.  So I can’t just fire it up and start migrating information.

I did a quick search for information…  A little troubleshooting…  And while it does seem to be possible to convince the servers to start replicating again, it also doesn’t appear to be a very good idea.  It looks like the recommended way to get things working again is to un-install and then re-install Active Directory on the server.  And since it’s got Exchange installed simply un-installing Active Directory is not going to work very well.

So I’m likely looking at a complete reformat & re-install of the entire OS, from scratch.  Which means we have to locate the media we used to install it in the first place - not an easy thing to do around here.  Discs seem to vanish into thin air.  And then all the updates will need to be downloaded and installed again.  All of which is supposed to somehow be squeezed in between jobs in an already short and busy week.

Yeah…  Fun…

I swear…  It’ll be a miracle if I make it ’til morning without a heart attack.

So, I got the best backup I could, and then I kicked off the MS SQL upgrade.  The software that runs on top of this database is flaky to say the absolute least, so my confidence in the success of this endeaver was not high.  I was watching that progress bar like it was a fuse slowly burning down.  I was waiting for the bomb to go off at any moment.  I thoroughly expect some kind of catastrophic error to occur.  Maybe one of the remaining HDDs would simply melt.

Well, the MS SQL upgrade takes some time, and I got bored.  So I wandered off to IM my wife and surf the web.  Then the server made on of those awful error beeps.  The kind of default system beep that usually means you’ve just done something very stupid, or lost a lot of data, or maybe both.  My blood froze, my heart sank.  I thought for certain that beep was heralding the destruction of my weekend.  I thought for certain I’d be up all night long trying to rebuild a dead database.

Turns out the installer just wanted disc 2.

The installation actually completed without any errors.  The tables all showed up in the management screen, all the jobs/agents/users/security/whatever looked like it should.  I was able to fire up the software and log in as the test user.  Everything appears to have gone fairly smoothly.

I also got the backup straightened out, and fixed their antivirus, and replaced the failed drive.  All without incident.

I left there around 9:45.  I have to cross some train tracks on the way home.  And just as I was driving across them I heard a train whistle…  I looked in the direction it was coming from and saw a train’s headlight about a block away.  Scared the hell out of me.  I thought for certain I was about to die.

The train wasn’t moving terribly fast though, and I had plenty of time to get across the tracks safely.  Still, it was a horrifying experience.

That’s three terrific scares I’ve had today.  Well above my normal quota.

I’m still at work, and it doesn’t look like I’ll be going home anytime soon…

There’s a medical office that we support.  They’re open every single day of the week.  They’re seeing patients from about 8:00 in the morning until 7:00 or so at night.  Every day.  So downtime is nearly impossible to schedule.  Which means that things kind of pile up and when we can schedule downtime, there’s a lot of work to be done.

So, they’ve got a HDD that’s failing, and we need to replace it.  We need to upgrade them to MS SQL 2005, so that they can upgrade their software, so that they’re compliant with some new registry or law or something.  Their backup has been doing odd things.  Their antivirus has been doing odd things.  They’ve been running out of space on their C: drive.  All sorts of stuff to do.

And tonight was the only time we could get downtime scheduled.  So I showed up at 6:00…  Waited for the last patient to leave…  Waited for the doctors to all go home…  And started in on their server.

Now, I should tell you that I’ve been dreading this job all week long.  Normally I don’t mind late night or weekend jobs…  Normally I don’t worry too much about working over here…  But, for some reason, I’ve been dreading this job.  Just a feeling that something wasn’t right.

Now I know why.

One of the first things I wanted to do, before touching anything else, is get a good backup of the whole system - in case anything breaks along the way.  Turns out their tapes are way over capacity.  They’ve got a DAT72 drive which typically holds about 60 gigs or so…  And they’ve got about 120 gigs of data.  So I can’t get a backup of the whole system.

The next problem is the speed of their backup.  I guess I’ve gotten spoiled by all our clients with LTO drives…  But I was amazed to see that even after I pared down their backup to the bare essentials it would still take about 5 hours to run.  There’s no way I’m going to be hanging around here until midnight.

So I’m dumping their databases to my USB HDD and sincerely hoping that nothing horrible happens.  With the way their backup has been acting lately, and the capacity issues, I’m not sure how much we’d be able to restore from a tape should the absolute worst happen.  And when you’re swapping out hard drives, even in a RAID, the absolute worst is entirely too possible.

Had some kind of weird, horrible dream last night.  Something about a killer disease/organism invading the ISS, and then the Space Shuttle, and then threatening to invade Earth.  Reminded me a bit of Dead Space.  Ordinarily that might have been a fun dream…  But something about the way it all was put together was absolutely horrific.  Very unsettling.

Woke up this morning with a raging headache.  And after that dream I was briefly convinced that I was infected with that alien disease and I was going to turn into a monster.  Great way to ruin the morning.

I popped several Excedrin and then hopped in the shower.  Shaved, and then washed my face with peppermint oil Castile soap.  I probably should have remembered what happened last time I tried that…  But I wasn’t thinking too clearly at that time…  Let me just say that the sensation of peppermint oil on a freshly-shaven face is a very effective way to wake yourself up in the morning.

At work, the phone immediately started ringing.  Assorted issues with various networks and servers around town.  Already scrambling to get everything done.  And on my first call of the morning I managed to bash my head on the sharp corner of some metal shelving.  Immediately gave myself a new headache to replace the one I started the day with.