Server and Blog back up – lessons learned…

Well, I think the server is back up and running correctly. So, the news…

Mail, the website and the blog are all back online, and I apologise to the blog subscribers that have been bombarded with old postings. This couldn’t be helped, there was no way to avoid it (well, there was, but it wasn’t really possible and more prone to error than just recreating the posts).

The server problem: At first I thought the server had been hacked as there seemed to be a very general minor error. However, as I tried to correct the problem (it first appeared in the mail system) the problem ‘crept’ in to the web server, the database, and finally in to the system.

A complete system rebuild didn’t solve the problem, and there were a number of strange disk errors reported (although the disk verified OK). This made me think there was a general disk failure. A new disk was put in, but this didn’t solve the problem. All very strange.

Finally, the RAM was replaced, and everything was fine. Disks were recognised and the operating system could be installed.

Once this was solved I thought the rest would be dead easy as the system could restored from the back-up. However, not so fast….

First, the local back-up wouldn’t mount. This suggested that the problem with the server had ‘propagated’ to the back-up system (which was just an attached synchronized disk). No problem, use the off-site back-up. Well, that wasn’t as ‘rock-solid’ as I had thought. I would guess it was 99% complete in terms of data, however, the ‘structure’ was not preserved so the connections between some data (such as blog posts and associated images) was not complete. Hence, it has taken longer to get things back up and running (and some images are still not back in the right place).

All this has taken the best part of a week, so a week of development work has been lost… and a lot can happen in a week, such as the release of two new iPhone Apps….