Since setting up this site, I’ve been researching, investigating and trying to implement various methods of disaster recovery in case the inevitable happens and I suffer either a hardware or software failure. Despite this looming possibility, I still haven’t managed to find a set up which works for my particular set up, in the way that I’d like it to. Today, I discovered that I really need to get my head around this problem once and for all…
In this post:
- In the beginning…
- A dive into web history…
- Taking back control
- It just works.
- A ten minute job
- What’s the learning from this experience?
In the beginning…
It would be an understatement to say that the World Wide Web has changed a lot since 1997, the year when I created my first, incredibly embarrassing website. In that era of the web, you would write some HTML on your computer, log in to the web space via FTP that had been provided by your ISP, copy the updated files across and off you go, your presence on the web was up and running. Within minutes you had a Comic Sans filled page, with a garish background, probably a midi file playing in the background and a scrolling message. Halcyon days.
A dive into web history…
If you’re not old enough to remember 1997 or have simply forgotten what it was like back then, allow me to explain or to remind you. The web was a really rather innocent place with open chat rooms on services like Yahoo, everyone had a personal or family homepage telling you about their life for no reason other than we all thought that’s what we should be doing and that’s what the web was for and your parents had no idea just how dangerous the internet could be for young people, or what they were getting up to.
You connected using your telephone line, heard those screeches and beeps that have since become a meme and browsed at a blistering 56kbps, or 5kb/s download speed. To put that in perspective, a picture on your phone is around 3MB, on a 1990’s internet connection, it’d take just over 10 minutes on a good day to download that picture – and you think Snapchat is slow sometimes.
If you were really unlucky, then you connected through AOL (America Online), a company notorious for having created so much plastic waste by spamming letterboxes, magazines and anywhere else they could find with free CD’s, they generated a bigger carbon footprint than Shell Oil.
Their idea of user friendly was to greet you with the voice of Joanna Lumley every time you logged in.
God help you if you’d left your PC speakers on loud and forgotten about it, only to be cut off from the internet at which point Joanna suddenly screams “GOODBYE” at you in her posh voice whilst you jump out of your skin.
I used to love writing HTML even though I was horrendous at design and always struggled to make anything which looked even remotely nice. For a few years I plodded along, updating my personal homepage in Edit Plus (free with a PC magazine), uploading updates and manually emailing people to tell them it had changed. Ultimately, times changed and “Web 2.0” came along and this meant dynamic websites.
Very quickly creating websites went from being so simple that anyone could make one, to being much more complicated and requiring a proper server for them to work – your free ISP webspace wasn’t going to cut it any more. At this time many personal websites started to drop off the web as their owners abandoned them, couldn’t keep up with how to create modern websites or simply lost access to their webspace.
The joy of the early web was the sense of community created by so many people sharing their personal interests in a public forum. Most of these sites existed because the barrier for entry was so low and the cost was effectively zero. The idea of paying for a website or server side abilities was something only businesses had to worry about.
Suddenly, this all changed. If you wanted a website you needed a server and this just wasn’t practical in a home environment. Computers were not energy efficient in the early 2000’s, they were large, loud and hungry so it just didn’t make sense that you’d leave a machine on 24/7 for the sake of a personal site. The alternative was a free site like a blog, which seemed like a great idea at the time, but eventually you realised you were being sucked into services owned by Google and other companies.
If you wanted to move your content it became extremely difficult. I have an archive of a blog I ran from 2003-2007 and it is next to useless. Extracting the content from it would involve writing a program to parse the XML and it just isn’t worth it.
In 2009 I began an educational website and chose to run it on Weebly for one simple reason – convenience. The drag and drop interface meant I could just smash out content and not have to care about page layouts, themes and so forth. It worked absolutely fine for many, many years but then it started to become unmanageable. When big updates or structural changes were needed I decided to migrate it to WordPress, or at least I thought I would.
Despite a great deal of tinkering, it just refused to migrate sensibly. You can’t extract your site from their service to run it elsewhere. Logically, why would they let you? They don’t want you to be able to take your business elsewhere. This is a huge, huge problem with the modern web and it is a real shame that the once open door to your own space on the web is now almost closed for the majority of people unless they want to lose control.
Taking back control
I missed having a website for many years and many times looked into creating something but usually concluded that I couldn’t for one reason or another – be it hardware, time or simply knowledge of what I was doing. More recently, I decided that if I was going to do it, now was the time and I needed to have complete ownership and control of my site. That meant creating my own web server.
My priority was low power consumption and silent running. I live in a house the size of a shoebox and due to space and networking constraints the only place I could put a machine completely out of the way was behind the sofa. I considered a Raspberry Pi, they use next to no energy and are silent. Lots of websites are actually hosted on these machines, but the downsides are too many – they run on memory cards and they like to die if they get hammered, the Pi isn’t a particularly powerful machine so performance isn’t great and they use awful flimsy micro-usb power connectors that have a habit of breaking.
After some research and popping my eyes back in their sockets having seen the prices people were charging for old knackers of computers, I settled on this:
The picture doesn’t do the ThinkCentre justice, it is the embodiment of the word tiny. It’s got a perfectly powerful processor for a little web server, runs silently and consumes very little power – perfect for my needs.
It initially took some messing around to get it working as it was refusing to boot certain types of operating system. I updated the BIOS/UEFI but only after discovering the instructions on the Lenovo site are wrong and I had to figure it out for myself, but eventually I had a machine that could boot and install Ubuntu.
Performance was good but loading and reboot times were so slow. I put this down to the fact it had a hard drive and not an SSD. This was to be a mistake later. I also upgraded the RAM to give the server a bit more headroom and that did seem to improve things slightly.
I spent a long time learning how to set up, secure and configure a Linux based web server and I have learned an absurd amount along the way. The only problem is that I didn’t document what I did and consequently I have absolutely no idea of the steps I’d have to take to replicate the configuration I have now should I need to. It quickly became apparent that I needed to create a backup should the worst happen.
It just works.
We are all used to technology that just does what it is meant to do. The times when you needed a reasonable level of technical competence to use computing equipment are now a distant memory for most every day users. We’ve truly made massive strides towards abstracting away the underlying complexity and just letting people get on with their lives. This is a good thing, but it is also an absolute nightmare when you scratch below the surface or have to start diagnosing problems. As soon as you step outside the normal, expected way of working, you find yourself thrown back 20 years.
I’d started to notice the odd slow down here and there, the occasional pause or need to reboot the web server. This isn’t normal, especially for a Linux machine – it wouldn’t be unusual to not shut down or restart for months on end and it would happily carry on working. I decided to at least clone the hard drive to an image just in case. I cracked out Clonezilla (it is superb, free and works) and rebooted the machine.
The backup over my network to a NAS worked really well at first, it was plain sailing. Then, during the verify stage it really threw a paddy. Initially it told me the image was broken, then it just started getting slower and slower until it would’ve taken several hours to just verify a small backup. I checked the size of the backup – 7.5gb. The server has 23gb used. Something was amiss.
I tried a second clone of the drive and this time got an image of 5gb. Clonezilla reported it had finished and then promptly crashed. At that point I had no idea if I had a working backup or not, but I was erring on the side of not. The best thing to do in this circumstance is to replace the drive entirely, so I bought an SSD from Amazon and waited a day for it to turn up.
A ten minute job
Cloning a hard drive in 2022 should take a matter of minutes. You plug both drives in to either a dock or a suitable PC, load up the software of your choice, tick one drive as source and the other as destination and off it goes. I have one piece of advice for you, dear reader, and that is – any time you need to do a hardware job with a quick turn around, what should take minutes will actually take you hours or even days for no explicable reason.
I made just this mistake today as I shut down the server, whipped the drive out and thought to myself “should only take a few minutes and we’ll be back up and running.” How wrong I was.
Things started badly almost instantly. I plugged the drives into my desktop PC and one wasn’t recognised, I think because of a SATA port that has just decided not to work any longer. Having rebooted a few times to fix that, I installed Macrium Reflect which is a well regarded disk cloning and backup application. Feeling confident it would all “just work” I clicked through the clone dialogues and pressed “go.”
Nothing happened.
I left it for 20 minutes, it just sat there on 0%. No error messages, no progress, nothing at all.
Fair enough, I thought, and installed Acronis True Image – same idea, different developer. Well, I say installed, it too decided to get half way and then just sit there failing to install. I was beginning to question how happy my desktop was, it should be fine as it’s basically new and never gets used. I hit the reset button as nothing was happening.
Windows 11 refused to boot.
Its times like these I genuinely dislike computing. I used to enjoy fixing computers and diagnosing Windows issues in the 90’s and 2000’s, now I cannot be bothered – not when there are children running around and other priorities that need my attention more than yet another Windows install.
The thing with Windows is that when something goes wrong it’s a horrible experience. Error messages are useless and it’ll happily go round in circles, failing in exactly the same way. It went through the automated repair process, then told me it couldn’t repair anything and would I like to try a reboot instead? Long story short, eventually, after about an hour, it suddenly came back to life.
I reinstalled Macrium Reflect and this time instead of just hanging it actually gave me an error “Cannot initialise GPT partition.” At this point, I gave up and went down the only route I could think of – try Clonezilla.
Why I didn’t just do this in the first place, I don’t know. In the space of 20 minutes I’d created a USB Clonezilla drive, booted, set the clone up and completed without any errors. I couldn’t believe my eyes. Tentatively I screwed the new SSD into the server, put it all back together and turned it on.
It worked.
Not only did it work, but it was now unbelievably fast. The time taken to boot the server now is around 10-15 seconds from cold. That’s insane performance from something so old and cheap. In all, this site had been down about 3-4 hours for a job that should’ve taken 30 mins to 1 hour. That isn’t great when you’re trying to build an audience. People don’t tend to come back when they try to visit your site and its stone dead.
What’s the learning from this experience?
I could just conclude that Windows drives me round the bend and I should have known better, but that’s not the real story here. I’m fairly convinced my desktop machine has a hardware issue that needs diagnosing when I’ve nothing better to do.
The real learning here is the importance of not only backup, but documentation. If I’d recorded the steps I took to set up my server, I’d be in a position where I could just recreate it on any other hardware, reinstall my site backup and get going again. I hugely regret not taking more care to leave a trail of breadcrumbs to follow in future.
Secondly, I’ve learned that it’s important to do your homework before diving in with both feet and not to have to learn on the fly as you’re in the middle of dealing with a stressful situation at the same time. I should have checked the health of the hard drive in the server before even starting to create and add content to a website. In hindsight, I haven’t a clue why I was so stupid as to not just buy a new drive in the beginning and set it all up together. We live and learn.
Finally, I have learned about a wonderful Linux utility called Timeshift. This tool allows you to really simply set up automatic backup jobs to an external drive that will allow you to completely restore a linux install to a previous state, even from a live USB install. That is absolute gold. My only problem is it won’t write to my NAS and I don’t have an external drive at the moment to use as a backup drive…
Technology. There’s always something.
Share this post:
