The bumpy road of upgrading 3 servers from Ubuntu 9.10 to 11.10.
The challenge
Three of the servers used for welcome.in needed to be upgraded badly. All three servers were running Ubuntu 9.10. (You can find out your Ubuntu version number with: lsb_release -a). The current version of Ubuntu server is, at this point in time, of course Ubuntu 11.10. The problem with running an older version is that fewer and fewer applications are (properly) supported or just don’t work at all.
First step
We start off by backing up everything on these servers. Back-ups are usually a good idea, however, in this case, even more needed. This upgrade is going to update the server from 9.10 to 11.10, that is quite the update isn’t it? 2 years worth of updates. The underlaying problem is these servers run as a system. The web server utilizes both database servers through a MySQL proxy. This proxy separates the MySQL read and write requests and forwards them to the appropriate server. It took a lot of effort and time to get this system up and running. The problem is, I’m currently running version 0.7.0 of the proxy software. The current version in the newer operating systems is 0.8.2, this might not seem to far away but most of the programming was re-done on this version. So I’ll have to get it working again with this new version.
Backup
When backing up the data from a server it’s often a good idea to not forget things. Forgetting to backup things can (and probably at some point will) get you into trouble. So when you backup a (Linux) server don’t forget to backup:
- You home folder
- The /etc/ folder (configs)
- Databases on the server
- Your web directory (/var/www/ on Ubuntu)
- Other subfolders in /var and /usr
- (These wont always contain useful data but some programs place there settings in there)
- Parameters for how you run your scripts (there not always physically saved on the server, so file backups won’t cover that
- Cron jobs
- Other self made scripts
Upgrades
First, we start by stopping apache. We don’t want visitors on the website doing things while your preforming maintenance, so: sudo /etc/init.d/apache2 stop. Alright, let’s see, our Varnish proxy now gives a sort of soft error page with a 503 service unavailable. This means the server is overloaded or in maintenance. After some final backup checks I start BYOBU (formally known as screen). This will help in case I disconnect during the upgrade, my session will be saved within BYOBU. This upgrade is, however, a terrible waste of the uptime counter: “up 391 days, 20:29, 1 user, load average: 0.19, 0.15, 0.06″.
Well, here goes nothing (and everything): sudo do-release-upgrade
I then realized that its going to just get the LTS versions so I edited /etc/update-manager/release-upgrades and set Prompt=normal So, back to the upgrading, it is going to upgrade to 10.04 first.
Do you want to start the upgrade?
3 packages are going to be removed. 45 new packages are going to be
installed. 388 packages are going to be upgraded.You have to download a total of 197M. This download will take about
25 minutes with a 1Mbit DSL connection and about 7 hours with a 56k
modem.
*sigh* Here we go… Upgrade time! … 1 hour later…. Well, that seemed to have gone smoothly.. now for the scariest part..
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select ‘y’ the system will be restarted.Continue [yN]
Guess what? it came back up! Yay. However, this is not the end, yet. Now we’re at version 10.04. This means we need another upgrade to get all the way to where we need to be. So here we go again. Problem was though: Checking for a new Ubuntu release, No new release found. What the hell? So I checked the status of the the /etc/update-manager/release-upgrades and guess what? It’s set to LTS again. So we fix this and upgrade it again. Upgrade done, Second reboot.
The whole process is the same for the 2 database servers, Update, Reboot, and repeat.
But then
That is the point where things started to go wrong. I checked about halfway into the upgrades (so after the 3rd system upgrade) if everything worked. And everything was fine. I could visit the website so the website and the database servers were up and running. However, when I was doing the final upgrade on my first database server from 11.04 to 11.10 it started to act up. At first I got some error messages. It couldn’t find some of the repositories. This problem remained after restarting my BYOBU session. I figured I would leave it be for a sec and started testing the webserver. At this point it should have been up and running. But when I visited the website I got a: “website not found” error, Thats strange..
Varnish
Varnish didn’t want to start for some reason. After checking the logs I found out that one of the lines in the configuration file was deprecated. So I commented out that line, and the next line with an error popped up. It seems that going from version 2.x of Varnish to 3.x you need to re-write your config file. So for now I will have to remove Varnish (leave it off) and just start Apache.
Apache
And of course, according to Murphy’s law, Apache didn’t want to start either. The error message it gave me:
No such file or directory: could not open transfer log file /etc/apache2/${APACHE_LOG_DIR}/other_vhosts_access.log.
So I went into the apache folder to find out what was wrong. Checked out the envvars files, my own file as well as the one that came with the update (the .dist file). And I figured: “ah what the hell” might as well use the Dist version. So I swapped the two, and tada!: Apache worked again.
MySQL-Proxy
Then the next problem showed its ugly head: MySQL-Proxy. When I did my halfway check, MySQL-Proxy was definitely still working. However, after the final update, Ubuntu updated it to version 0.8. And again, like Varnish, most of my configuration wasn’t supported anymore. To make things worse: one of the database servers was at this point completely unresponsive. I tweaked the settings so that only the working database server would be used. The whole system was designed to be able to run on just one of the two database servers. So, the website is up and running again and it works just fine.
Conclusion
The moral of this story is, updating a server shouldn’t be done unless necessary (security updates excluded of course). It often breaks software(configurations) when you go to a newer version (from 2.x to 3.x for instance). On the other hand it’s not good to be running an OS that is a couple of years old, some packages just won’t be supported anymore (the reason why I upgraded). The most important thing when you plan an update like this is to think about it before you act. Start off with full backups (including configuration files). Then form a contingency plan, in case things go pear shaped. In my case, I had plans to run on 1 database server or even without one if really needed. Last piece of advice: don’t forget anything! During maintenance we found out there was another domain that also ran on the webserver. So that one went offline without notice *cough* oops! Luckily, it was one of our own domains and it didn’t have a lot of impact.