It’s rare that I’m moved to write a blog post about something that happens to me while I’m at work. This is mostly because I assume everybody in the world who uses Plone has had the same problems before. Once in a while though you come across something that it seems like you really should be sharing with the world.
Zeoserver will continue to ‘function’ normally even without a Data.fs and regardless of zeoclient restarts.
This happened to us recently. Now I’m sure that this is not something that you will come across very frequently, if at all. However, if you do have this happen to you, the consequences could be disastrous.
So how did you find that out and what triggered it?
Here’s a hypothetical situation for you – let’s imagine that you are developing on a site and you have made some changes to some of the products that you use in that site. It comes time to try installing these new products, so you make a backup of your Data.fs (sensible, well done!) and then you re-buildout, and restart your zeoclient instances one by one, to avoid downtime.
You reinstall your products and add some data into the site, looking out for problems. UH-OH! Something unexpected has happened and you need to roll back!
You roll-back your buildout, replace your Data.fs with that backup you cleverly made earlier, and restart your zeoclient instances. JOY! Your site is back to how it was before, nothing is broken and everything is totally fine. Phew! That was close.
So you carry on working, data gets entered into your site and life goes on. Now at some point in the future, you hit a situation which requires you to restart your ENTIRE site. Zeoclients and Zeoserver. You do so, and when you restart your instances, suddenly you have been flung back in time! All the information you entered into your site since that rollback has disappeared! WTF?!
There were no clues in the system that this was going to happen:
- Your transaction log / undo history were all running fine and recording what had happened
- You weren’t getting any errors whenever you saved data to the system
- You zeoclients were restarting fine
Maybe. Maybe, if you were running backups using repozo you might have noticed that the diffs were not as you might expect, but that system works, it’s just there in case of emergency right? Well all that time, your repozo has been backing up the restored backup of the database, but your zeoserver has been storing everything to a patch of hard drive which was being held by the now non-existent Data.fs that you replaced when rolling back.
What can I do?
Nothing. Unless you get very lucky, scraping your hard drive, you wont recover that information. You’ll have to put it all in again.
The Lesson?
If you ever have to replace your Data.fs, restart your zeoserver as soon as you do so.
Here’s a picture we drew:
A diagram of a disappearing Data.fs
