New backup and restore facility

[Matt]

Now that the Quota Tracker provision is somewhat finished, it's time to proceed onward to the Referral and Backup/Restore Systems. I've expecting to complete the Referral System within the next 1 - 2 weeks, after which time I will begin work exclusively on revamping the existing backup/restore facility to a much more usable and user-friendly interface.

Before we discuss what things to implement or change in this new system, let me give you a bit of background information on how the current system, borrowed from Ensim, works.

Old Backup Process:
When a user requests a backup, two things are created. First an XML file with all of the site's Ensim configuration is generated. This contains information about aliases, disk space, bandwidth, sub-users, sub-user features, and all of the basic Ensim information. Of course this excludes a lot of esprit-specific features and capabilities such as domains added through the "Multiple Domain Manager", custom cronjobs, log profiles, Urchin data, PostgreSQL data, and so on. In other words, if the feature appears in esprit, but not in apnscp, there's a good chance the backup facility won't back it up.

To make matters even worse, when a backup request is made, all files are backed up, including ones that you wouldn't use from a restore. This includes the hardlinked files owned by root. For example, cat, grep, ps, library files under /usr/lib/, all of those, which are dependent upon the server configuration, rather than account, are included. This results in enormous backups. For example, what should be a 25 MB backup file turns into a 500 MB+ backup. Backups come in one flavor, tarball + gzip or tar.gz extension.

Proposed New Backup Process:
Ideally I want to keep an XML configuration similar to Ensim, but add the additional domain configuration, Urchin, log profiles, and so on to make a restore seamless, should the need ever arise for a user to rollback.

Further, the backups will backup just the files that you have created. This would include Ruby gems, your e-mail folders, Urchin profiles, MySQL and PostgreSQL data, your HTML files, user information (e.g. password file, user privileges), and nothing more than what is unique to your site. This helps cut down on a lot of needless junk in the backup and also significantly reduces file backup sizes.

I would like to introduce some other compression options, e.g. zip and bzip2 to managing the backup a little bit more convenient as well.

Aside from those few changes, what other ideas do you want to see in the new backup system?

[dkozinn]

Originally Posted by Matt

Aside from those few changes, what other ideas do you want to see in the new backup system?

I'd like to see an on-server directory of the contents of each backup stored in such a way that I can figure out which backup contains the right file. (I'm assuming that backups would be run and downloaded off the server, as I think they are now; I haven't gone to the old control panel for such a long time that I've forgotten ).

I would like to see the ability to exclude certain files/directories from backup. (e.g., I don't need to back up the photos from Gallery, because if I need to restore them I can just re-upload.)

I'd like to see the ability to run the backup process unattended, with a couple of options: 1) Provide a server/id/password to send the completed backup to (via FTP, SCP, etc.) or 2) Save the backup locally (of course it would count against your quota) for manual retrieval at a later time. (The basic reason for this might be to run a backup before running a cron job that may make significant changes to the environment. Obviously, the information to be modified could/should be backed up by the process anyway, but if the facility exists to do the backups, it's easier than re-inventing the wheel while minimizing the possibility of forgetting to back up something critical.)

[Matt]

All right, taking into consideration your feedback, here is what we have thus far.

• SCP/SFTP, I will hold off on that for the time being and just go with FTP.
• There will be an esprit backup/restore and cPanel restore facility for the initial release.
• I would like to eventually either add a rsync or diff backup engine to transfer/store incremental changes between files.
• Include and exclude filters will follow glob-style matching patterns.
• The configuration options are borrowed from the One-Click framework.

That's all for tonight though.

esprit Backup Facility

[dkozinn]

Looks good to me. What's the "Backup Filter" dropdown at the top do?

[Matt]

It changes the engine used to generate the backup or process a restore.

[cbergy]

I was about to ask how to backup user accounts and email on the server, but I have a feeling this thread will address that and more. How's the backup coming along?

When I check in esprit, the only 'Backups' which jump out at me are for MySQL and Postgre.

[Matt]

Originally Posted by cbergy

I was about to ask how to backup user accounts and email on the server, but I have a feeling this thread will address that and more. How's the backup coming along?

When I check in esprit, the only 'Backups' which jump out at me are for MySQL and Postgre.

Sorry for the delay, but yesterday was Amazing Legal Threat Friday from... mothers... who blog. Yes, sound paradoxical? Don't worry there's nothing right about it -- it was absolutely baseless. The good thing to know from all of this is that I will protect every critical work so long as it falls under fair use. Having said that, backups exist in apnscp, but not esprit. They are coming once the new one-click framework is released, which should have been done this weekend, but due to unforeseen meritless threats that has been pushed back to sometime this week. If you would like though, I can setup a basic backup system to make a tarball of a certain set of directories every x days.

[cbergy]

Oh lovely, that sounds like a great Friday. Good to hear it was more a waste of time than anything worse.

I've got a few migrations to complete, moving some packages from Kernel to Gauss in the near future. Before I jump into it, I was hoping to backup everything (for my client's peace of mind, and my own). No huge rush though, I can definitely wait till sometime this week for the roll-out of backups in esprit.

[Matt]

It will probably be much longer than mid-week since the new backup facility has a few components written, but it's far from ready. Drop me a PM with what you want backed up and when.