Apis Networks

Access Control Panel
Control Panel Login close
Username
Domain
Password
 
Contact Us
create account
Reply
 
Thread Tools Display Modes
  #1  
Old 04-20-2006, 09:36 PM
Matt's Avatar
Matt Matt is offline
Tech Liaison
 
Join Date: Jun 2005
Location: Atlanta, Georgia
Posts: 1,035
Send a message via AIM to Matt Send a message via MSN to Matt
New backup and restore facility

Now that the Quota Tracker provision is somewhat finished, it's time to proceed onward to the Referral and Backup/Restore Systems. I've expecting to complete the Referral System within the next 1 - 2 weeks, after which time I will begin work exclusively on revamping the existing backup/restore facility to a much more usable and user-friendly interface.

Before we discuss what things to implement or change in this new system, let me give you a bit of background information on how the current system, borrowed from Ensim, works.

Old Backup Process:
When a user requests a backup, two things are created. First an XML file with all of the site's Ensim configuration is generated. This contains information about aliases, disk space, bandwidth, sub-users, sub-user features, and all of the basic Ensim information. Of course this excludes a lot of esprit-specific features and capabilities such as domains added through the "Multiple Domain Manager", custom cronjobs, log profiles, Urchin data, PostgreSQL data, and so on. In other words, if the feature appears in esprit, but not in apnscp, there's a good chance the backup facility won't back it up.

To make matters even worse, when a backup request is made, all files are backed up, including ones that you wouldn't use from a restore. This includes the hardlinked files owned by root. For example, cat, grep, ps, library files under /usr/lib/, all of those, which are dependent upon the server configuration, rather than account, are included. This results in enormous backups. For example, what should be a 25 MB backup file turns into a 500 MB+ backup. Backups come in one flavor, tarball + gzip or tar.gz extension.

Proposed New Backup Process:
Ideally I want to keep an XML configuration similar to Ensim, but add the additional domain configuration, Urchin, log profiles, and so on to make a restore seamless, should the need ever arise for a user to rollback.

Further, the backups will backup just the files that you have created. This would include Ruby gems, your e-mail folders, Urchin profiles, MySQL and PostgreSQL data, your HTML files, user information (e.g. password file, user privileges), and nothing more than what is unique to your site. This helps cut down on a lot of needless junk in the backup and also significantly reduces file backup sizes.

I would like to introduce some other compression options, e.g. zip and bzip2 to managing the backup a little bit more convenient as well.

Aside from those few changes, what other ideas do you want to see in the new backup system?
Reply With Quote
  #2  
Old 06-02-2006, 01:28 PM
dkozinn's Avatar
dkozinn dkozinn is offline
Apis Networks Savant
 
Join Date: Nov 2005
Location: River Vale, NJ
Posts: 92
Quote:
Originally Posted by Matt
Aside from those few changes, what other ideas do you want to see in the new backup system?
I'd like to see an on-server directory of the contents of each backup stored in such a way that I can figure out which backup contains the right file. (I'm assuming that backups would be run and downloaded off the server, as I think they are now; I haven't gone to the old control panel for such a long time that I've forgotten ).

I would like to see the ability to exclude certain files/directories from backup. (e.g., I don't need to back up the photos from Gallery, because if I need to restore them I can just re-upload.)

I'd like to see the ability to run the backup process unattended, with a couple of options: 1) Provide a server/id/password to send the completed backup to (via FTP, SCP, etc.) or 2) Save the backup locally (of course it would count against your quota) for manual retrieval at a later time. (The basic reason for this might be to run a backup before running a cron job that may make significant changes to the environment. Obviously, the information to be modified could/should be backed up by the process anyway, but if the facility exists to do the backups, it's easier than re-inventing the wheel while minimizing the possibility of forgetting to back up something critical.)
Reply With Quote
  #3  
Old 06-17-2006, 01:35 AM
Matt's Avatar
Matt Matt is offline
Tech Liaison
 
Join Date: Jun 2005
Location: Atlanta, Georgia
Posts: 1,035
Send a message via AIM to Matt Send a message via MSN to Matt
All right, taking into consideration your feedback, here is what we have thus far.
  • SCP/SFTP, I will hold off on that for the time being and just go with FTP.
  • There will be an esprit backup/restore and cPanel restore facility for the initial release.
  • I would like to eventually either add a rsync or diff backup engine to transfer/store incremental changes between files.
  • Include and exclude filters will follow glob-style matching patterns.
  • The configuration options are borrowed from the One-Click framework.
That's all for tonight though.

esprit Backup Facility
Reply With Quote
  #4  
Old 06-17-2006, 03:59 PM
dkozinn's Avatar
dkozinn dkozinn is offline
Apis Networks Savant
 
Join Date: Nov 2005
Location: River Vale, NJ
Posts: 92
Looks good to me. What's the "Backup Filter" dropdown at the top do?
Reply With Quote
  #5  
Old 06-17-2006, 04:15 PM
Matt's Avatar
Matt Matt is offline
Tech Liaison
 
Join Date: Jun 2005
Location: Atlanta, Georgia
Posts: 1,035
Send a message via AIM to Matt Send a message via MSN to Matt
It changes the engine used to generate the backup or process a restore.
Reply With Quote
  #6  
Old 10-27-2006, 02:19 PM
cbergy's Avatar
cbergy cbergy is offline
Apis Networks Diehard
 
Join Date: Apr 2006
Posts: 101
I was about to ask how to backup user accounts and email on the server, but I have a feeling this thread will address that and more. How's the backup coming along?

When I check in esprit, the only 'Backups' which jump out at me are for MySQL and Postgre.
Reply With Quote
  #7  
Old 10-28-2006, 10:53 PM
Matt's Avatar
Matt Matt is offline
Tech Liaison
 
Join Date: Jun 2005
Location: Atlanta, Georgia
Posts: 1,035
Send a message via AIM to Matt Send a message via MSN to Matt
Quote:
Originally Posted by cbergy
I was about to ask how to backup user accounts and email on the server, but I have a feeling this thread will address that and more. How's the backup coming along?

When I check in esprit, the only 'Backups' which jump out at me are for MySQL and Postgre.
Sorry for the delay, but yesterday was Amazing Legal Threat Friday from... mothers... who blog. Yes, sound paradoxical? Don't worry there's nothing right about it -- it was absolutely baseless. The good thing to know from all of this is that I will protect every critical work so long as it falls under fair use. Having said that, backups exist in apnscp, but not esprit. They are coming once the new one-click framework is released, which should have been done this weekend, but due to unforeseen meritless threats that has been pushed back to sometime this week. If you would like though, I can setup a basic backup system to make a tarball of a certain set of directories every x days.
Reply With Quote
  #8  
Old 10-29-2006, 06:35 PM
cbergy's Avatar
cbergy cbergy is offline
Apis Networks Diehard
 
Join Date: Apr 2006
Posts: 101
Oh lovely, that sounds like a great Friday. Good to hear it was more a waste of time than anything worse.

I've got a few migrations to complete, moving some packages from Kernel to Gauss in the near future. Before I jump into it, I was hoping to backup everything (for my client's peace of mind, and my own). No huge rush though, I can definitely wait till sometime this week for the roll-out of backups in esprit.
Reply With Quote
  #9  
Old 10-30-2006, 11:49 AM
Matt's Avatar
Matt Matt is offline
Tech Liaison
 
Join Date: Jun 2005
Location: Atlanta, Georgia
Posts: 1,035
Send a message via AIM to Matt Send a message via MSN to Matt
It will probably be much longer than mid-week since the new backup facility has a few components written, but it's far from ready. Drop me a PM with what you want backed up and when.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump