Difference between revisions of "Server backup and restore"

From Biowikifarm Metawiki
Jump to: navigation, search
(Created page with '==Strategies== Wikis: Both the media folders and mysql databases need regular backup and validation of restore. We can run a backup to a separate disk, which is mounted at /mnt...')
 
(→‎Settings backup: some adjustment to the settings backup (wiki settings now included))
 
(9 intermediate revisions by 4 users not shown)
Line 2: Line 2:
  
 
Wikis: Both the media folders and mysql databases need regular backup and validation of restore.  
 
Wikis: Both the media folders and mysql databases need regular backup and validation of restore.  
−
We can run a backup to a separate disk, which is mounted at /mnt/dump and mapped; with /var/backups mapped to /mnt/dump/var/backups. Thus writing a backup to /var/backups will end up on this disk.
+
We can run a backup to a separate disk, which is mounted at /mnt/backup/ and is also linked from /var/backups. Thus writing a backup to /var/backups will end up on this disk.  
  
−
Backup scripts are currently stored at: /var/backupscripts/
+
Backup scripts are currently stored at: /etc/backupscripts/
  
 
We want a combination of rsync for heavily used resources (especially the media folders or wikis, and other media storage) and versioned, 7-zipped backup (to protect against user-error as well as hardware failure). The versioning schema is planned to be 7 daily versions (cyclically overwritten), 5 weekly ones (cyclically overwritten), 12 monthly ones (cyclically overwritten), yearly ones (not overwritten).
 
We want a combination of rsync for heavily used resources (especially the media folders or wikis, and other media storage) and versioned, 7-zipped backup (to protect against user-error as well as hardware failure). The versioning schema is planned to be 7 daily versions (cyclically overwritten), 5 weekly ones (cyclically overwritten), 12 monthly ones (cyclically overwritten), yearly ones (not overwritten).
Line 11: Line 11:
 
Versioned backup should be /etc and /home weekly, and /var (without backup, mysql db and media folders) daily.
 
Versioned backup should be /etc and /home weekly, and /var (without backup, mysql db and media folders) daily.
  
−
== Current MySQL backup ==
+
== Initiating the backup ==
 +
The backup is initiated by a cron job specified in /etc/crontab. At 2:06 it calls <code>daily_bak_entry.sh</code>. The entry script only calls <code>daily_bak_master.sh</code> and forwards the output (stdout and stderr) into a temporary file which is afterwards moved to <code>/var/log/backups/</code>  so it is possible to determine what when wrong if somethings goes wrong.
 +
 
 +
 
 +
The master script first checks if the backup drive /mnt/backup is mounted. If it is not (this could be caused by a mounting issue with the NFS drive), it uses the secondary local hard drive mounted at /mnt/dump as alternative. The backups would in this case be saved at <code>/mnt/dump/var/backups/</code>.
 +
 
 +
Next, the main script removes the outdated backups as outlined above and then calls the task specific backup scripts
 +
* daily_bak_mysql.sh
 +
* daily_bak_settings.sh
 +
* daily_bak_svn.sh
 +
* daily_bak_wiki_media.sh
 +
* daily_bak_special.sh
 +
 
 +
After the backups are done, the resulting files from the mysql, settings and svn (not wiki and special though) backups are copied to <code>/var/backups/WEEKLY</code> or <code>/var/backups/MONTHLY</code> if it is the first day of the week (Monday) or of the month respectively.
 +
 
 +
== MySQL backup ==
  
 
The daily backup of all of mysql databases is performed by the script  
 
The daily backup of all of mysql databases is performed by the script  
−
  /var/backupscripts/mysql_bak_manol.sh
+
  /etc/backupscripts/daily_bak_mysql.sh
−
which is executed as cron job at 03:12 every night. The backup files is located in  
+
The backup files are located in  
−
  /mnt/dump/var/backups/mysql/
+
  /var/backups/DAILY/mysql/
 
and is named as:
 
and is named as:
−
  ttYY-MM-DD_DBNAME.sql.7z
+
  YY-MM-DD_DBNAME.sql.7z
  
−
In the script there is a variable '''DBSKIP''' that contains names of databases to skip, for example:
+
By default it exports all the databases within mysql. In the script there is a variable '''DBSKIP''' that contains names of databases to skip, for example:
−
  DBSKIP="information_schema|otherdb1|otherdb2"
+
  DBSKIP="information_schema|wmf_commons|wmf_dewiki"
  
−
== Current MediaWiki media folder backup ==
+
== SVN backup ==
 +
The daily backup of svn repository is performed by the script
 +
/etc/backupscripts/daily_bak_svn.sh
 +
The backup files are located in
 +
/var/backups/DAILY/mysql/
 +
and is named as:
 +
YY-MM-DD_var_lib_svn.tar.7z
  
 +
It backups everything within <code>/var/lib/svn/</code>.
 +
 +
 +
== Settings backup ==
 +
The daily backup of various settings is performed by the script
 +
/etc/backupscripts/daily_bak_settings.sh
 +
The backup files are located in
 +
/var/backups/DAILY/settings/
 +
and is named as:
 +
YY-MM-DD_settings.tar.7z
 +
 +
It backups everything within <code>/etc/</code>, <code>/var/spool/cron/crontabs/</code> and the individual wiki settings. For the wiki settings, the script looks for the locations of LocalSettings.php files within <code>/var/www</code> and for each one it finds, it copies all of the files in the corresponding folder, with the exception of *.zip, *.bz and *.log files. Subfolders and symbolic links are not included. This reduces of all the backuped files within <code>/var/www</code> to about 1 MB after compression while still covering almost all of the system relevant settings
 +
 +
== MediaWiki media folder and settings backup ==
 
The daily backup of media wikis is performed by the script  
 
The daily backup of media wikis is performed by the script  
−
  /var/backupscripts/mediawiki_bak_manol.sh
+
  /etc/backupscripts/daily_bak_wiki_media.sh
−
which is executed as cron job at 03:22 every night.
+
  
−
It rsync files from media dirs of wikis to the corresponding subdir in <pre>/mnt/dump/var/backups/wikis/</pre>
+
It searches for all the files called <code>LocalSettings.php</code> within <code>/var/www</code> and all its subdirectories, regardless of the depth. For each directory in which a <code>LocalSettings.php</code> it backups all the <code>LocalSetting*</code> files one tar and it synchronizes the media directory.
 +
 
 +
The media directories are synchronized using <code>rsync</code>to the corresponding subdir in <pre>/var/backups/DAILY/wikis/</pre>
 +
 
 +
The backup of all the <code>LocalSetting*</code> files as well as all the <code>*.php</code> files within <code>/var/www</code> (without subdirectories)
 +
and stored in <pre>/var/backups/DAILY/wikis/</pre> in a file called
 +
wikisettingsYY-MM-DD.tar.7z
 +
 
 +
It is possible to specify wikis which should not be backed up like this. This could be useful for wikis that have their own backup strategy, like practical plans. In order to have a wiki be ignored, add the path to the variable '''WIKIEXCLUDELIST''' at the beginning of the script (space separated). Any wikis below this path will be ignored, even if they are within subdirectories:
 +
WIKIEXCLUDELIST="v-practicalplants v-mfn/w/extensions2/WYSIWYG v-mfn/sammlung/extensions2.BACKUP"
 +
 
 +
== Special backup ==
 +
The daily backup of special tasks is performed by the script
 +
/etc/backupscripts/daily_bak_special.sh
 +
 
 +
It contains the backup of the directory <code>/var/www/v-practicalplants</code>. This directory is deliberately not covered by the script <code>daily_bak_wiki_media.sh</code>.
 +
 
 +
Another archive is created for <code>/var/www</code> which contains all the files and directories except the various <code>cache/</code> and <code>media/</code> directories and <code>v-practicalplants</code>. Unlike the wikisettings backup above, this one contains also files from non-wiki folders and customized wiki files, (e.g. extensions2/).
 +
 
 +
The resulting files is stored in <code><back_up_dir>/DAILY/special</code>.
 +
 
 +
Both backups are not part of the daily/weekly/monthly replication, due to their size. However the most recent backup is coped into <code><back_up_dir>/DAILY/special/old</code> before the next backup is started, so that one valid backup remains, in case the backup process fails.
  
−
It also makes tar backup with all LocalSetting* in <pre>/mnt/dump/var/backups/wikis/</pre>. The name of tars is:
 
−
ttYY-MM-DD_settings.tar
 
  
 
== Remote backup ==
 
== Remote backup ==
 +
Since internal backup has beeen running satisfactorily for the past 2 years (to date November 2014), external backups are no longer required. The script can be removed from crontab. The "backup" section in /etc/crontab should look like this:
 +
 +
<pre>
 +
### backups:
 +
6 6 * * * root /etc/backupscripts/daily_bak_entry.sh
 +
</pre>
 +
 +
The file /etc/backupscripts/remote_bak_manol.sh can be removed.
 +
 +
See also: [[Testing /etc/crontab]]
 +
 +
== Filling disk with null ==
 +
 +
cd /mnt/backup/; sudo dd if=/dev/zero bs=2048 of=/mnt/backup/file1
  
−
The daily remote backup of these dirs:
+
Reason: if backup shall be compressed, this will drastically increase compressibility. Danger: watch this with df -h, else it will fill each mount completely, potentially crashing the server.
−
/mnt/dump/var/backups/wikis/
+
−
/mnt/dump/var/backups/mysql/
+
−
is performed by the script
+
−
/var/backupscripts/remote_bak_manol.sh
+
  
−
It is executed as cron job at 03:33 every night and make rsync of specified dirs to remote server
+
== Backup Size ==
−
k2n.bikam.com
+
Based on the current data that is backed up (as of July 2016), the complete backup will take up around 320 GB.  
  
 +
'''243.9 GB DAILY (kept for 8 days)'''
 +
  20.5 GB DAILY/mysql
 +
  0.1 GB DAILY/settings
 +
  3.8 GB DAILY/special
 +
  0.5 GB DAILY/svn
 +
219.0 GB DAILY/wikis
 +
 +
''' 13.2 GB WEEKLY (keep for 5 month)'''
 +
  12.8 GB WEEKLY/mysql
 +
  0.1 GB WEEKLY/settings
 +
  0.3 GB WEEKLY/svn
 +
 +
''' 63.6 GB MONTHLY (kept for 24 month)'''
 +
  61.6 GB MONTHLY/mysql
 +
  0.5 GB MONTHLY/settings
 +
  1.5 GB MONTHLY/svn
 +
 +
'''320  GB Total'''
  
 
[[Category:Software documentation]]
 
[[Category:Software documentation]]

Latest revision as of 13:50, 22 September 2017

Strategies

Wikis: Both the media folders and mysql databases need regular backup and validation of restore. We can run a backup to a separate disk, which is mounted at /mnt/backup/ and is also linked from /var/backups. Thus writing a backup to /var/backups will end up on this disk.

Backup scripts are currently stored at: /etc/backupscripts/

We want a combination of rsync for heavily used resources (especially the media folders or wikis, and other media storage) and versioned, 7-zipped backup (to protect against user-error as well as hardware failure). The versioning schema is planned to be 7 daily versions (cyclically overwritten), 5 weekly ones (cyclically overwritten), 12 monthly ones (cyclically overwritten), yearly ones (not overwritten). That is after the first year, one has 7 + 5 + 12 + years = 24 + yearly copies.

Versioned backup should be /etc and /home weekly, and /var (without backup, mysql db and media folders) daily.

Initiating the backup

The backup is initiated by a cron job specified in /etc/crontab. At 2:06 it calls daily_bak_entry.sh. The entry script only calls daily_bak_master.sh and forwards the output (stdout and stderr) into a temporary file which is afterwards moved to /var/log/backups/ so it is possible to determine what when wrong if somethings goes wrong.


The master script first checks if the backup drive /mnt/backup is mounted. If it is not (this could be caused by a mounting issue with the NFS drive), it uses the secondary local hard drive mounted at /mnt/dump as alternative. The backups would in this case be saved at /mnt/dump/var/backups/.

Next, the main script removes the outdated backups as outlined above and then calls the task specific backup scripts

  • daily_bak_mysql.sh
  • daily_bak_settings.sh
  • daily_bak_svn.sh
  • daily_bak_wiki_media.sh
  • daily_bak_special.sh

After the backups are done, the resulting files from the mysql, settings and svn (not wiki and special though) backups are copied to /var/backups/WEEKLY or /var/backups/MONTHLY if it is the first day of the week (Monday) or of the month respectively.

MySQL backup

The daily backup of all of mysql databases is performed by the script 

/etc/backupscripts/daily_bak_mysql.sh

The backup files are located in 

/var/backups/DAILY/mysql/

and is named as: 

YY-MM-DD_DBNAME.sql.7z

By default it exports all the databases within mysql. In the script there is a variable DBSKIP that contains names of databases to skip, for example: 

DBSKIP="information_schema|wmf_commons|wmf_dewiki"

SVN backup

The daily backup of svn repository is performed by the script 

/etc/backupscripts/daily_bak_svn.sh

The backup files are located in 

/var/backups/DAILY/mysql/

and is named as: 

YY-MM-DD_var_lib_svn.tar.7z

It backups everything within /var/lib/svn/.


Settings backup

The daily backup of various settings is performed by the script 

/etc/backupscripts/daily_bak_settings.sh

The backup files are located in 

/var/backups/DAILY/settings/

and is named as: 

YY-MM-DD_settings.tar.7z

It backups everything within /etc/, /var/spool/cron/crontabs/ and the individual wiki settings. For the wiki settings, the script looks for the locations of LocalSettings.php files within /var/www and for each one it finds, it copies all of the files in the corresponding folder, with the exception of *.zip, *.bz and *.log files. Subfolders and symbolic links are not included. This reduces of all the backuped files within /var/www to about 1 MB after compression while still covering almost all of the system relevant settings

MediaWiki media folder and settings backup

The daily backup of media wikis is performed by the script 

/etc/backupscripts/daily_bak_wiki_media.sh

It searches for all the files called LocalSettings.php within /var/www and all its subdirectories, regardless of the depth. For each directory in which a LocalSettings.php it backups all the LocalSetting* files one tar and it synchronizes the media directory.

The media directories are synchronized using rsyncto the corresponding subdir in 
/var/backups/DAILY/wikis/

The backup of all the LocalSetting* files as well as all the *.php files within /var/www (without subdirectories)

and stored in 
/var/backups/DAILY/wikis/
in a file called 
wikisettingsYY-MM-DD.tar.7z

It is possible to specify wikis which should not be backed up like this. This could be useful for wikis that have their own backup strategy, like practical plans. In order to have a wiki be ignored, add the path to the variable WIKIEXCLUDELIST at the beginning of the script (space separated). Any wikis below this path will be ignored, even if they are within subdirectories: 

WIKIEXCLUDELIST="v-practicalplants v-mfn/w/extensions2/WYSIWYG v-mfn/sammlung/extensions2.BACKUP"

Special backup

The daily backup of special tasks is performed by the script 

/etc/backupscripts/daily_bak_special.sh

It contains the backup of the directory /var/www/v-practicalplants. This directory is deliberately not covered by the script daily_bak_wiki_media.sh.

Another archive is created for /var/www which contains all the files and directories except the various cache/ and media/ directories and v-practicalplants. Unlike the wikisettings backup above, this one contains also files from non-wiki folders and customized wiki files, (e.g. extensions2/).

The resulting files is stored in <back_up_dir>/DAILY/special.

Both backups are not part of the daily/weekly/monthly replication, due to their size. However the most recent backup is coped into <back_up_dir>/DAILY/special/old before the next backup is started, so that one valid backup remains, in case the backup process fails.


Remote backup

Since internal backup has beeen running satisfactorily for the past 2 years (to date November 2014), external backups are no longer required. The script can be removed from crontab. The "backup" section in /etc/crontab should look like this: 

### backups:
6 6 * * * root /etc/backupscripts/daily_bak_entry.sh

The file /etc/backupscripts/remote_bak_manol.sh can be removed.

See also: Testing /etc/crontab

Filling disk with null

cd /mnt/backup/; sudo dd if=/dev/zero bs=2048 of=/mnt/backup/file1

Reason: if backup shall be compressed, this will drastically increase compressibility. Danger: watch this with df -h, else it will fill each mount completely, potentially crashing the server.

Backup Size

Based on the current data that is backed up (as of July 2016), the complete backup will take up around 320 GB. 

243.9 GB DAILY (kept for 8 days)
 20.5 GB DAILY/mysql
  0.1 GB DAILY/settings
  3.8 GB DAILY/special
  0.5 GB DAILY/svn
219.0 GB DAILY/wikis

 13.2 GB WEEKLY (keep for 5 month)
 12.8 GB WEEKLY/mysql
  0.1 GB WEEKLY/settings
  0.3 GB WEEKLY/svn

 63.6 GB MONTHLY (kept for 24 month)
 61.6 GB MONTHLY/mysql 
  0.5 GB MONTHLY/settings
  1.5 GB MONTHLY/svn

320   GB Total
Retrieved from "https://biowikifarm.net/metawiki/index.php?title=Server_backup_and_restore&oldid=5992"