Home>

Recently, I want to regularly back up important internal server data to storage.Next, through this article, I will share the regular synchronization backup of rsync+ crontab under centos7.Interested friends follow us to see it

Recently, I want to regularly back up important internal server data to storage.By the way take a note

Previously, a cwrsync (client) + rsync (server:storage) bat script under win

This time a whole script sh under linux to perform regular automatic backup data

Client:192.168.0.100 (rsync-linux)

Server:192.168.0.252 (storage)

Introduction to rsync:

rsync is a data mirror backup tool for Linux systems.Use the fast incremental backup tool remote sync to synchronize remotely.Supports local replication,Or synchronize with other ssh and rsync hosts.

1. Server configuration:

The server of rsync is mainly the storage of Qunhui.So there is an interface configuration directly,So I wo n’t explain it in detail here.As shown below:

2. Client configuration:

Create a password file,/etc/rsyncd/rsyncd.pass and modify it to 600 permissions

[[email protected] ~] #mkdir rsyncd
[[email protected] ~] #vim /rsyncd/rsyncd.pass
[[email protected] ~] #chmod 600 /rsyncd/rsyncd.pass

▲ Description:The password in /rsyncd/rsyncd.pass needs to be consistent with the password of the user specified in /etc/rsyncd/rsyncd.pass or stored rsync

The client transmits files to the server,If it is port 873, you can remove --port

rsync -arvz --progress/data [email protected] ::log --password-file =/rsyncd/rsyncd.pass

You can also pull files from the server if needed,Need to remove write only=yes from /etc/rsyncd/rsyncd.conf of the server

#Pull the entire directory
rsync -arvz --progress --password-file =/rsyncd/rsyncd.pass --port=873 [email protected] ::log/var/log
#Pull a single file
rsync -arvz --progress --password-file =/rsyncd/rsyncd.pass --port=873 [email protected] ::log/a.log/var/log

rsync server configuration parameters:

Global parameter
All parameters before [module] in the file are global parameters,Of course, you can also define module parameters in the global parameters section,At this time, the value of this parameter is the default value of all modules.
port
Specify the port number used by the daemon,The default is 873.
motd file
The "motd file" parameter is used to specify a message file,The content of the file is displayed to the client when the client connects to the server,There is no motd file by default.
log file
"log file" specifies the log file for rsync,Without sending logs to syslog. For example, it can be specified as "/var/log/rsyncd.log".
pid file
Specify the rsync pid file, usually "/var/run/rsyncd.pid".
syslog facility
Specifies the message level when rsync sends log messages to syslog,Common message levels are:uth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, security, sys-log, user, uucp, local0, local1, local2, local3, local4, local5, local6, and local7. The default is daemon.
Module parameters
It is mainly to define which directory of the server is to be synchronized.
The format must be of the form "[module]". This name is the name you see on the rsync client.
It's a bit like the share name provided by the samba server.
The data that is actually synchronized by the server is specified by path.
We can according to our needs,To specify multiple modules,The following parameters can be defined in the module:
comment
Assign a description to the module,This description is displayed to the client along with the module name when the client connects to get a list of modules.
There is no description definition by default.
path
Specify the directory tree path for this module for backup,This parameter must be specified.
use chroot
If "use chroot" is specified as true, rsync first chroots to the directory specified by the path parameter before transferring the file.
The reason for this is to achieve additional security,However, the disadvantage is that roots permissions are required, and directory files pointed to by external symbolic links cannot be backed up.
The chroot value is true by default.
uid
This option specifies the uid that the daemon should have when the module transfers files. Use with the gid option to determine which file permissions can be accessed.
The default value is "nobody".
gid
This option specifies the gid that the daemon should have when the module transfers files. The default value is "nobody".
max connections
Specify the maximum number of concurrent connections for this module to protect the server,Connection requests that exceed the limit will be told to try again later.
The default value is 0, which means there is no limit.
list
This option specifies that when a client requests a list of modules that can be used,Whether the module should be listed.
If this option is set to false, hidden modules can be created.
The default value is true.
read only
This option sets whether to allow customers to upload files.
If true then any upload request will fail,If false and the server directory read and write permissions allow then upload is allowed.
The default is true.
exclude
Used to specify multiple files or directories (relative paths) separated by spaces and add them to the exclude list. This is equivalent to using --exclude in the client command to specify the mode,A module can only specify one exclude option. One thing to note is that this option has some security issues,Customers are likely to bypass the exclude list, and if i want to ensure that specific files cannot be accessed, it is best to use it with the uid/gid option.
exclude from
Specify a filename containing the definition of the exclude pattern,The server reads the exclude list definition from this file.
include
Used to specify files or directories that are not excluded from the requirements.
This is equivalent to using --include in the client command to specify the mode,Combining include and exclude can define complex exclude/include rules.
include from
Specify a file name containing the definition of the include pattern,The server reads the include list definition from this file.
auth users
This option specifies a list of usernames separated by spaces or commas.
Only these users are allowed to connect to the module.
The user here has nothing to do with the system user.
If "auth users" is set, then the client sends a connection request to this module and will be challenged by rsync request to verify the identity challenge/response authentication protocol used here.
The user name and password are stored in clear text in the file specified by the "secrets file" option.
Modules can be connected without a password by default (that is, anonymously).
secrets file
This option specifies a file containing defined username:password pairs.
Only when "auth users" are defined,This file is only useful.
Each line of the file contains a username:passwd pair. Generally, the password should not exceed 8 characters. There is no default secures file name, and you need to specify one limit (for example:/etc/rsyncd.passwd). Note:The permissions of this file must be 600, otherwise the client will not be able to connect to the server.
strict modes
This option specifies whether to monitor password file permissions,If the value of this option is true, the password file can only be accessed by the user running as the rsync server.
No other users can access the file.
The default is true.
hosts allow
This option specifies which IP clients are allowed to connect to the module.
A customer schema definition can take the following form:
A single IP address, for example:192.167.0.1
The entire network segment,For example:192.168.0.0/24, or 192.168.0.0/255.255.255.0
Multiple IPs or network segments need to be separated by spaces.
"*" Means all,The default is to allow all hosts to connect.
hosts deny
Specify the machines that are not allowed to connect to the rsync server,You can use the definition of hosts allow to define.
The default is no hosts deny definition.
ignore errors
Specifies that rsyncd ignores io errors on the server when determining whether to perform a delete operation during transmission. Generally speaking, rsync will skip the --delete operation when an io error occurs to prevent temporary resource shortage or other io errors. Serious Problem.
ignore nonreadable
Specifies that the rysnc server completely ignores files that the user does not have access to.
This makes sense if there are files in the directory that need to be backed up that should not be available to the backup person.
lock file
Specify a lock file that supports the max connections parameter,The default value is /var/run/rsyncd.lock.
transfer logging
Make the rsync server use ftp format files to record download and upload operations in its own separate log.
log format
With this option, users can customize the fields of the log file when using transfer logging.
Its format is a string containing a format specifier,The format specifiers that can be used are as follows:
%h remote host name
%a remote ip address
%l file length characters
%p process id of this rsync session
%o action type:"send" or "recv"
%f file name
%p module path
%m module name
%t current time
%u authenticated username (null if anonymous)
%b number of bytes actually transferred
%c When sending a file,This field records the checksum of the file
The default log format is:"%o%h [%a]%m (%u)%f%l", in general,"%T [%p]" is added to the head of each line. A perl script called rsyncstats is also released in the source code to count log files in this format.
timeout
This option allows you to override the IP timeout period specified by the customer.
This option ensures that the rsync server does not wait forever for a crashed client.
The timeout is in seconds.
0 means no timeout is defined,This is also the default.
For anonymous rsync servers,An ideal number is 600.
refuse options
This option allows you to define a list of command parameters that the customer is not allowed to use for this module.
The full name of the command must be used here,It cannot be abbreviated.
However, when a command is rejected, the server reports an error message and then exits.
To prevent the use of compression,It should be:"dont compress=*".
dont compress
Used to specify files that are not compressed and then transferred.
The default value is * .gz * .tgz * .zip * .z * .rpm * .deb * .iso * .bz2 * .tbz

rsync common commands:

-v, --verbose verbose mode output
 -q, --quiet reduced output mode
 -c, --checksum turn on the check switch,Force verification of file transfers
 -a, --archive archive mode,Means to transfer files recursively,And keep all file attributes,Equal to -rlptgod
 -r, --recursive handle subdirectories in recursive mode
 -r, --relative use relative path information
 -b, --backup create a backup,That is, when the same file name already exists for the purpose,Rename the old file to ~ filename. You can use the --suffix option to specify a different backup file prefix.
 --backup-dir Store backup files (such as ~ filename) in a directory.
 -suffix=suffix defines the backup file prefix
 -u, --update just update,That is, skip all files that already exist in dst, and the file time is later than the file to be backed up.
(Do not overwrite updated files)
 -l, --links keep soft links
 -l, --copy-links handle soft links like regular files
 --copy-unsafe-links Copy only links outside the directory tree of the src path
 --safe-links ignore links pointing outside the src directory tree
 -h, --hard-links keep hard links
 -p, --perms keep file permissions
 -o, --owner keep file owner information
 -g, --group keep file group information
 -d, --devices keep device file information
 -t, --times keep file time information
 -s, --sparse special processing for sparse files to save dst space
 -n, --dry-run show which files will be transferred
 -w, --whole-file copy files,No incremental detection
 -x, --one-file-system do not cross file system boundaries
 -b, --block-size=size block size used by the test algorithm,700 bytes by default
 -e, --rsh=command specify data synchronization using rsh and ssh
 --rsync-path=path specifies the path information of the rsync command on the remote server
 -c, --cvs-exclude automatically ignore files in the same way as cvs,Used to exclude unwanted files
 --existing updates only those files that already exist in dst, without backing up newly created files
 --delete delete files that src does not have in dst
 --delete-excluded also deletes files on the receiving end that are specified by this option
 --delete-after Delete after transmission
 --ignore-errors delete io errors in time
 --max-delete=num delete up to num files
 --partial keeps files that are not completely transferred for some reason,To speed up subsequent retransmissions
 --force forcibly delete the directory,Even if not empty
 --numeric-ids do not match numeric user and group ids to username and group names
 --timeout=time ip timeout time,Unit is second
 -i, --ignore-times do not skip files with the same time and length
 --size-only When deciding whether to back up files,Just look at file size without considering file time
 --modify-window=num timestamp window used to determine if the files are the same time,Default is 0
 -t --temp-dir=dir create temporary files in dir
 --compare-dest=dir also compares the files in dir to determine if a backup is required
 -p is equivalent to --partial
 --progress show backup progress
 -z, --compress Compress backup files during transfer
 --exclude=pattern specifies the pattern to exclude files that do not need to be transferred
 --include=pattern specify file patterns to be transferred without exclusion
 --exclude-from=file exclude files with the specified pattern in file
 --include-from=file does not exclude files that match the pattern specified by file
 --version print version information
 --address bind to a specific address
 --config=file specifies another configuration file,Do not use the default rsyncd.conf file
 --port=port specify another rsync service port
 --blocking-io use blocking io for remote shell
 -stats gives the transfer status of some files
 --progress displays the transmission process during transmission
 --log-format=format specifies the log file format
 --password-file=file get password from file
 --bwlimit=kbps limit i/o bandwidth, kbytes per second
 -h, --help display help information

crontab introduction:

Crond is a daemon process under Linux that periodically performs certain tasks or waits to process certain events.
Similar to scheduled tasks under windows,When the installation is complete,This service tool is installed by default,And it will automatically start the crond process. The crond process will periodically check whether there are tasks to be performed,If there are tasks to be performed,This task is performed automatically.
Task scheduling under Linux is divided into two categories:system task scheduling and user task scheduling.
System task scheduling:the work to be performed periodically by the system,Such as writing cached data to the hard disk, log cleaning, and so on.
There is a crontab file in the/etc directory. This is the configuration file for system task scheduling.
The/etc/crontab file includes the following lines:
[[email protected] ~] #cat/etc/crontab
shell =/bin/bash
path =/sbin:/bin:/usr/sbin:/usr/bin
#run-parts
51 * * * * root run-parts /etc/cron.hourly
24 7 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

User Task Scheduling:Work that users regularly perform,Such as user data backup, regular email reminders, etc.Users can use the crontab tool to customize their own scheduled tasks.All user-defined crontab files are stored in the/var/spool/cron directory. Its file name is the same as the user name.

crontab file meaning:

In the crontab file created by the user, each line represents a task,Each field on each line represents a setting,Its format is divided into six fields.The first five segments are time-setting segments.The sixth paragraph is the command paragraph to be executed,The format is as follows:

minute hour day month week command

among them:

minute:indicates the minute,Can be any integer from 0 to 59.

hour:indicates the hour,Can be any integer from 0 to 23.

day:indicates the date,Can be any integer from 1 to 31.

month:indicates the month,Can be any integer from 1 to 12.

week:indicates the day of the week,Can be any integer from 0 to 7,Here 0 or 7 stands for Sunday.

command:the command to be executed,Can be a system command,It can also be a script file written by yourself.

In each of the above fields,You can also use the following special characters:

Asterisk (*):represents all possible values,For example, if the month field is an asterisk,It means that the command operation is executed every month after the constraints of other fields are met.

Comma (,):you can specify a range of lists with comma-separated values,For example, "1,2,5,7,8,9"

Middle bar (-):You can use the middle bar between integers to represent a range of integers,For example "2-6" means "2,3,4,5,6"

Forward slash (/):You can use the forward slash to specify the time interval frequency,For example, "0-23/2" means execute every two hours.Also forward slashes can be used with asterisks,For example, */10, if used in the minute field, it means that it is executed every ten minutes.

crontab common commands:

1. List crontab files

To list crontab files, use:

[[email protected] ~] #crontab -l
 0,15,30,45,18-06 * * */bin/echo `date`>dev/tty

You will see something similar to the above.You can use this method to make a backup of the crontab file in the $home directory:

[[email protected] ~] #crontab -l>$home/mycron

In this way, once the crontab file is accidentally deleted by mistake, you can use the method described in the previous section to quickly restore it.

2. Edit the crontab file

If i want to add, delete, or edit entries in the crontab file,And the editing environment variable is set to vi, then you can use vi to edit the crontab file. The corresponding command is:

[[email protected] ~] #crontab -e

You can modify the crontab file and exit as you would any other file with vi.If some entries are modified or new entries are added,Then when saving the file, c r o n will perform the necessary integrity checks on it.If one of these fields has a value outside the allowed range,It will prompt you.

When we edit the crontab file, we may add new entries.For example, add the following:

#dt:delete core files, at 3.30am on 1,7,14,21,26,26 days of each month (note)
 30 3 1,7,14,21,26 * */bin/find -name "core" -exec rm {} \;

Save and exit now.It is best to add a comment above each entry in the crontab file,In this way, you can know its function, running time,More importantly,Know which user's job this is.

Now let's list all its information using the crontab -l command mentioned earlier:

[[email protected] ~] #crontab -l
 #(crondave installed on tue may 4 13:07:43 1999)
 #dt:ech the date to the console every 30 minites
 0,15,30,45 18-06 * * */bin/echo `date`>/dev/tty1
 #dt:delete core files, at 3.30am on 1,7,14,21,26,26 days of each month
 30 3 1,7,14,21,26 * */bin/find -name "core" -exec rm {} \;

3. Delete the crontab file

To delete the crontab file, you can use:

[[email protected] ~] #crontab -r

crontab usage examples

Example 1:execute command every 1 minute

command:

* * * * * command

Example 2:Execute rsync-sh script every Sunday night at 00:00

command:

[[email protected] ~] #crontab -e
#dt:execute rsync-sh script every sunday at 00:00 (note)
0 0 * * 7 sh /root/rsyncd/rsync-gitbak.sh

The rsync client automatically synchronizes with the rsync server:

First, let's make a shell script

[[email protected] rsyncd] #vim rsyncd.sh
#!/bin/bash
rsync -arvz --progress/data [email protected] ::log --password-file =/rsyncd/rsyncd.pass

Command:crontab -e to edit join scheduled tasks

[[email protected] ~] #crontab -e
#dt:execute rsync-sh script every sunday at 00:00 (note)
0 0 * * 7 sh /root/rsyncd/rsyncd.sh

Use crontab -l to view the scheduled tasks joined

[[email protected] ~] #crontab -l
0 0 * * 7 sh /root/rsyncd/rsyncd.sh

note:

1. When the program runs at the time you specify,A letter will be sent to you.Show what the program is running.If you do not wish to receive such a letter,Please add a space after each line with>/dev/null 2 ​​>&1.

2.%is considered to be newline in crontab, and it is necessary to use \ for escape. For example, crontab execution line,If there is "date +%y%m%d", it must be replaced with:"date + \%y \%m \%d"

Problems encountered and solutions:

Edited the sh script on windows and uploaded it to linux, and executed an error:

bad interpreter:no such file or directory

vim rsyncd.sh use the command:set ff?to see if it is doc or unix format, if it is dos format, use the command:set ff=unix to convert to unix

  • Previous Analysis of conditional rendering instructions in Vuejs
  • Next Python push box game