When we are migrating a site, rsync is our friend. Let's get intimate and hands on with file synchronization.
...
Using Drush's built-in RSYNC
Info |
---|
note that drush rsync can be funky, especially with older versions of drush. Real rsync in generally recommended. |
Basic Usage
Drush aliases makes operations destinations clear.
Here is some basic usage. We pull images down from dev to our local. Drupal will find the path to each instance's %files directory. Magic.
Note that the slash appears slash appears on the source, but not the target destination.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush rsync @mysite.dev:%files/images/ @mysite.local:%files/images |
Here, we pull down all files from live to local, but exclude drupal's private files
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush rsync --exclude=private @mysite.live:%files/ @mysite.local:%files |
Info |
---|
note that you cannot use drush to sync 2 remote instances, like live to test. That would be an on-server operation. |
Safety first: passing flags
...
Drush has some option flags of its own we can send to test operations before committing. In this case we are going from our local to staging:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush --simulate --debug rsync --mode=razoglp @sitename.local:%files/images/ @sitename.staging:%files/images |
...
Excluding Paths
Drush provides the ability to leverage parts of rsync, but will not easily exclude files based on patterns. We can exclude paths, however:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush rsync -y --mode=razog --exclude-paths=%files/private:%files/css:%files/js:%files/styles:%files/ctools:%files/backup_migrate @mysite.localdev:%files/ @mysite.devlocal:%files/ |
Using native rsync for full control of file
...
transfers
Here is a verbose exclude list using straight rsync:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ rsync -razv --delete --exclude=css --exclude=*_cache --exclude=js --exclude=googleanalytics --exclude=xmlsitemap \
--exclude=backup_migrate --exclude=styles user@domain.com:/path/to/remote/files/ \
/path/to/local/files/ |
We exclude a number of directories here and also delete anything on our local that does not exist on the remote.
setting hosts
With native rsync, we don't use drush aliases. However, we can set up shortcuts to hosts in .ssh/config.
Code Block | ||
---|---|---|
| ||
# mysite Host mysite.dev Hostname mysite.com port 2222 Compression yes User username Host mysite.test Hostname test.mysite.com port 2222 Compression yes User username |
These greatly simplify logging in via SSH. Now, instead of
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ ssh -p 2222 username@test.mysite.com |
you can simply do:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ ssh mysite.test |
-e 'ssh -axp 1022'
By default, symbolic links are not transferred at all.
We can replace multiple exclude patterns in the rsync command with a reference to an external file, using a flag, like so: --exclude-from '/var/www/backup/rsync-exclude.txt'
Here are files and directories to exclude from drupal's files directory.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
.DS_Store css civicrm ctools js *_cache googleanalytics xmlsitemap backup_migrate | ||||||
Code Block | ||||||
| ||||||
-v, --verbose increase verbosity -q, --quiet suppress non-error messages --no- tmp styles .DS_Store .DS_Store* .sass-cache ehthumbs.db Icon? Thumbs.db ._* *.bak *_cache *.tgz *.un~ .buildpath .project .settings *.sublime-project *.sublime-workspace *.sublime-projectcompletions nbproject |
rsync to pantheon
Grab the pantheon credentials from the desitnation account and modify to fit.
Log in to the client server to run this operation, though you can sync a local directory to Pantheon in much the same way.
You will need to get the ssh key for the machine performing the rsync operation to Pantheon up to your control panel if it is not already there.
Code Block | ||
---|---|---|
| ||
# rsync the files to Panthon dev
rsync -trazog --verbose --exclude-from='/var/www/backup/rsync-files-exclude.txt' \
/var/www/mysite/htdocs/sites/default/files/ \
--ipv4 -e 'ssh -p 2222' \
dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX@appserver.dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX.drush.in:files;
# enter pantheon dashboard password when prompted |
...
Resources
Code Block | ||||
---|---|---|---|---|
| ||||
-v, --verbose increase verbosity -q, --quiet suppress non-error messages --no-motd suppress daemon-mode MOTD (see caveat) -c, --checksum skip based on checksum, not mod-time & size -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) --no-OPTION turn off an implied OPTION (e.g. --no-D) -r, --recursive recurse into directories -R, --relative use relative path names --no-implied-dirs don't send implied dirs with --relative -b, --backup make backups (see --suffix & --backup-dir) --backup-dir=DIR make backups into hierarchy based in DIR --suffix=SUFFIX backup suffix (default ~ w/o --backup-dir) -u, --update skip files that are newer on the receiver --inplace update destination files in-place --append append data onto shorter files --append-verify --append w/old data in file checksum -d, --dirs transfer directories without recursing -l, --links copy symlinks as symlinks -L, --copy-links transform symlink into referent file/dir --copy-unsafe-links only "unsafe" symlinks are transformed --safe-links ignore symlinks that point outside the tree -k, --copy-dirlinks transform symlink to dir into referent dir -K, --keep-dirlinks treat symlinked dir on receiver as dir -H, --hard-links preserve hard links -p, --perms preserve permissions -E, --executability preserve executability --chmod=CHMOD affect file and/or directory permissions -A, --acls preserve ACLs (implies -p) -X, --xattrs preserve extended attributes -o, --owner preserve owner (super-user only) -g, --group preserve group --devices preserve device files (super-user only) --specials preserve special files -D same as --devices --specials -t, --times preserve modification times -O, --omit-dir-times omit directories from --times --super receiver attempts super-user activities --fake-super store/recover privileged attrs using xattrs -S, --sparse handle sparse files efficiently -n, --dry-run perform a trial run with no changes made -W, --whole-file copy files whole (w/o delta-xfer algorithm) -x, --one-file-system don't cross filesystem boundaries -B, --block-size=SIZE force a fixed checksum block-size -e, --rsh=COMMAND specify the remote shell to use --rsync-path=PROGRAM specify the rsync to run on remote machine --existing skip creating new files on receiver --ignore-existing skip updating files that exist on receiver --remove-source-files sender removes synchronized files (non-dir) --del an alias for --delete-during --delete delete extraneous files from dest dirs --delete-before receiver deletes before transfer (default) --delete-during receiver deletes during xfer, not before --delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not before --delete-excluded also delete excluded files from dest dirs --ignore-errors delete even if there are I/O errors --force force deletion of dirs even if not empty --max-delete=NUM don't delete more than NUM files --max-size=SIZE don't transfer any file larger than SIZE --min-size=SIZE don't transfer any file smaller than SIZE --partial keep partially transferred files --partial-dir=DIR put a partially transferred file into DIR --delay-updates put all updated files into place at end -m, --prune-empty-dirs prune empty directory chains from file-list --numeric-ids don't map uid/gid values by user/group name --timeout=SECONDS set I/O timeout in seconds --contimeout=SECONDS set daemon connection timeout in seconds -I, --ignore-times don't skip files that match size and time --size-only skip files that match in size --modify-window=NUM compare mod-times with reduced accuracy -T, --temp-dir=DIR create temporary files in directory DIR -y, --fuzzy find similar file for basis if no dest file --compare-dest=DIR also compare received files relative to DIR --copy-dest=DIR ... and include copies of unchanged files --link-dest=DIR hardlink to files in DIR when unchanged -z, --compress compress file data during the transfer --compress-level=NUM explicitly set compression level --skip-compress=LIST skip compressing files with suffix in LIST -C, --cvs-exclude auto-ignore files in the same way CVS does -f, --filter=RULE add a file-filtering RULE -F same as --filter='dir-merge /.rsync-filter' repeated: --filter='- .rsync-filter' --exclude=PATTERN exclude files matching PATTERN --exclude-from=FILE read exclude patterns from FILE --include=PATTERN don't exclude files matching PATTERN --includeexclude-from=FILE read includeexclude patterns from FILE --files-frominclude=FILEPATTERN readdon't listexclude offiles source-filematching namesPATTERN from FILE -0, --from0 all *from/filter files are delimited by 0s -s, --protectinclude-argsfrom=FILE read include patterns no space-splitting; wildcard chars onlyfrom FILE --files-addressfrom=ADDRESSFILE bindread addresslist forof outgoingsource-file socketnames tofrom daemonFILE -0, --from0 --port=PORT specify double-colon alternate port number --sockopts=OPTIONSall *from/filter files are delimited by 0s -s, --protect-args specify custom TCP options no space-splitting; wildcard chars only --blocking-io --address=ADDRESS use blockingbind I/Oaddress for outgoing thesocket remoteto shelldaemon --statsport=PORT give some file-transfer stats -8,specify --8-bit-output double-colon alternate port number --sockopts=OPTIONS leave high-bit chars unescaped in outputspecify custom -h, --human-readableTCP options --blocking-io output numbers in a human-readable format use --progress blocking I/O for the remote shell --stats show progress during transfer -P give some file-transfer stats -8, --8-bit-output leave high-bit samechars asunescaped --partial --progressin output -ih, --itemizehuman-changesreadable output numbers in a change-summary for all updateshuman-readable format --progress --out-format=FORMAT show outputprogress updatesduring usingtransfer the specified-P FORMAT --log-file=FILE log what we're doing to the specified FILE same as --partial --log-file-format=FMTprogress -i, --itemize-changes logoutput updatesa usingchange-summary thefor specifiedall FMTupdates --passwordout-fileformat=FILEFORMAT output readupdates daemon-accessusing passwordthe fromspecified FILEFORMAT --listlog-onlyfile=FILE log what we're doing listto the filesspecified insteadFILE of copying them --bwlimit=KBPS--log-file-format=FMT log updates using limit I/O bandwidth; KBytes per secondthe specified FMT --writepassword-batchfile=FILE read daemon-access write a batched update topassword from FILE --only-write-batch=FILE like --write-batch but w/o updating destlist-only --read-batch=FILE list the files instead of copying readthem a batched update from FILE --protocolbwlimit=NUM KBPS force an olderlimit protocol version to be usedI/O bandwidth; KBytes per second --write-iconv=CONVERT_SPECbatch=FILE write requesta charsetbatched conversionupdate ofto filenamesFILE --checksumonly-write-seedbatch=NUMFILE like --write-batch but set block/file checksum seed (advanced) -4, --ipv4w/o updating dest --read-batch=FILE read a batched update from FILE prefer IPv4 -6, --ipv6protocol=NUM force an older protocol version to be used prefer IPv6 --versioniconv=CONVERT_SPEC request charset conversion of filenames print version number (-h) --help --checksum-seed=NUM set block/file checksum seed (advanced) -4, --ipv4 show this help (see below for -h comment) |
Resources
...
prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (see below for -h comment) |