When we are migrating a site, rsync is our friend. Let's get intimate and hands on with file synchronization.
...
Using Drush's built-in RSYNC
Info |
---|
note that drush rsync can be funky, especially with older versions of drush. Real rsync in generally recommended. |
Basic Usage
Drush aliases makes operations destinations clear.
Here is some basic usage. We pull images down from dev to our local. Drupal will find the path to each instance's %files directory. Magic.
Note that the slash appears slash appears on the source, but not the target destination.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush rsync @mysite.dev:%files/images/ @mysite.local:%files/images |
Here, we pull down all files from live to local, but exclude drupal's private files
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush rsync --exclude=private @mysite.live:%files/ @mysite.local:%files |
Info |
---|
note that you cannot use drush to sync 2 remote instances, like live to test. That would be an on-server operation. |
Safety first: passing flags
...
Drush has some option flags of its own we can send to test operations before committing. In this case we are going from our local to staging:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush --simulate --debug rsync --mode=razoglp @sitename.local:%files/images/ @sitename.staging:%files/images |
...
Excluding Paths
Drush provides the ability to leverage parts of rsync, but will not easily exclude files based on patterns. We can exclude paths, however:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ drush rsync -y --mode=razog --exclude-paths=%files/private:%files/css:%files/js:%files/styles:%files/ctools:%files/backup_migrate @mysite.localdev:%files/ @mysite.devlocal:%files/ |
Using native rsync for full control of file
...
transfers
Here is a verbose exclude list using straight rsync:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ rsync -razv --delete --exclude=css --exclude=*_cache --exclude=js --exclude=googleanalytics --exclude=xmlsitemap \
--exclude=backup_migrate --exclude=styles user@domain.com:/path/to/remote/files/ \
/path/to/local/files/ |
We exclude a number of directories here and also delete anything on our local that does not exist on the remote.
setting hosts
With native rsync, we don't use drush aliases. However, we can set up shortcuts to hosts in .ssh/config.
Code Block | ||
---|---|---|
| ||
# mysite Host mysite.dev Hostname mysite.com port 2222 Compression yes User username Host mysite.test Hostname test.mysite.com port 2222 Compression yes User username |
These greatly simplify logging in via SSH. Now, instead of
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ ssh -p 2222 username@test.mysite.com |
you can simply do:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ ssh mysite.test |
-e 'ssh -axp 1022'
By default, symbolic links are not transferred at all.
We can replace multiple exclude patterns in the rsync command with a reference to an external file, using a flag, like so: --exclude-from '/var/www/backup/rsync-exclude.txt'
Here are files and directories to exclude from drupal's files directory.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
.DS_Store css civicrm ctools js *_cache googleanalytics xmlsitemap backup_migrate | ||||||
Code Block | ||||||
| ||||||
-v, --verbose increase verbosity -q, --quiet suppress non-error messages --no-motd suppress daemon-mode MOTD (see caveat) -c, --checksum skip based on checksum, not mod-time & size -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) --no-OPTION tmp styles .DS_Store .DS_Store* .sass-cache ehthumbs.db Icon? Thumbs.db ._* *.bak *_cache *.tgz *.un~ .buildpath .project .settings *.sublime-project *.sublime-workspace *.sublime-projectcompletions nbproject |
rsync to pantheon
Grab the pantheon credentials from the desitnation account and modify to fit.
Log in to the client server to run this operation, though you can sync a local directory to Pantheon in much the same way.
You will need to get the ssh key for the machine performing the rsync operation to Pantheon up to your control panel if it is not already there.
Code Block | ||
---|---|---|
| ||
# rsync the files to Panthon dev
rsync -trazog --verbose --exclude-from='/var/www/backup/rsync-files-exclude.txt' \
/var/www/mysite/htdocs/sites/default/files/ \
--ipv4 -e 'ssh -p 2222' \
dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX@appserver.dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX.drush.in:files;
# enter pantheon dashboard password when prompted |
...
Resources
Code Block | ||||
---|---|---|---|---|
| ||||
-v, --verbose increase verbosity turn off an implied OPTION (e.g. --no-D) -r, --recursive-q, --quiet suppress non-error messages recurse into directories -R, -no-relativemotd suppress daemon-mode MOTD (see caveat) use relative path names-c, --checksum --no-implied-dirs skip based don't send implied dirs with --relativeon checksum, not mod-time & size -ba, --backuparchive archive makemode; backupsequals (see --suffixrlptgoD &(no -H,-backupA,-dirX) --backup-dir=DIRno-OPTION make backups into hierarchyturn basedoff inan DIRimplied OPTION (e.g. --no-D) -r, --suffix=SUFFIXrecursive backup suffix (default ~recurse w/o --backup-dir)into directories -uR, --updaterelative use relative path names skip files that are newer on the receiver --inplace --no-implied-dirs don't send implied dirs with --relative -b, --backup update destination files in-place make backups (see --suffix & --appendbackup-dir) --backup-dir=DIR make backups appendinto datahierarchy ontobased shorterin filesDIR --append-verifysuffix=SUFFIX backup suffix (default --append~ w/old data in file checksum -do --backup-dir) -u, --dirsupdate skip files transferthat directoriesare withoutnewer recursingon the -l, --linksreceiver --inplace copy symlinks as symlinksupdate destination -L,files in--copy-linksplace transform symlink into referent file/dir--append --copy-unsafe-links onlyappend "unsafe"data symlinksonto areshorter transformedfiles --safeappend-linksverify --append w/old data ignorein file checksum symlinks that point outside the tree -k, --copy-dirlinks-d, --dirs transform symlink to dirtransfer intodirectories referentwithout dirrecursing -Kl, --keep-dirlinkslinks treat symlinked dir oncopy receiversymlinks as dirsymlinks -HL, --hardcopy-links preservetransform hardsymlink linksinto -p, --permsreferent file/dir --copy-unsafe-links only "unsafe" symlinks are transformed preserve permissions -E, --executabilitysafe-links ignore symlinks preservethat executabilitypoint outside the tree -k, --chmod=CHMODcopy-dirlinks transform symlink affectto filedir and/orinto directoryreferent permissionsdir -AK, --aclskeep-dirlinks treat symlinked dir on receiver as dir preserve ACLs (implies -p) -X, --xattrs-H, --hard-links preserve extendedhard attributeslinks -op, --ownerperms preserve owner (super-user only) -gpermissions -E, --groupexecutability preserve executability --chmod=CHMOD preserve group --devices affect file and/or directory permissions -A, --acls preserve device files (super-user only) --specialspreserve ACLs (implies -p) -X, --xattrs preserve special files -D preserve extended attributes -o, --owner samepreserve asowner (super--devices --specialsuser only) -tg, --timesgroup preserve modificationgroup times -O, --omit-dir-timesdevices omitpreserve directoriesdevice fromfiles --times(super-user only) --superspecials preserve special files -D receiver attempts super-user activities --fake-super store/recover privilegedsame attrsas using xattrs--devices --specials -St, --sparsetimes handle sparsepreserve filesmodification efficientlytimes -nO, --dry-run omit-dir-times perform a trial run with no changes madeomit directories -W,from --whole-filetimes --super copy files whole (w/o delta-xfer algorithm) -x, --one-file-system receiver attempts super-user activities don't cross filesystem boundaries -B, --block-size=SIZEfake-super forcestore/recover aprivileged fixedattrs checksumusing block-sizexattrs -eS, --rsh=COMMANDsparse specify the remote shell to usehandle sparse files efficiently -n, --rsync-path=PROGRAMdry-run specify the rsync to run on remote machineperform a trial run with no --existingchanges made -W, --whole-file skip creating newcopy files on receiver whole (w/o delta-xfer algorithm) -x, --ignoreone-file-existingsystem skipdon't updatingcross filesfilesystem thatboundaries exist on receiver --remove-source-files -B, --block-size=SIZE sender removes synchronized files (non-dir) --delforce a fixed checksum block-size -e, --rsh=COMMAND specify the remote shell to use an alias for --delete-duringrsync-path=PROGRAM specify the rsync to --deleterun on remote machine --existing delete extraneous files from dest dirs skip creating new files --delete-beforeon receiver --ignore-existing receiver deletes before transfer (default) skip updating files that exist --delete-duringon receiver --remove-source-files receiver deletessender duringremoves xfer,synchronized not beforefiles (non-dir) --delete-delaydel find deletions during, delete after an alias for --delete-afterduring --delete receiver deletes after transfer, not before --delete-excluded delete extraneous files from dest dirs also --delete excluded-before files from dest dirs receiver --ignore-errorsdeletes before transfer (default) --delete-during even if there are I/O errors receiver deletes during --forcexfer, not before --delete-delay force deletionfind ofdeletions dirsduring, evendelete ifafter not empty --max-delete=NUMdelete-after don'treceiver deletedeletes moreafter thantransfer, NUMnot filesbefore --maxdelete-size=SIZEexcluded also delete don'texcluded transferfiles anyfrom filedest larger thandirs SIZE --minignore-size=SIZEerrors don'tdelete transfereven anyif filethere smallerare thanI/O SIZEerrors --partialforce force deletion of dirs keepeven partiallyif transferrednot filesempty --partialmax-dirdelete=DIRNUM put adon't partiallydelete transferredmore filethan intoNUM DIRfiles --delaymax-updatessize=SIZE put all updated files into place at end -m, --prune-empty-dirsdon't transfer any file larger than SIZE --min-size=SIZE don't prunetransfer emptyany directoryfile chainssmaller fromthan file-listSIZE --numeric-idspartial don't map uid/gid valueskeep bypartially user/grouptransferred namefiles --timeout=SECONDS set I/O timeout in secondspartial-dir=DIR put a partially transferred file into DIR --delay-updates --contimeout=SECONDS put all setupdated daemonfiles connectioninto timeoutplace inat secondsend -Im, --ignoreprune-empty-timesdirs prune empty directory chains don't skip files that match size and time --size-onlyfrom file-list --numeric-ids don't map skipuid/gid filesvalues that match in sizeby user/group name --modify-windowtimeout=NUMSECONDS compare mod-times withset reducedI/O accuracytimeout in -T, --temp-dir=DIRseconds --contimeout=SECONDS set createdaemon temporaryconnection filestimeout in directoryseconds DIR -yI, --fuzzyignore-times find similar file for basis if no dest filedon't skip files that match size and time --comparesize-dest=DIRonly also compare received files relative to DIR skip files that match --copy-dest=DIRin size --modify-window=NUM ... and includecompare copiesmod-times ofwith unchangedreduced filesaccuracy -T, --linktemp-destdir=DIR hardlink create totemporary files in directory DIR when unchanged -zy, --compressfuzzy compressfind similar file for basis dataif duringno thedest transferfile --compresscompare-level=NUMdest=DIR also compare received files relative to DIR explicitly set compression level--copy-dest=DIR --skip-compress=LIST skip compressing... and include copies of unchanged files with suffix in LIST -C, --cvslink-excludedest=DIR hardlink to auto-ignore files in theDIR samewhen wayunchanged CVS does -fz, --filter=RULEcompress add a file-filtering RULE -F compress file data during the transfer --compress-level=NUM explicitly set compression level same as --skip-filter='dir-merge /.rsync-filter'compress=LIST skip compressing files with suffix in LIST -C, --cvs-exclude auto-ignore files in the same way CVS repeated:does --filter='- .rsync-filter'f, --filter=RULE add a file--exclude=PATTERNfiltering RULE -F exclude files matching PATTERN --exclude-from=FILE read exclude patterns from FILE same as --include=PATTERN filter='dir-merge /.rsync-filter' don't exclude files matching PATTERN --include-from=FILE read include patterns from FILE repeated: --files-from=FILE filter='- .rsync-filter' read list of source-file names from FILE -0, --from0 --exclude=PATTERN exclude files matching PATTERN --exclude-from=FILE read exclude allpatterns *from/filter filesFILE are delimited by 0s -s, --protect-args -include=PATTERN don't no space-splitting; wildcard chars onlyexclude files matching PATTERN --include-addressfrom=ADDRESSFILE read include bindpatterns addressfrom forFILE outgoing socket to daemon --files-portfrom=PORTFILE read list of source-file names from FILE specify double-colon alternate port number -0, --from0 --sockopts=OPTIONS specify custom TCP options all *from/filter files are delimited by 0s -s, --blockingprotect-ioargs no use blocking I/O for the remote shellspace-splitting; wildcard chars only --statsaddress=ADDRESS bind address for outgoing socket to daemon give some file-transfer stats -8, --8-bit-output-port=PORT leavespecify high-bit chars unescaped in output -h, --human-readabledouble-colon alternate port number --sockopts=OPTIONS specify custom TCP options output numbers in a human-readable format--blocking-io use --progress blocking I/O for the remote shell --stats show progress during transfer -P give some file-transfer stats -8, --8-bit-output leave high-bit samechars asunescaped --partial --progressin output -ih, --itemizehuman-changesreadable output numbers in a change-summary for all updateshuman-readable format --progress --out-format=FORMAT show progress outputduring updatestransfer using the-P specified FORMAT --log-file=FILE log what we're doing to the specified FILE same as --partial --progress -i, -log-file-format=FMTitemize-changes logoutput updatesa usingchange-summary thefor specifiedall FMTupdates --passwordout-fileformat=FILEFORMAT read daemon-access password from FILEoutput updates using the specified FORMAT --listlog-onlyfile=FILE log what we're doing listto the filesspecified insteadFILE of copying them --bwlimit=KBPS--log-file-format=FMT log updates using the limit I/O bandwidth; KBytes per secondspecified FMT --writepassword-batchfile=FILE read daemon-access write a batched update topassword from FILE --only-write-batch=FILE like --write-batch but w/o updating destlist-only --read-batch=FILE list the files instead of copying readthem a batched update from FILE --protocolbwlimit=NUM KBPS force an olderlimit protocol version to be usedI/O bandwidth; KBytes per second --write-iconv=CONVERT_SPECbatch=FILE write requesta charsetbatched conversionupdate ofto filenamesFILE --checksumonly-write-seedbatch=NUMFILE like --write-batch but set block/file checksum seed (advanced) -4, --ipv4w/o updating dest --read-batch=FILE read a batched update from FILE prefer IPv4 -6, --ipv6protocol=NUM force an older protocol version to be used prefer IPv6 --versioniconv=CONVERT_SPEC request charset conversion of filenames print version number (-h) --help checksum-seed=NUM set block/file checksum show this help (see below for -h comment) |
Resources
...
seed (advanced)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (see below for -h comment) |