Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Exclude styles directory when rsyncing.

When we are migrating a site, rsync is our friend. Let's get intimate and hands on with file synchronization.

...

Using Drush's built-in RSYNC

Info

note that drush rsync can be funky, especially with older versions of drush. Real rsync in generally recommended.

Basic Usage

Drush aliases makes operations destinations clear. 

Here is some basic usage. We pull images down from dev to our local. Drupal will find the path to each instance's %files directory. Magic.

Note that the slash appears slash appears on the source, but not the target destination.

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync @mysite.dev:%files/images/ @mysite.local:%files/images

Here, we pull down all files from live to local, but exclude drupal's private files

Code Block
language
languagebash
themeFadeToGreybash
$ drush rsync --exclude=private @mysite.live:%files/ @mysite.local:%files


Info

note that you cannot use drush to sync 2 remote instances, like live to test. That would be an on-server operation.

Safety first: passing flags

...

Drush has some option flags of its own we can send to test operations before committing. In this case we are going from our local to staging:

Code Block
language
languagebash
themeFadeToGreybash
$ drush --simulate --debug rsync --mode=razoglp @sitename.local:%files/images/ @sitename.staging:%files/images

...


Excluding Paths

Drush provides the ability to leverage parts of rsync, but will not easily exclude files based on patterns. We can exclude paths, however:

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync -y --mode=razog --exclude-paths=%files/private:%files/css:%files/js:%files/styles:%files/ctools:%files/backup_migrate @mysite.localdev:%files/ @mysite.devlocal:%files/ 

 


Using native rsync for full control of file

...

transfers

 


Here is a verbose exclude list using straight rsync:

Code Block
languagebash
themeFadeToGrey
languagebash
$ rsync -razv --delete --exclude=css --exclude=*_cache --exclude=js --exclude=googleanalytics --exclude=xmlsitemap \
  --exclude=backup_migrate --exclude=styles user@domain.com:/path/to/remote/files/ \
  /path/to/local/files/

We exclude a number of directories here and also delete anything on our local that does not exist on the remote.

 


setting hosts

With native rsync, we don't use drush aliases. However, we can set up shortcuts to hosts in .ssh/config.

Code Block
themeFadeToGrey
# mysite
Host mysite.dev
  Hostname mysite.com
  port 2222
  Compression yes
  User username
Host mysite.test
  Hostname test.mysite.com
  port 2222
  Compression yes
  User username

 


These greatly simplify logging in via SSH. Now, instead of

Code Block
languagebash
themeFadeToGrey
languagebash
$ ssh -p 2222 username@test.mysite.com

you can simply do:

Code Block
languagebash
themeFadeToGrey
languagebash
$ ssh mysite.test


    -e 'ssh -axp 1022'  


By default, symbolic links are not transferred at all.

 


We can replace multiple exclude patterns in the rsync command with a reference to an external file, using a flag, like so: --exclude-from '/var/www/backup/rsync-exclude.txt'

Here are files and directories to exclude from drupal's files directory.

Code Block
languagebash
themeFadeToGrey
languagebash
.DS_Store
css
civicrm
ctools
js
*_cache
googleanalytics
xmlsitemap
backup_migrate
Code Block
titlersync options
collapsetrue
 -v, --verbose               increase verbosity
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION
tmp
styles
.DS_Store
.DS_Store*
.sass-cache
ehthumbs.db
Icon?
Thumbs.db
._*
*.bak
*_cache
*.tgz
*.un~
.buildpath
.project
.settings
*.sublime-project
*.sublime-workspace
*.sublime-projectcompletions
nbproject


rsync to pantheon 

Grab the pantheon credentials from the desitnation account and modify to fit.

Log in to the client server to run this operation, though you can sync a local directory to Pantheon in much the same way.

You will need to get the ssh key for the machine performing the rsync operation to Pantheon up to your control panel if it is not already there.

Code Block
languagebash
# rsync the files to Panthon dev
rsync -trazog --verbose --exclude-from='/var/www/backup/rsync-files-exclude.txt' \
  /var/www/mysite/htdocs/sites/default/files/ \
  --ipv4 -e 'ssh -p 2222' \
  dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX@appserver.dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX.drush.in:files;
# enter pantheon dashboard password when prompted


...

Resources



Code Block
titlersync options
collapsetrue
 -v, --verbose               increase verbosity
 turn off an implied OPTION (e.g. --no-D)
 -r, --recursive-q, --quiet                 suppress non-error messages
 recurse into directories  -R, -no-relativemotd               suppress daemon-mode MOTD (see caveat)
  use relative path names-c, --checksum         --no-implied-dirs     skip based don't send implied dirs with --relativeon checksum, not mod-time & size
 -ba, --backuparchive               archive makemode; backupsequals (see --suffixrlptgoD &(no -H,-backupA,-dirX)
     --backup-dir=DIRno-OPTION          make backups into hierarchyturn basedoff inan DIRimplied OPTION (e.g. --no-D)
 -r, --suffix=SUFFIXrecursive          backup suffix (default ~recurse w/o --backup-dir)into directories
 -uR, --updaterelative              use relative path names
skip files that are newer on the receiver
     --inplace        --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup            update destination files in-place make backups (see --suffix & --appendbackup-dir)
     --backup-dir=DIR        make backups appendinto datahierarchy ontobased shorterin filesDIR
     --append-verifysuffix=SUFFIX         backup suffix (default --append~ w/old data in file checksum
 -do --backup-dir)
 -u, --dirsupdate                skip files transferthat directoriesare withoutnewer recursingon the -l, --linksreceiver
     --inplace            copy symlinks as symlinksupdate destination -L,files in--copy-linksplace
           transform symlink into referent file/dir--append           --copy-unsafe-links     onlyappend "unsafe"data symlinksonto areshorter transformedfiles
     --safeappend-linksverify         --append w/old data ignorein file checksum
symlinks that point outside the tree
 -k, --copy-dirlinks-d, --dirs               transform symlink to dirtransfer intodirectories referentwithout dirrecursing
 -Kl, --keep-dirlinkslinks              treat symlinked dir oncopy receiversymlinks as dirsymlinks
 -HL, --hardcopy-links            preservetransform hardsymlink linksinto  -p, --permsreferent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
  preserve permissions  -E, --executabilitysafe-links            ignore symlinks preservethat executabilitypoint outside the tree
 -k, --chmod=CHMODcopy-dirlinks         transform symlink affectto filedir and/orinto directoryreferent permissionsdir
 -AK, --aclskeep-dirlinks         treat symlinked dir on receiver as   dir
preserve ACLs (implies -p)
 -X, --xattrs-H, --hard-links                preserve extendedhard attributeslinks
 -op, --ownerperms                 preserve owner (super-user only)
 -gpermissions
 -E, --groupexecutability         preserve executability
     --chmod=CHMOD preserve group      --devices   affect file and/or directory permissions
 -A, --acls        preserve device files (super-user only)      --specialspreserve ACLs (implies -p)
 -X, --xattrs        preserve special files  -D    preserve extended attributes
 -o, --owner                 samepreserve asowner (super--devices --specialsuser only)
 -tg, --timesgroup                 preserve modificationgroup
  times  -O, --omit-dir-timesdevices               omitpreserve directoriesdevice fromfiles --times(super-user only)
     --superspecials              preserve special files
 -D    receiver attempts super-user activities      --fake-super            store/recover privilegedsame attrsas using xattrs--devices --specials
 -St, --sparsetimes                handle sparsepreserve filesmodification efficientlytimes
 -nO, --dry-run omit-dir-times              perform a trial run with no changes madeomit directories -W,from --whole-filetimes
     --super      copy files whole (w/o delta-xfer algorithm)  -x, --one-file-system   receiver attempts super-user activities
don't cross filesystem boundaries  -B, --block-size=SIZEfake-super            forcestore/recover aprivileged fixedattrs checksumusing block-sizexattrs
 -eS, --rsh=COMMANDsparse           specify the remote shell to usehandle sparse files efficiently
 -n, --rsync-path=PROGRAMdry-run        specify the rsync to run on remote machineperform a trial run with no --existingchanges made
 -W, --whole-file          skip creating newcopy files on receiver
    whole (w/o delta-xfer algorithm)
 -x, --ignoreone-file-existingsystem       skipdon't updatingcross filesfilesystem thatboundaries
exist on receiver
     --remove-source-files -B, --block-size=SIZE    sender removes synchronized files (non-dir)
     --delforce a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
  an alias for --delete-duringrsync-path=PROGRAM    specify the rsync to --deleterun on remote machine
     --existing       delete extraneous files from dest dirs  skip creating new files --delete-beforeon receiver
     --ignore-existing  receiver deletes before transfer (default) skip updating files that exist --delete-duringon receiver
     --remove-source-files  receiver deletessender duringremoves xfer,synchronized not beforefiles (non-dir)
     --delete-delaydel            find deletions during, delete after   an alias for --delete-afterduring
     --delete     receiver deletes after transfer, not before      --delete-excluded  delete extraneous files from dest dirs
    also --delete excluded-before files from dest dirs     receiver --ignore-errorsdeletes before transfer (default)
     --delete-during even if there are I/O errors   receiver deletes during --forcexfer, not before
     --delete-delay         force deletionfind ofdeletions dirsduring, evendelete ifafter
not empty      --max-delete=NUMdelete-after          don'treceiver deletedeletes moreafter thantransfer, NUMnot filesbefore
     --maxdelete-size=SIZEexcluded       also delete don'texcluded transferfiles anyfrom filedest larger thandirs
SIZE      --minignore-size=SIZEerrors         don'tdelete transfereven anyif filethere smallerare thanI/O SIZEerrors
     --partialforce                 force deletion of dirs keepeven partiallyif transferrednot filesempty
     --partialmax-dirdelete=DIRNUM       put adon't partiallydelete transferredmore filethan intoNUM DIRfiles
     --delaymax-updatessize=SIZE         put all updated files into place at end
 -m, --prune-empty-dirsdon't transfer any file larger than SIZE
     --min-size=SIZE         don't prunetransfer emptyany directoryfile chainssmaller fromthan file-listSIZE
     --numeric-idspartial            don't map uid/gid valueskeep bypartially user/grouptransferred namefiles
     --timeout=SECONDS       set I/O timeout in secondspartial-dir=DIR       put a partially transferred file into DIR
     --delay-updates       --contimeout=SECONDS  put all setupdated daemonfiles connectioninto timeoutplace inat secondsend
 -Im, --ignoreprune-empty-timesdirs      prune empty directory chains don't skip files that match size and time
     --size-onlyfrom file-list
     --numeric-ids           don't map skipuid/gid filesvalues that match in sizeby user/group name
     --modify-windowtimeout=NUMSECONDS     compare mod-times withset reducedI/O accuracytimeout in -T, --temp-dir=DIRseconds
     --contimeout=SECONDS    set createdaemon temporaryconnection filestimeout in directoryseconds
DIR
 -yI, --fuzzyignore-times                 find similar file for basis if no dest filedon't skip files that match size and time
     --comparesize-dest=DIRonly      also compare received files relative to DIR skip files that match  --copy-dest=DIRin size
     --modify-window=NUM   ... and includecompare copiesmod-times ofwith unchangedreduced filesaccuracy
    -T, --linktemp-destdir=DIR         hardlink create totemporary files in directory DIR
when unchanged
 -zy, --compressfuzzy                 compressfind similar file for basis dataif duringno thedest transferfile
     --compresscompare-level=NUMdest=DIR      also compare received files relative to DIR
     explicitly set compression level--copy-dest=DIR         --skip-compress=LIST    skip compressing... and include copies of unchanged files
with suffix in LIST
 -C, --cvslink-excludedest=DIR         hardlink to auto-ignore files in theDIR samewhen wayunchanged
CVS does
 -fz, --filter=RULEcompress           add a file-filtering RULE
 -F          compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
   same as --skip-filter='dir-merge /.rsync-filter'compress=LIST    skip compressing files with suffix in LIST
 -C, --cvs-exclude           auto-ignore files in the same way CVS repeated:does
 --filter='- .rsync-filter'f, --filter=RULE           add a file--exclude=PATTERNfiltering RULE
 -F    exclude files matching PATTERN      --exclude-from=FILE     read exclude patterns from FILE    same as --include=PATTERN filter='dir-merge /.rsync-filter'
     don't exclude files matching PATTERN      --include-from=FILE     read include patterns from FILE     repeated: --files-from=FILE filter='- .rsync-filter'
     read list of source-file names from FILE
 -0, --from0    --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude allpatterns *from/filter filesFILE
are delimited by 0s  -s, --protect-args  -include=PATTERN       don't no space-splitting; wildcard chars onlyexclude files matching PATTERN
      --include-addressfrom=ADDRESSFILE     read include bindpatterns addressfrom forFILE
outgoing socket to daemon      --files-portfrom=PORTFILE       read list of source-file names from FILE
specify double-colon alternate port number -0, --from0          --sockopts=OPTIONS       specify custom TCP options
    all *from/filter files are delimited by 0s
 -s, --blockingprotect-ioargs          no use blocking I/O for the remote shellspace-splitting; wildcard chars only
        --statsaddress=ADDRESS       bind address for outgoing socket to daemon
   give some file-transfer stats
 -8, --8-bit-output-port=PORT             leavespecify high-bit chars unescaped in output
 -h, --human-readabledouble-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
output  numbers in a human-readable format--blocking-io           use --progress     blocking I/O for the remote shell
     --stats    show progress during transfer  -P        give some file-transfer stats
 -8, --8-bit-output          leave high-bit samechars asunescaped --partial --progressin output
 -ih, --itemizehuman-changesreadable        output numbers in a change-summary for all updateshuman-readable format
     --progress           --out-format=FORMAT   show progress outputduring updatestransfer
using the-P specified FORMAT      --log-file=FILE         log what we're doing to the specified FILE  same as --partial --progress
 -i, -log-file-format=FMTitemize-changes       logoutput updatesa usingchange-summary thefor specifiedall FMTupdates
     --passwordout-fileformat=FILEFORMAT     read daemon-access password from FILEoutput updates using the specified FORMAT
     --listlog-onlyfile=FILE         log what we're doing listto the filesspecified insteadFILE
of copying them      --bwlimit=KBPS--log-file-format=FMT   log updates using the    limit I/O bandwidth; KBytes per secondspecified FMT
     --writepassword-batchfile=FILE    read daemon-access write a batched update topassword from FILE
     --only-write-batch=FILE like --write-batch but w/o updating destlist-only            --read-batch=FILE list the files instead of copying readthem
a batched update from FILE      --protocolbwlimit=NUM  KBPS        force an olderlimit protocol version to be usedI/O bandwidth; KBytes per second
     --write-iconv=CONVERT_SPECbatch=FILE      write requesta charsetbatched conversionupdate ofto filenamesFILE
     --checksumonly-write-seedbatch=NUMFILE like --write-batch but  set block/file checksum seed (advanced)
 -4, --ipv4w/o updating dest
     --read-batch=FILE       read a batched update from FILE
 prefer IPv4  -6, --ipv6protocol=NUM          force an older protocol version to be used
prefer IPv6      --versioniconv=CONVERT_SPEC    request charset conversion of filenames
      print version number
(-h) --help         checksum-seed=NUM     set block/file checksum  show this help (see below for -h comment)

 

Resources

 

...

seed (advanced)
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (see below for -h comment)