Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Exclude styles directory when rsyncing.

Status
colourYellow
titleDRAFT
When we are migrating a site, rsync is our friend. Let's get intimate and hands on with file synchronization.

...

Using Drush's built-in RSYNC

Info

note that drush rsync can be funky, especially with older versions of drush. Real rsync in generally recommended.

Basic Usage

Drush aliases makes operations destinations clear. 

Here is some basic usage. We pull images down from dev to our local. Drupal will find the path to each instance's %files directory. Magic.

Note that the slash appears slash appears on the source, but not the target destination.

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync @mysite.dev:%files/images/ @mysite.local:%files/images

Here, we pull down all files from live to local, but exclude drupal's private files

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync --exclude=private @mysite.live:%files/ @mysite.local:%files


Info

note that you cannot use drush to sync 2 remote instances, like live to test. That would be an on-server operation.

Safety first: passing flags

...

Drush has some option flags of its own we can send to test operations before committing. In this case we are going from our local to staging:

Code Block
themelanguageFadeToGreybash
languagethemebashFadeToGrey
$ drush --simulate --debug rsync --mode=razoglp @sitename.local:%files/images/ @sitename.staging:%files/images

 


Excluding Paths

Drush provides the ability to leverage parts of rsync, but will not easily exclude files based on patterns. We can exclude paths, however:

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync -y --mode=razog --exclude-paths=%files/private:%files/css:%files/js:%files/styles:%files/ctools:%files/backup_migrate @mysite.localdev:%files/ @mysite.devlocal:%files/ 

 


Using native rsync for full control of file

...

transfers


Here is a verbose exclude list using straight rsync:

Code Block
languagebash
themeFadeToGrey
languagebash
$ rsync -razv --delete --exclude=css --exclude=*_cache --exclude=js --exclude=googleanalytics --exclude=xmlsitemap \
  --exclude=backup_migrate --exclude=styles user@domain.com:/path/to/remote/files/ \
  /path/to/local/files/

We exclude a number of directories here and also delete anything on our local that does not exist on the remote.

 


setting hosts

With native rsync, we don't use drush aliases. However, we can set up shortcuts to hosts in .ssh/config.

Code Block
themeFadeToGrey
# mysite
Host mysite.dev
  Hostname mysite.com
  port 2222
  Compression yes
  User username
Host mysite.test
  Hostname test.mysite.com
  port 2222
  Compression yes
  User username

...


These greatly simplify logging in via SSH. Now, instead of

Code Block
languagebash
themeFadeToGrey
languagebash
$ ssh -p 2222 username@test.mysite.com

you can simply do:

Code Block
languagebash
themeFadeToGrey
languagebash
$ ssh mysite.test

...


  -e 'ssh -axp 1022' 

 


By default, symbolic links are not transferred at all. 


We can replace multiple exclude patterns in the rsync command with a reference to an external file, using a flag, like so: --exclude-from '/var/www/backup/rsync-exclude.txt'

Here are files and directories to exclude from drupal's files directory.

title
Code Block
languagebash
themeFadeToGrey
languagebash
css
civicrm
ctools
js
googleanalytics
xmlsitemap
backup_migrate
tmp
styles
.DS_Store
.DS_Store*
.sass-cache
ehthumbs.db
Icon?
Thumbs.db
._*
*.bak
*_cache
*.tgz
*.un~
.buildpath
.project
.settings
*.sublime-project
*.sublime-workspace
*.sublime-projectcompletions
nbproject
Code Block


rsync

...

to pantheon 

Grab the pantheon credentials from the desitnation account and modify to fit.

Log in to the client server to run this operation, though you can sync a local directory to Pantheon in much the same way.

You will need to get the ssh key for the machine performing the rsync operation to Pantheon up to your control panel if it is not already there.

Code Block
languagebash
# rsync the files to Panthon dev
rsync -trazog --verbose --exclude-from='/var/www/backup/rsync-files-exclude.txt' \
  /var/www/mysite/htdocs/sites/default/files/ \
  --ipv4 -e 'ssh -p 2222' \
  dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX@appserver.dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX.drush.in:files;
# enter pantheon dashboard password when prompted


...

Resources



Code Block
titlersync options
collapsetrue
 -v, --verbose             turn off anincrease implied OPTION (e.g. --no-D)
 -rverbosity
 -q, --recursivequiet             recurse into directories  -R,suppress non--relativeerror messages
     --no-motd       use relative path names     suppress daemon--no-implied-dirsmode MOTD (see caveat)
 -c, --checksum       don't send implied dirs with --relative     skip based on checksum, not mod-time & size
 -ba, --backuparchive               archive make backups (seemode; equals --suffixrlptgoD &(no -H,-backupA,-dirX)
     --backupno-dir=DIROPTION        make backups into hierarchy based inturn DIRoff an implied    --suffix=SUFFIX         backup suffix (default ~ w/oOPTION (e.g. --backupno-dirD)
 -ur, --updaterecursive             recurse into directories
skip files that are newer on the receiver-R, --relative           --inplace   use relative path names
     --no-implied-dirs   update destination files in-place don't send implied dirs with --appendrelative
 -b, --backup             append data onto shortermake filesbackups (see --suffix &   --appendbackup-verifydir)
        --backup-append w/old data in file checksum
 -d, --dirs      dir=DIR        make backups into hierarchy based in DIR
       --suffix=SUFFIX         backup transfer directories without recursingsuffix (default ~ w/o --backup-dir)
 -lu, --linksupdate                skip files that are newer on the copyreceiver
symlinks as symlinks  -L, --copy-linksinplace            transform symlink into referentupdate file/dirdestination files in-place
   -  -copy-unsafe-links-append            only "unsafe" symlinks are transformedappend data onto shorter files
 --safe-links    --append-verify        ignore symlinks that point outside the tree--append w/old data in file checksum
 -kd, --copy-dirlinksdirs         transform symlink to dir into referent dir  -K, --keep-dirlinks transfer directories without recursing
 -l, --links              treat symlinked dir oncopy receiversymlinks as dirsymlinks
 -HL, --hardcopy-links            preservetransform hardsymlink linksinto  -p, --permsreferent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
  preserve permissions  -E, -safe-executabilitylinks         preserve executability
     --chmod=CHMOD   ignore symlinks that point outside the tree
 -k, --copy-dirlinks         transform symlink affectto filedir and/orinto directoryreferent permissionsdir
 -AK, --aclskeep-dirlinks         treat symlinked dir on receiver as dir
  preserve ACLs (implies -p)
 -X, --xattrs-H, --hard-links            preserve hard links
 preserve extended attributes
 -o-p, --ownerperms                 preserve owner (super-user only)
 -gpermissions
 -E, --group  executability               preserve groupexecutability
     --deviceschmod=CHMOD           affect file and/or directory preservepermissions
device files (super-user only)-A, --acls       --specials           preserve ACLs (implies preserve-p)
special files-X,  --Dxattrs                preserve          same as --devices --specials
 -textended attributes
 -o, --timesowner                 preserve modification timesowner (super-user only)
 -Og, --omit-dir-timesgroup         omit directories from --times     preserve --supergroup
     --devices           receiver attempts super   preserve device files (super-user activitiesonly)
     --fake-superspecials            store/recover privileged attrspreserve usingspecial xattrsfiles
 -S, --sparseD                       handle sparse files efficientlysame as --n,devices --dry-runspecials
 -t, --times            perform a trial run with nopreserve changesmodification madetimes
 -WO, --whole-fileomit-dir-times        omit directories from --times
 copy files whole (w/o delta-xfer algorithm)
 -x, --one-file-system--super              don't cross filesystem boundariesreceiver attempts -B,super-user activities
     --block-size=SIZEfake-super        force a fixed checksum block-size store/recover privileged attrs using xattrs
 -eS, --rsh=COMMANDsparse           specify the remote shell to usehandle sparse files efficiently
 -n, --rsyncdry-path=PROGRAMrun    specify the rsync to run on remote machine    perform a --existingtrial run with no changes made
 -W, --whole-file      skip creating new files on receiver copy files whole (w/o  --ignore-existing    delta-xfer algorithm)
 -x, --one-file-system   skip updating files that existdon't oncross receiverfilesystem boundaries
   -B, --remove-source-filesblock-size=SIZE   sender removes synchronized files (non-dir)force a fixed checksum block-size
 --dele, --rsh=COMMAND           specify the remote shell to use
  an alias for --delete-duringrsync-path=PROGRAM    specify the rsync to run on remote machine
     --deleteexisting              skip creating delete extraneousnew files fromon dest dirsreceiver
     --deleteignore-beforeexisting       skip updating receiverfiles deletesthat beforeexist transferon (default)receiver
     --deleteremove-source-duringfiles   sender removes synchronized files (non-dir)
 receiver deletes during xfer, not--del before      --delete-delay          find deletions during,an deletealias afterfor      --delete-afterduring
     --delete    receiver deletes after transfer, not before      --delete-excluded       also delete excludedextraneous files from dest dirs
     --ignoredelete-errorsbefore         deletereceiver evendeletes ifbefore there are I/O errorstransfer (default)
      --forcedelete-during         receiver deletes during xfer, not before
   force deletion of--delete-delay dirs even if not empty      --max-delete=NUM   find deletions during, delete after
    don't --delete-after more than NUM files      receiver deletes after transfer, not before
     --maxdelete-size=SIZEexcluded       also delete don'texcluded transferfiles anyfrom filedest largerdirs
than SIZE      --minignore-size=SIZEerrors         don'tdelete transfereven anyif filethere smallerare thanI/O SIZEerrors
     --partialforce               keep partially transferred files force deletion of dirs even if not empty
     --partialmax-dirdelete=DIRNUM       put adon't partiallydelete transferredmore filethan intoNUM DIRfiles
     --delay-updatesmax-size=SIZE         don't transfer putany allfile updatedlarger filesthan intoSIZE
place at end  -m, --prune-empty-dirsmin-size=SIZE       prune empty directorydon't chainstransfer fromany file-list smaller than SIZE
     --numeric-idspartial            don't map uid/gid valueskeep bypartially user/grouptransferred namefiles
     --partial-timeoutdir=SECONDSDIR       put set I/O timeout in secondsa partially transferred file into DIR
     --delay-contimeout=SECONDSupdates    set daemon  connection timeout input secondsall updated -I, --ignore-times      files into place at end
 -m, --prune-empty-dirs    don't skip filesprune thatempty matchdirectory sizechains andfrom timefile-list
     --sizenumeric-onlyids           don't map skipuid/gid filesvalues thatby matchuser/group inname
size      --modify-windowtimeout=NUMSECONDS     compare mod-times withset reducedI/O accuracytimeout in -T, --temp-dir=DIRseconds
     --contimeout=SECONDS    set createdaemon temporaryconnection filestimeout in directoryseconds
DIR  -yI, --ignore-fuzzytimes          don't skip files that match size and findtime
similar file for basis if no--size-only dest file      --compare-dest=DIR     skip alsofiles comparethat receivedmatch filesin relative tosize
DIR      --copymodify-destwindow=DIRNUM     compare mod-times with reduced ...accuracy
and include copies of unchanged files
     --link-dest -T, --temp-dir=DIR         hardlink tocreate temporary files in directory DIR
when unchanged
 -zy, --compressfuzzy                 find compresssimilar file data during the transferfor basis if no dest file
     --compresscompare-level=NUM    explicitly set compression leveldest=DIR      also compare received files relative to DIR
     --skipcopy-compressdest=LISTDIR    skip compressing files with suffix in... LISTand  -C,include copies of unchanged files
     --cvslink-excludedest=DIR         hardlink  auto-ignoreto files in theDIR samewhen way CVSunchanged
does  -fz, --filter=RULEcompress           add a file-filtering RULEcompress file -Fdata during the transfer
     --compress-level=NUM    explicitly set compression level
      --skip-compress=LIST   same as --filter='dir-merge /.rsync-filter'
      skip compressing files with suffix in LIST
 -C, --cvs-exclude           auto-ignore files in the same way CVS does
    repeated: -f, --filter='- .rsync-filter'RULE       --exclude=PATTERN    add a  exclude files matching PATTERNfile-filtering RULE
 -F      --exclude-from=FILE     read exclude patterns from FILE      --include=PATTERN     same as don't exclude files matching PATTERN--filter='dir-merge /.rsync-filter'
        --include-from=FILE     read include patterns from FILE      --files-from=FILE       read list of source-file names from FILE
 -0, --from0repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
      all *from/filter files are delimited by 0s
 -s, --protect-args --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN   no space-splitting; wildcard chars onlydon't exclude files matching PATTERN
 --address=ADDRESS    --include-from=FILE   bind address forread outgoinginclude socketpatterns tofrom daemonFILE
     --portfiles-from=PORTFILE       read list     specify double-colon alternate port number
     --sockopts=OPTIONSof source-file names from FILE
 -0, --from0         specify custom TCP options     all *from/filter files are delimited by 0s
 -s, --blockingprotect-ioargs          no use blocking I/O for the remote shellspace-splitting; wildcard chars only
        --statsaddress=ADDRESS       bind address for outgoing socket to daemon
   give some file-transfer stats
 -8, --8-bit-output--port=PORT             specify double-colon alternate leave high-bit chars unescaped in output
 -h, --human-readableport number
     --sockopts=OPTIONS      specify custom outputTCP numbersoptions
in a human-readable format      --blocking-progressio           use blocking I/O showfor progressthe duringremote transfershell
 -P    --stats                 give     same as --partial --progresssome file-transfer stats
 -i8, --8-itemizebit-changesoutput       output a change-summary forleave allhigh-bit updateschars unescaped in output
 -h, --out-format=FORMAThuman-readable        output updatesnumbers usingin thea specifiedhuman-readable FORMATformat
     --log-file=FILEprogress          log what we're doing toshow theprogress specifiedduring FILEtransfer
 -P    --log-file-format=FMT   log updates using the specified FMT      --password-file=FILE    read daemon-access password from FILEsame as --partial --progress
 -i, --listitemize-onlychanges       output a change-summary for all updates
list the files instead of copying them
     --bwlimit=KBPS --out-format=FORMAT     output updates using the  limit I/O bandwidth; KBytes per secondspecified FORMAT
        --writelog-batchfile=FILE      write a batched update to    log what we're doing to the specified FILE
     --onlylog-writefile-batchformat=FILEFMT like --write-batch but w/o updating dest log updates using the specified FMT
     --readpassword-batchfile=FILE       read a batched updatedaemon-access password from FILE
     --protocol=NUMlist-only            force anlist olderthe protocolfiles versioninstead toof becopying usedthem
     --iconvbwlimit=CONVERT_SPECKBPS    request charset conversion of filenames  limit I/O bandwidth;  --checksum-seed=NUM   KBytes per second
 set block/file checksum seed (advanced)  -4, -write-ipv4batch=FILE      write a batched update to FILE
      prefer IPv4
 -6, --ipv6    --only-write-batch=FILE like --write-batch but w/o updating dest
     --read-batch=FILE       read prefer IPv6a batched update from FILE
     --versionprotocol=NUM          force an older protocol version to be used
 print version number (-h) --helpiconv=CONVERT_SPEC    request charset conversion of filenames
     --checksum-seed=NUM     set showblock/file thischecksum helpseed (seeadvanced)
below for-4, --h comment)

 

Resources

 

...

ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (see below for -h comment)