Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Exclude styles directory when rsyncing.

When we are migrating a site, rsync is our friend. Let's get intimate and hands on with file synchronization.

...

Using Drush's built-in RSYNC

Info

note that drush rsync can be funky, especially with older versions of drush. Real rsync in generally recommended.

Basic Usage

Drush aliases makes operations destinations clear. 

Here is some basic usage. We pull images down from dev to our local. Drupal will find the path to each instance's %files directory. Magic.

Note that the slash appears slash appears on the source, but not the target destination.

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync @mysite.dev:%files/images/ @mysite.local:%files/images

Here, we pull down all files from live to local, but exclude drupal's private files

Code Block
language
languagebash
themeFadeToGreybash
$ drush rsync --exclude=private @mysite.live:%files/ @mysite.local:%files


Info

note that you cannot use drush to sync 2 remote instances, like live to test. That would be an on-server operation.

Safety first: passing flags

...

Drush has some option flags of its own we can send to test operations before committing. In this case we are going from our local to staging:

Code Block
language
languagebash
themeFadeToGreybash
$ drush --simulate --debug rsync --mode=razoglp @sitename.local:%files/images/ @sitename.staging:%files/images

...


Excluding Paths

Drush provides the ability to leverage parts of rsync, but will not easily exclude files based on patterns. We can exclude paths, however:

Code Block
languagebash
themeFadeToGrey
languagebash
$ drush rsync -y --mode=razog --exclude-paths=%files/private:%files/css:%files/js:%files/styles:%files/ctools:%files/backup_migrate @mysite.localdev:%files/ @mysite.devlocal:%files/ 

 


Using native rsync for full control of file

...

transfers

 


Here is a verbose exclude list using straight rsync:

Code Block
languagebash
themeFadeToGrey
languagebash
$ rsync -razv --delete --exclude=css --exclude=*_cache --exclude=js --exclude=googleanalytics --exclude=xmlsitemap \
  --exclude=backup_migrate --exclude=styles user@domain.com:/path/to/remote/files/ \
  /path/to/local/files/

We exclude a number of directories here and also delete anything on our local that does not exist on the remote.

 


setting hosts

With native rsync, we don't use drush aliases. However, we can set up shortcuts to hosts in .ssh/config.

Code Block
themeFadeToGrey
# mysite
Host mysite.dev
  Hostname mysite.com
  port 2222
  Compression yes
  User username
Host mysite.test
  Hostname test.mysite.com
  port 2222
  Compression yes
  User username

 


These greatly simplify logging in via SSH. Now, instead of

Code Block
languagebash
themeFadeToGrey
languagebash
$ ssh -p 2222 username@test.mysite.com

you can simply do:

Code Block
languagebash
themeFadeToGrey
languagebash
$ ssh mysite.test


    -e 'ssh -axp 1022'  


By default, symbolic links are not transferred at all.

 


We can replace multiple exclude patterns in the rsync command with a reference to an external file, using a flag, like so: --exclude-from '/var/www/backup/rsync-exclude.txt'

Here are files and directories to exclude from drupal's files directory.

Code Block
languagebash
themeFadeToGrey
languagebash
.DS_Store
css
civicrm
ctools
js
*_cache
googleanalytics
xmlsitemap
backup_migrate
Code Block
titlersync options
collapsetrue
 -v, --verbose               increase verbosity
 -q, --quiet                 suppress non-error messages
     --no-
tmp
styles
.DS_Store
.DS_Store*
.sass-cache
ehthumbs.db
Icon?
Thumbs.db
._*
*.bak
*_cache
*.tgz
*.un~
.buildpath
.project
.settings
*.sublime-project
*.sublime-workspace
*.sublime-projectcompletions
nbproject


rsync to pantheon 

Grab the pantheon credentials from the desitnation account and modify to fit.

Log in to the client server to run this operation, though you can sync a local directory to Pantheon in much the same way.

You will need to get the ssh key for the machine performing the rsync operation to Pantheon up to your control panel if it is not already there.

Code Block
languagebash
# rsync the files to Panthon dev
rsync -trazog --verbose --exclude-from='/var/www/backup/rsync-files-exclude.txt' \
  /var/www/mysite/htdocs/sites/default/files/ \
  --ipv4 -e 'ssh -p 2222' \
  dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX@appserver.dev.b37d48af-d7ab-4d1e-b6cf-XXXXXXXXXXX.drush.in:files;
# enter pantheon dashboard password when prompted


...

Resources



Code Block
titlersync options
collapsetrue
 -v, --verbose               increase verbosity
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place
     --append                append data onto shorter files
     --append-verify         --append w/old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the tree
 -k, --copy-dirlinks         transform symlink to dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve executability
     --chmod=CHMOD           affect file and/or directory permissions
 -A, --acls                  preserve ACLs (implies -p)
 -X, --xattrs                preserve extended attributes
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
     --super                 receiver attempts super-user activities
     --fake-super            store/recover privileged attrs using xattrs
 -S, --sparse                handle sparse files efficiently
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (w/o delta-xfer algorithm)
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that exist on receiver
     --remove-source-files   sender removes synchronized files (non-dir)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from dest dirs
     --delete-before         receiver deletes before transfer (default)
     --delete-during         receiver deletes during xfer, not before
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not before
     --delete-excluded       also delete excluded files from dest dirs
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of dirs even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at end
 -m, --prune-empty-dirs      prune empty directory chains from file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match size and time
     --size-only             skip files that match in size
     --modify-window=NUM     compare mod-times with reduced accuracy
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare received files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with suffix in LIST
 -C, --cvs-exclude           auto-ignore files in the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --includeexclude-from=FILE     read includeexclude patterns from FILE
     --files-frominclude=FILEPATTERN       readdon't listexclude offiles source-filematching namesPATTERN
from FILE
 -0, --from0                 all *from/filter files are delimited by 0s  -s, --protectinclude-argsfrom=FILE     read include patterns   no space-splitting; wildcard chars onlyfrom FILE
       --files-addressfrom=ADDRESSFILE       bindread addresslist forof outgoingsource-file socketnames tofrom daemonFILE
 -0, --from0   --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONSall *from/filter files are delimited by 0s
 -s, --protect-args      specify custom TCP options no space-splitting; wildcard chars only
--blocking-io     --address=ADDRESS      use blockingbind I/Oaddress for outgoing thesocket remoteto shelldaemon
     --statsport=PORT                 give some file-transfer stats  -8,specify --8-bit-output   double-colon alternate port number
     --sockopts=OPTIONS leave high-bit chars unescaped in outputspecify custom -h, --human-readableTCP options
     --blocking-io  output numbers in a human-readable format    use  --progress    blocking I/O for the remote shell
     --stats    show progress during transfer  -P        give some file-transfer stats
 -8, --8-bit-output          leave high-bit samechars asunescaped --partial --progressin output
 -ih, --itemizehuman-changesreadable        output numbers in a change-summary for all updateshuman-readable format
     --progress          --out-format=FORMAT    show outputprogress updatesduring usingtransfer
the specified-P FORMAT      --log-file=FILE         log what we're doing to the specified FILE   same as --partial --log-file-format=FMTprogress
 -i, --itemize-changes       logoutput updatesa usingchange-summary thefor specifiedall FMTupdates
     --passwordout-fileformat=FILEFORMAT     output readupdates daemon-accessusing passwordthe fromspecified FILEFORMAT
     --listlog-onlyfile=FILE         log what we're doing listto the filesspecified insteadFILE
of copying them      --bwlimit=KBPS--log-file-format=FMT   log updates using     limit I/O bandwidth; KBytes per secondthe specified FMT
     --writepassword-batchfile=FILE    read daemon-access write a batched update topassword from FILE
     --only-write-batch=FILE like --write-batch but w/o updating destlist-only            --read-batch=FILE list the files instead of copying readthem
a batched update from FILE
     --protocolbwlimit=NUM  KBPS        force an olderlimit protocol version to be usedI/O bandwidth; KBytes per second
     --write-iconv=CONVERT_SPECbatch=FILE      write requesta charsetbatched conversionupdate ofto filenamesFILE
     --checksumonly-write-seedbatch=NUMFILE like --write-batch but  set block/file checksum seed (advanced)
 -4, --ipv4w/o updating dest
     --read-batch=FILE       read a batched update from FILE
 prefer IPv4  -6, --ipv6protocol=NUM          force an older protocol version to be used
prefer IPv6      --versioniconv=CONVERT_SPEC    request charset conversion of filenames
      print version number
(-h) --help      --checksum-seed=NUM     set block/file checksum seed (advanced)
 -4, --ipv4            show this help (see below for -h comment)

 

Resources

 

...

 prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (see below for -h comment)