doveadm-sync
DOVEADM-SYNC(1) Dovecot DOVEADM-SYNC(1)
NAME
doveadm-sync - Dovecot's two-way mailbox synchronization utility
doveadm-backup - Dovecot's one-way mailbox synchronization utility
SYNOPSIS
doveadm [-Dv] sync [-u user|-A|-F file] [-S socket_path] [-1fPRU] [-l
secs] [-r rawlog_path] [-m mailbox] [-g mailbox_guid] [-n namespace|-N]
[-x exclude] [-s state] [-T secs] [-t start date] [-e end date] -d|des-
tination
doveadm [-Dv] backup [-u user|-A|-F file] [-S socket_path] [-fPRU] [-l
secs] [-r rawlog_path] [-m mailbox] [-g mailbox_guid] [-n namespace|-N]
[-x exclude] [-s state] [-T secs] [-t start date] [-e end date] -d|des-
tination
DESCRIPTION
dsync is Dovecot's mailbox synchronization utility. It can be used for
several different use cases: Two-way synchronization of mailboxes, cre-
ating backups of mails, and convert mailboxes from/to different mailbox
formats. All of these can be used within the same server or between
different servers (via ssh(1) or tcp connections). Remote mailboxes
can be accessed also via IMAP protocol, which allows using dsync for
mailbox migration purposes.
You can run dsync in one of three modes:
o doveadm backup performs one-way synchronization. If
there are any changes in the destination they will be
deleted, so the destination will look exactly like the
source.
o doveadm sync performs two-way synchronization. It merges
all changes without losing anything. Both the mailboxes
will end up looking identical after the synchronization
is finished.
o doveadm sync -1 performs one-way synchronization, but it
merges the changes in destination without deleting any-
thing. This doesn't currently work perfectly, so its use
should be limited. Its main purpose is that during mail-
box migration you can run doveadm backup multiple times,
then switch mails to be delivered to the new mailbox and
run doveadm sync -1 once more to transfer any last new
mails from the old mailbox.
The one-way algorithm is the same as two-way dsync algo-
rithm except the source account is not modified. It
fetches the message's GUID (Global UID), which is used to
identify any conflicting UIDs in messages. As long as the
source and destination side has matching UID<->GUID map-
ping, those emails are assumed to be synced correctly.
Only after the first mismatch will changes begin.
Example: Source mailbox has messages UID 1..5; source
mailbox is sync'd using doveadm backup to the destina-
tion. Subsequently, UID 6 is delivered to the source
mailbox and UID 1 is expunged from the destination mail-
box. In this example, UID 1 is kept removed (in destina-
tion) because UID 1..5 have identical Date+Message-ID
headers. UID 6 is not seen in destination so it's copied.
If both source and destination have UID 6, but the mes-
sages are different, the headers don't match and both the
messages are kept in the destination but they're given
new UIDs 7 and 8 just to be sure any client didn't get
confused about what UID 11 actually was. Thus, one-way
sync begins to quickly diverge from the source mailbox
once changes start to occur on either side; one-way sync
should therefore normally only be used within a short pe-
riod of time after a doveadm backup or doveadm sync com-
mand was used to synchronize the mailboxes.
There are also three different synchronization algorithms:
o Full synchronization (-f parameter) scans through all the
messages in all the mailboxes. This guarantees that ev-
erything will be synchronized, but it's unnecessarily
slow for incremental synchronization.
o Fast synchronization (default) first attempts to find
mailboxes that have changed, and synchronize only those.
This is done by checking the mailboxes' metadata (NEXTUID
and HIGHESTMODSEQ). Usually this works fine, especially
with one-way synchronization, but if both sides do ex-
actly the same number of changes, the metadata may end up
containing the same values even if the changes were dif-
ferent.
o Stateful synchronization (-s parameter) is the most effi-
cient way to synchronize mailboxes. It relies on having
the earlier dsync run's state saved somewhere and being
passed to the next dsync run. Based on this state dsync
can send only the changes that happened after the previ-
ous dsync run. As long as the state or the mailboxes
aren't corrupted this algorithm should work perfectly.
The replicator process uses this internally to perform
most of the synchronization.
The syncing is done as perfectly as possible: an IMAP or a POP3 client
shouldn't be able to notice any differences between the two mailboxes.
Two-way syncing means that it's safe to do any kind of modifications in
both sides, and dsync will merge the changes without losing any changes
done on either side. This is possible because dsync can access Dove-
cot's index logs that keep track of changes. It's of course possible
to have conflicts during merging, these are resolved in a safe way.
See the dsync design document for more information.
dsync uses the same configuration files as the rest of Dovecot (via
doveconf(1) binary). The entire configuration can be changed by giving
-c parameter to another configuration file, or using -o parameter to
override specific settings. When executing a remote dsync program it
works the same way: it uses its own local configuration.
dsync can be run completely standalone. It doesn't require any Dovecot
server processes to be running, except when using -u parameter to do a
userdb lookup from auth process.
dsync can sync either one or multiple users using the -u or -A parame-
ters. For continuous replication you can use the Dovecot replicator
process, which automatically runs dsync whenever messages have changed.
OPTIONS
Global doveadm(1) options:
-D Enables verbosity and debug messages.
-o setting=value
Overrides the configuration setting from /etc/dovecot/dove-
cot.conf and from the userdb with the given value. In order to
override multiple settings, the -o option may be specified mul-
tiple times.
-v Enables verbosity, including progress counter.
Command specific options:
-1 Do one-way synchronization instead of two-way synchronization.
-A If the -A option is present, the command will be performed for
all users. Using this option in combination with system users
from userdb { driver = passwd } is not recommended, because it
contains also users with a lower UID than the one configured
with the first_valid_uid setting.
When the SQL userdb module is used make sure that the iter-
ate_query setting in /etc/dovecot/dovecot-sql.conf.ext matches
your database layout. When using the LDAP userdb module, make
sure that the iterate_attrs and iterate_filter settings in
/etc/dovecot/dovecot-ldap.conf.ext match your LDAP schema. Oth-
erwise doveadm(1) will be unable to iterate over all users.
-F file
Execute the command for all the users in the file. This is sim-
ilar to the -A option, but instead of getting the list of users
from the userdb, they are read from the given file. The file
contains one username per line.
-N Synchronize all the available namespaces. By default only name-
spaces that don't have explicit location setting are synchro-
nized.
-P Run a doveadm-purge(1) for the destination (remote) storage af-
ter synchronization.
-R Do a reverse sync. Normally, messages would be pushed from the
local system to the destination (remote). This option reverses
the flow, and will instead pull messages from the remote to the
local storage.
-S socket_path
The option's argument is either an absolute path to a local UNIX
domain socket, or a hostname and port (hostname:port), in order
to connect a remote host via a TCP socket.
This allows an administrator to execute doveadm(1) mail commands
through the given socket.
-T secs
Specify the time in seconds, how long doveadm(1) should wait for
stalled I/O operations. The default timeout is 600 seconds.
-U This is used internally by replicator to have dsync notify it
when the synchronization is finished.
-d Use the default destination, which is looked up from the
mail_replica userdb extra field.
-g mailbox_guid
Same as -m, but find the mailbox to be synchronized by its GUID
instead of by name.
-l secs
Lock the dsync for this user. Wait for maximum secs before giv-
ing up. This parameter should be used to avoid broken synchro-
nization if it's possible that dsync is being run concurrently
for the same user.
-m mailbox
Synchronize only this mailbox name.
-n namespace
Synchronize only the specified namespace. This parameter can be
used multiple times.
-r rawlog_path
Running dsync remotely, write the remote input/output traffic to
the specified log file.
-s previous_state
Use stateful synchronization. If the previous state is unknown,
use an empty string. The new state is always printed to stan-
dard output.
-u user/mask
Run the command only for the given user. It's also possible to
use '*' and '?' wildcards (e.g. -u *@example.org).
When neither the -A option, nor the -F file option, nor the
-u user was specified, the command will be executed with the en-
vironment of the currently logged in user.
-x mailbox_mask
Exclude the specified mailbox name/mask. The mask may contain
"?" and "*" wildcards. This parameter can be used multiple
times.
ARGUMENTS
destination
This argument specifies the synchronized destination. It can be
one of:
location
Same as mail_location setting, e.g. maildir:~/Maildir
remote:login@host
Uses dsync_remote_cmd setting to connect to the remote
host (usually via ssh)
remoteprefix:login@host
This is the same as remote, except "user@domain\n" is
sent before dsync protocol starts. This allows imple-
menting a trusted wrapper script that runs doveadm
dsync-server by reading the username from the first line.
tcp:host[:port]
Connects to remote doveadm server via TCP. The default
port is specified by doveadm_port setting.
tcps:host[:port]
This is the same as tcp, but with SSL.
EXIT STATUS
dsync will exit with one of the following values:
0 Synchronization was done perfectly.
2 Synchronization was done without errors, but some changes couldn't
be done, so the mailboxes aren't perfectly synchronized. Running
dsync again usually fixes this. Typically this occurs for message
modification sequences with newly created mailboxes. It can also
occur if one of the mailboxes change during the syncing.
1, >2
Synchronization failed.
EXAMPLE
SYNCHRONIZATION
Synchronize mailboxes with a remote server. Any errors are written to
stderr.
doveadm sync -u username@example.com remote:server-replica.example.com
If you need more complex parameters to ssh, you can use e.g.:
doveadm sync -u username@example.com ssh -i id_dsa.dovecot \
mailuser@example.com doveadm dsync-server -u username@example.com
CONVERTING
Assuming that the mail_location setting in /etc/dove-
cot/conf.d/10-mail.conf is set to: mail_location = mdbox:~/mdbox, a
logged in system user may convert her/his mails from its Maildir in
her/his home directory to the mdbox mailbox format. The user has to
execute the command:
doveadm sync maildir:~/Maildir
If you want to do this without any downtime, you can do the conversion
one user at a time. Initially:
o Configuration uses mail_location = maildir:~/Maildir
o Set up the possibility of doing per-user mail location using
userdb extra fields.
Then for each user:
1. Run doveadm sync once to do the initial conversion.
2. Run doveadm sync again, because the initial conversion could
have taken a while and new changes could have occurred during
it. This second time only applies changes, so it should be
fast.
3. Update mail extra field in userdb to mdbox:~/mdbox. If you're
using auth cache, you need to flush it, e.g. doveadm auth
cache flush.
4. Wait for a few seconds and then kill (doveadm kick) the user's
all existing imap and pop3 sessions (that are still using
maildir).
5. Run doveadm sync once more to apply final changes that were
possibly done. After this there should be no changes to
Maildir, because the user's mail location has been changed and
all existing processes using it have been killed.
Once all users have been converted, you can set the default mail_loca-
tion to mdbox and remove the per-user mail locations from userdb.
REPORTING BUGS
Report bugs, including doveconf -n output, to the Dovecot Mailing List
<dovecot@dovecot.org>. Information about reporting bugs is available
at: http://dovecot.org/bugreport.html
SEE ALSO
doveadm(1), doveadm-auth(1), doveadm-kick(1), doveadm-purge(1), dove-
conf(1)
Additional resources:
dsync design
http://wiki2.dovecot.org/Design/Dsync
Dovecot v2.3 2015-08-31 DOVEADM-SYNC(1)
Man Pages Copyright Respective Owners. Site Copyright (C) 1994 - 2024
Hurricane Electric.
All Rights Reserved.