Back to Projects
ZCS Lightning Mailbox Move

ZCS Lightning Mailbox Move

**ZCS Lightning Mailbox Move** is a high-performance Python-based tool designed for migrating Zimbra Collaboration Suite (ZCS) mailboxes between different mail stores with minimal downtime. The tool handles the complete migration process including data synchronization, database migration, and configuration updates.

zimbramailbox movezmmboxmove
View on GitHub

License: GPL v2

ZCS Lightning Mailbox Move

Overview

ZCS Lightning Mailbox Move is a high-performance Python-based tool designed for migrating Zimbra Collaboration Suite (ZCS) mailboxes between different mail stores with minimal downtime. The tool handles the complete migration process including data synchronization, database migration, and configuration updates.

Features

  • Fast Mailbox Migration: Optimized process for moving mailboxes between Zimbra stores
  • Minimal Downtime: Uses pre-sync and final sync approach to minimize service interruption
  • Batch Processing: Support for single user or CSV file-based bulk migrations
  • SSH Tunneling: Secure database connections through SSH tunnels
  • Automatic Rollback: Database rollback on migration failure
  • Logging: Comprehensive logging with optional per-user log files
  • Data Integrity: Validates data migration by comparing record counts
  • Cache Management: Automatic cache flushing across all proxy and store servers

Architecture

The tool operates on the source mailbox server and performs:

  1. LDAP queries for user account information
  2. MySQL database operations for mailbox data migration
  3. rsync for file system synchronization (store and index folders)
  4. SSH operations for remote server management
  5. SOAP API calls for Zimbra provisioning

Requirements

System Requirements

  • Python 3.6 or higher
  • pip3
  • Root SSH access between source and destination servers
  • SSH key-based authentication configured
  • Access to Zimbra LDAP server
  • Access to Zimbra MySQL databases

Python Dependencies

PyMySQL~=1.0.2
paramiko~=3.1.0
ldap3~=2.9.1
ozpy~=1.0.1
configobj~=5.0.8

Installation

  1. Clone the repository:

    git clone https://github.com/ScaleNix/zcslightningmove.git
cd zcslightningmove

  2. Install Python dependencies:

    pip3 install -r requirements.txt

  3. Configure SSH access:

    • Set up SSH key-based authentication from source to destination server
    • Test connection and validate fingerprint: 
      ssh root@destination-server

  4. Prepare destination stores (run as zimbra user):

    for i in `seq 1 100` ; do sed 's/${DATABASE_NAME}/'"mboxgroup$i/g" /opt/zimbra/db/create_database.sql | mysql ; done

  5. Configure the tool: Edit mailboxmove.conf with your environment settings (see Configuration section)

Configuration

Configuration File: mailboxmove.conf

[global]
log_file='/tmp/mailbox_move.log'

[zimbra-config]
offset_mailbox_id=90000
ldap_server = 'ldap://172.16.0.4'
ldap_port=389
ldap_user = 'uid=zimbra,cn=admins,cn=zimbra'
ldap_password = 'YOUR_LDAP_PASSWORD'
ZADMIN_USERNAME = 'admin_service@yourdomain.com'
ZADMIN_PASSWORD = 'YOUR_ADMIN_PASSWORD'
ZSTORE = "localhost"

[store-config]
    [[store1.domain.com]]
        host='127.0.0.1'
        user='zimbra'
        password='MYSQL_PASSWORD'
        port='17306'
    [[store2.domain.com]]
        host='127.0.0.1'
        user='zimbra'
        password='MYSQL_PASSWORD'
        port='17307'

Configuration Parameters

Global Section

  • log_file: Path to the main log file

Zimbra Config Section

  • offset_mailbox_id: Offset value added to original mailbox ID to create new mailbox ID (prevents ID conflicts)
  • ldap_server: LDAP server URL
  • ldap_port: LDAP server port (default: 389)
  • ldap_user: LDAP admin user DN
  • ldap_password: LDAP admin password
  • ZADMIN_USERNAME: Zimbra admin username for SOAP API
  • ZADMIN_PASSWORD: Zimbra admin password
  • ZSTORE: Store server hostname

Store Config Section

Define each store server with:

  • host: MySQL host (use 127.0.0.1 when using SSH tunnel)
  • user: MySQL username (typically 'zimbra')
  • password: MySQL password
  • port: Local port for SSH tunnel

Usage

Command Line Arguments

usage: main.py [-h] [-a EMAIL] [-d DESTINATION] [-f CSV_FILE] [-l] [-s]

Fast mailbox move

optional arguments:
  -h, --help            show this help message and exit
  -a EMAIL, --email EMAIL
                        define email addr to move
  -d DESTINATION, --destination DESTINATION
                        define destination store
  -f CSV_FILE, --csv-file CSV_FILE
                        define csv file
  -l, --listen          active tunneling mode
  -s, --separate-log    separate logs per user

Starting SSH Tunnel

Before performing migrations, start the SSH tunnel in a separate terminal:

python3 main.py -l

This creates SSH tunnels to all configured store servers for secure MySQL access.

Migrating a Single User

python3 main.py -a user@example.com -d destination-store.domain.com

Migrating Multiple Users from CSV

Create a CSV file with one email per line:

user1@example.com
user2@example.com
user3@example.com

Run the migration:

python3 main.py -f users.csv -d destination-store.domain.com

Separate Logs per User

To create individual log files for each migrated user:

python3 main.py -a user@example.com -d destination-store.domain.com -s

This creates a log file at /tmp/user@example.com.log

Migration Process

Pre-Migration Phase

  1. Validation: Checks script is running on the correct source server
  2. LDAP Query: Retrieves user's zimbraId and current zimbraMailHost
  3. Database Query: Gets mailbox_id and mailbox_group from source database
  4. ID Calculation: Calculates new mailbox_id (original + offset) ensuring no conflicts
  5. Folder Creation: Creates destination store and index folders
  6. Pre-Sync: Initial rsync of store and index data while user is active

Migration Phase

  1. Maintenance Mode: Sets account to maintenance status
  2. Final Sync: Performs final rsync to catch any changes
  3. Database Dump: Exports mailbox data from source database
  4. ID Transformation: Updates mailbox_id in SQL dumps
  5. Database Import: Imports transformed data to destination
  6. Validation: Verifies record counts match between source and destination

Post-Migration Phase

  1. Account Update: Updates zimbraMailHost attribute
  2. Transport Update: Updates zimbraMailTransport for mail routing
  3. Cache Flush: Flushes cache on all store servers
  4. Memcached Flush: Flushes memcached on all proxy servers
  5. Activation: Removes maintenance mode

Rollback on Failure

If migration fails, the tool automatically:

  • Removes entries from destination database
  • Keeps source data intact
  • Removes maintenance mode
  • Logs detailed error information

Database Tables Migrated

The following tables are migrated:

  • appointment
  • data_source_item
  • imap_folder
  • imap_message
  • mail_item
  • tag
  • tagged_item
  • open_conversation
  • pop3_message
  • purged_conversations
  • purged_messages
  • revision
  • revision_dumpster
  • mail_item_dumpster
  • appointment_dumpster
  • tombstone

File System Structure

Source Paths

  • Store: /opt/zimbra/store/{shift_id}/{mailbox_id}/
  • Index: /opt/zimbra/index/{shift_id}/{mailbox_id}/

Destination Paths

  • Store: /opt/zimbra/store/{shift_id}/{new_mailbox_id}/
  • Index: /opt/zimbra/index/{shift_id}/{new_mailbox_id}/

Where shift_id = mailbox_id >> 12 (bit shift operation)

Logging

Log Locations

  • Main log: /tmp/mailbox_move.log (configurable)
  • Per-user logs: /tmp/{email}.log (when using -s flag)
  • Temporary SQL files: /var/tmp/{zimbraId}.{table}.sql

Log Levels

  • INFO: General migration progress
  • DEBUG: Detailed operation information including SQL queries
  • WARNING: Non-critical issues
  • ERROR: Critical failures requiring attention

Temporary Files

The tool creates temporary files in /var/tmp/:

  • SQL dump files: {zimbraId}.{table}.sql.orig
  • Modified SQL files: {zimbraId}.{table}.sql
  • Bash scripts: dump_sql_{db}_{table}_{zimbraId}.sh
  • Injection scripts: inject_sql_{db}_{table}_{zimbraId}.sh

Error Handling

Common Issues

  1. Wrong Server Location

    • Error: "Wrong location of store"
    • Solution: Run script on the source mailbox server, not destination

  2. Duplicate Mailbox ID

    • Error: "mailbox_id present in destination"
    • Solution: Increase offset_mailbox_id in configuration

  3. SSH Connection Failed

    • Error: Connection timeout or refused
    • Solution: Verify SSH keys and network connectivity

  4. Database Connection Failed

    • Error: "Can't connect to MySQL server"
    • Solution: Check SSH tunnel is running with -l flag

  5. Cache Flush Failed

    • Behavior: Manual intervention required
    • Solution: Run zmprov fc all and flush memcached manually

Best Practices

  1. Test Environment: Always test migrations in a development environment first
  2. Backup: Ensure backups are current before migration
  3. Off-Peak Hours: Schedule migrations during low-usage periods
  4. Batch Size: For bulk migrations, process in reasonable batches
  5. Monitor: Watch logs actively during migration
  6. Verification: Verify user can access mailbox after migration
  7. Cleanup: Remove temporary SQL files periodically

Security Considerations

  • Store passwords securely in configuration file
  • Restrict file permissions on mailboxmove.conf: 
    chmod 600 mailboxmove.conf
  • Use SSH key authentication without passphrases for automation
  • Run with minimum required privileges
  • Audit log files for sensitive information

Performance Tuning

Factors Affecting Performance

  • Mailbox size (data volume)
  • Network bandwidth between servers
  • MySQL database performance
  • Storage I/O performance

Optimization Tips

  • Use high-speed network connections
  • Ensure adequate disk I/O on both servers
  • Consider running pre-sync during business hours
  • Perform final migration during maintenance window
  • Monitor system resources during migration

Troubleshooting

Enable Debug Logging

Debug logging is enabled by default. Check log files for detailed information.

Verify Prerequisites

# Check Python version
python3 --version

# Verify SSH connectivity
ssh root@destination-server hostname

# Test LDAP connection
ldapsearch -x -H ldap://ldap-server -D "uid=zimbra,cn=admins,cn=zimbra" -W

# Verify tunnel
netstat -tlnp | grep 17306

Manual Rollback

If automatic rollback fails:

# Connect to destination database
mysql -h 127.0.0.1 -P 17306 -u zimbra -p

# Delete from zimbra database
DELETE FROM zimbra.mailbox WHERE id={new_mailbox_id};

# Delete from mailbox group tables
DELETE FROM mboxgroup{group}.mail_item WHERE mailbox_id={new_mailbox_id};
# Repeat for all tables

API Reference

Main Classes

User Class

Represents a mailbox user being migrated.

Attributes:

  • email: User's email address
  • zimbraId: Zimbra unique identifier
  • zimbraMailHost: Current mail host
  • mailbox_id: Current mailbox ID
  • mailbox_group: Mailbox group number
  • new_mailbox_id: Calculated new mailbox ID
  • new_zimbraMailHost: Destination mail host

Methods:

  • init(new_mailHost): Initialize user for migration
  • move(): Execute the migration process
  • activate_maintenance(): Put account in maintenance mode
  • disable_maintenance(): Remove maintenance mode
  • presync(): Perform pre-synchronization
  • final_sync(): Perform final synchronization and migration

Key Functions

  • get_account(email): Retrieve account attributes from Zimbra
  • modify_account(zimbraId, **attr): Modify account attributes
  • rsync(src_server, src_dir, dst_server, dst_dir, user): Synchronize files
  • get_dump(store_mysql_conn, db, table, where, user): Export database table
  • inject_dump(store_mysql_conn, db, table, data_dump, user): Import database table
  • validate_db_migration(user): Verify migration integrity
  • flush_cache_all_stores(): Flush Zimbra cache on all stores
  • flush_all_memcached(): Flush memcached on all proxies

Contributing

Contributions are welcome! Please follow these guidelines:

  1. Fork the repository
  2. Create a feature branch
  3. Test thoroughly in a lab environment
  4. Submit a pull request with detailed description

Support

For issues, questions, or contributions:

  • GitHub Issues: https://github.com/ScaleNix/zcslightningmove/issues
  • Repository: https://github.com/ScaleNix/zcslightningmove

Changelog

Current Version

  • Initial release with core migration functionality
  • Support for single and bulk user migrations
  • SSH tunnel support for secure database access
  • Automatic rollback on failure
  • Comprehensive logging

Credits

Developed by ScaleNix

Disclaimer

This tool performs critical mailbox migration operations. Always:

  • Test in a non-production environment first
  • Maintain current backups
  • Have a rollback plan
  • Monitor the migration process
  • Verify results after migration

The authors are not responsible for data loss or service disruption resulting from the use of this tool.