Migrating Using Rsync

Learn how to migrate files and applications to KloudBean using Rsync, a powerful and efficient Linux tool for copying files from source to destination or fetching data on the destination server.

Overview

Rsync is a wonderful Linux tool to copy files from source to destination or fetch data on the destination server. It's ideal for migrating large amounts of data with features like incremental updates, compression, and resume capability. Rsync efficiently synchronizes files and directories between locations, making it perfect for server-to-server migrations.

Prerequisites

Before starting migration using Rsync, ensure you have:

SSH access to KloudBean server (admin credentials)
SSH access to source server (if migrating from another server)
Rsync installed on both servers (usually pre-installed on Linux)
Command line access and basic Linux knowledge
IP whitelisting configured (if required for access)

What is Rsync?

Key Features

Efficient Transfer: Only transfers changed files, saving bandwidth and time
Incremental Sync: Syncs only differences between source and destination
Compression: Compresses data during transfer to reduce bandwidth usage
Resume Capability: Can resume interrupted transfers without starting over
Preserve Permissions: Maintains file permissions, timestamps, and ownership
Exclude Options: Exclude specific files/directories from transfer
Bandwidth Control: Limit bandwidth usage during transfer

When to Use Rsync

✅ Ideal For:

Large file migrations (GBs or TBs of data)
Server-to-server transfers
Regular synchronizations
Maintaining file permissions and structure
Efficient bandwidth usage
Resuming interrupted transfers

❌ Not Ideal For:

Small one-time uploads (use FileManager instead)
Users unfamiliar with command line
When GUI tools are preferred

Getting SSH Access to KloudBean Server

Before you can use Rsync, you need SSH access to your KloudBean server.

Step 1: Get Server Access Credentials

To get SSH access credentials:

Navigate to Server Administration: Log in to your KloudBean dashboard and open the server administration page
Go to Admin Access: Navigate to "Admin Access" in the server settings menu
Open Admin Credentials: Click on "Admin Credentials" tab within the Admin Access section
View Credentials: You'll see:
- Public IP: The public IP address of your server
- Username: The admin username (typically starts with admin_)
- Password: The password for the admin user

For detailed instructions on getting server access credentials, see: Connecting to Server via SSH or SFTP - Getting Server Access Credentials

Step 2: SSH into KloudBean Server

Once you have the credentials:

# Connect to KloudBean server
ssh admin_username@server_ip

# Example
ssh [email protected]

Enter the password when prompted. You're now connected to your KloudBean server.

Important: Make sure your IP address is whitelisted for SSH access. Review Server SSH Access Control to manage SSH access.

Migration Methods

Method 1: Fetch Data on KloudBean Server Using Rsync

This method involves SSH into your KloudBean server and running rsync commands to pull data from your source server into your project directory on KloudBean.

Step 1: SSH into KloudBean Server

Get SSH credentials from Server Administration → Admin Access → Admin Credentials
Connect via SSH:
```
ssh admin_username@kloudbean_server_ip
```
Enter password when prompted
Navigate to your project directory:
```
cd /home/admin/hosted-sites/<app_system_user>/app-html/
```
Note: Replace <app_system_user> with your actual application system user.

Step 2: Ensure Access is Enabled

note

Important: Make sure access to the source server is enabled or your current KloudBean server IP is whitelisted on the source server.

If source server requires IP whitelisting: Add your KloudBean server's public IP to the source server's whitelist
If source server has firewall rules: Ensure SSH access is allowed from KloudBean server IP
Test connection: Verify you can SSH from KloudBean server to source server

Step 3: Run Rsync Command

Once you're in your project directory on KloudBean server, run rsync to fetch data:

Basic Rsync Command:

rsync -avz --progress \
  source_username@source_server_ip:/path/to/source/files/ \
  /home/admin/hosted-sites/<app_system_user>/app-html/

Detailed Example:

# Fetch WordPress files from source server to KloudBean
rsync -avz --progress \
  --exclude='wp-config.php' \
  --exclude='.htaccess' \
  --exclude='wp-content/cache/' \
  --exclude='wp-content/upgrade/' \
  [email protected]:/var/www/html/wordpress/ \
  /home/admin/hosted-sites/<app_system_user>/app-html/

Command Breakdown:

-a: Archive mode (preserves permissions, timestamps, symlinks, etc.)
-v: Verbose (shows detailed progress)
-z: Compress during transfer (saves bandwidth)
--progress: Shows transfer progress for each file
--exclude: Exclude specific files or directories

Advanced Options:

# With bandwidth limit and resume capability
rsync -avz --progress --partial --bwlimit=1000 \
  --exclude='node_modules' \
  --exclude='.git' \
  --exclude='*.log' \
  source_user@source_server:/source/path/ \
  /home/admin/hosted-sites/<app_system_user>/app-html/

Options Explained:

--partial: Keep partially transferred files (allows resume)
--bwlimit=1000: Limit bandwidth to 1000 KB/s
--exclude='pattern': Exclude files matching pattern

Step 4: Monitor Progress

Rsync will show:

File list: Files being transferred
Progress: Transfer speed and percentage
Summary: Total files, size transferred, time taken

Example Output:

sending incremental file list
file1.php
        123,456  100%    2.34MB/s    0:00:00
file2.php
        234,567  100%    3.45MB/s    0:00:00
...
sent 1,234,567 bytes  received 5,678,901 bytes  456.78 bytes/sec
total size is 12,345,678  speedup is 1.23

Step 5: Verify Transfer

After transfer completes:

# Check file count
find /home/admin/hosted-sites/<app_system_user>/app-html/ -type f | wc -l

# Check directory structure
ls -la /home/admin/hosted-sites/<app_system_user>/app-html/

# Verify specific files
ls -lh /home/admin/hosted-sites/<app_system_user>/app-html/important-file.php

Method 2: Push Data to KloudBean Server

This method involves going to your current (source) server and running rsync commands to push data directly to your KloudBean server.

Step 1: SSH into Your Current Server

Connect to your source server:
```
ssh username@current_server_ip
```
Navigate to source directory:
```
cd /path/to/source/files
```

Step 2: Ensure Access is Enabled

note

Important: Make sure your current server's IP is whitelisted on KloudBean server for SSH access.

Whitelist IP on KloudBean:
- Go to KloudBean Server Administration → Firewall → SSH Access Control
- Add your current server's public IP address
- Save changes
Test connection: Verify you can SSH from current server to KloudBean server:
```
ssh admin_username@kloudbean_server_ip
```

Step 3: Run Rsync Command to Push Data

From your current server, run rsync to push data to KloudBean:

Basic Push Command:

rsync -avz --progress \
  /local/source/path/ \
  kloudbean_username@kloudbean_server_ip:/home/admin/hosted-sites/<app_system_user>/app-html/

Detailed Example:

# Push WordPress files to KloudBean
rsync -avz --progress \
  --exclude='wp-config.php' \
  --exclude='.htaccess' \
  --exclude='wp-content/cache/' \
  --exclude='wp-content/upgrade/' \
  --exclude='node_modules' \
  --exclude='.git' \
  /var/www/html/wordpress/ \
  admin_username@kloudbean_server_ip:/home/admin/hosted-sites/<app_system_user>/app-html/

With Advanced Options:

# Push with bandwidth limit and resume capability
rsync -avz --progress --partial --bwlimit=2000 \
  --exclude='*.log' \
  --exclude='tmp/' \
  --exclude='cache/' \
  /source/path/ \
  admin_username@kloudbean_server_ip:/home/admin/hosted-sites/<app_system_user>/app-html/

Exclude Multiple Patterns:

# Using exclude file
rsync -avz --progress \
  --exclude-from='exclude-list.txt' \
  /source/path/ \
  admin_username@kloudbean_server_ip:/home/admin/hosted-sites/<app_system_user>/app-html/

Create exclude-list.txt:

.git
node_modules
*.log
cache/
tmp/
.DS_Store
wp-config.php
.htaccess

Step 4: Enter Password (If Using Password Authentication)

If you're using password authentication:

Enter the KloudBean admin password when prompted
For automated scripts, consider using SSH keys instead

Using SSH Keys (Recommended):

# Generate SSH key on source server (if not exists)
ssh-keygen -t rsa -b 4096

# Copy key to KloudBean server
ssh-copy-id admin_username@kloudbean_server_ip

# Now rsync will work without password
rsync -avz --progress /source/ admin_username@kloudbean_server_ip:/destination/

Step 5: Monitor and Verify

Monitor progress: Watch the transfer progress
Wait for completion: Let the transfer complete

Verify on KloudBean: SSH into KloudBean server and verify files:

ssh admin_username@kloudbean_server_ip
ls -la /home/admin/hosted-sites/<app_system_user>/app-html/

Common Rsync Options

Basic Options

-a or --archive: Archive mode (preserves permissions, timestamps, symlinks, etc.)
-v or --verbose: Verbose output (shows files being transferred)
-z or --compress: Compress data during transfer
-h or --human-readable: Human-readable file sizes
--progress: Show progress for each file

Advanced Options

--partial: Keep partially transferred files (allows resume)
--delete: Delete files in destination not in source (use with caution!)
--exclude='pattern': Exclude files matching pattern
--exclude-from='file': Exclude patterns from file
--bwlimit=KBPS: Limit bandwidth usage (in KB/s)
--dry-run: Test run without actually transferring
--timeout=SECONDS: Set I/O timeout

Useful Combinations

# Safe test run (see what would be transferred)
rsync -avz --dry-run --progress source/ destination/

# Resume interrupted transfer
rsync -avz --partial --progress source/ destination/

# Limit bandwidth and exclude patterns
rsync -avz --bwlimit=1000 --exclude='*.log' source/ destination/

# Mirror source to destination (delete extra files)
rsync -avz --delete source/ destination/

Rsync Alternative Tools

While Rsync is excellent, there are alternative tools you can use for file migration. Here are some popular alternatives:

1. SCP (Secure Copy)

What it is: SCP is a simple command-line tool for securely copying files over SSH.

Usage:

# Copy single file
scp /local/file.txt user@remote:/remote/path/

# Copy directory recursively
scp -r /local/directory/ user@remote:/remote/path/

# With specific port
scp -P 2222 /local/file.txt user@remote:/remote/path/

Pros:

✅ Simple and straightforward
✅ Built into most Linux/Unix systems
✅ Secure (uses SSH encryption)
✅ Easy to use for one-time transfers

Cons:

❌ No incremental sync (always transfers full files)
❌ No resume capability
❌ Slower for large migrations
❌ No progress indication by default
❌ Less efficient than rsync

Best For: Small to medium file transfers, simple one-time copies

2. SFTP (SSH File Transfer Protocol)

What it is: SFTP is an interactive file transfer protocol over SSH, often used with GUI clients.

Usage:

# Interactive SFTP session
sftp user@remote

# Once connected:
put /local/file.txt /remote/path/
get /remote/file.txt /local/path/
mput /local/directory/* /remote/path/
mget /remote/directory/* /local/path/

GUI Tools: FileZilla, WinSCP, Cyberduck

Pros:

✅ User-friendly GUI clients available
✅ Secure (SSH encryption)
✅ Interactive file browsing
✅ Good for manual file management

Cons:

❌ No incremental sync
❌ Slower than rsync
❌ Manual process (not automated)
❌ No resume capability

Best For: Manual file management, GUI users, small file transfers

3. Tar over SSH

What it is: Using tar to create archives and transfer them over SSH.

Usage:

# Create tar archive and transfer in one command
tar czf - /source/path/ | ssh user@remote "cd /destination && tar xzf -"

# With progress (using pv if available)
tar czf - /source/path/ | pv | ssh user@remote "cd /destination && tar xzf -"

Pros:

✅ Single command operation
✅ Preserves permissions and structure
✅ Can compress during transfer
✅ Works well for entire directories

Cons:

❌ No incremental sync (transfers everything)
❌ Requires enough disk space for archive
❌ Slower for large datasets
❌ No resume capability

Best For: One-time full directory transfers, when rsync is not available

4. Rclone

What it is: Rclone is a command-line program to manage files on cloud storage and sync between different storage systems.

Usage:

# Install rclone
curl https://rclone.org/install.sh | sudo bash

# Sync from source to destination
rclone sync /source/path remote:destination/path --progress

# Copy with options
rclone copy /source/path remote:destination/path \
  --progress --transfers=4 --checkers=8

Pros:

✅ Supports many cloud storage providers
✅ Incremental sync capability
✅ Resume capability
✅ Bandwidth limiting
✅ Good for cloud-to-cloud transfers

Cons:

❌ Requires installation
❌ More complex configuration
❌ Overkill for simple server-to-server transfers

Best For: Cloud storage migrations, multi-cloud scenarios

5. Unison

What it is: Unison is a file synchronization tool that can keep two replicas of a collection of files synchronized.

Usage:

# Install unison
sudo apt-get install unison  # Debian/Ubuntu
sudo yum install unison      # CentOS/RHEL

# Sync two directories
unison /source/path ssh://user@remote//destination/path

Pros:

✅ Bidirectional sync
✅ Conflict resolution
✅ Handles file updates intelligently
✅ Good for keeping directories in sync

Cons:

❌ Requires installation
❌ More complex than rsync
❌ Slower than rsync
❌ Less commonly used

Best For: Bidirectional synchronization, keeping multiple locations in sync

Comparison Table

Tool	Incremental Sync	Resume	Speed	Ease of Use	Best For
Rsync	✅ Yes	✅ Yes	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	Large migrations, server-to-server
SCP	❌ No	❌ No	⭐⭐⭐	⭐⭐⭐⭐⭐	Simple file copies
SFTP	❌ No	❌ No	⭐⭐	⭐⭐⭐⭐⭐	Manual file management
Tar over SSH	❌ No	❌ No	⭐⭐⭐	⭐⭐⭐	Full directory transfers
Rclone	✅ Yes	✅ Yes	⭐⭐⭐⭐	⭐⭐⭐	Cloud storage migrations
Unison	✅ Yes	✅ Yes	⭐⭐⭐	⭐⭐	Bidirectional sync

Best Practices

Use Dry Run First: Always test with --dry-run to see what would be transferred
Exclude Unnecessary Files: Exclude cache, logs, node_modules, .git, etc.
Preserve Permissions: Use -a flag for archive mode
Monitor Progress: Use --progress to see transfer status
Compress Transfer: Use -z flag to save bandwidth
Resume Capability: Use --partial for large transfers
Limit Bandwidth: Use --bwlimit if needed to avoid network congestion
Verify After: Always verify files after migration
Backup First: Backup before migration
Use SSH Keys: Use SSH keys instead of passwords for better security

Troubleshooting

Permission Denied Errors

# Check source permissions
ls -la /source/path

# Check destination permissions
ls -la /destination/path

# Use sudo if needed (be careful)
sudo rsync -avz source/ destination/

Connection Timeout

# Increase timeout
rsync -avz --timeout=300 source/ destination/

# Or use SSH keepalive
ssh -o ServerAliveInterval=60 user@server

Large File Issues

# Use --partial for large files
rsync -avz --partial --progress source/ destination/

# Or split into smaller transfers

Network Issues

# Limit bandwidth to avoid network issues
rsync -avz --bwlimit=500 source/ destination/

# Use compression
rsync -avz -z source/ destination/

IP Not Whitelisted

Problem: Cannot connect to KloudBean server from source server.

Solution:

Go to KloudBean Server Administration → Firewall → SSH Access Control
Add your source server's public IP address
Save changes
Try connecting again

Next Steps

Learn about Migrating WordPress for complete WordPress migration
Explore Migrating Databases for database migration
Review Migrating Using FileManager for GUI-based migration
Check Connecting to Server via SSH or SFTP for SSH access details

Overview​

Prerequisites​

What is Rsync?​

Key Features​

When to Use Rsync​

Getting SSH Access to KloudBean Server​

Step 1: Get Server Access Credentials​

Step 2: SSH into KloudBean Server​

Migration Methods​

Method 1: Fetch Data on KloudBean Server Using Rsync​

Step 1: SSH into KloudBean Server​

Step 2: Ensure Access is Enabled​

Step 3: Run Rsync Command​

Step 4: Monitor Progress​

Step 5: Verify Transfer​

Method 2: Push Data to KloudBean Server​

Step 1: SSH into Your Current Server​

Step 2: Ensure Access is Enabled​

Step 3: Run Rsync Command to Push Data​

Step 4: Enter Password (If Using Password Authentication)​

Step 5: Monitor and Verify​

Common Rsync Options​

Basic Options​

Advanced Options​

Useful Combinations​

Rsync Alternative Tools​

1. SCP (Secure Copy)​

2. SFTP (SSH File Transfer Protocol)​

3. Tar over SSH​

4. Rclone​

5. Unison​

Comparison Table​

Best Practices​

Troubleshooting​

Permission Denied Errors​

Connection Timeout​

Large File Issues​

Network Issues​

IP Not Whitelisted​

Next Steps​

Overview

Prerequisites

What is Rsync?

Key Features

When to Use Rsync

Getting SSH Access to KloudBean Server

Step 1: Get Server Access Credentials

Step 2: SSH into KloudBean Server

Migration Methods

Method 1: Fetch Data on KloudBean Server Using Rsync

Step 1: SSH into KloudBean Server

Step 2: Ensure Access is Enabled

Step 3: Run Rsync Command

Step 4: Monitor Progress

Step 5: Verify Transfer

Method 2: Push Data to KloudBean Server

Step 1: SSH into Your Current Server

Step 2: Ensure Access is Enabled

Step 3: Run Rsync Command to Push Data

Step 4: Enter Password (If Using Password Authentication)

Step 5: Monitor and Verify

Common Rsync Options

Basic Options

Advanced Options

Useful Combinations

Rsync Alternative Tools

1. SCP (Secure Copy)

2. SFTP (SSH File Transfer Protocol)

3. Tar over SSH

4. Rclone

5. Unison

Comparison Table

Best Practices

Troubleshooting

Permission Denied Errors

Connection Timeout

Large File Issues

Network Issues

IP Not Whitelisted

Next Steps