Next Steps After Installation

Congratulations on installing Hubio Sync! Here’s how to get from installation to your first successful data migration.

1. Configure Your First Data Source

Before you can sync data, you need to configure at least one data source.

Create Configuration File

Hubio Sync reads configuration from ~/.config/hubio-sync/config.toml (Linux/macOS) or %APPDATA%\hubio-sync\config.toml (Windows).

Create the configuration directory:

# Linux/macOS
mkdir -p ~/.config/hubio-sync

# Windows (PowerShell)
New-Item -ItemType Directory -Path "$env:APPDATA\hubio-sync" -Force

Example Configuration

Create config.toml with your data source connection:

# config.toml - Example MySQL source configuration

[source]
type = "mysql"
host = "localhost"
port = 3306
database = "myapp_production"
username = "readonly_user"
password = "your_secure_password"

# Connection pool settings
max_connections = 10
connection_timeout = 30

[destination]
type = "s3"
bucket = "my-data-lake"
region = "us-east-1"
prefix = "mysql-exports/"

# AWS credentials (or use IAM role)
access_key_id = "YOUR_AWS_ACCESS_KEY"
secret_access_key = "YOUR_AWS_SECRET_KEY"

[sync]
# Schedule: Run daily at 2 AM
schedule = "0 2 * * *"

# Tables to sync (or use "*" for all)
tables = ["users", "orders", "products"]

# Incremental sync based on updated_at column
incremental_column = "updated_at"

Supported Data Sources

Hubio Sync supports:

  • Databases: MySQL, PostgreSQL, SQLite, SQL Server, Oracle
  • APIs: REST, GraphQL, gRPC
  • Files: CSV, JSON, Parquet, Avro
  • Cloud: AWS S3, Google Cloud Storage, Azure Blob Storage
  • Warehouses: Snowflake, BigQuery, Redshift, Databricks

See Configuration Reference for all options.

2. Run Your First Migration

Validate Configuration

Before running a full sync, validate your configuration:

hubio-sync validate

This will:

  • ✅ Check configuration syntax
  • ✅ Test source connection
  • ✅ Test destination connection
  • ✅ Validate table/schema access
  • ✅ Estimate data volume

Run Example Migration

Try the example migration to verify everything works:

hubio-sync validate --example

This runs a small test migration using sample data to ensure your setup is correct.

Run Full Migration

Once validation passes, run your first migration:

# One-time sync
hubio-sync run

# Dry run (preview without executing)
hubio-sync run --dry-run

# Specific table
hubio-sync run --table users

# Force full refresh (ignore incremental state)
hubio-sync run --full-refresh

Monitor Progress

Watch the migration in real-time:

# Show live progress
hubio-sync run --verbose

# Show detailed logs
hubio-sync run --log-level debug

3. Explore Advanced Features

Scheduling

Set up automated syncs using cron syntax:

[sync]
schedule = "0 2 * * *"  # Daily at 2 AM

Or use the built-in scheduler:

# Start scheduler daemon
hubio-sync scheduler start

# View scheduled jobs
hubio-sync scheduler list

# Stop scheduler
hubio-sync scheduler stop

Transformations

Apply transformations during sync:

[[transformations]]
table = "users"
transform = "anonymize"
columns = ["email", "phone", "ssn"]

[[transformations]]
table = "orders"
transform = "filter"
condition = "created_at >= NOW() - INTERVAL 90 DAY"

Monitoring

Track sync performance and health:

# View sync status
hubio-sync status

# Show metrics
hubio-sync metrics

# Export metrics to Prometheus
hubio-sync metrics --format prometheus

Webhooks

Get notified when syncs complete:

[notifications]
webhook_url = "https://your-org.webhook.office.com/webhookb2/YOUR/WEBHOOK/URL"
notify_on = ["success", "failure", "warning"]

Quick Reference

Essential Commands

# Validate configuration
hubio-sync validate

# Run migration
hubio-sync run

# View status
hubio-sync status

# Show version
hubio-sync --version

# Get help
hubio-sync --help

Common Workflows

Initial Setup:

mkdir -p ~/.config/hubio-sync
vi ~/.config/hubio-sync/config.toml  # Create config
hubio-sync validate                   # Test connection
hubio-sync run --dry-run              # Preview
hubio-sync run                        # Execute

Daily Operations:

hubio-sync status                     # Check health
hubio-sync run                        # Manual sync
hubio-sync metrics                    # View performance

Troubleshooting:

hubio-sync validate --verbose         # Detailed validation
hubio-sync run --log-level debug      # Debug mode
hubio-sync run --dry-run              # Preview changes

Additional Resources

For support, contact support@hubio.com.

What’s Next?

Ready to dive deeper? Explore these resources:

Happy syncing! 🎉