Upgrade to Airbyte v2 using abtcl

1. Stop Legacy Airbyte Installation

If you are upgrading from Docker Compose, stop the existing instance to free up resources and ports:

docker compose stop

or reset Airbyte v1 installation

abctl local uninstall

2. Install the abctl Tool

Install the Airbyte CLI tool using the following command:

curl -LsfS https://get.airbyte.com | bash -

3. Configure values.yaml

Create a file named values.yaml in your working directory. This configuration is optimized for high-performance workloads and prevents memory crashes by setting explicit resource limits. It also uses an external postgres database.

Note on Server Capacity: The resource values provided in the values.yaml below are optimized for a high-capacity server (e.g., 4 CPUs and 64GB RAM). If your server has different specifications, please adjust the CPU and memory requests and limits accordingly.

Create values.yaml with this content:

postgresql:
  enabled: false

global:
  env_vars:
    WORKLOAD_HEARTBEAT_TIMEOUT_MS: 1800000
    DD_ENABLED: "false"
    DD_TRACE_ENABLED: "false"
    PUBLISH_METRICS: "false"
    # Optimizes Java heap and ensures the process exits on OOM to trigger a restart
    CONTAINER_ORCHESTRATOR_JAVA_OPTS: "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=85.0"

  database:
    type: external
    host: xxx
    port: '5432'
    name: airbyte
    user: xxx
    password: xxx
  auth:
    cookieSecureSetting: 'false'
  workloads:
    resources:
      mainContainer:
        memory:
          request: 2Gi
          limit: 32Gi
        cpu:
          request: 500m
          limit: 4000m
      replication:
        maxConcurrentJobs: "4"
        memory:
          request: 2Gi
          limit: 32Gi
        cpu:
          request: 500m
          limit: 4000m
      discover:
        memory:
          request: 2Gi
          limit: 8Gi
        cpu:
          request: 500m
          limit: 2000m
  jobs:
    resources:
      requests:
        memory: 2Gi
        cpu: 1000m
      limits:
        cpu: 2000m
        memory: 8Gi
  worker:
    maxCheckWorkers: "5"
    maxSyncWorkers: "5"

Remote Server Migration (Custom Host & Port)

If you are deploying on a remote server and need to specify a host (HTTP/HTTPS) and avoid browser pop-ups:

For HTTP without domain:

abctl local install --no-browser --insecure-cookies --values ./values.yaml

For HTTP (Port 80) with domain:

abctl local install --no-browser --insecure-cookies --values ./values.yaml --host your.domain.com --port 80

For HTTPS with domain:

abctl local install --no-browser --values ./values.yaml --host your.domain.com --port 443

5. Set Identity Credentials

Configure your access credentials for the Airbyte UI:

# Set Email
abctl local credentials --email [email protected]

# Set Password
abctl local credentials --password your_secure_password

6. Accessing via PuTTY SSH Tunneling

If your Airbyte instance is on a remote server, use PuTTY to forward the traffic to your local machine.

1. Open PuTTY and enter your server IP.
2. Navigate to Connection > SSH > Tunnels.
3. Source port: 8000 (the port you will use in your local browser).
4. Destination: localhost:8000 (or localhost:80 depending on your server configuration).
5. Click Add, then click Open.
6. Once logged into the shell, open your browser and go to: http://localhost:8000

7. Troubleshooting

If you need to reset the installation or re-apply a modified values.yaml without losing your data, use the uninstall command with the persisted flag:

# Removes the installation but keeps your data intact
abctl local uninstall --persisted

# Reinstall with updated values
abctl local install --values ./values.yaml

For more information, visit the Official Airbyte OSS Documentation.