Migrations

osctld has support for migrating containers between different vpsAdminOS nodes with SSH used as a transport channel. Each vpsAdminOS node has a system user called migration. The source node is connecting to the migration user on the destination node. Authentication is based on public/private keys.

Setup

On the source node, a public/private key pair is needed. It can be generated by osctl migration key gen, or the keys can be manually installed to paths given by osctl migration key path public and osctl migration key path private.

source-node $ osctl migration key gen -t ecdsa

source-node $ osctl migration key path public
/tank/conf/migration/key.pub

source-node $ cat /tank/conf/migration/key.pub
ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYT... root@source-node

Through another communication channel, picked at your discretion, the public key of the source node must be transfered to the destination node and authorized to migrate containers to that node. Once transfered, the key can be authorized using osctl migration authorized-keys add, the key is entered on a single line to the standard input:

destination-node $ osctl migration authorized-keys add
ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYT... root@source-node

Now the source node is authorized to migrate containers to the destination node. You can verify that by running SSH directly:

source-node $ ssh -T -i `osctl migration key path private` -l migration destination-node
Usage:
  receive skel [pool]
  receive base [pool:]<id> <dataset> [snapshot]
  receive incremental [pool:]<id> <dataset> [snapshot]
  receive transfer [pool:]<id>
  receive cancel [pool:]<id>

As you can see, the SSH connection is limited to several commands that are handled by osctld. The meaning of those commands will become clearer once you read the section below, but it's enough to know the shell is restricted for migrations only.

Migrating containers

The container migration itself consists of several steps:

  • osctl ct migrate stage is used to prepare environment on the destination node and copy configuration
  • osctl ct migrate sync sends over the container's rootfs
  • osctl ct migrate transfer stops the container on the source node, performs another rootfs sync and finally starts the container on the destination node
  • osctl ct migrate cleanup is used to remove the container from the source node

Up until osctl ct migrate transfer, the migration can be cancelled using osctl ct migrate cancel, which resets the container's migration state on the source node and removes the partially transfered container from the destination node.

osctl ct migrate now will perform all necessary migration steps in succession. Use this when you don't care when is a particular migration step run. Otherwise, you can choose when to run specific migration steps to optimize for minimum downtime at reasonable hours. Consider the following example:

# 1) Prepare the migration
source-node $ osctl ct migrate stage myct01 destination-node

# 2) Make an initial copy the container's rootfs, this will be the most
#    time-consuming operation
source-node $ osctl ct migrate sync myct01

# 3) Having the bulk of the container's data transfered, you can now wait for
#    when the downtime will be least unwelcome, e.g. during the night. Run the
#    following command when ready.
source-node $ osctl ct migrate transfer myct01

# 4) Cleanup is executed only on the source node, the destination node is already
#    finished
source-node $ osctl ct migrate cleanup myct01