Initial push
This commit is contained in:
@@ -0,0 +1,315 @@
|
||||
# docker-migrate
|
||||
|
||||
Move Docker containers — and their data — from one host to another, from a web UI, in a few clicks.
|
||||
|
||||
It handles the whole container, not just the image: named volumes, anonymous volumes, bind mounts,
|
||||
user-defined networks, published ports, environment, capabilities, restart policy, healthchecks and
|
||||
resource limits. You pick what travels, per container and per mount.
|
||||
|
||||
Two ways to move things:
|
||||
|
||||
| Mode | What happens | When to use it |
|
||||
| --- | --- | --- |
|
||||
| **Host to host over SSH** | The source connects to the target over SSH and streams everything straight into the target's `docker cp` / `docker load`. Nothing touches disk in between. | The two hosts can reach each other. |
|
||||
| **Migration package** | Builds a self-contained folder or `.tar` holding the data, the images, and a plain-bash `install.sh`. Carry it on a disk, run the script on the target. | Air-gapped targets, or when you want the move reviewed and replayed later. |
|
||||
|
||||
The target needs **nothing installed**: no agent, no Python, no Go. Just `sshd`, a working `docker`
|
||||
CLI, `bash` and `gzip`.
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
### Run it in a container (recommended)
|
||||
|
||||
On the **source** host:
|
||||
|
||||
```bash
|
||||
git clone <this repo> docker-migrate && cd docker-migrate
|
||||
docker compose up -d --build
|
||||
docker compose logs docker-migrate # prints the URL, including the access token
|
||||
```
|
||||
|
||||
Then open the printed URL. It binds to `127.0.0.1` only; reach it from your laptop with a tunnel:
|
||||
|
||||
```bash
|
||||
ssh -L 8080:127.0.0.1:8080 you@source-host
|
||||
```
|
||||
|
||||
### Run it bare-metal
|
||||
|
||||
The binary is fully static and embeds the web UI, so there is nothing to install alongside it.
|
||||
|
||||
```bash
|
||||
make build # needs Go 1.25+ and Node 20+ ... or just `go build .` if you skip the UI rebuild
|
||||
./docker-migrate serve
|
||||
```
|
||||
|
||||
Prebuilt for several platforms:
|
||||
|
||||
```bash
|
||||
make release # dist/docker-migrate-linux-amd64, -linux-arm64, -darwin-arm64, -windows-amd64
|
||||
```
|
||||
|
||||
`docker-migrate` needs access to the Docker socket on the source host, so run it as a user in the
|
||||
`docker` group (or as root).
|
||||
|
||||
---
|
||||
|
||||
## Using it
|
||||
|
||||
1. **Containers tab** — everything on the source host, grouped by compose project.
|
||||
Tick the ones to move. Use *select all*, *select running*, or the compose-project checkbox for
|
||||
batch selection.
|
||||
2. **Expand a row** (`▸`) to choose per-container details: the name on the target, whether the image
|
||||
is pulled or transferred, whether networks and ports come along, and — per mount — whether to
|
||||
**copy the data**, **create it empty**, or **not mount it at all**. Bind mounts can be relocated to
|
||||
a different path on the target; named volumes can be renamed.
|
||||
3. **Apply to selected** in the toolbar does the same thing to every selected container at once
|
||||
(*copy all data*, *skip binds*, *image: pull on target*, …).
|
||||
4. **Right panel** — add the target host, hit *connect*, review the options, then either
|
||||
**migrate over SSH** or **build a package**.
|
||||
5. **Preview the commands** shows the exact `docker` invocations that will run on the target. Nothing
|
||||
is hidden.
|
||||
6. **Jobs tab** — live progress per container and per mount, with the full command log.
|
||||
|
||||
Start with **dry run** ticked. It performs every check and prints every command without changing
|
||||
anything on the target.
|
||||
|
||||
---
|
||||
|
||||
## How the data is actually moved
|
||||
|
||||
The interesting part is that there is exactly **one** mechanism for every kind of data location:
|
||||
|
||||
```
|
||||
source daemon ──CopyFromContainer(/mount/path)──▶ tar stream ──gzip──▶ ssh ──▶ docker cp -a - ctr:/parent
|
||||
```
|
||||
|
||||
The source container's own mount path is read through the Docker archive API — the same thing
|
||||
`docker cp` uses. That means:
|
||||
|
||||
- named volumes, anonymous volumes and bind mounts are all handled identically;
|
||||
- no helper image is pulled, and the container's image does not need `tar` inside it;
|
||||
- it works whether the container is running or stopped;
|
||||
- file ownership, permissions, symlinks and hardlinks are preserved (`docker cp -a`).
|
||||
|
||||
On the target the container is **created first, started last**. Creating it is what makes Docker
|
||||
materialise the named volumes and bind directories; the data is then copied into the stopped
|
||||
container, and only then is it started.
|
||||
|
||||
A mount the container declares **read-only** cannot be written through the container itself. For
|
||||
those, a throwaway container is created (never started) with the same volume attached writable, the
|
||||
data is copied into it, and it is removed straight after.
|
||||
|
||||
### What is faithfully reproduced
|
||||
|
||||
Image (by pull or by layer transfer), command, entrypoint, environment, labels, working directory,
|
||||
user, hostname, published and exposed ports, all mount types, user-defined networks with their
|
||||
subnets and the container's aliases, DNS settings, extra hosts, capabilities, devices, sysctls,
|
||||
ulimits, security options, restart policy, stop signal and timeout, healthcheck, log driver and
|
||||
options, memory/CPU/pids limits, privileged, read-only rootfs, init, and the PID/IPC/UTS/userns
|
||||
modes.
|
||||
|
||||
Settings that come from the **image** are deliberately not re-emitted — the recreated container
|
||||
carries only genuine run-time overrides, so it stays readable and keeps working when the image is
|
||||
later updated.
|
||||
|
||||
### What it will not do for you
|
||||
|
||||
- **`--rm` is never reapplied.** A migrated container that deletes itself cannot be inspected.
|
||||
- **`--volumes-from` is not reproduced.** You are warned; migrate the other container and mount
|
||||
explicitly.
|
||||
- **`--network container:other`** requires the other container to be migrated too. You are warned.
|
||||
- **Swarm services** are out of scope. This tool moves plain containers.
|
||||
- **Live databases**: copying a running database's files gives you a crash-consistent snapshot at
|
||||
best. The default is to stop the source container while copying — leave it on. For anything you
|
||||
really care about, take a dump instead and migrate that.
|
||||
- **Cross-architecture moves**: an `amd64` image will not run on an `arm64` target. The preflight
|
||||
shows the target's architecture; check it.
|
||||
|
||||
---
|
||||
|
||||
## Safety
|
||||
|
||||
The tool can stop containers and read every volume on the host, so it is treated as a privileged
|
||||
admin tool:
|
||||
|
||||
- It binds to **`127.0.0.1` by default**. Binding anywhere else automatically generates an access
|
||||
token and prints it.
|
||||
- **SSH host keys are verified** exactly like OpenSSH. An unknown key is refused until you approve
|
||||
the fingerprint in the UI; a *changed* key is refused outright until you explicitly replace it.
|
||||
Trusted keys go to `<data-dir>/known_hosts`.
|
||||
- **Credentials are not persisted unless you ask.** By default the password or key lives in memory
|
||||
for the session. Ticking *remember* writes it to `<data-dir>/connections.json`, mode `0600`.
|
||||
- **Nothing on the target is overwritten by default.** If a container name already exists the item
|
||||
fails; you choose *skip*, *rename* or *replace* explicitly. An existing **volume** is reused and
|
||||
merged into, never silently deleted, unless you pick *replace*.
|
||||
- Every command that runs on the target is echoed into the job log.
|
||||
- Bind mounts of `/var/run/docker.sock`, `/proc`, `/sys`, `/dev` and `/` are flagged, and a mount at
|
||||
`/` is refused outright.
|
||||
|
||||
---
|
||||
|
||||
## The migration package
|
||||
|
||||
`build package` produces:
|
||||
|
||||
```
|
||||
shop-migration/
|
||||
install.sh self-contained bash; read it, it is the whole contract
|
||||
manifest.json machine-readable description of everything inside
|
||||
README.txt instructions for whoever runs it
|
||||
images/ docker image archives (.tar.gz)
|
||||
data/ one archive per mount, per container
|
||||
```
|
||||
|
||||
On the target:
|
||||
|
||||
```bash
|
||||
./install.sh --dry-run # print every command, change nothing
|
||||
./install.sh # restore
|
||||
```
|
||||
|
||||
The installer never parses the manifest — every command is written out literally, so it can be read
|
||||
and audited before running. It checksums each payload before feeding it to Docker, and supports:
|
||||
|
||||
```
|
||||
--dry-run print every command without changing anything
|
||||
--yes do not ask for confirmation
|
||||
--no-start create the containers but leave them stopped
|
||||
--conflict MODE fail (default) | skip | replace | rename
|
||||
--rename-suffix S suffix used by --conflict rename
|
||||
--skip-verify do not checksum the payloads
|
||||
--only NAME[,NAME...] restore only these containers
|
||||
--docker CMD docker command to use
|
||||
--sudo prefix docker with sudo -n
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Target host requirements
|
||||
|
||||
| Requirement | Why |
|
||||
| --- | --- |
|
||||
| `sshd`, reachable from the source | transport |
|
||||
| `docker` CLI + a working daemon | everything |
|
||||
| the login user can use docker | either in the `docker` group, or tick *run docker through `sudo -n`* |
|
||||
| `gzip` | compressed transfers; without it the tool falls back to uncompressed |
|
||||
| `bash` | only for the migration package installer |
|
||||
|
||||
The **connect** button runs a preflight and tells you which of these are missing, plus the target's
|
||||
free disk space and architecture.
|
||||
|
||||
---
|
||||
|
||||
## Command line
|
||||
|
||||
```
|
||||
docker-migrate [serve] [flags] start the web interface (default)
|
||||
docker-migrate inspect [flags] print the source inventory as JSON
|
||||
docker-migrate version
|
||||
```
|
||||
|
||||
`serve` flags:
|
||||
|
||||
```
|
||||
--addr string address to listen on (default "127.0.0.1:8080")
|
||||
--token string require this token on every request; "auto" generates one
|
||||
--data-dir string connections and trusted host keys (default: OS config dir)
|
||||
--package-dir string where migration packages are written (default <data-dir>/packages)
|
||||
--docker-host string source docker daemon (default: the DOCKER_HOST environment)
|
||||
-v verbose logging
|
||||
```
|
||||
|
||||
`inspect` is handy for scripting and for reporting bugs:
|
||||
|
||||
```bash
|
||||
docker-migrate inspect --sizes | jq '.containers[] | {name, image, mounts}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HTTP API
|
||||
|
||||
Everything the UI does is available over HTTP. Pass the token as `X-Auth-Token` when one is set.
|
||||
|
||||
```
|
||||
GET /api/health
|
||||
GET /api/source inventory + default selections
|
||||
GET /api/source/sizes volume sizes (slow)
|
||||
GET /api/connections
|
||||
POST /api/connections
|
||||
DELETE /api/connections/{id}
|
||||
POST /api/connections/{id}/probe read the SSH host key fingerprint
|
||||
POST /api/connections/{id}/trust approve that fingerprint
|
||||
POST /api/connections/{id}/test preflight the target
|
||||
GET /api/connections/{id}/inventory what is already on the target
|
||||
POST /api/plan/preview render the commands, run nothing
|
||||
POST /api/migrate/ssh start a host-to-host migration
|
||||
POST /api/migrate/package start a package build
|
||||
GET /api/jobs, /api/jobs/{id}
|
||||
GET /api/jobs/{id}/events server-sent events, live progress
|
||||
POST /api/jobs/{id}/cancel
|
||||
GET /api/packages, /api/packages/{name}/download
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
```
|
||||
web/ React + TypeScript UI (vite)
|
||||
internal/spec/ the transport model: a container, and how to render it back into docker flags
|
||||
internal/dkr/ source Docker daemon: inventory, archive streams, image save
|
||||
internal/sshx/ SSH transport, host key trust, driving the target's docker CLI
|
||||
internal/migrate/ the two engines: SSH streaming, and package + installer generation
|
||||
internal/job/ progress tracking for long-running work
|
||||
internal/api/ HTTP handlers and SSE
|
||||
internal/webui/ the built UI, embedded into the binary
|
||||
```
|
||||
|
||||
```bash
|
||||
make test # go test ./...
|
||||
make vet
|
||||
make ui # rebuild the embedded UI
|
||||
cd web && npm run dev # UI dev server on :5173, proxying /api to :8080
|
||||
```
|
||||
|
||||
### Verifying it works
|
||||
|
||||
On a Linux host with Docker (a VM is fine):
|
||||
|
||||
```bash
|
||||
go test ./... # unit tests: command rendering, plan resolution,
|
||||
# and the generated installer, checked with bash
|
||||
|
||||
go test -tags e2e ./test/... -v # end to end, against the real daemon:
|
||||
# creates a container with a named volume, a
|
||||
# read-only bind mount and an anonymous volume,
|
||||
# writes files into all three, builds a package,
|
||||
# runs the generated install.sh, then reads the
|
||||
# files back out of the restored container
|
||||
```
|
||||
|
||||
The e2e tests restore onto the same daemon under a suffixed name and clean up after
|
||||
themselves, so a single machine is enough:
|
||||
|
||||
- `TestPackageRoundTrip` — builds a package and runs the generated `install.sh` for real.
|
||||
- `TestSSHMigration` — drives the host-to-host engine over a genuine SSH connection to
|
||||
`127.0.0.1`, so the whole transport (ssh, gzip streaming, the target's docker CLI, the
|
||||
staging container for read-only mounts, the verify step) is exercised. It needs
|
||||
`DM_SSH_HOST`, `DM_SSH_USER` and `DM_SSH_KEY`, and skips without them:
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -N '' -f ~/.ssh/dm_loop
|
||||
cat ~/.ssh/dm_loop.pub >> ~/.ssh/authorized_keys
|
||||
DM_SSH_HOST=127.0.0.1 DM_SSH_USER=root DM_SSH_KEY=~/.ssh/dm_loop \
|
||||
go test -tags e2e ./test/... -run TestSSHMigration -v
|
||||
```
|
||||
|
||||
Both were run against Debian 13 with Docker 29.7.2, alongside a browser pass over the web
|
||||
UI covering the trust prompt, a batch migration and a package build.
|
||||
|
||||
`internal/webui/dist` is committed so that a plain `go build .` produces a working binary without a
|
||||
Node toolchain. Rerun `make ui` after changing anything under `web/`.
|
||||
Reference in New Issue
Block a user