156 lines
5.8 KiB
Markdown
156 lines
5.8 KiB
Markdown
# Deploying stratoflights-predictor
|
|
|
|
The predictor is a single static Go binary with no database and no required
|
|
external services. It downloads NOAA GFS/GEFS wind data to **node-local disk**
|
|
and serves the REST API (see `/docs` or `api/rest/predictor.swagger.yml`).
|
|
|
|
It is an **internal backend**: the public entrypoint is the stratoflights API
|
|
gateway, which calls the predictor over an internal overlay network. The
|
|
predictor enforces no auth of its own.
|
|
|
|
## Environments
|
|
|
|
| Environment | File | Notes |
|
|
|---|---|---|
|
|
| Local dev | `docker-compose.yml` | one instance, metrics off, named volume |
|
|
| Staging (single host) | `docker-compose.staging.yml` | all features + bundled Prometheus |
|
|
| Production (Swarm) | `docker-compose.swarm.yml` | node-pinned, replicated, metrics |
|
|
|
|
```bash
|
|
# Local
|
|
docker compose up --build
|
|
curl localhost:8080/ready
|
|
|
|
# Staging (single host, exercises the metrics pipeline)
|
|
docker compose -f docker-compose.staging.yml up --build
|
|
# Prometheus at :9090, predictor target should be UP
|
|
|
|
# Production — see below
|
|
```
|
|
|
|
## Production (Docker Swarm)
|
|
|
|
### Storage and node placement — the important part
|
|
|
|
The wind dataset is ~8.9 GiB (0.5°) and must live on **local disk, never NFS**.
|
|
To bound the number of copies, the service is pinned to nodes carrying the
|
|
`predictor.data=true` label; **label at most two nodes**. Each labelled node
|
|
keeps exactly one copy under a node-local bind mount.
|
|
|
|
On **each** labelled node, provision the local directories and a writable owner
|
|
for the non-root container (uid:gid `65532:65532`):
|
|
|
|
```bash
|
|
sudo mkdir -p /srv/predictor/data /srv/predictor/elevation
|
|
sudo chown -R 65532:65532 /srv/predictor
|
|
# (optional) seed the elevation dataset so descent terminates at ground level:
|
|
# python3 scripts/build_elevation.py /srv/predictor/elevation/ruaumoko-dataset
|
|
```
|
|
|
|
Label the two storage nodes:
|
|
|
|
```bash
|
|
docker node update --label-add predictor.data=true <node-a>
|
|
docker node update --label-add predictor.data=true <node-b>
|
|
```
|
|
|
|
Replicas are spread one-per-node by default (redundancy across both copies).
|
|
Scaling to multiple replicas **per** node is safe: they share the node-local
|
|
volume and coordinate the download with an exclusive `flock`, so only one
|
|
process per node fetches the dataset — the others wait and load the committed
|
|
file. To scale: `docker service scale predictor_predictor=4` (≤2 per node).
|
|
|
|
### Network
|
|
|
|
The gateway and Prometheus reach the predictor over a shared overlay. Create it
|
|
once and have the gateway stack join the same external network:
|
|
|
|
```bash
|
|
docker network create -d overlay --attachable stratoflights-net
|
|
```
|
|
|
|
The service is published only on that network under the alias `predictor`
|
|
(`http://predictor:8080`). No public Traefik router — the gateway is the edge.
|
|
|
|
### Deploy
|
|
|
|
Via the CI pipeline (recommended): push a `v*` tag → the image is built and the
|
|
stack is deployed through the Swarmpit API. Manually:
|
|
|
|
```bash
|
|
TAG=v1.0.0 docker stack deploy -c docker-compose.swarm.yml --with-registry-auth predictor
|
|
```
|
|
|
|
or import `docker-compose.swarm.yml` into Swarmpit and set `TAG`.
|
|
|
|
### Configuration
|
|
|
|
All settings are env vars (file/env/flag precedence; see README). Production
|
|
defaults are in `docker-compose.swarm.yml`:
|
|
|
|
| Variable | Purpose |
|
|
|---|---|
|
|
| `PREDICTOR_DATA_DIR=/data` | node-local dataset dir (bind mount) |
|
|
| `PREDICTOR_ELEVATION_DATASET=/srv/ruaumoko-dataset` | optional terrain data |
|
|
| `PREDICTOR_SOURCE=gfs-0p50-3h` | `gfs-0p50-3h`, `gfs-0p25-3h`, `gfs-0p25-1h`, `gefs-0p50-3h` |
|
|
| `PREDICTOR_DOWNLOAD_PARALLEL=16` | concurrent GRIB downloads |
|
|
| `PREDICTOR_UPDATE_INTERVAL=6h` | forecast refresh cadence |
|
|
| `PREDICTOR_METRICS_ENABLED=true` | expose `/metrics` |
|
|
|
|
No Docker secrets are needed — the predictor has no database or credentials.
|
|
|
|
### Health
|
|
|
|
- `GET /health` — liveness (always 200 while the process runs). The container
|
|
`HEALTHCHECK` calls the binary's `-healthcheck` mode (no curl in the image).
|
|
- `GET /ready` — readiness (200 only once a dataset is loaded). The gateway
|
|
should gate traffic on this; Swarm does **not** kill a container that is still
|
|
performing its first download thanks to the 120s `start_period`.
|
|
|
|
### Metrics
|
|
|
|
`/metrics` exposes Prometheus counters (`predictor_predictions_total`,
|
|
`predictor_downloads_total`, `predictor_download_bytes_total`) and the
|
|
`predictor_active_dataset_epoch_seconds` gauge. The service carries
|
|
`prometheus.scrape/port/path` deploy labels for Swarm service discovery; point
|
|
your central Prometheus at the `stratoflights-net` network.
|
|
|
|
## CI/CD (Forgejo → Swarmpit)
|
|
|
|
`.forgejo/workflows/ci-cd.yml`:
|
|
|
|
1. **test** (every push/PR): `gofmt` check, `go vet`, `go build`, `go test -race`.
|
|
2. **build** (develop branch and `v*` tags): buildx `linux/amd64` image pushed to
|
|
`git.intra.yksa.space/web/predictor` (`:develop`, or `:<version>` + `:latest`).
|
|
3. **deploy-staging** (develop) / **deploy-production** (`v*` tags): deploy
|
|
`docker-compose.swarm.yml` to the environment's Swarmpit stack via
|
|
`deploy/swarmpit-deploy.sh`.
|
|
|
|
Configure runner secrets (scope staging/production via Forgejo environments):
|
|
|
|
- `REGISTRY_USERNAME`, `REGISTRY_PASSWORD` — container registry
|
|
- `SWARMPIT_URL`, `SWARMPIT_TOKEN`, `STACK_NAME` — Swarmpit deploy target
|
|
- `CA_CERTIFICATES` — optional PEM bundle if Swarmpit uses a private CA
|
|
|
|
Cut a release:
|
|
|
|
```bash
|
|
git tag v1.0.0 && git push origin v1.0.0
|
|
```
|
|
|
|
## Operations
|
|
|
|
```bash
|
|
docker service ls --filter label=com.docker.stack.namespace=predictor
|
|
docker service logs -f predictor_predictor
|
|
docker service scale predictor_predictor=2 # ≤2 per labelled node
|
|
docker service rollback predictor_predictor
|
|
```
|
|
|
|
Trigger a dataset refresh or inspect jobs through the admin API:
|
|
|
|
```bash
|
|
curl -X POST http://predictor:8080/api/v1/admin/datasets -d '{"latest":true}'
|
|
curl http://predictor:8080/api/v1/admin/jobs
|
|
curl http://predictor:8080/api/v1/admin/status
|
|
```
|