Skip to content

Runbook — visual catalog publication

Internal-only documentation site for the ciano-lake catalog. Owner: data platform. Nothing here touches quality gates, live data, or other services.

What it is

One publisher job turns five read-only sources into one versioned snapshot (catalog_context.json) and renders, from that single snapshot:

  • a MkDocs Material static site (declared semantics, verbatim operator notes, observed runs including the latest failed attempt, materialized coverage per environment, deep-documented tables),
  • llms.txt (AI-context index), and
  • optional Postgres comments on the declared publish surface (gold_bacen schema).

The analyst entry point is data-access.md, with recipes in data-access-recipes.md. When present in the repository, the publisher copies these source pages into the generated MkDocs project, links the home/dataset/table pages back to the guide, and checks the final HTML route plus the CDA/BACEN stable IDs in both catalog_context.json and llms.txt. Generated site files are never edited manually.

Site and Postgres publication are tracked separately in the publication record; they share the snapshot id but are not transactionally synchronized with each other.

Publication (local build, one command)

uv run python -m ciano_lake publish-catalog \
  --inventory vps=/path/to/vps_inventory.jsonl \
  [--inventory local=/path/to/local_inventory.jsonl] \
  [--no-pg-comments]

Defaults: output root {CIANO_META_ROOT}/catalog_site; Postgres comments are attempted only when CIANO_PG_DSN is set (skip with --no-pg-comments).

Output layout under the output root:

path meaning
versions/{snapshot_id}/ the served HTML for that snapshot (content-addressed; identical inputs → identical id)
versions/{snapshot_id}.project/ raw render (mkdocs project, snapshot JSON, llms.txt) — audit trail
current atomic pointer to the served version (symlink where supported, else CURRENT file)
_latest_publication.json last publication record: site status, PG status, runtime

Guarantees: the site is validated before promotion; promotion is a single rename into a never-pre-existing dir; the pointer flips atomically. Any failure before the flip leaves the previous site fully served.

The JSONL inventory comes from the coverage collector (run it on the data VPS with explicit roots):

uv run python tools/coverage_inventory.py collect \
  --root <lake-root> --repo-root <repo> --configuration-root <repo> \
  --environment vps --output-dir <out>

If no inventory is passed, every page says materialized state is unknown — the site never infers it.

Guia de acesso — rota e rede

Após um publish autorizado, a página fica disponível dentro do site em runbooks/data-access.html, portanto a rota interna completa é https://100.70.145.60/catalogo/runbooks/data-access.html. O host é Tailscale-only e a URL usa o IP do VPS; é esperado que um navegador mostre aviso de certificado porque o certificado não foi emitido para esse IP. Não há promessa de DNS, TLS público ou acesso fora da rede interna.

Deploying to the internal host (additive; touches nothing else)

deploy/catalog-site/publish-catalog-site.sh \
  <output-root>/versions/<snapshot_id> <host> [remote-root]

The script rsyncs the new version dir (unique name, --delete never used), validates index.html on the destination, then flips current atomically. Rollback = relink current to any retained version (command in the script header). On hosts without local rsync, the same contract is served by tar -C <version-dir> -cf - . | ssh <host> "tar -C <versions/<id>> -xf -" (transport only; the atomicity contract is unchanged).

Executed serving (2026-09-15, option (a), user-approved): the tools VPS (vps-dockers) serves the site through the existing Traefik (ciano-stack):

  • staging root: /home/plinio/catalog-site (/opt needs sudo; $HOME is plinio-owned; same atomic contract);
  • container ciano-stack-catalogo-site-1 (nginx:alpine, additive service in /opt/ciano-stack/docker-compose.yml, network ciano-stack_ciano-net) bind-mounts the site root read-only and serves root /srv/catalog/current — the current symlink resolves per request, so publisher pointer flips propagate live without touching the container;
  • Traefik router: PathPrefix(/catalogo) + tailscale-only@file (100.64.0.0/10 allowlist) + catalogo-strip strip-prefix middleware appended to the watched /opt/ciano-stack/traefik/middlewares.yml;
  • verified 2026-09-15: index / a real table page / llms.txt all 200 through the real Traefik HTTPS route from a tailnet-range source; loopback 403 (middleware confirmed applying);
  • reachable now at https://<tools-VPS tailnet IP>/catalogo/ (100.70.145.60 — certificate-name warning expected: the ACME cert is for bi.cianoinvestimentos.space). The FQDN path (https://bi.cianoinvestimentos.space/catalogo/) did not resolve from a tailnet client at deployment time (MagicDNS "server failed"; public DNS returns a proxy IP) — a DNS decision for the user, not a serving defect;
  • backups made before any edit (restore = copy back): /opt/ciano-stack/docker-compose.yml.bak-catalogo-20260915, /opt/ciano-stack/traefik/middlewares.yml.bak-catalogo-20260915;
  • full rollback: remove the catalogo-site service block + docker compose rm -sf catalogo-site, delete the catalogo-strip block from middlewares.yml (file is watched — Traefik reloads), relink current for a site-version rollback.

Postgres comments

Attempted only with CIANO_PG_DSN set. Tables are verified first (to_regclass): absent tables are skipped (not_present), never created, never errors. Per-table status lands in the publication record. Comments come from the same snapshot (tabela_id, declared chave_status, chave, deep-doc summary, first safe-join warning).

Refresh cadence — executed state (2026-09-15)

Not installed yet; one exact gap remains, user-owned: the sudo timer installation. The transport leg is proven and the site is serving the latest vps build.

  1. Transport vps-db → vps-dockers — PROVEN 2026-09-15 (after the user saved the tailnet ACL allowing ssh from tag:infra). Two facts make it work: the dedicated keypair on vps-db (~/.ssh/id_ed25519_catalogo, private key never read or copied by the agent; public half appended to vps-dockers' authorized_keys), and the explicit remote user plinio@ — vps-db's local user is dbadmin, and the default-user connection is refused by the tailnet policy (does not permit you to SSH as user "dbadmin"). The service unit carries Environment=CIANO_CATALOG_HOST=plinio@vps-dockers; the full job (collect → publish → transport with atomic flip) ran end-to-end manually on vps-db and the served snapshot flipped to the vps build. The latest manual run produced snapshot 194c08533b2ab049 (92 HTML pages, 78 Markdown pages, 77 table pages, 10.46 s) and the destination retained four version directories for rollback.
  2. The systemd timer needs sudo. A user-level timer was rejected deliberately: without lingering the user manager dies at session end, so an unattended schedule would silently stop. The prepared system units are the design; install them with:
# on vps-db (no push; repo synced to 4e6a7e8 via git bundle):
sudo cp /opt/ciano-lake/deploy/catalog-site/ciano-lake-catalog-publish.service \
        /opt/ciano-lake/deploy/catalog-site/ciano-lake-catalog-publish.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now ciano-lake-catalog-publish.timer

The job itself (deploy/catalog-site/publish-catalog-job.sh) runs collect → publish → best-effort transport with the discovered, non-secret roots (CIANO_LAKE_ROOT=/data/lake, repo /opt/ciano-lake) and the recorded WP4 coverage scope; it never aborts on a failed stage (a failed run still refreshes the site, showing the failed attempt distinctly) and logs to /opt/ciano-lake/meta/logs/catalog-publish.log. The Bruin timer was verified not-found/inactive on vps-db — no competing schedule. PG comments stay off (--no-pg-comments) until the user supplies CIANO_PG_DSN via a root-readable drop-in.

Runtime

Measured locally on the real catalog (74 catalog tables + 3 publish-surface pages, 89 HTML pages): ~2–3.5 s end-to-end on a laptop, dominated by the mkdocs build. The vps-db manual job measured 10.46 s for 92 HTML pages (78 Markdown pages, 77 table pages); the slower number includes the fresh inventory and publication stages.

Prerequisites

  • mkdocs + mkdocs-material (dev-dependencies; the publisher hard-fails without them — it never half-builds).
  • CIANO_META_ROOT pointing at the real meta root on the publish host (the snapshot records whether the env var was set, so a wrong default is visible on the page).
  • Coverage inventory JSONLs for materialized claims (else "unknown").