Prefect Runbook (K8s Dev Worker)

This runbook describes the current Prefect workflow for rapid in-cluster development.

Current Model

  • prefect-server runs inside the namespace and exposes API/UI on port 4200.
  • prefect-dev-worker runs as a long-lived worker pod using work pool dev-process (process type).
  • Deployments in prefect.yaml run git_clone on each flow run, so code is pulled from Git (branch dev) instead of baked /app source.

This means you do not need to rebuild the image for normal code changes; commit and push is enough.

Required Components

  1. Helm release with Prefect enabled and secret injection configured.
  2. A dev-process work pool in Prefect.
  3. Valid Oracle credentials in secrets.existingSecret (e.g. specify-secret).
  4. S3 report uploads: Flows such as Migrate Users and Migrate MUSIT Actors only write report.json when S3_BUCKET is set (plus S3/MinIO credentials). Reports use S3_MIGRATION_REPORTS_PREFIX (default migration-reports), not S3_PREFIX / oracle-schema. If S3_BUCKET is absent, runs succeed but report_uploaded is false — see Migration reports on S3.
  5. Migration image containing runtime dependencies:
    • prefect
    • python-oracledb with Oracle Instant Client for thick mode

Helm Configuration Notes

In charts/specify7/staging.values.yaml:

  • prefect.server.enabled: true
  • prefect.devWorker.enabled: true
  • prefect.devWorker.workPool: "dev-process"
  • prefect.devWorker.image.* points to your migration image tag.
  • secrets.existingSecret points to the env secret with Oracle and Prefect vars.

Daily Dev Loop

  1. Start API access:
kubectl port-forward svc/specify7-prefect-server 4200:4200
  1. In another terminal:
source .venv/bin/activate
export PREFECT_API_URL=http://127.0.0.1:4200/api

  1. Commit and push code changes to dev (the branch configured in prefect.yaml).

  2. Register/update deployment:

prefect deploy --all

Non-interactive deploys: This repo includes a root prefect.toml with [cli] prompt = false, so Prefect loads PREFECT_CLI_PROMPT=false when your shell’s current working directory is the project root (see prefect config view). You should not get per-deployment confirmation prompts. If you run Prefect from elsewhere, either cd into the repo first, use an explicit flag, or set the variable for that shell:

prefect deploy --no-prompt --all
# or
PREFECT_CLI_PROMPT=false prefect deploy --all

To persist the setting in a Prefect profile instead of prefect.toml, use prefect config set PREFECT_CLI_PROMPT=false (writes to the active profile; see Settings and profiles).

  1. Run PROD connectivity check (Oracle + S3 preflight):
prefect deployment run "Infrastructure Prod Check/infrastructure-prod-check-dev"

Optional: run Oracle schema snapshot export (uploads JSON/CSV to S3):

prefect deployment run "Oracle Schema Snapshot/oracle-schema-snapshot-dev"

Schema snapshot artifacts include schema_catalog.json, CSV extracts, and schema.dbml.

Optional: sync Specify hierarchy from YAML (post-bootstrap, idempotent; default is dry run):

prefect deployment run "Sync Specify structure/sync-specify-structure-dev" --param dry_run=false

See Specify structure sync for the YAML format and recorded outcomes.

Optional: migrate MUSIT ACTOR + PERSON_NAME into Specify Agent (Phase 1.1; default is dry run):

prefect deployment run "Migrate MUSIT Actors/migrate-musit-agents-dev"

See MUSIT collection agents migration for parameters and scope.

Optional: migrate application users from Oracle USD_METADATA into Specify SpecifyUser + Agent (Phase 1.4; default is dry run):

prefect deployment run "Migrate Users/migrate-users-dev" --param dry_run=false

See User migration report for the report format and recorded outcomes.

  1. Inspect results:
prefect flow-run ls
prefect flow-run logs <FLOW_RUN_ID>
kubectl logs -f -l component=prefect-dev-worker

Known Oracle Failure Patterns

  • DPY-6005 ... [Errno 111] Connection refused
    Network path or listener is not reachable from the cluster.

  • DPY-6001 ... service is not registered (ORA-12514-like)
    Host/port are reachable, but ORACLE_*_SERVICE is wrong for that listener.

  • DPY-3001 ... only supported in thick mode
    Server requires native network encryption/integrity; thick mode is required.

  • DPI-1047 ... cannot locate libclntsh.so
    Oracle Instant Client library is missing/invisible in image or stale image tag is still running.

  • S3 upload errors
    Verify S3_BUCKET, credentials, endpoint/region, and path-style settings in your secret. For MinIO/proxy setups with XAmzContentSHA256Mismatch, set S3_PAYLOAD_SIGNING_ENABLED=false.

Practical Tips

  • Use explicit image tags (not only latest) for reproducibility.
  • Keep prefect.yaml pull branch aligned with your active branch.
  • If runs are stuck in Scheduled, verify worker health and in-namespace connectivity to prefect-server:4200.
This site uses Just the Docs, a documentation theme for Jekyll.