---
title: "Operations, retention and recovery"
description: "Nextvision Status — Operations, retention and recovery"
---

> Documentation Index
> Fetch the complete documentation index at: https://status.nextvisions.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations, retention and recovery

## Health

`/healthz` checks the API process. `/readyz` checks PostgreSQL and Redis. Object-store failure degrades attachment/report/snapshot operations; monitor MinIO separately. Workers log scheduler, evaluator, retention and delivery failures to stdout/stderr.

Observe probe last-seen timestamps, due-job age, pending/failed deliveries, PostgreSQL storage/partition growth and Redis memory. Redis is configured with no eviction. Snapshot age is visible on the public page and is not a substitute for an external monitor of this system.

## Retention

Free keeps raw checks/metrics seven days and daily/hourly aggregates 90 days. Business retention is configured by platform admin. Raw tables use monthly PostgreSQL partitions; the worker prepares adjacent months and deletes tenant-expired samples. Empty historical partitions can be removed during database maintenance after confirming no retained rows remain.

## Backups

Use your platform's encrypted backup target and retention policy. Example local logical backup:

```sh
docker compose exec -T postgres pg_dump -U postgres -Fc status > status.dump
```

Back up MinIO objects, versioning configuration and encryption keys separately. Redis persistence improves queue/cache recovery, but PostgreSQL is the authoritative source for jobs, outbox events and delivery attempts. Never include `.env` or raw production credentials in source control or diagnostic bundles.

## Restore drill

Provision a separate PostgreSQL database and private MinIO bucket. Restore the dump with `pg_restore --no-owner`, restore bucket objects and secrets, then run the migrator. Start API/workers against the restored dependencies and check tenant isolation, published history, attachments and snapshot timestamps before changing traffic.

The `scripts/backup-restore.sh` script performs a disposable local PostgreSQL restore drill without replacing the live database. Production recovery still needs a separate object-store restore test and recovery-point review.

## Failure behavior

When Redis restarts, workers rescan database pending rows. When a worker dies, expired job/delivery leases are reclaimed. If the public endpoint cannot verify the page in PostgreSQL, it bypasses Redis and reads the published MinIO snapshot. The response is always marked `stale: true`; its original `updated_at`, published branding and any suspension notice are preserved. Once the database recovers, the endpoint resumes serving live data.

Fallback has a separate seven-second budget after the live database/cache budget (up to five seconds). It makes at most two complete object-read attempts, each limited to three seconds, with a short delay between attempts. The storage SDK may retry HTTP exchanges within each attempt. Interrupted bodies, network timeouts and temporary storage errors can be retried; missing/forbidden objects and invalid snapshots fail immediately. The API validates the complete JSON, page slug, timestamp, status and required collections, with a 64 MiB size limit. It returns 503 if no valid snapshot can be read; the UI retains its last loaded data with an error.

Look for `public snapshot read failed` in API logs: `error_kind`, `storage_code`, `storage_status`, `attempt`, `elapsed_ms` and `retry` identify the failure without recording credentials or response bodies. `public snapshot read recovered` records a successful second attempt. Monitor these failures and snapshot age separately from `/readyz`. A concurrent image build is not proof of the cause of a storage error; check actual storage/network diagnostics and host capacity.

## Capacity validation

Use the included benchmark command and load profile. Synthetic benchmark observations do not prove real Internet endpoint performance or a 24-hour production soak. Record machine resources, image digests, throughput, scheduling lag and API percentiles. Do not advertise a 10,000-monitor SLA before a representative 24-hour run completes.

Source: https://status.nextvisions.ai/docs/en/operations/index.mdx