# Slide viewer tile-server operations

The tile server is a source-bound pixel service. cBioPortal remains the owner
of WSI hierarchy and slide-access metadata; the tile server receives an exact
source URL and short-lived capability from the backend.

## Development capacity controls

- Beta runs one warm replica and accepts node/pod downtime. The dedicated
  node group keeps a one-node steady-state floor and a four-node ceiling;
  scale to the ceiling manually for an announced load event until workload
  autoscaling is added.
- The beta disruption budget allows the single pod to be voluntarily
  disrupted; production should use a separate multi-replica policy.
  Gunicorn worker count and the remaining runtime settings come from the
  private ConfigMap/environment.
- The checked-in container manifest reserves 8Gi and caps memory at 16Gi per
  pod. CPU and ephemeral-storage requests/limits remain unset until sustained
  production load establishes realistic values.
- Pods select and tolerate `workload=tile-viewer`. That node group is managed
  outside this repository and must exist before the Argo Application is synced.
- `MAX_IMAGE_OPERATIONS=2` bounds blocking tile/thumbnail work per worker;
  one replica and two Gunicorn workers provide four bounded image operation
  slots across beta.
- `CACHE_MISS_RATE_LIMIT_PER_MINUTE` is a shared Redis sliding-window limit
  for extraction leaders per capability subject and source. Redis cache hits
  are not application-rate-limited.
- Redis locks coalesce identical cache misses across workers and replicas.
- Cold-read guardrails use a renewable 120-second distributed lock lease, a
  30-second follower wait, a 2-second image-operation queue bound, and a
  60-second Gunicorn worker timeout. ECS slide opens use bounded client
  connect/read timeouts and retries. These values do not increase worker,
  image-operation, or open-slide concurrency.
- Ingress retains a 100 requests/second limit, burst multiplier 5, and a
  300-connection limit per client (six browser connections for 50 users behind
  one NAT address). Requests carrying `X-WSI-Source` are hashed
  to keep a slide's cold-open cache on one pod.

The Redis instance used by `REDIS_URL` must have bounded memory, an LRU/LFU
eviction policy, and alerts for evictions, memory pressure, and unavailable
connections. Redis is an optimization; a Redis outage must leave the service
authorized and bounded by the ingress and image-operation controls.

Before promotion, record the live Redis service/CIDR, ECS endpoint/CIDR, and
cluster DNS labels. The current policy intentionally remains ingress-only
until those destinations are verified; the promotion gate is to add
default-deny egress with only DNS, Redis port 6379, and ECS port 9020 allowed.
Do not broaden the policy to `0.0.0.0/0` as a substitute for that inventory.

## Monitoring

Datadog scrapes `/metrics` through the pod IP using OpenMetrics. The public
ingress intentionally routes only `/wsi/tiles`, `/wsi/thumbnails`,
`/wsi/health`, and `/wsi/ready`; `/metrics` is not externally exposed. The
network policy permits the monitoring namespace to scrape the metrics route.

Alert on tile/thumbnail p95 and p99 latency, hierarchy/access latency from the
backend, cache hit rate, distributed lock outcomes, Redis errors, image queue
time, 429/503 rates, pod restarts, memory, and block-cache volume usage.

## Release procedure

1. Build and publish the tile-server image, then replace the tag in the
   deployment with the exact immutable `image@sha256:<digest>` reference
   before promotion. The deployment is not promotion-ready while it contains
   only a tag.
2. Build the green ClickHouse database with the WSI access projection and
   import the complete `meta_wsi.txt`/`data_wsi.txt` snapshots.
3. Verify WSI row counts, `can_serve_tiles`, projection materialization, and
   ClickHouse query plans before switching cBioPortal to the green database.
4. Ensure the external `tile-viewer` node group and private configuration are
   available, deploy the tile server, warm representative slides, and monitor
   memory, Redis, and latency dashboards during the canary.

Rollback is the previous backend/tile image pair plus the previous green
ClickHouse database. The projection is additive and does not require a live
mutation of the active database.