328 lines
10 KiB
Markdown
328 lines
10 KiB
Markdown
# Scaling Roadmap — BD FHIR National
|
||
|
||
**Audience:** DGHS infrastructure team, future system architects
|
||
**Current phase:** Pilot (Phase 1)
|
||
|
||
---
|
||
|
||
## Phase thresholds
|
||
|
||
| Metric | Phase 1 (Pilot) | Phase 2 (Regional) | Phase 3 (National) |
|
||
|--------|-----------------|---------------------|---------------------|
|
||
| Vendors | <50 | <500 | >500 |
|
||
| Resources/day | <10,000 | <100,000 | >1,000,000 |
|
||
| Resources total | <1M | <10M | >10M |
|
||
| HAPI replicas | 1 | 3 | 5-10+ |
|
||
| Orchestrator | docker-compose | docker-compose | Kubernetes |
|
||
| PostgreSQL | Single instance | Primary + replica | Patroni HA cluster |
|
||
| Estimated trigger | Now | 6-18 months | 18-36 months |
|
||
|
||
---
|
||
|
||
## Phase 1 → Phase 2 changes
|
||
|
||
### 1. Scale HAPI replicas to 3
|
||
|
||
No configuration changes required — the architecture was designed for this from day one.
|
||
|
||
```bash
|
||
# On the production Ubuntu server
|
||
cd /opt/bd-fhir-national
|
||
docker-compose --env-file .env up -d --scale hapi=3
|
||
```
|
||
|
||
**Verify after scaling:**
|
||
```bash
|
||
# All 3 replicas healthy
|
||
docker-compose ps hapi
|
||
|
||
# nginx is load balancing across all 3
|
||
# (check HAPI logs — requests should appear in all replica logs)
|
||
docker-compose logs --tail=50 hapi
|
||
|
||
# pgBouncer pool has sufficient capacity
|
||
# 3 replicas × 5 HikariCP connections = 15 connections
|
||
# pgBouncer pool_size=20 — 5 headroom remaining. Acceptable.
|
||
```
|
||
|
||
**pgBouncer adjustment at 3+ replicas:**
|
||
At 5 replicas (5 × 5 = 25 connections), the current pgBouncer pool_size=20 becomes
|
||
a bottleneck. Update docker-compose.yml:
|
||
|
||
```yaml
|
||
# pgbouncer-fhir environment:
|
||
PGBOUNCER_DEFAULT_POOL_SIZE: "30" # was 20
|
||
# And increase postgres-fhir max_connections in postgresql.conf:
|
||
# max_connections = 40 # was 30
|
||
```
|
||
|
||
### 2. Add PostgreSQL streaming replication (read replica)
|
||
|
||
For read-heavy workloads (FHIR search, bulk export), add a read replica.
|
||
HAPI supports separate read and write datasource URLs.
|
||
|
||
```yaml
|
||
# Add to docker-compose.yml:
|
||
postgres-fhir-replica:
|
||
image: postgres:15-alpine
|
||
environment:
|
||
POSTGRES_DB: fhirdb
|
||
PGUSER: replicator
|
||
POSTGRES_PASSWORD: ${FHIR_REPLICA_PASSWORD}
|
||
volumes:
|
||
- postgres-fhir-replica-data:/var/lib/postgresql/data
|
||
- ./postgres/fhir/replica.conf:/etc/postgresql/postgresql.conf:ro
|
||
command: >
|
||
bash -c "
|
||
until pg_basebackup -h postgres-fhir -U replicator -D /var/lib/postgresql/data -P -Xs -R; do
|
||
sleep 5;
|
||
done && postgres -c config_file=/etc/postgresql/postgresql.conf"
|
||
networks:
|
||
- backend-fhir
|
||
```
|
||
|
||
Add `HAPI_DATASOURCE_READ_URL` environment variable pointing to the replica,
|
||
and update `DataSourceConfig.java` to configure a separate read datasource.
|
||
|
||
### 3. Add Redis for distributed JWKS cache
|
||
|
||
Currently each HAPI replica maintains an independent in-memory JWKS cache.
|
||
At 3 replicas, a Keycloak key rotation triggers 3 independent JWKS re-fetches
|
||
within the same second. This is acceptable. At 10+ replicas, add Redis for
|
||
a shared JWKS cache to reduce Keycloak load.
|
||
|
||
```yaml
|
||
# Add to docker-compose.yml:
|
||
redis:
|
||
image: redis:7-alpine
|
||
networks:
|
||
- frontend
|
||
- backend-fhir
|
||
command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
|
||
```
|
||
|
||
Update `KeycloakJwtInterceptor` to use Spring Cache with Redis backend for JWKS storage.
|
||
|
||
---
|
||
|
||
## Phase 2 → Phase 3 changes
|
||
|
||
### Move to Kubernetes
|
||
|
||
At national scale, docker-compose is not the correct orchestrator. Kubernetes
|
||
provides:
|
||
- Horizontal Pod Autoscaler (scale on CPU/RPS automatically)
|
||
- Rolling deployments (zero-downtime IG version upgrades)
|
||
- Pod Disruption Budgets (maintain minimum replicas during node maintenance)
|
||
- Namespace isolation (separate FHIR, audit, monitoring namespaces)
|
||
|
||
**Kubernetes equivalents:**
|
||
|
||
| docker-compose service | Kubernetes resource |
|
||
|------------------------|---------------------|
|
||
| hapi (--scale N) | Deployment + HPA |
|
||
| postgres-fhir | StatefulSet (or external Patroni) |
|
||
| postgres-audit | StatefulSet (or external Patroni) |
|
||
| pgbouncer-fhir | Deployment (sidecar or standalone) |
|
||
| nginx | Ingress (nginx-ingress-controller) |
|
||
|
||
### Partition HAPI JPA tables
|
||
|
||
At 5M+ resources in `HFJ_RESOURCE`, evaluate partitioning (see V1 migration comments).
|
||
|
||
Prerequisites before partitioning HAPI JPA tables:
|
||
1. HAPI must be stopped during the migration (ALTER TABLE is not online in PostgreSQL 15)
|
||
2. Foreign key references to HFJ_RESOURCE from all SPIDX tables must be updated
|
||
3. The partition key must be included in all primary keys
|
||
4. Hibernate DDL validation must be disabled during migration, then re-enabled
|
||
|
||
This is a planned maintenance window operation — minimum 4-hour downtime window
|
||
for a database with 5M resources. At 10,000 resources/day, you have
|
||
approximately 18 months to plan this migration from initial deployment.
|
||
|
||
**Trigger:** Run `EXPLAIN ANALYZE` on a representative FHIR search query.
|
||
When sequential scans on HFJ_RESOURCE appear in the plan despite indexes,
|
||
partitioning is overdue.
|
||
|
||
---
|
||
|
||
## Partition maintenance — monthly cron job
|
||
|
||
The audit tables are partitioned by month with partitions pre-created through 2027.
|
||
**A missing partition causes INSERT to fail with a hard error** — no graceful degradation.
|
||
|
||
### Setup (run once on the audit PostgreSQL host)
|
||
|
||
```bash
|
||
# Create a login user for the maintenance function
|
||
# (audit_maintainer_login was created by postgres/audit/init.sql)
|
||
|
||
# Add to crontab on the Ubuntu host (or in a scheduled container):
|
||
crontab -e
|
||
|
||
# Run on the 20th of each month at 00:00 UTC — creates next month's partition
|
||
0 0 20 * * docker exec bd-postgres-audit psql \
|
||
-U audit_maintainer_login \
|
||
-d auditdb \
|
||
-c "SELECT audit.create_next_month_partitions();" \
|
||
>> /var/log/bd-fhir-partition-maintenance.log 2>&1
|
||
```
|
||
|
||
### Verify partition creation
|
||
|
||
```bash
|
||
# After the cron runs, verify the new partition exists
|
||
docker exec bd-postgres-audit psql -U postgres -d auditdb -c "
|
||
SELECT
|
||
c.relname AS partition_name,
|
||
pg_get_expr(c.relpartbound, c.oid) AS partition_range
|
||
FROM pg_class c
|
||
JOIN pg_inherits i ON i.inhrelid = c.oid
|
||
JOIN pg_class p ON p.oid = i.inhparent
|
||
JOIN pg_namespace n ON n.oid = p.relnamespace
|
||
WHERE n.nspname = 'audit'
|
||
AND p.relname = 'audit_events'
|
||
ORDER BY c.relname DESC
|
||
LIMIT 3;
|
||
"
|
||
# Should show the three most recent monthly partitions
|
||
```
|
||
|
||
### Monitor for missing partitions
|
||
|
||
Add this check to your monitoring system (Prometheus alerting or cron):
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
# check_audit_partitions.sh
|
||
# Alert if the next month's partition does not exist by the 25th
|
||
|
||
NEXT_MONTH=$(date -d "+1 month" +%Y_%m)
|
||
PARTITION="audit_events_${NEXT_MONTH}"
|
||
|
||
RESULT=$(docker exec bd-postgres-audit psql -U postgres -d auditdb -tAc "
|
||
SELECT COUNT(*) FROM pg_class c
|
||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||
WHERE n.nspname = 'audit' AND c.relname = '${PARTITION}';")
|
||
|
||
if [ "$RESULT" -eq "0" ]; then
|
||
echo "ALERT: Missing audit partition for next month: ${PARTITION}"
|
||
# Send to your alerting system (PagerDuty, Slack, email)
|
||
exit 1
|
||
fi
|
||
echo "OK: Partition ${PARTITION} exists"
|
||
exit 0
|
||
```
|
||
|
||
---
|
||
|
||
## Monitoring — key metrics to track
|
||
|
||
These metrics indicate when scaling actions are needed.
|
||
|
||
### PostgreSQL — fhir
|
||
|
||
```sql
|
||
-- Connection utilisation (should be <80% of max_connections)
|
||
SELECT count(*) as active_connections,
|
||
max_conn,
|
||
round(100.0 * count(*) / max_conn, 1) as utilisation_pct
|
||
FROM pg_stat_activity, (SELECT setting::int as max_conn FROM pg_settings WHERE name='max_connections') mc
|
||
WHERE state = 'active'
|
||
GROUP BY max_conn;
|
||
|
||
-- Table bloat (trigger VACUUM if dead_tuple_ratio > 10%)
|
||
SELECT relname, n_live_tup, n_dead_tup,
|
||
round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 1) as dead_pct
|
||
FROM pg_stat_user_tables
|
||
WHERE relname IN ('hfj_resource', 'hfj_spidx_token', 'hfj_res_ver')
|
||
ORDER BY dead_pct DESC;
|
||
|
||
-- Index usage (trigger REINDEX if idx_scan is 0 for a non-new index)
|
||
SELECT relname, indexrelname, idx_scan, idx_tup_read
|
||
FROM pg_stat_user_indexes
|
||
WHERE relname LIKE 'hfj_%'
|
||
ORDER BY idx_scan ASC
|
||
LIMIT 10;
|
||
```
|
||
|
||
### PostgreSQL — audit
|
||
|
||
```sql
|
||
-- Partition sizes (plan next archive when any partition exceeds 10GB)
|
||
SELECT
|
||
c.relname as partition,
|
||
pg_size_pretty(pg_relation_size(c.oid)) as size
|
||
FROM pg_class c
|
||
JOIN pg_inherits i ON i.inhrelid = c.oid
|
||
JOIN pg_class p ON p.oid = i.inhparent
|
||
JOIN pg_namespace n ON n.oid = p.relnamespace
|
||
WHERE n.nspname = 'audit' AND p.relname = 'audit_events'
|
||
ORDER BY c.relname DESC;
|
||
|
||
-- Rejection rate by vendor (flag vendors with >10% rejection rate)
|
||
SELECT
|
||
client_id,
|
||
COUNT(*) as total_events,
|
||
SUM(CASE WHEN outcome = 'REJECTED' THEN 1 ELSE 0 END) as rejections,
|
||
ROUND(100.0 * SUM(CASE WHEN outcome = 'REJECTED' THEN 1 ELSE 0 END) / COUNT(*), 1) as rejection_pct
|
||
FROM audit.audit_events
|
||
WHERE event_time > NOW() - INTERVAL '7 days'
|
||
AND event_type IN ('OPERATION', 'VALIDATION_FAILURE')
|
||
GROUP BY client_id
|
||
ORDER BY rejection_pct DESC;
|
||
```
|
||
|
||
### HAPI — Prometheus metrics
|
||
|
||
Key metrics exposed at `/actuator/prometheus`:
|
||
|
||
| Metric | Alert threshold |
|
||
|--------|-----------------|
|
||
| `hikaricp_connections_pending` | >0 for >30s → pool exhaustion |
|
||
| `hikaricp_connection_timeout_total` | Any increment → pool exhaustion |
|
||
| `http_server_requests_seconds_max` | >30s → OCL timeout or slow validation |
|
||
| `jvm_memory_used_bytes / jvm_memory_max_bytes` | >85% → OOM risk, increase container memory |
|
||
| `process_uptime_seconds` | Resets → unexpected container restart |
|
||
|
||
---
|
||
|
||
## IG upgrade procedure
|
||
|
||
When BD Core IG advances from v0.2.1 to v0.3.0:
|
||
|
||
```bash
|
||
# 1. On CI machine: place new package.tgz in src/main/resources/packages/
|
||
cp bd.gov.dghs.core-0.3.0.tgz hapi-overlay/src/main/resources/packages/
|
||
|
||
# 2. Remove old package (one IG version per image)
|
||
rm hapi-overlay/src/main/resources/packages/bd.gov.dghs.core-0.2.1.tgz
|
||
|
||
# 3. Update application.yaml / docker-compose env vars:
|
||
# HAPI_IG_PACKAGE_CLASSPATH=classpath:packages/bd.gov.dghs.core-0.3.0.tgz
|
||
# HAPI_IG_VERSION=0.3.0
|
||
|
||
# 4. Build and push new image
|
||
docker build \
|
||
--build-arg IG_PACKAGE=bd.gov.dghs.core-0.3.0.tgz \
|
||
--build-arg BUILD_VERSION=1.1.0 \
|
||
--build-arg GIT_COMMIT=$(git rev-parse --short HEAD) \
|
||
-t your-registry.dghs.gov.bd/bd-fhir-hapi:1.1.0 \
|
||
-f hapi-overlay/Dockerfile .
|
||
|
||
docker push your-registry.dghs.gov.bd/bd-fhir-hapi:1.1.0
|
||
|
||
# 5. Update HAPI_IMAGE in .env on production server
|
||
# 6. Rolling redeploy
|
||
docker-compose --env-file .env pull hapi
|
||
docker-compose --env-file .env up -d --no-deps hapi
|
||
|
||
# 7. Verify new IG version is active
|
||
curl -s https://fhir.dghs.gov.bd/fhir/metadata | jq '.software.version'
|
||
# Expected: "0.3.0" or the configured HAPI_IG_VERSION value
|
||
```
|
||
|
||
**Vendor notification:** IG upgrades that change SHALL constraints require
|
||
vendor notification at least 30 days in advance. Vendors must test against
|
||
the staging environment before production deployment.
|