895 lines
27 KiB
Markdown
895 lines
27 KiB
Markdown
# BD FHIR National — Production Deployment Guide
|
|
|
|
**Target OS:** Ubuntu 22.04 LTS
|
|
**Audience:** DGHS infrastructure team
|
|
**Estimated time:** 90 minutes first deployment, 15 minutes subsequent upgrades
|
|
|
|
---
|
|
|
|
## Prerequisites checklist
|
|
|
|
Before starting, confirm all of the following:
|
|
|
|
- [ ] Ubuntu 22.04 LTS server provisioned with minimum 8GB RAM, 4 vCPU, 100GB disk
|
|
- [ ] Server has outbound HTTPS access to:
|
|
- `auth.dghs.gov.bd` (Keycloak)
|
|
- `tr.ocl.dghs.gov.bd` (OCL)
|
|
- `icd11.dghs.gov.bd` (cluster validator)
|
|
- Your private Docker registry
|
|
- [ ] TLS certificates provisioned at paths matching `.env` `TLS_CERT_PATH` / `TLS_KEY_PATH`
|
|
- [ ] Keycloak `hris` realm configured per `ops/keycloak-setup.md`
|
|
- [ ] BD Core IG `bd.gov.dghs.core-0.2.1.tgz` present in `hapi-overlay/src/main/resources/packages/` on CI machine
|
|
- [ ] CI machine has built and pushed the Docker image to private registry
|
|
- [ ] `.env` file prepared from `.env.example` with all secrets filled in
|
|
|
|
---
|
|
|
|
## Part 1 — Server preparation
|
|
|
|
### 1.1 — Install Docker Engine
|
|
|
|
```bash
|
|
# Remove any conflicting packages
|
|
for pkg in docker.io docker-doc docker-compose docker-compose-v2 \
|
|
podman-docker containerd runc; do
|
|
sudo apt-get remove -y $pkg 2>/dev/null
|
|
done
|
|
|
|
# Add Docker's official GPG key
|
|
sudo apt-get update
|
|
sudo apt-get install -y ca-certificates curl
|
|
sudo install -m 0755 -d /etc/apt/keyrings
|
|
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
|
|
-o /etc/apt/keyrings/docker.asc
|
|
sudo chmod a+r /etc/apt/keyrings/docker.asc
|
|
|
|
# Add Docker repository
|
|
echo \
|
|
"deb [arch=$(dpkg --print-architecture) \
|
|
signed-by=/etc/apt/keyrings/docker.asc] \
|
|
https://download.docker.com/linux/ubuntu \
|
|
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
|
|
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
|
|
|
|
# Install Docker Engine and Compose plugin
|
|
sudo apt-get update
|
|
sudo apt-get install -y docker-ce docker-ce-cli containerd.io \
|
|
docker-buildx-plugin docker-compose-plugin
|
|
|
|
# Verify
|
|
docker --version # Docker Engine 25.x or higher
|
|
docker compose version # Docker Compose v2.x or higher
|
|
```
|
|
|
|
### 1.2 — Configure Docker daemon
|
|
|
|
```bash
|
|
# Create daemon config: limit log size, set storage driver
|
|
sudo tee /etc/docker/daemon.json <<'EOF'
|
|
{
|
|
"log-driver": "json-file",
|
|
"log-opts": {
|
|
"max-size": "100m",
|
|
"max-file": "5"
|
|
},
|
|
"storage-driver": "overlay2",
|
|
"live-restore": true
|
|
}
|
|
EOF
|
|
|
|
sudo systemctl restart docker
|
|
sudo systemctl enable docker
|
|
|
|
# Add your deploy user to the docker group (avoids sudo on every docker command)
|
|
sudo usermod -aG docker $USER
|
|
# Log out and back in for group membership to take effect
|
|
```
|
|
|
|
### 1.3 — Create application directory
|
|
|
|
```bash
|
|
sudo mkdir -p /opt/bd-fhir-national
|
|
sudo chown $USER:$USER /opt/bd-fhir-national
|
|
cd /opt/bd-fhir-national
|
|
```
|
|
|
|
### 1.4 — Deploy project files
|
|
|
|
Copy the entire project directory to the server. Recommended approach:
|
|
|
|
```bash
|
|
# From your CI/deployment machine:
|
|
rsync -avz --exclude='.git' \
|
|
--exclude='hapi-overlay/target' \
|
|
--exclude='hapi-overlay/src' \
|
|
./bd-fhir-national/ \
|
|
deploy@your-server:/opt/bd-fhir-national/
|
|
|
|
# The server needs:
|
|
# /opt/bd-fhir-national/
|
|
# ├── docker-compose.yml
|
|
# ├── .env ← you create this (see 1.5)
|
|
# ├── nginx/nginx.conf
|
|
# ├── postgres/fhir/postgresql.conf
|
|
# ├── postgres/fhir/init.sql
|
|
# ├── postgres/audit/postgresql.conf
|
|
# └── postgres/audit/init.sql
|
|
#
|
|
# The hapi-overlay/ source tree does NOT need to be on the production server.
|
|
# Only the Docker image (pre-built and pushed to registry) is needed.
|
|
```
|
|
|
|
### 1.5 — Create .env file
|
|
|
|
```bash
|
|
cd /opt/bd-fhir-national
|
|
cp .env.example .env
|
|
chmod 600 .env # restrict to owner only — contains secrets
|
|
|
|
# Edit .env with actual values
|
|
nano .env
|
|
```
|
|
|
|
**Required values in .env:**
|
|
|
|
```bash
|
|
# Docker image — must match what CI pushed
|
|
HAPI_IMAGE=your-registry.dghs.gov.bd/bd-fhir-hapi:1.0.0
|
|
|
|
# FHIR database
|
|
FHIR_DB_NAME=fhirdb
|
|
FHIR_DB_SUPERUSER=postgres
|
|
FHIR_DB_SUPERUSER_PASSWORD=$(openssl rand -base64 32)
|
|
FHIR_DB_APP_USER=hapi_app
|
|
FHIR_DB_APP_PASSWORD=$(openssl rand -base64 32)
|
|
|
|
# Audit database
|
|
AUDIT_DB_NAME=auditdb
|
|
AUDIT_DB_SUPERUSER=postgres
|
|
AUDIT_DB_SUPERUSER_PASSWORD=$(openssl rand -base64 32)
|
|
AUDIT_DB_WRITER_USER=audit_writer_login
|
|
AUDIT_DB_WRITER_PASSWORD=$(openssl rand -base64 32)
|
|
AUDIT_DB_MAINTAINER_USER=audit_maintainer_login
|
|
AUDIT_DB_MAINTAINER_PASSWORD=$(openssl rand -base64 32)
|
|
|
|
# TLS certificate paths (absolute paths on this server)
|
|
TLS_CERT_PATH=/etc/ssl/dghs/fhir.dghs.gov.bd.crt
|
|
TLS_KEY_PATH=/etc/ssl/dghs/fhir.dghs.gov.bd.key
|
|
```
|
|
|
|
> **Security:** Never commit `.env` to version control. Store the filled
|
|
> `.env` in your secrets vault (HashiCorp Vault, AWS SSM, or encrypted backup).
|
|
> Verify permissions after creation: `ls -la .env` should show `-rw-------`.
|
|
|
|
### 1.6 — Fix PostgreSQL init script password injection
|
|
|
|
The `postgres/audit/init.sql` file contains placeholder passwords.
|
|
PostgreSQL's Docker entrypoint does not perform variable substitution in
|
|
`.sql` init files — only `.sh` files. Replace the init SQL with a shell script:
|
|
|
|
```bash
|
|
# Create shell-based init script for audit database
|
|
cat > /opt/bd-fhir-national/postgres/audit/init.sh <<'INITSCRIPT'
|
|
#!/bin/bash
|
|
set -e
|
|
|
|
# Load passwords from environment variables
|
|
# (These env vars are set in docker-compose.yml from .env)
|
|
WRITER_USER="${AUDIT_DB_WRITER_USER:-audit_writer_login}"
|
|
WRITER_PASS="${AUDIT_DB_WRITER_PASSWORD}"
|
|
MAINTAINER_USER="${AUDIT_DB_MAINTAINER_USER:-audit_maintainer_login}"
|
|
MAINTAINER_PASS="${AUDIT_DB_MAINTAINER_PASSWORD}"
|
|
|
|
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
|
|
-- Create writer login user
|
|
DO \$\$
|
|
BEGIN
|
|
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${WRITER_USER}') THEN
|
|
CREATE USER ${WRITER_USER}
|
|
WITH NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT LOGIN
|
|
CONNECTION LIMIT 20
|
|
PASSWORD '${WRITER_PASS}';
|
|
END IF;
|
|
END
|
|
\$\$;
|
|
|
|
-- Create maintainer login user
|
|
DO \$\$
|
|
BEGIN
|
|
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${MAINTAINER_USER}') THEN
|
|
CREATE USER ${MAINTAINER_USER}
|
|
WITH NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT LOGIN
|
|
CONNECTION LIMIT 5
|
|
PASSWORD '${MAINTAINER_PASS}';
|
|
END IF;
|
|
END
|
|
\$\$;
|
|
|
|
GRANT CONNECT ON DATABASE ${POSTGRES_DB} TO ${WRITER_USER};
|
|
GRANT CONNECT ON DATABASE ${POSTGRES_DB} TO ${MAINTAINER_USER};
|
|
EOSQL
|
|
INITSCRIPT
|
|
|
|
chmod +x /opt/bd-fhir-national/postgres/audit/init.sh
|
|
```
|
|
|
|
Update `docker-compose.yml` to mount `init.sh` instead of `init.sql` for
|
|
the `postgres-audit` service:
|
|
|
|
```yaml
|
|
# In postgres-audit volumes: section, change:
|
|
# - ./postgres/audit/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
|
|
# To:
|
|
- ./postgres/audit/init.sh:/docker-entrypoint-initdb.d/init.sh:ro
|
|
```
|
|
|
|
Also pass the audit user environment variables to `postgres-audit`:
|
|
|
|
```yaml
|
|
# In postgres-audit environment: section, add:
|
|
AUDIT_DB_WRITER_USER: ${AUDIT_DB_WRITER_USER}
|
|
AUDIT_DB_WRITER_PASSWORD: ${AUDIT_DB_WRITER_PASSWORD}
|
|
AUDIT_DB_MAINTAINER_USER: ${AUDIT_DB_MAINTAINER_USER}
|
|
AUDIT_DB_MAINTAINER_PASSWORD: ${AUDIT_DB_MAINTAINER_PASSWORD}
|
|
```
|
|
|
|
Similarly for `postgres-fhir`, create `init.sh`:
|
|
|
|
```bash
|
|
cat > /opt/bd-fhir-national/postgres/fhir/init.sh <<'INITSCRIPT'
|
|
#!/bin/bash
|
|
set -e
|
|
|
|
APP_USER="${FHIR_DB_APP_USER:-hapi_app}"
|
|
APP_PASS="${FHIR_DB_APP_PASSWORD}"
|
|
|
|
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
|
|
DO \$\$
|
|
BEGIN
|
|
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${APP_USER}') THEN
|
|
CREATE USER ${APP_USER}
|
|
WITH NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT LOGIN
|
|
CONNECTION LIMIT 30
|
|
PASSWORD '${APP_PASS}';
|
|
END IF;
|
|
END
|
|
\$\$;
|
|
|
|
GRANT CONNECT ON DATABASE ${POSTGRES_DB} TO ${APP_USER};
|
|
GRANT USAGE ON SCHEMA public TO ${APP_USER};
|
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
|
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO ${APP_USER};
|
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
|
GRANT USAGE, SELECT ON SEQUENCES TO ${APP_USER};
|
|
EOSQL
|
|
INITSCRIPT
|
|
|
|
chmod +x /opt/bd-fhir-national/postgres/fhir/init.sh
|
|
```
|
|
|
|
### 1.7 — Authenticate with private Docker registry
|
|
|
|
```bash
|
|
# Log in to your private registry
|
|
docker login your-registry.dghs.gov.bd \
|
|
--username ${REGISTRY_USER} \
|
|
--password-stdin <<< "${REGISTRY_PASSWORD}"
|
|
|
|
# Verify the login persisted
|
|
cat ~/.docker/config.json | jq '.auths | keys'
|
|
# Should include "your-registry.dghs.gov.bd"
|
|
```
|
|
|
|
---
|
|
|
|
## Part 2 — First deployment
|
|
|
|
### 2.1 — Pull images
|
|
|
|
```bash
|
|
cd /opt/bd-fhir-national
|
|
|
|
# Pull all images declared in docker-compose.yml
|
|
docker compose --env-file .env pull
|
|
|
|
# Verify images are present locally
|
|
docker images | grep -E "hapi|postgres|pgbouncer|nginx"
|
|
```
|
|
|
|
### 2.2 — Start infrastructure services first
|
|
|
|
Start databases before HAPI. HAPI's `depends_on` with `condition: service_healthy`
|
|
handles this automatically, but starting manually in stages helps isolate
|
|
any first-run issues.
|
|
|
|
```bash
|
|
# Start databases
|
|
docker compose --env-file .env up -d postgres-fhir postgres-audit
|
|
|
|
# Wait for health checks to pass (up to 60 seconds)
|
|
echo "Waiting for PostgreSQL to be ready..."
|
|
until docker compose --env-file .env ps postgres-fhir \
|
|
| grep -q "healthy"; do
|
|
sleep 3
|
|
echo -n "."
|
|
done
|
|
echo ""
|
|
echo "postgres-fhir: healthy"
|
|
|
|
until docker compose --env-file .env ps postgres-audit \
|
|
| grep -q "healthy"; do
|
|
sleep 3
|
|
echo -n "."
|
|
done
|
|
echo ""
|
|
echo "postgres-audit: healthy"
|
|
```
|
|
|
|
### 2.3 — Verify PostgreSQL user creation
|
|
|
|
```bash
|
|
# Verify FHIR app user was created
|
|
docker exec bd-postgres-fhir psql -U postgres -d fhirdb -c \
|
|
"SELECT rolname, rolcanlogin FROM pg_roles WHERE rolname = 'hapi_app';"
|
|
# Expected: hapi_app | t
|
|
|
|
# Verify audit writer user was created
|
|
docker exec bd-postgres-audit psql -U postgres -d auditdb -c \
|
|
"SELECT rolname, rolcanlogin FROM pg_roles WHERE rolname = 'audit_writer_login';"
|
|
# Expected: audit_writer_login | t
|
|
```
|
|
|
|
### 2.4 — Start pgBouncer
|
|
|
|
```bash
|
|
docker compose --env-file .env up -d pgbouncer-fhir pgbouncer-audit
|
|
|
|
# Verify pgBouncer is healthy
|
|
until docker compose --env-file .env ps pgbouncer-fhir \
|
|
| grep -q "healthy"; do
|
|
sleep 3
|
|
done
|
|
echo "pgbouncer-fhir: healthy"
|
|
```
|
|
|
|
### 2.5 — Start HAPI (first replica)
|
|
|
|
```bash
|
|
docker compose --env-file .env up -d hapi
|
|
|
|
# Follow startup logs — this takes 60-120 seconds on first run
|
|
# Watch for these key log events in order:
|
|
# 1. "Running FHIR Flyway migrations" — V1 schema creation
|
|
# 2. "Running Audit Flyway migrations" — V2 audit schema creation
|
|
# 3. "Advisory lock acquired" — IG package initialisation begins
|
|
# 4. "BD Core IG package loaded successfully" — IG loaded
|
|
# 5. "BdTerminologyValidationSupport initialised" — OCL integration ready
|
|
# 6. "KeycloakJwtInterceptor initialised" — JWT validation ready
|
|
# 7. "HAPI RestfulServer interceptors registered" — server ready
|
|
# 8. Spring Boot startup completion message with port 8080
|
|
|
|
docker compose --env-file .env logs -f hapi
|
|
# Press Ctrl+C when you see the startup completion message
|
|
```
|
|
|
|
**Expected startup log sequence (key lines only):**
|
|
```
|
|
INFO o.f.core.internal.command.DbMigrate - Running FHIR Flyway migrations
|
|
INFO o.f.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "public"
|
|
INFO o.f.core.internal.command.DbMigrate - Running Audit Flyway migrations
|
|
INFO o.f.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "audit"
|
|
INFO b.g.d.f.init.IgPackageInitializer - Advisory lock acquired: lockKey=... waitedMs=...
|
|
INFO b.g.d.f.init.IgPackageInitializer - BD Core IG package loaded successfully: version=0.2.1
|
|
INFO b.g.d.f.t.BdTerminologyValidationSupport - BdTerminologyValidationSupport initialised
|
|
INFO b.g.d.f.i.KeycloakJwtInterceptor - KeycloakJwtInterceptor initialised
|
|
INFO b.g.d.f.c.SecurityConfig - HAPI RestfulServer interceptors registered
|
|
INFO o.s.b.w.e.t.TomcatWebServer - Tomcat started on port(s): 8080
|
|
INFO b.g.d.f.BdFhirApplication - Started BdFhirApplication in XX.XXX seconds
|
|
```
|
|
|
|
### 2.6 — Start nginx
|
|
|
|
```bash
|
|
docker compose --env-file .env up -d nginx
|
|
|
|
# Verify nginx started without config errors
|
|
docker compose --env-file .env logs nginx | tail -20
|
|
# Should NOT contain: [emerg] or [crit] — only [notice] lines
|
|
|
|
# Verify nginx health
|
|
docker compose --env-file .env ps nginx
|
|
# Status should be: Up (healthy)
|
|
```
|
|
|
|
### 2.7 — Verify full stack health
|
|
|
|
```bash
|
|
# Internal health check (bypasses nginx, hits HAPI directly)
|
|
docker exec $(docker compose --env-file .env ps -q hapi | head -1) \
|
|
curl -s http://localhost:8080/actuator/health | jq .
|
|
|
|
# Expected output:
|
|
# {
|
|
# "status": "UP",
|
|
# "components": {
|
|
# "db": { "status": "UP" },
|
|
# "auditDb": { "status": "UP" },
|
|
# "ocl": { "status": "UP" },
|
|
# "livenessState": { "status": "UP" },
|
|
# "readinessState": { "status": "UP" }
|
|
# }
|
|
# }
|
|
|
|
# External health check (through nginx + TLS)
|
|
curl -s https://fhir.dghs.gov.bd/actuator/health/liveness | jq .
|
|
# Expected: { "status": "UP" }
|
|
|
|
# FHIR metadata endpoint (unauthenticated)
|
|
curl -s https://fhir.dghs.gov.bd/fhir/metadata | jq '{
|
|
resourceType,
|
|
fhirVersion,
|
|
software: .software,
|
|
implementation: .implementation
|
|
}'
|
|
# Expected:
|
|
# {
|
|
# "resourceType": "CapabilityStatement",
|
|
# "fhirVersion": "4.0.1",
|
|
# "software": { "name": "BD FHIR National Repository", "version": "0.2.1" }
|
|
# }
|
|
```
|
|
|
|
---
|
|
|
|
## Part 3 — Phase 2 acceptance tests
|
|
|
|
Run all seven tests before declaring the deployment production-ready.
|
|
Each test includes the expected HTTP status, expected response body shape,
|
|
and what to check in the audit log if the test fails.
|
|
|
|
### Setup: obtain a vendor test token
|
|
|
|
```bash
|
|
VENDOR_TOKEN=$(curl -s -X POST \
|
|
"https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
|
|
-d "grant_type=client_credentials" \
|
|
-d "client_id=fhir-vendor-TEST-FAC-001" \
|
|
-d "client_secret=${TEST_VENDOR_SECRET}" \
|
|
| jq -r '.access_token')
|
|
|
|
echo "Token obtained: ${VENDOR_TOKEN:0:20}..."
|
|
```
|
|
|
|
---
|
|
|
|
### Test 1 — Valid Condition with valid ICD-11 code → 201
|
|
|
|
Submits a BD Core IG-compliant `bd-condition` resource with a valid
|
|
ICD-11 Diagnosis-class code.
|
|
|
|
```bash
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${VENDOR_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{
|
|
"resourceType": "Condition",
|
|
"meta": {
|
|
"profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
|
|
},
|
|
"clinicalStatus": {
|
|
"coding": [{
|
|
"system": "http://terminology.hl7.org/CodeSystem/condition-clinical",
|
|
"code": "active"
|
|
}]
|
|
},
|
|
"verificationStatus": {
|
|
"coding": [{
|
|
"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status",
|
|
"code": "confirmed"
|
|
}]
|
|
},
|
|
"code": {
|
|
"coding": [{
|
|
"system": "http://id.who.int/icd/release/11/mms",
|
|
"code": "1C62.0",
|
|
"display": "Typhoid fever"
|
|
}]
|
|
},
|
|
"subject": {
|
|
"reference": "Patient/test-patient-001"
|
|
},
|
|
"recordedDate": "2025-03-01"
|
|
}'
|
|
```
|
|
|
|
**Expected:** `HTTP 201 Created` with `Location` header containing the new resource URL.
|
|
|
|
**If 422 instead:**
|
|
- Check OCL connectivity: `curl https://tr.ocl.dghs.gov.bd/api/fhir/CodeSystem/$validate-code?system=http://id.who.int/icd/release/11/mms&code=1C62.0`
|
|
- Check IG is loaded: `curl http://localhost:8080/actuator/health` — OCL component should be UP
|
|
- Check HAPI logs for profile validation errors
|
|
|
|
---
|
|
|
|
### Test 2 — Invalid ICD-11 code → 422
|
|
|
|
```bash
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${VENDOR_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{
|
|
"resourceType": "Condition",
|
|
"meta": {
|
|
"profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
|
|
},
|
|
"clinicalStatus": {
|
|
"coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-clinical", "code": "active"}]
|
|
},
|
|
"verificationStatus": {
|
|
"coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status", "code": "confirmed"}]
|
|
},
|
|
"code": {
|
|
"coding": [{
|
|
"system": "http://id.who.int/icd/release/11/mms",
|
|
"code": "INVALID-CODE-99999",
|
|
"display": "This code does not exist"
|
|
}]
|
|
},
|
|
"subject": {"reference": "Patient/test-patient-001"},
|
|
"recordedDate": "2025-03-01"
|
|
}'
|
|
```
|
|
|
|
**Expected:** `HTTP 422 Unprocessable Entity` with OperationOutcome containing:
|
|
- `issue[0].severity`: `error`
|
|
- `issue[0].diagnostics`: contains "INVALID-CODE-99999" and rejection reason
|
|
- `issue[0].expression`: contains `Condition.code`
|
|
|
|
**Verify in audit table:**
|
|
```sql
|
|
SELECT rejection_code, rejection_reason, invalid_code, element_path
|
|
FROM audit.fhir_rejected_submissions
|
|
ORDER BY submission_time DESC LIMIT 1;
|
|
-- Expected: TERMINOLOGY_INVALID_CODE | OCL rejected code... | INVALID-CODE-99999 | Condition.code...
|
|
```
|
|
|
|
---
|
|
|
|
### Test 3 — Device-class ICD-11 code in Condition.code → 422
|
|
|
|
Device-class codes are valid ICD-11 codes but are not in the
|
|
`bd-condition-icd11-diagnosis-valueset` (restricted to Diagnosis + Finding).
|
|
|
|
```bash
|
|
# XA7RE2 is an example Device-class code in ICD-11 MMS
|
|
# Verify it is Device-class in OCL before running this test:
|
|
# curl "https://tr.ocl.dghs.gov.bd/api/fhir/CodeSystem/$lookup?system=http://id.who.int/icd/release/11/mms&code=XA7RE2"
|
|
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${VENDOR_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{
|
|
"resourceType": "Condition",
|
|
"meta": {
|
|
"profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
|
|
},
|
|
"clinicalStatus": {
|
|
"coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-clinical", "code": "active"}]
|
|
},
|
|
"verificationStatus": {
|
|
"coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status", "code": "confirmed"}]
|
|
},
|
|
"code": {
|
|
"coding": [{
|
|
"system": "http://id.who.int/icd/release/11/mms",
|
|
"code": "XA7RE2",
|
|
"display": "Device code — should be rejected"
|
|
}]
|
|
},
|
|
"subject": {"reference": "Patient/test-patient-001"},
|
|
"recordedDate": "2025-03-01"
|
|
}'
|
|
```
|
|
|
|
**Expected:** `HTTP 422` with OperationOutcome.
|
|
Rejection code in audit: `TERMINOLOGY_INVALID_CLASS`.
|
|
|
|
**If 201 instead (code accepted):**
|
|
- OCL ValueSet class restriction is not enforcing correctly
|
|
- Verify the ValueSet collection in OCL has correct concept_class filter
|
|
- Run: `python version_upgrade.py --verify-class-restriction`
|
|
|
|
---
|
|
|
|
### Test 4 — Profile violation (missing required field) → 422
|
|
|
|
Submits a Condition missing `clinicalStatus` which is required by `bd-condition` profile.
|
|
|
|
```bash
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${VENDOR_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{
|
|
"resourceType": "Condition",
|
|
"meta": {
|
|
"profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
|
|
},
|
|
"code": {
|
|
"coding": [{
|
|
"system": "http://id.who.int/icd/release/11/mms",
|
|
"code": "1C62.0"
|
|
}]
|
|
},
|
|
"subject": {"reference": "Patient/test-patient-001"}
|
|
}'
|
|
```
|
|
|
|
**Expected:** `HTTP 422` with OperationOutcome referencing missing `clinicalStatus`.
|
|
|
|
**If 201 instead:**
|
|
- BD Core IG is not loaded or profile is not enforcing `clinicalStatus` as required
|
|
- Check startup logs for IG load success
|
|
- Verify: `curl http://localhost:8080/fhir/StructureDefinition/bd-condition`
|
|
|
|
---
|
|
|
|
### Test 5 — No Bearer token → 401
|
|
|
|
```bash
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{"resourceType": "Condition"}'
|
|
```
|
|
|
|
**Expected:** `HTTP 401` with `WWW-Authenticate` header and OperationOutcome.
|
|
|
|
```bash
|
|
# Verify WWW-Authenticate header is present
|
|
curl -s -I \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{"resourceType":"Condition"}' \
|
|
| grep -i "www-authenticate"
|
|
# Expected: WWW-Authenticate: Bearer realm="BD FHIR National Repository"...
|
|
```
|
|
|
|
---
|
|
|
|
### Test 6 — Valid token but missing mci-api role → 401
|
|
|
|
Create a test client WITHOUT `mci-api` role in Keycloak for this test.
|
|
Or use a token from a different realm.
|
|
|
|
```bash
|
|
# Token from a client without mci-api role
|
|
NO_ROLE_TOKEN=$(curl -s -X POST \
|
|
"https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
|
|
-d "grant_type=client_credentials" \
|
|
-d "client_id=fhir-test-no-role" \
|
|
-d "client_secret=${TEST_NO_ROLE_SECRET}" \
|
|
| jq -r '.access_token')
|
|
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${NO_ROLE_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{"resourceType": "Condition"}'
|
|
```
|
|
|
|
**Expected:** `HTTP 401`.
|
|
|
|
**Verify in audit log:**
|
|
```sql
|
|
SELECT event_type, outcome_detail, client_id
|
|
FROM audit.audit_events
|
|
WHERE event_type = 'AUTH_FAILURE'
|
|
ORDER BY event_time DESC LIMIT 1;
|
|
-- Expected: AUTH_FAILURE | Required role 'mci-api' not present... | fhir-test-no-role
|
|
```
|
|
|
|
---
|
|
|
|
### Test 7 — Expired token → 401
|
|
|
|
```bash
|
|
# An expired token is one whose 'exp' claim is in the past.
|
|
# Easiest approach: obtain a token, wait for it to expire (default: 5 minutes),
|
|
# then use it.
|
|
#
|
|
# For automated testing, forge an expired token manually:
|
|
# (This requires knowing the signing key — use only in test environments)
|
|
#
|
|
# Alternative: Use a token from a deactivated Keycloak client
|
|
# (revoke the client's credentials, existing tokens become invalid)
|
|
|
|
# Or simply wait:
|
|
echo "Waiting 6 minutes for token to expire..."
|
|
EXPIRED_TOKEN=$(curl -s -X POST \
|
|
"https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
|
|
-d "grant_type=client_credentials" \
|
|
-d "client_id=fhir-vendor-TEST-FAC-001" \
|
|
-d "client_secret=${TEST_VENDOR_SECRET}" \
|
|
| jq -r '.access_token')
|
|
|
|
sleep 360 # wait for 5-minute Keycloak default expiry
|
|
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${EXPIRED_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{"resourceType": "Condition"}'
|
|
```
|
|
|
|
**Expected:** `HTTP 401` — "Token has expired".
|
|
|
|
---
|
|
|
|
### Test 8 — Cluster expression: raw postcoordinated code without extension → 422
|
|
|
|
```bash
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X POST https://fhir.dghs.gov.bd/fhir/Condition \
|
|
-H "Authorization: Bearer ${VENDOR_TOKEN}" \
|
|
-H "Content-Type: application/fhir+json" \
|
|
-d '{
|
|
"resourceType": "Condition",
|
|
"meta": {
|
|
"profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
|
|
},
|
|
"clinicalStatus": {
|
|
"coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-clinical", "code": "active"}]
|
|
},
|
|
"verificationStatus": {
|
|
"coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status", "code": "confirmed"}]
|
|
},
|
|
"code": {
|
|
"coding": [{
|
|
"system": "http://id.who.int/icd/release/11/mms",
|
|
"code": "1C62.0&has_severity=mild",
|
|
"display": "Raw postcoordinated string — prohibited"
|
|
}]
|
|
},
|
|
"subject": {"reference": "Patient/test-patient-001"},
|
|
"recordedDate": "2025-03-01"
|
|
}'
|
|
```
|
|
|
|
**Expected:** `HTTP 422` with OperationOutcome diagnosing:
|
|
`"ICD-11 postcoordinated expression in Condition.code.coding[0] must use the icd11-cluster-expression extension"`
|
|
|
|
Rejection code in audit: `CLUSTER_STEM_MISSING_EXTENSION`.
|
|
|
|
---
|
|
|
|
### Test 9 — Cache flush endpoint requires fhir-admin role
|
|
|
|
```bash
|
|
# Attempt with vendor token (mci-api only) — should be 403
|
|
curl -s -w "\n--- HTTP %{http_code} ---\n" \
|
|
-X DELETE https://fhir.dghs.gov.bd/admin/terminology/cache \
|
|
-H "Authorization: Bearer ${VENDOR_TOKEN}"
|
|
# Expected: 403 (blocked by nginx IP restriction OR TerminologyCacheManager role check)
|
|
|
|
# Attempt with fhir-admin token — should be 200
|
|
ADMIN_TOKEN=$(curl -s -X POST \
|
|
"https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
|
|
-d "grant_type=client_credentials" \
|
|
-d "client_id=fhir-admin-pipeline" \
|
|
-d "client_secret=${FHIR_ADMIN_CLIENT_SECRET}" \
|
|
| jq -r '.access_token')
|
|
|
|
# Note: /admin/ is restricted to 172.20.0.0/16 in nginx.
|
|
# Run this from within the Docker network or from the server itself:
|
|
docker exec $(docker compose --env-file .env ps -q hapi | head -1) \
|
|
curl -s -X DELETE \
|
|
-H "Authorization: Bearer ${ADMIN_TOKEN}" \
|
|
http://localhost:8080/admin/terminology/cache | jq .
|
|
# Expected: 200 with { "status": "flushed", "entriesEvicted": N, ... }
|
|
```
|
|
|
|
---
|
|
|
|
## Part 4 — Subsequent deployments (image upgrade)
|
|
|
|
When a new Docker image is built and pushed (new IG version, code changes):
|
|
|
|
```bash
|
|
cd /opt/bd-fhir-national
|
|
|
|
# 1. Update image tag in .env
|
|
nano .env
|
|
# Change: HAPI_IMAGE=your-registry.dghs.gov.bd/bd-fhir-hapi:1.0.0
|
|
# To: HAPI_IMAGE=your-registry.dghs.gov.bd/bd-fhir-hapi:1.1.0
|
|
|
|
# 2. Pull new image
|
|
docker compose --env-file .env pull hapi
|
|
|
|
# 3. Rolling restart — replaces containers one at a time
|
|
# At 1 replica (pilot): brief downtime expected (~30s)
|
|
docker compose --env-file .env up -d --no-deps hapi
|
|
|
|
# At 3 replicas (Phase 2): true rolling update — scale up then scale down
|
|
docker compose --env-file .env up -d --no-deps --scale hapi=4 hapi
|
|
# Wait for new replica to be healthy:
|
|
sleep 30
|
|
docker compose --env-file .env up -d --no-deps --scale hapi=3 hapi
|
|
|
|
# 4. Verify startup
|
|
docker compose --env-file .env logs --tail=50 hapi
|
|
|
|
# 5. Run acceptance tests (at minimum Tests 1, 2, 5)
|
|
```
|
|
|
|
---
|
|
|
|
## Part 5 — Operational runbook
|
|
|
|
### View logs
|
|
|
|
```bash
|
|
# All services
|
|
docker compose --env-file .env logs -f
|
|
|
|
# HAPI only (structured JSON — pipe through jq)
|
|
docker compose --env-file .env logs -f hapi | jq -R 'try fromjson'
|
|
|
|
# nginx access log
|
|
docker compose --env-file .env logs -f nginx
|
|
|
|
# Filter for rejected submissions in HAPI logs
|
|
docker compose --env-file .env logs hapi | \
|
|
jq -R 'try fromjson | select(.message | contains("rejected"))'
|
|
```
|
|
|
|
### Restart a specific service
|
|
|
|
```bash
|
|
docker compose --env-file .env restart hapi
|
|
docker compose --env-file .env restart nginx
|
|
```
|
|
|
|
### Emergency: full stack restart
|
|
|
|
```bash
|
|
docker compose --env-file .env down
|
|
docker compose --env-file .env up -d
|
|
```
|
|
|
|
### Query rejected submissions
|
|
|
|
```bash
|
|
docker exec bd-postgres-audit psql -U postgres -d auditdb -c "
|
|
SELECT
|
|
submission_time,
|
|
resource_type,
|
|
rejection_code,
|
|
LEFT(rejection_reason, 100) as reason,
|
|
client_id
|
|
FROM audit.fhir_rejected_submissions
|
|
ORDER BY submission_time DESC
|
|
LIMIT 20;"
|
|
```
|
|
|
|
### Check pgBouncer pool status
|
|
|
|
```bash
|
|
# Connect to pgBouncer admin interface
|
|
docker exec -it bd-pgbouncer-fhir \
|
|
psql -h localhost -p 5432 -U pgbouncer pgbouncer -c "SHOW POOLS;"
|
|
```
|
|
|
|
### Monitor disk usage
|
|
|
|
```bash
|
|
# PostgreSQL data volumes
|
|
docker system df -v | grep -E "postgres|audit"
|
|
|
|
# Log volume
|
|
docker system df -v | grep hapi-logs
|
|
```
|