Files

Dr. B. M. Riazul Islam c11f0bd5bc first commit

2026-03-16 00:02:58 +06:00

27 KiB

Raw Blame History

BD FHIR National — Production Deployment Guide

Target OS: Ubuntu 22.04 LTS
Audience: DGHS infrastructure team
Estimated time: 90 minutes first deployment, 15 minutes subsequent upgrades

Prerequisites checklist

Before starting, confirm all of the following:

Ubuntu 22.04 LTS server provisioned with minimum 8GB RAM, 4 vCPU, 100GB disk
Server has outbound HTTPS access to:
- auth.dghs.gov.bd (Keycloak)
- tr.ocl.dghs.gov.bd (OCL)
- icd11.dghs.gov.bd (cluster validator)
- Your private Docker registry
TLS certificates provisioned at paths matching .env TLS_CERT_PATH / TLS_KEY_PATH
Keycloak hris realm configured per ops/keycloak-setup.md
BD Core IG bd.gov.dghs.core-0.2.1.tgz present in hapi-overlay/src/main/resources/packages/ on CI machine
CI machine has built and pushed the Docker image to private registry
.env file prepared from .env.example with all secrets filled in

Part 1 — Server preparation

1.1 — Install Docker Engine

# Remove any conflicting packages
for pkg in docker.io docker-doc docker-compose docker-compose-v2 \
           podman-docker containerd runc; do
  sudo apt-get remove -y $pkg 2>/dev/null
done

# Add Docker's official GPG key
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) \
  signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine and Compose plugin
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

# Verify
docker --version          # Docker Engine 25.x or higher
docker compose version    # Docker Compose v2.x or higher

1.2 — Configure Docker daemon

# Create daemon config: limit log size, set storage driver
sudo tee /etc/docker/daemon.json <<'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "5"
  },
  "storage-driver": "overlay2",
  "live-restore": true
}
EOF

sudo systemctl restart docker
sudo systemctl enable docker

# Add your deploy user to the docker group (avoids sudo on every docker command)
sudo usermod -aG docker $USER
# Log out and back in for group membership to take effect

1.3 — Create application directory

sudo mkdir -p /opt/bd-fhir-national
sudo chown $USER:$USER /opt/bd-fhir-national
cd /opt/bd-fhir-national

1.4 — Deploy project files

Copy the entire project directory to the server. Recommended approach:

# From your CI/deployment machine:
rsync -avz --exclude='.git' \
  --exclude='hapi-overlay/target' \
  --exclude='hapi-overlay/src' \
  ./bd-fhir-national/ \
  deploy@your-server:/opt/bd-fhir-national/

# The server needs:
# /opt/bd-fhir-national/
# ├── docker-compose.yml
# ├── .env                        ← you create this (see 1.5)
# ├── nginx/nginx.conf
# ├── postgres/fhir/postgresql.conf
# ├── postgres/fhir/init.sql
# ├── postgres/audit/postgresql.conf
# └── postgres/audit/init.sql
#
# The hapi-overlay/ source tree does NOT need to be on the production server.
# Only the Docker image (pre-built and pushed to registry) is needed.

1.5 — Create .env file

cd /opt/bd-fhir-national
cp .env.example .env
chmod 600 .env   # restrict to owner only — contains secrets

# Edit .env with actual values
nano .env

Required values in .env:

# Docker image — must match what CI pushed
HAPI_IMAGE=your-registry.dghs.gov.bd/bd-fhir-hapi:1.0.0

# FHIR database
FHIR_DB_NAME=fhirdb
FHIR_DB_SUPERUSER=postgres
FHIR_DB_SUPERUSER_PASSWORD=$(openssl rand -base64 32)
FHIR_DB_APP_USER=hapi_app
FHIR_DB_APP_PASSWORD=$(openssl rand -base64 32)

# Audit database
AUDIT_DB_NAME=auditdb
AUDIT_DB_SUPERUSER=postgres
AUDIT_DB_SUPERUSER_PASSWORD=$(openssl rand -base64 32)
AUDIT_DB_WRITER_USER=audit_writer_login
AUDIT_DB_WRITER_PASSWORD=$(openssl rand -base64 32)
AUDIT_DB_MAINTAINER_USER=audit_maintainer_login
AUDIT_DB_MAINTAINER_PASSWORD=$(openssl rand -base64 32)

# TLS certificate paths (absolute paths on this server)
TLS_CERT_PATH=/etc/ssl/dghs/fhir.dghs.gov.bd.crt
TLS_KEY_PATH=/etc/ssl/dghs/fhir.dghs.gov.bd.key

Security: Never commit .env to version control. Store the filled .env in your secrets vault (HashiCorp Vault, AWS SSM, or encrypted backup). Verify permissions after creation: ls -la .env should show -rw-------.

1.6 — Fix PostgreSQL init script password injection

The postgres/audit/init.sql file contains placeholder passwords. PostgreSQL's Docker entrypoint does not perform variable substitution in .sql init files — only .sh files. Replace the init SQL with a shell script:

# Create shell-based init script for audit database
cat > /opt/bd-fhir-national/postgres/audit/init.sh <<'INITSCRIPT'
#!/bin/bash
set -e

# Load passwords from environment variables
# (These env vars are set in docker-compose.yml from .env)
WRITER_USER="${AUDIT_DB_WRITER_USER:-audit_writer_login}"
WRITER_PASS="${AUDIT_DB_WRITER_PASSWORD}"
MAINTAINER_USER="${AUDIT_DB_MAINTAINER_USER:-audit_maintainer_login}"
MAINTAINER_PASS="${AUDIT_DB_MAINTAINER_PASSWORD}"

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    -- Create writer login user
    DO \$\$
    BEGIN
        IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${WRITER_USER}') THEN
            CREATE USER ${WRITER_USER}
                WITH NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT LOGIN
                CONNECTION LIMIT 20
                PASSWORD '${WRITER_PASS}';
        END IF;
    END
    \$\$;

    -- Create maintainer login user
    DO \$\$
    BEGIN
        IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${MAINTAINER_USER}') THEN
            CREATE USER ${MAINTAINER_USER}
                WITH NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT LOGIN
                CONNECTION LIMIT 5
                PASSWORD '${MAINTAINER_PASS}';
        END IF;
    END
    \$\$;

    GRANT CONNECT ON DATABASE ${POSTGRES_DB} TO ${WRITER_USER};
    GRANT CONNECT ON DATABASE ${POSTGRES_DB} TO ${MAINTAINER_USER};
EOSQL
INITSCRIPT

chmod +x /opt/bd-fhir-national/postgres/audit/init.sh

Update docker-compose.yml to mount init.sh instead of init.sql for the postgres-audit service:

# In postgres-audit volumes: section, change:
# - ./postgres/audit/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
# To:
- ./postgres/audit/init.sh:/docker-entrypoint-initdb.d/init.sh:ro

Also pass the audit user environment variables to postgres-audit:

# In postgres-audit environment: section, add:
AUDIT_DB_WRITER_USER:       ${AUDIT_DB_WRITER_USER}
AUDIT_DB_WRITER_PASSWORD:   ${AUDIT_DB_WRITER_PASSWORD}
AUDIT_DB_MAINTAINER_USER:   ${AUDIT_DB_MAINTAINER_USER}
AUDIT_DB_MAINTAINER_PASSWORD: ${AUDIT_DB_MAINTAINER_PASSWORD}

Similarly for postgres-fhir, create init.sh:

cat > /opt/bd-fhir-national/postgres/fhir/init.sh <<'INITSCRIPT'
#!/bin/bash
set -e

APP_USER="${FHIR_DB_APP_USER:-hapi_app}"
APP_PASS="${FHIR_DB_APP_PASSWORD}"

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    DO \$\$
    BEGIN
        IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = '${APP_USER}') THEN
            CREATE USER ${APP_USER}
                WITH NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT LOGIN
                CONNECTION LIMIT 30
                PASSWORD '${APP_PASS}';
        END IF;
    END
    \$\$;

    GRANT CONNECT ON DATABASE ${POSTGRES_DB} TO ${APP_USER};
    GRANT USAGE ON SCHEMA public TO ${APP_USER};
    ALTER DEFAULT PRIVILEGES IN SCHEMA public
        GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO ${APP_USER};
    ALTER DEFAULT PRIVILEGES IN SCHEMA public
        GRANT USAGE, SELECT ON SEQUENCES TO ${APP_USER};
EOSQL
INITSCRIPT

chmod +x /opt/bd-fhir-national/postgres/fhir/init.sh

1.7 — Authenticate with private Docker registry

# Log in to your private registry
docker login your-registry.dghs.gov.bd \
  --username ${REGISTRY_USER} \
  --password-stdin <<< "${REGISTRY_PASSWORD}"

# Verify the login persisted
cat ~/.docker/config.json | jq '.auths | keys'
# Should include "your-registry.dghs.gov.bd"

Part 2 — First deployment

2.1 — Pull images

cd /opt/bd-fhir-national

# Pull all images declared in docker-compose.yml
docker compose --env-file .env pull

# Verify images are present locally
docker images | grep -E "hapi|postgres|pgbouncer|nginx"

2.2 — Start infrastructure services first

Start databases before HAPI. HAPI's depends_on with condition: service_healthy handles this automatically, but starting manually in stages helps isolate any first-run issues.

# Start databases
docker compose --env-file .env up -d postgres-fhir postgres-audit

# Wait for health checks to pass (up to 60 seconds)
echo "Waiting for PostgreSQL to be ready..."
until docker compose --env-file .env ps postgres-fhir \
    | grep -q "healthy"; do
  sleep 3
  echo -n "."
done
echo ""
echo "postgres-fhir: healthy"

until docker compose --env-file .env ps postgres-audit \
    | grep -q "healthy"; do
  sleep 3
  echo -n "."
done
echo ""
echo "postgres-audit: healthy"

2.3 — Verify PostgreSQL user creation

# Verify FHIR app user was created
docker exec bd-postgres-fhir psql -U postgres -d fhirdb -c \
  "SELECT rolname, rolcanlogin FROM pg_roles WHERE rolname = 'hapi_app';"
# Expected: hapi_app | t

# Verify audit writer user was created
docker exec bd-postgres-audit psql -U postgres -d auditdb -c \
  "SELECT rolname, rolcanlogin FROM pg_roles WHERE rolname = 'audit_writer_login';"
# Expected: audit_writer_login | t

2.4 — Start pgBouncer

docker compose --env-file .env up -d pgbouncer-fhir pgbouncer-audit

# Verify pgBouncer is healthy
until docker compose --env-file .env ps pgbouncer-fhir \
    | grep -q "healthy"; do
  sleep 3
done
echo "pgbouncer-fhir: healthy"

2.5 — Start HAPI (first replica)

docker compose --env-file .env up -d hapi

# Follow startup logs — this takes 60-120 seconds on first run
# Watch for these key log events in order:
#   1. "Running FHIR Flyway migrations" — V1 schema creation
#   2. "Running Audit Flyway migrations" — V2 audit schema creation
#   3. "Advisory lock acquired" — IG package initialisation begins
#   4. "BD Core IG package loaded successfully" — IG loaded
#   5. "BdTerminologyValidationSupport initialised" — OCL integration ready
#   6. "KeycloakJwtInterceptor initialised" — JWT validation ready
#   7. "HAPI RestfulServer interceptors registered" — server ready
#   8. Spring Boot startup completion message with port 8080

docker compose --env-file .env logs -f hapi
# Press Ctrl+C when you see the startup completion message

Expected startup log sequence (key lines only):

INFO  o.f.core.internal.command.DbMigrate - Running FHIR Flyway migrations
INFO  o.f.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "public"
INFO  o.f.core.internal.command.DbMigrate - Running Audit Flyway migrations
INFO  o.f.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "audit"
INFO  b.g.d.f.init.IgPackageInitializer - Advisory lock acquired: lockKey=... waitedMs=...
INFO  b.g.d.f.init.IgPackageInitializer - BD Core IG package loaded successfully: version=0.2.1
INFO  b.g.d.f.t.BdTerminologyValidationSupport - BdTerminologyValidationSupport initialised
INFO  b.g.d.f.i.KeycloakJwtInterceptor - KeycloakJwtInterceptor initialised
INFO  b.g.d.f.c.SecurityConfig - HAPI RestfulServer interceptors registered
INFO  o.s.b.w.e.t.TomcatWebServer - Tomcat started on port(s): 8080
INFO  b.g.d.f.BdFhirApplication - Started BdFhirApplication in XX.XXX seconds

2.6 — Start nginx

docker compose --env-file .env up -d nginx

# Verify nginx started without config errors
docker compose --env-file .env logs nginx | tail -20
# Should NOT contain: [emerg] or [crit] — only [notice] lines

# Verify nginx health
docker compose --env-file .env ps nginx
# Status should be: Up (healthy)

2.7 — Verify full stack health

# Internal health check (bypasses nginx, hits HAPI directly)
docker exec $(docker compose --env-file .env ps -q hapi | head -1) \
  curl -s http://localhost:8080/actuator/health | jq .

# Expected output:
# {
#   "status": "UP",
#   "components": {
#     "db": { "status": "UP" },
#     "auditDb": { "status": "UP" },
#     "ocl": { "status": "UP" },
#     "livenessState": { "status": "UP" },
#     "readinessState": { "status": "UP" }
#   }
# }

# External health check (through nginx + TLS)
curl -s https://fhir.dghs.gov.bd/actuator/health/liveness | jq .
# Expected: { "status": "UP" }

# FHIR metadata endpoint (unauthenticated)
curl -s https://fhir.dghs.gov.bd/fhir/metadata | jq '{
  resourceType,
  fhirVersion,
  software: .software,
  implementation: .implementation
}'
# Expected:
# {
#   "resourceType": "CapabilityStatement",
#   "fhirVersion": "4.0.1",
#   "software": { "name": "BD FHIR National Repository", "version": "0.2.1" }
# }

Part 3 — Phase 2 acceptance tests

Run all seven tests before declaring the deployment production-ready. Each test includes the expected HTTP status, expected response body shape, and what to check in the audit log if the test fails.

Setup: obtain a vendor test token

VENDOR_TOKEN=$(curl -s -X POST \
  "https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=fhir-vendor-TEST-FAC-001" \
  -d "client_secret=${TEST_VENDOR_SECRET}" \
  | jq -r '.access_token')

echo "Token obtained: ${VENDOR_TOKEN:0:20}..."

Test 1 — Valid Condition with valid ICD-11 code → 201

Submits a BD Core IG-compliant bd-condition resource with a valid ICD-11 Diagnosis-class code.

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${VENDOR_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{
    "resourceType": "Condition",
    "meta": {
      "profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
    },
    "clinicalStatus": {
      "coding": [{
        "system": "http://terminology.hl7.org/CodeSystem/condition-clinical",
        "code": "active"
      }]
    },
    "verificationStatus": {
      "coding": [{
        "system": "http://terminology.hl7.org/CodeSystem/condition-ver-status",
        "code": "confirmed"
      }]
    },
    "code": {
      "coding": [{
        "system": "http://id.who.int/icd/release/11/mms",
        "code": "1C62.0",
        "display": "Typhoid fever"
      }]
    },
    "subject": {
      "reference": "Patient/test-patient-001"
    },
    "recordedDate": "2025-03-01"
  }'

Expected: HTTP 201 Created with Location header containing the new resource URL.

If 422 instead:

Check OCL connectivity: curl https://tr.ocl.dghs.gov.bd/api/fhir/CodeSystem/$validate-code?system=http://id.who.int/icd/release/11/mms&code=1C62.0
Check IG is loaded: curl http://localhost:8080/actuator/health — OCL component should be UP
Check HAPI logs for profile validation errors

Test 2 — Invalid ICD-11 code → 422

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${VENDOR_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{
    "resourceType": "Condition",
    "meta": {
      "profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
    },
    "clinicalStatus": {
      "coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-clinical", "code": "active"}]
    },
    "verificationStatus": {
      "coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status", "code": "confirmed"}]
    },
    "code": {
      "coding": [{
        "system": "http://id.who.int/icd/release/11/mms",
        "code": "INVALID-CODE-99999",
        "display": "This code does not exist"
      }]
    },
    "subject": {"reference": "Patient/test-patient-001"},
    "recordedDate": "2025-03-01"
  }'

Expected: HTTP 422 Unprocessable Entity with OperationOutcome containing:

issue[0].severity: error
issue[0].diagnostics: contains "INVALID-CODE-99999" and rejection reason
issue[0].expression: contains Condition.code

Verify in audit table:

SELECT rejection_code, rejection_reason, invalid_code, element_path
FROM audit.fhir_rejected_submissions
ORDER BY submission_time DESC LIMIT 1;
-- Expected: TERMINOLOGY_INVALID_CODE | OCL rejected code... | INVALID-CODE-99999 | Condition.code...

Test 3 — Device-class ICD-11 code in Condition.code → 422

Device-class codes are valid ICD-11 codes but are not in the bd-condition-icd11-diagnosis-valueset (restricted to Diagnosis + Finding).

# XA7RE2 is an example Device-class code in ICD-11 MMS
# Verify it is Device-class in OCL before running this test:
# curl "https://tr.ocl.dghs.gov.bd/api/fhir/CodeSystem/$lookup?system=http://id.who.int/icd/release/11/mms&code=XA7RE2"

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${VENDOR_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{
    "resourceType": "Condition",
    "meta": {
      "profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
    },
    "clinicalStatus": {
      "coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-clinical", "code": "active"}]
    },
    "verificationStatus": {
      "coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status", "code": "confirmed"}]
    },
    "code": {
      "coding": [{
        "system": "http://id.who.int/icd/release/11/mms",
        "code": "XA7RE2",
        "display": "Device code — should be rejected"
      }]
    },
    "subject": {"reference": "Patient/test-patient-001"},
    "recordedDate": "2025-03-01"
  }'

Expected: HTTP 422 with OperationOutcome. Rejection code in audit: TERMINOLOGY_INVALID_CLASS.

If 201 instead (code accepted):

OCL ValueSet class restriction is not enforcing correctly
Verify the ValueSet collection in OCL has correct concept_class filter
Run: python version_upgrade.py --verify-class-restriction

Test 4 — Profile violation (missing required field) → 422

Submits a Condition missing clinicalStatus which is required by bd-condition profile.

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${VENDOR_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{
    "resourceType": "Condition",
    "meta": {
      "profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
    },
    "code": {
      "coding": [{
        "system": "http://id.who.int/icd/release/11/mms",
        "code": "1C62.0"
      }]
    },
    "subject": {"reference": "Patient/test-patient-001"}
  }'

Expected: HTTP 422 with OperationOutcome referencing missing clinicalStatus.

If 201 instead:

BD Core IG is not loaded or profile is not enforcing clinicalStatus as required
Check startup logs for IG load success
Verify: curl http://localhost:8080/fhir/StructureDefinition/bd-condition

Test 5 — No Bearer token → 401

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Content-Type: application/fhir+json" \
  -d '{"resourceType": "Condition"}'

Expected: HTTP 401 with WWW-Authenticate header and OperationOutcome.

# Verify WWW-Authenticate header is present
curl -s -I \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Content-Type: application/fhir+json" \
  -d '{"resourceType":"Condition"}' \
  | grep -i "www-authenticate"
# Expected: WWW-Authenticate: Bearer realm="BD FHIR National Repository"...

Test 6 — Valid token but missing mci-api role → 401

Create a test client WITHOUT mci-api role in Keycloak for this test. Or use a token from a different realm.

# Token from a client without mci-api role
NO_ROLE_TOKEN=$(curl -s -X POST \
  "https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=fhir-test-no-role" \
  -d "client_secret=${TEST_NO_ROLE_SECRET}" \
  | jq -r '.access_token')

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${NO_ROLE_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{"resourceType": "Condition"}'

Expected: HTTP 401.

Verify in audit log:

SELECT event_type, outcome_detail, client_id
FROM audit.audit_events
WHERE event_type = 'AUTH_FAILURE'
ORDER BY event_time DESC LIMIT 1;
-- Expected: AUTH_FAILURE | Required role 'mci-api' not present... | fhir-test-no-role

Test 7 — Expired token → 401

# An expired token is one whose 'exp' claim is in the past.
# Easiest approach: obtain a token, wait for it to expire (default: 5 minutes),
# then use it.
#
# For automated testing, forge an expired token manually:
# (This requires knowing the signing key — use only in test environments)
#
# Alternative: Use a token from a deactivated Keycloak client
# (revoke the client's credentials, existing tokens become invalid)

# Or simply wait:
echo "Waiting 6 minutes for token to expire..."
EXPIRED_TOKEN=$(curl -s -X POST \
  "https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=fhir-vendor-TEST-FAC-001" \
  -d "client_secret=${TEST_VENDOR_SECRET}" \
  | jq -r '.access_token')

sleep 360  # wait for 5-minute Keycloak default expiry

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${EXPIRED_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{"resourceType": "Condition"}'

Expected: HTTP 401 — "Token has expired".

Test 8 — Cluster expression: raw postcoordinated code without extension → 422

curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X POST https://fhir.dghs.gov.bd/fhir/Condition \
  -H "Authorization: Bearer ${VENDOR_TOKEN}" \
  -H "Content-Type: application/fhir+json" \
  -d '{
    "resourceType": "Condition",
    "meta": {
      "profile": ["https://fhir.dghs.gov.bd/core/StructureDefinition/bd-condition"]
    },
    "clinicalStatus": {
      "coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-clinical", "code": "active"}]
    },
    "verificationStatus": {
      "coding": [{"system": "http://terminology.hl7.org/CodeSystem/condition-ver-status", "code": "confirmed"}]
    },
    "code": {
      "coding": [{
        "system": "http://id.who.int/icd/release/11/mms",
        "code": "1C62.0&has_severity=mild",
        "display": "Raw postcoordinated string — prohibited"
      }]
    },
    "subject": {"reference": "Patient/test-patient-001"},
    "recordedDate": "2025-03-01"
  }'

Expected: HTTP 422 with OperationOutcome diagnosing: "ICD-11 postcoordinated expression in Condition.code.coding[0] must use the icd11-cluster-expression extension"

Rejection code in audit: CLUSTER_STEM_MISSING_EXTENSION.

Test 9 — Cache flush endpoint requires fhir-admin role

# Attempt with vendor token (mci-api only) — should be 403
curl -s -w "\n--- HTTP %{http_code} ---\n" \
  -X DELETE https://fhir.dghs.gov.bd/admin/terminology/cache \
  -H "Authorization: Bearer ${VENDOR_TOKEN}"
# Expected: 403 (blocked by nginx IP restriction OR TerminologyCacheManager role check)

# Attempt with fhir-admin token — should be 200
ADMIN_TOKEN=$(curl -s -X POST \
  "https://auth.dghs.gov.bd/realms/hris/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=fhir-admin-pipeline" \
  -d "client_secret=${FHIR_ADMIN_CLIENT_SECRET}" \
  | jq -r '.access_token')

# Note: /admin/ is restricted to 172.20.0.0/16 in nginx.
# Run this from within the Docker network or from the server itself:
docker exec $(docker compose --env-file .env ps -q hapi | head -1) \
  curl -s -X DELETE \
    -H "Authorization: Bearer ${ADMIN_TOKEN}" \
    http://localhost:8080/admin/terminology/cache | jq .
# Expected: 200 with { "status": "flushed", "entriesEvicted": N, ... }

Part 4 — Subsequent deployments (image upgrade)

When a new Docker image is built and pushed (new IG version, code changes):

cd /opt/bd-fhir-national

# 1. Update image tag in .env
nano .env
# Change: HAPI_IMAGE=your-registry.dghs.gov.bd/bd-fhir-hapi:1.0.0
# To:     HAPI_IMAGE=your-registry.dghs.gov.bd/bd-fhir-hapi:1.1.0

# 2. Pull new image
docker compose --env-file .env pull hapi

# 3. Rolling restart — replaces containers one at a time
# At 1 replica (pilot): brief downtime expected (~30s)
docker compose --env-file .env up -d --no-deps hapi

# At 3 replicas (Phase 2): true rolling update — scale up then scale down
docker compose --env-file .env up -d --no-deps --scale hapi=4 hapi
# Wait for new replica to be healthy:
sleep 30
docker compose --env-file .env up -d --no-deps --scale hapi=3 hapi

# 4. Verify startup
docker compose --env-file .env logs --tail=50 hapi

# 5. Run acceptance tests (at minimum Tests 1, 2, 5)

Part 5 — Operational runbook

View logs

# All services
docker compose --env-file .env logs -f

# HAPI only (structured JSON — pipe through jq)
docker compose --env-file .env logs -f hapi | jq -R 'try fromjson'

# nginx access log
docker compose --env-file .env logs -f nginx

# Filter for rejected submissions in HAPI logs
docker compose --env-file .env logs hapi | \
  jq -R 'try fromjson | select(.message | contains("rejected"))'

Restart a specific service

docker compose --env-file .env restart hapi
docker compose --env-file .env restart nginx

Emergency: full stack restart

docker compose --env-file .env down
docker compose --env-file .env up -d

Query rejected submissions

docker exec bd-postgres-audit psql -U postgres -d auditdb -c "
SELECT
    submission_time,
    resource_type,
    rejection_code,
    LEFT(rejection_reason, 100) as reason,
    client_id
FROM audit.fhir_rejected_submissions
ORDER BY submission_time DESC
LIMIT 20;"

Check pgBouncer pool status

# Connect to pgBouncer admin interface
docker exec -it bd-pgbouncer-fhir \
  psql -h localhost -p 5432 -U pgbouncer pgbouncer -c "SHOW POOLS;"

Monitor disk usage

# PostgreSQL data volumes
docker system df -v | grep -E "postgres|audit"

# Log volume
docker system df -v | grep hapi-logs

27 KiB Raw Blame History