Runbook: Keycloak Realm Recovery

Restore a broken or misconfigured Keycloak realm on the Microtec ERP platform.

Audience: Platform engineers, DevOps
Prerequisites: Keycloak admin credentials, az CLI authenticated
Container: mic-erp-be-{env}-keycloak in the Public CAE

Overview

Microtec ERP uses Keycloak with custom SPIs (Service Provider Interfaces) called keycloak-seeding. These SPIs run automatically on every Keycloak startup and are responsible for:

Creating realms (microtec for ERP, businessowner for BO portal)
Configuring OIDC clients per service
Setting authentication flow priorities
Enabling/disabling specific clients

Because seeding runs on startup, most realm recovery scenarios are solved by restarting the Keycloak container.

Realm Structure

Realm	Audience	Key Clients
`microtec`	ERP users (all modules)	`apps-portal`, `inventory`, `hr`, `accounting`, `gateway`
`businessowner`	Tenant admin portal	`business-owners-app`, `business-owners-api`

Scenario 1: Realm is Missing or Corrupted

Symptoms

Login page returns "Realm not found"
Keycloak admin shows the realm is absent or incomplete
Services return 401 with "Invalid realm" in logs

Resolution — Restart Keycloak (seeding auto-recovers)

bash

export ENV="dev"   # or stage, preprod, uat, prod
export RG="mic-erp-be-${ENV}-containers-rg"
export KC_APP="mic-erp-be-${ENV}-keycloak"

# [ACTION] Get the active revision
REVISION=$(az containerapp revision list \
  --name "${KC_APP}" --resource-group "${RG}" \
  --query "[?properties.active==\`true\`].name" -o tsv)

# [ACTION] Restart — seeding SPI runs on startup and recreates the realm
az containerapp revision restart \
  --name "${KC_APP}" \
  --resource-group "${RG}" \
  --revision "${REVISION}"

echo "Keycloak restarting... allow 60-90 seconds for full startup + seeding"

bash

# [VERIFY] Confirm realm exists after restart
sleep 90
curl -sf "https://{domain}/auth/realms/microtec" | jq .realm
# Expected: "microtec"

curl -sf "https://{domain}/auth/realms/businessowner" | jq .realm
# Expected: "businessowner"

Scenario 2: Authentication Flow Priorities Broken

Symptoms

Users can log in with password but MFA or IdP-linked login fails
Cookie-based session not honoured
MAC (Microtec Auth Cookie) flow not triggered

Background

Keycloak authentication flows use a priority order. The Microtec platform requires:

Flow	Priority	Purpose
MAC (Microtec Auth Cookie)	10	Honour existing Microtec session cookie
Cookie	20	Standard Keycloak cookie-based SSO
IdP Redirect	30	Identity Provider (external) login
Forms	40+	Username/password form

[WARNING] If priorities are wrong (e.g., IdP=10 when it should be 30), IdP login will intercept all requests and password login may become unreachable.

Fix — Restart triggers priority restoration

The seeding SPI's ConfigureRealmClients step restores priorities on startup. Restart Keycloak as described in Scenario 1.

Manual fix via Keycloak Admin API (if restart is not possible)

bash

export KC_URL="https://{domain}/auth"
export REALM="microtec"

# [ACTION] Get admin token
ADMIN_TOKEN=$(curl -s -X POST \
  "${KC_URL}/realms/master/protocol/openid-connect/token" \
  -d "client_id=admin-cli" \
  -d "username=admin" \
  -d "password=${KC_ADMIN_PASS}" \
  -d "grant_type=password" | jq -r .access_token)

# [ACTION] List authentication flows
curl -s -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  "${KC_URL}/admin/realms/${REALM}/authentication/flows" | jq '.[] | {alias, id}'

# [ACTION] Update MAC flow execution priority to 10
# (Replace FLOW_ID and EXECUTION_ID with values from the list above)
curl -s -X PUT \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  "${KC_URL}/admin/realms/${REALM}/authentication/flows/${FLOW_ID}/executions" \
  -d '{"id":"EXECUTION_ID","priority":10}'

Scenario 3: Client Disabled or Missing

Symptoms

Specific service returns 401 with "Client not found" or "Client disabled"
Keycloak admin shows the client as disabled

Background

The seeding SPI's ConfigureRealmClients step:

Disables account-console (not needed in ERP)
Ensures account client is enabled (required for token introspection)
Enables all ERP service clients

SetClientEnabledAsync is called per client during seeding. Again, restart recovers this automatically.

Manual fix via Admin API

bash

# [ACTION] Get client ID (internal UUID)
CLIENT_UUID=$(curl -s -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  "${KC_URL}/admin/realms/${REALM}/clients?clientId=apps-portal" | jq -r '.[0].id')

# [ACTION] Enable the client
curl -s -X PUT \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  "${KC_URL}/admin/realms/${REALM}/clients/${CLIENT_UUID}" \
  -d "{\"enabled\": true}"

echo "Client apps-portal re-enabled"

Scenario 4: Backchannel Logout Broken

Symptoms

User logout from one app does not propagate to other apps (SSO session persists)
Logs show "Backchannel logout URL unreachable"

Background

Backchannel logout is already configured in the seeding SPI (ConfigureBackchannelLogout). The logout URL for each service is set to {service-internal-url}/auth/logout.

Verify the internal FQDN is reachable from the Keycloak container:

bash

# [ACTION] Execute inside Keycloak container
az containerapp exec \
  --name "${KC_APP}" \
  --resource-group "${RG}" \
  --command "curl -s http://mic-erp-be-dev-apps-portal/health"
# Expected: {"status":"Healthy"}

If unreachable, the Private CAE mTLS or DNS resolution is misconfigured — escalate to the platform team.

Scenario 5: Full Realm Data Loss

[WARNING] This is a destructive recovery. All realm customisations not covered by the seeding SPI will be lost.

bash

# [ACTION] Delete the broken realm
curl -s -X DELETE \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  "${KC_URL}/admin/realms/${REALM}"

# [ACTION] Restart Keycloak — seeding recreates the realm from scratch
az containerapp revision restart \
  --name "${KC_APP}" --resource-group "${RG}" \
  --revision "${REVISION}"

After restart, verify both realms are recreated and all clients are present.

Keycloak Admin Console Access

Environment	Admin Console URL
dev	https://dev.microtec-test.com/auth/admin
stage	https://stage.microtecstage.com/auth/admin
prod	https://onlinemicrotec.com.sa/auth/admin

[INFO] Admin credentials are stored in Key Vault under Keycloak--AdminPassword.

bash

# [ACTION] Retrieve admin password from KV
az keyvault secret show \
  --vault-name "mic-erp-be-dev-skv" \
  --name "Keycloak--AdminPassword" \
  --query "value" -o tsv

Startup Sequence Reference

When Keycloak starts, the seeding SPI executes in this order:

1. CreateRealm              — Create realm if not exists
2. ConfigureRealmSettings   — Token lifetime, brute force protection
3. CreateClients            — Register all OIDC clients
4. SetClientEnabledAsync    — Enable/disable specific clients
5. ConfigureRealmClients    — Set flow priorities, disable account-console
6. ConfigureAuthFlows       — Create/update MAC / Cookie / IdP flows
7. ConfigureBackchannelLogout — Set logout URLs
8. CreateDefaultUsers       — Seed test users (non-prod only)

Monitoring Keycloak

bash

# [INFO] Follow Keycloak startup logs
az containerapp logs show \
  --name "${KC_APP}" --resource-group "${RG}" \
  --follow --tail 200

# Look for these log lines to confirm successful seeding:
# INFO  [...keycloak-seeding] Realm 'microtec' seeding completed
# INFO  [...keycloak-seeding] Realm 'businessowner' seeding completed

Incident Response — if Keycloak issues caused a production incident
Key Rotation — rotate Keycloak client secrets

Runbook: Keycloak Realm Recovery ​

Overview ​

Realm Structure ​

Scenario 1: Realm is Missing or Corrupted ​

Symptoms ​

Resolution — Restart Keycloak (seeding auto-recovers) ​

Scenario 2: Authentication Flow Priorities Broken ​

Symptoms ​

Background ​

Fix — Restart triggers priority restoration ​

Manual fix via Keycloak Admin API (if restart is not possible) ​

Scenario 3: Client Disabled or Missing ​

Symptoms ​

Background ​

Manual fix via Admin API ​

Scenario 4: Backchannel Logout Broken ​

Symptoms ​

Background ​

Scenario 5: Full Realm Data Loss ​

Keycloak Admin Console Access ​

Startup Sequence Reference ​

Monitoring Keycloak ​

Related Runbooks ​

Runbook: Keycloak Realm Recovery

Overview

Realm Structure

Scenario 1: Realm is Missing or Corrupted

Symptoms

Resolution — Restart Keycloak (seeding auto-recovers)

Scenario 2: Authentication Flow Priorities Broken

Symptoms

Background

Fix — Restart triggers priority restoration

Manual fix via Keycloak Admin API (if restart is not possible)

Scenario 3: Client Disabled or Missing

Symptoms

Background

Manual fix via Admin API

Scenario 4: Backchannel Logout Broken

Symptoms

Background

Scenario 5: Full Realm Data Loss

Keycloak Admin Console Access

Startup Sequence Reference

Monitoring Keycloak

Related Runbooks