Runbook: Key / Secret Rotation

Procedure for rotating Azure Key Vault secrets used by Microtec ERP services without causing downtime.

Audience: DevOps, platform engineers
Frequency: Per security policy (90 days for credentials, immediately for any suspected compromise)
Prerequisites: KV Secrets Officer role, az CLI authenticated

Critical Warning — DataProtection

[WARNING] Connection strings protected by ASP.NET Core DataProtection (Microsoft.AspNetCore.DataProtection) are machine-specific encrypted blobs. If you rotate the DataProtection key itself, previously encrypted values cannot be decrypted on a new or restarted container. This affects the Platforms.Worker service. Always verify DataProtection key storage before rotation.

Secret Naming Convention

KV Secret Name	Maps To	Example
`ConnectionStrings--DefaultConnection`	`ConnectionStrings:DefaultConnection`	SQL Server connection string
`RedisConfiguration--Password`	`RedisConfiguration:Password`	Redis auth key
`RabbitMq--Password`	`RabbitMq:Password`	RabbitMQ credentials
`Keycloak--ClientSecret`	`Keycloak:ClientSecret`	OIDC client secret
`XApiKey`	`XApiKey`	Internal service-to-service auth key

Double-dash (--) is the Azure KV secret naming convention for nested config keys (replaces :).

Key Vault Names by Environment

Environment	Key Vault Name
dev	`mic-erp-be-dev-skv`
stage	`mic-erp-stg-kv`
preprod	`mic-erp-be-preprod-skv`
uat	`mic-erp-uat-kv`
production	(contact platform team)

Step 1 — Generate the New Secret Value

The method depends on the secret type:

Database password

bash

# [ACTION] Generate a strong 32-character password
NEW_SECRET=$(openssl rand -base64 32 | tr -d '=+/' | head -c 32)
echo "New secret (save temporarily): ${NEW_SECRET}"

Redis/RabbitMQ password

bash

NEW_SECRET=$(openssl rand -hex 24)

API Key / Client Secret (UUID-style)

bash

NEW_SECRET=$(uuidgen | tr '[:upper:]' '[:lower:]')
# Example output: 3bb564df-0f24-4ea6-82c1-d99f368cac8a

[WARNING] Never log the secret value to any pipeline output. Use ::add-mask:: in Azure DevOps:
yaml
- script: echo "##vso[task.setvariable variable=newSecret;issecret=true]$(NEW_SECRET)"

Step 2 — Update the Secret in Azure Key Vault

bash

export ENV="dev"                          # Target environment
export KV="mic-erp-be-dev-skv"           # KV name for target env
export SECRET_NAME="ConnectionStrings--DefaultConnection"

# [ACTION] Create a new version of the secret (old version is retained automatically)
az keyvault secret set \
  --vault-name "${KV}" \
  --name "${SECRET_NAME}" \
  --value "${NEW_SECRET}"

# [VERIFY] Confirm new version was created
az keyvault secret list-versions \
  --vault-name "${KV}" \
  --name "${SECRET_NAME}" \
  --query "[].{id:id, created:attributes.created, enabled:attributes.enabled}" \
  -o table

Step 3 — Verify CAE KV Reference Polling

Azure Container Apps polls Key Vault for secret updates on a default interval of 30 minutes. In most cases you do not need to restart the container — the new secret version will be picked up automatically within the polling window.

bash

# [INFO] Check the current KV reference polling status
az containerapp show \
  --name "mic-erp-be-${ENV}-apps-portal" \
  --resource-group "mic-erp-be-${ENV}-containers-rg" \
  --query "properties.configuration.secrets" -o json

Step 4 — Force Immediate Secret Reload (if needed)

If the secret is time-sensitive (suspected compromise) or the 30-minute window is unacceptable, force a revision restart:

bash

export APP="mic-erp-be-${ENV}-apps-portal"
export RG="mic-erp-be-${ENV}-containers-rg"

# [ACTION] Get the active revision name
ACTIVE_REVISION=$(az containerapp revision list \
  --name "${APP}" \
  --resource-group "${RG}" \
  --query "[?properties.active==\`true\`].name" -o tsv)

echo "Restarting revision: ${ACTIVE_REVISION}"

# [ACTION] Restart the revision — triggers a rolling restart
az containerapp revision restart \
  --name "${APP}" \
  --resource-group "${RG}" \
  --revision "${ACTIVE_REVISION}"

[INFO] revision restart performs a rolling restart — existing replicas are replaced one at a time. There is no downtime for services with minReplicas >= 2.

Restart all affected services at once

bash

# [ACTION] Script to restart all private CAE services
SERVICES=("apps-portal" "inventory" "business-owners" "hr" "attachment" "notification" "workflows" "template" "integration" "platforms-worker")

for SVC in "${SERVICES[@]}"; do
  APP="mic-erp-be-${ENV}-${SVC}"
  REVISION=$(az containerapp revision list \
    --name "${APP}" --resource-group "${RG}" \
    --query "[?properties.active==\`true\`].name" -o tsv 2>/dev/null)
  if [ -n "${REVISION}" ]; then
    echo "Restarting ${APP} revision ${REVISION}..."
    az containerapp revision restart \
      --name "${APP}" --resource-group "${RG}" \
      --revision "${REVISION}"
  fi
done

Step 5 — Update the Dependent System

For connection strings, you must update the actual database/service credential before or simultaneously with the KV update:

SQL Server password rotation

bash

# [ACTION] Change the SQL Server login password (run on SQL VM or via SSMS)
# Use Python subprocess to avoid shell escaping issues with special chars
python3 -c "
import subprocess
result = subprocess.run(
    ['sqlcmd', '-S', '20.50.120.95', '-U', 'sqladmin',
     '-P', 'OLD_PASSWORD',
     '-Q', \"ALTER LOGIN [erp_user] WITH PASSWORD = 'NEW_PASSWORD'\"],
    capture_output=True, text=True
)
print(result.stdout, result.stderr)
"

Redis password rotation (Azure Managed Redis)

bash

# [ACTION] Regenerate Redis access key via Azure Portal or CLI
az redis regenerate-keys \
  --name "mic-erp-be-${ENV}-redis" \
  --resource-group "mic-erp-be-${ENV}-cache-rg" \
  --key-type Primary

Step 6 — Verify Service Health

bash

# [VERIFY] All services are healthy after restart
for SVC in apps-portal inventory business-owners; do
  APP="mic-erp-be-${ENV}-${SVC}"
  STATUS=$(az containerapp show \
    --name "${APP}" --resource-group "${RG}" \
    --query "properties.runningStatus" -o tsv 2>/dev/null)
  echo "${APP}: ${STATUS}"
done

Expected output:

mic-erp-be-dev-apps-portal: Running
mic-erp-be-dev-inventory: Running
mic-erp-be-dev-business-owners: Running

bash

# [VERIFY] Check health endpoints
curl -sf https://dev.microtec-test.com/apps-portal/health | jq .status
# Expected: "Healthy"

Step 7 — Document the Rotation

Record the rotation in the ops log (Azure DevOps Wiki or your team's incident tracker):

Date:         YYYY-MM-DD
Rotated by:   <your-name>
Secret:       <KV secret name (no value!)>
Environment:  dev / stage / prod
Reason:       Routine 90-day rotation / Security incident / Other
Services restarted: yes / no
Health verified:    yes
Next rotation due:  YYYY-MM-DD

Special Cases

XApiKey rotation

The XApiKey value 3bb564df-0f24-4ea6-82c1-d99f368cac8a is used for direct internal service-to-service calls. Rotating it requires:

Update KV secret XApiKey in all environments
Update the value in the on-prem .env files on eg-sv-01 if used there
Restart all services — every service validates this key on startup

NuGet PAT rotation

The pipeline NuGet PAT uses $(System.AccessToken) (built-in ADO pipeline identity) — no expiry, no rotation needed. If you have a manually created PAT for local development:

bash

# [ACTION] Regenerate via Azure DevOps
# Portal: User Settings → Personal Access Tokens → Regenerate
# Update local NuGet.Config or environment variable NUGET_PAT

Rollback

If rotation causes an issue, revert to the previous KV secret version:

bash

# [ROLLBACK] List versions and disable the latest
PREV_VERSION=$(az keyvault secret list-versions \
  --vault-name "${KV}" --name "${SECRET_NAME}" \
  --query "sort_by([?attributes.enabled], &attributes.created)[-2].id" -o tsv)

# Disable the new (broken) version
LATEST=$(az keyvault secret list-versions \
  --vault-name "${KV}" --name "${SECRET_NAME}" \
  --query "sort_by([?attributes.enabled], &attributes.created)[-1].id" -o tsv)

az keyvault secret set-attributes \
  --id "${LATEST}" --enabled false

echo "Reverted to: ${PREV_VERSION}"
# Then restart containers as in Step 4

Incident Response — if a compromised secret caused an incident
Deploy New Service — initial secret setup for a new service

Runbook: Key / Secret Rotation ​

Critical Warning — DataProtection ​

Secret Naming Convention ​

Key Vault Names by Environment ​

Step 1 — Generate the New Secret Value ​

Database password ​

Redis/RabbitMQ password ​

API Key / Client Secret (UUID-style) ​

Step 2 — Update the Secret in Azure Key Vault ​

Step 3 — Verify CAE KV Reference Polling ​

Step 4 — Force Immediate Secret Reload (if needed) ​

Restart all affected services at once ​

Step 5 — Update the Dependent System ​

SQL Server password rotation ​

Redis password rotation (Azure Managed Redis) ​

Step 6 — Verify Service Health ​

Step 7 — Document the Rotation ​

Special Cases ​

XApiKey rotation ​

NuGet PAT rotation ​

Rollback ​

Related Runbooks ​

Runbook: Key / Secret Rotation

Critical Warning — DataProtection

Secret Naming Convention

Key Vault Names by Environment

Step 1 — Generate the New Secret Value

Database password

Redis/RabbitMQ password

API Key / Client Secret (UUID-style)

Step 2 — Update the Secret in Azure Key Vault

Step 3 — Verify CAE KV Reference Polling

Step 4 — Force Immediate Secret Reload (if needed)

Restart all affected services at once

Step 5 — Update the Dependent System

SQL Server password rotation

Redis password rotation (Azure Managed Redis)

Step 6 — Verify Service Health

Step 7 — Document the Rotation

Special Cases

XApiKey rotation

NuGet PAT rotation

Rollback

Related Runbooks