Appearance
Runbooks
Operational procedures for the Microtec ERP platform. These runbooks are the authoritative reference for routine operations, incident management, and recovery procedures.
Audience: Platform engineers, DevOps, on-call engineers
Scope: Azure Container Apps–based production platform + on-prem environments
Available Runbooks
| # | Runbook | Description |
|---|---|---|
| 1 | Deploy New Service | End-to-end guide for adding a new microservice from .NET project creation through first production deployment |
| 2 | Scale a Service | Configure KEDA autoscaling rules (CPU, memory, HTTP, cron) and apply them without downtime |
| 3 | Key Rotation | Safely rotate Azure Key Vault secrets and force CAE pods to pick up the new values |
| 4 | Incident Response | P0/P1 incident playbook — assess, contain, rollback, and write a post-mortem |
| 5 | Keycloak Realm Recovery | Recover a broken Keycloak realm, restore flow priorities, and re-enable clients |
| 6 | Database Maintenance | SQL Server backup, restore, index rebuild, and statistics update procedures |
| 7 | Certificate Renewal | Renew TLS certificates for Azure Front Door custom domains and on-prem NGINX |
| 8 | Disaster Recovery | Full-environment recovery from a region outage or critical data loss event |
| 9 | On-Prem Service Management | Manage Docker Compose services on eg-sv-01, eg-sv-vip, and build agents |
Runbook Conventions
Severity Labels
Runbook steps are labelled with one of:
| Label | Meaning |
|---|---|
[INFO] | Informational — no action required |
[ACTION] | Required step — execute exactly as written |
[VERIFY] | Verification step — confirm before proceeding |
[WARNING] | Risk of data loss or outage if skipped |
[ROLLBACK] | Execute only when rolling back a failed operation |
Environment Shortcodes
| Shortcode | Environment | Domain |
|---|---|---|
dev | Development | microtec-test.com |
stg | Staging | microtecstage.com |
preprod | Pre-production | — |
uat | User Acceptance Testing | microtec-uat.com |
prod | Production | onlinemicrotec.com.sa |
Common Variables
Most runbooks reference these shell variables. Export them before running any commands:
bash
export ENV="dev" # Target environment
export SVC="my-service" # Service short name
export RG="mic-erp-be-${ENV}-apps-private-rg"
export CAE_NAME="mic-erp-be-${ENV}-cae-private"
export ACR="micerpbe${ENV}acr"On-Call Reference
| Resource | URL / Contact |
|---|---|
| Azure Portal | https://portal.azure.com |
| Azure DevOps | https://dev.azure.com/microtec |
| App Insights (dev) | Azure Portal → mic-erp-be-dev-appinsights |
| Seq (dev) | http://localhost:1234 (via VPN) |
| RabbitMQ UI | http://localhost:15672 (via VPN) |
| SonarQube | http://192.168.120.254:8095 |
| NPM Proxy | http://192.168.120.12:8100 |
VPN required for all on-prem URLs. Contact your team lead for VPN credentials.
Maintenance Windows
Standard maintenance windows (all times UTC+3 / Riyadh):
| Type | Schedule | Duration |
|---|---|---|
| Routine deployments | Sun–Thu, 22:00–02:00 | Up to 4 h |
| Database maintenance | Friday, 01:00–05:00 | Up to 4 h |
| Emergency patches | Any time — notify stakeholders first | Minimum required |