OpenTelemetry

OpenTelemetry provides distributed tracing and metrics collection for all Microtec ERP backend services. It instruments HTTP requests, database queries, message bus operations, and custom business metrics using the OTLP (OpenTelemetry Protocol) exporter.

---

OTLP Endpoints

Protocol	Port	Format	Used By
gRPC	4317	Protobuf	.NET services (default)
HTTP/Protobuf	4318	Protobuf over HTTP	Services behind HTTP-only proxies

Local OpenTelemetry Collector receives on both ports and routes signals to Jaeger (traces), Prometheus (metrics), and Application Insights (cloud).

---

Configuration via appsettings.json

{
  "OpenTelemetry": {
    "EnableTracing": true,
    "EnableMetrics": true,
    "ServiceName": "accounting-api",
    "ServiceVersion": "1.0.0",
    "OtlpEndpoint": "http://localhost:4317",
    "OtlpProtocol": "grpc",
    "SamplingRatio": 1.0
  }
}

For production, SamplingRatio is set to 0.1 (10% sampling) to manage volume and cost.

---

Service Registration (Program.cs)

// In Shared.Web — OpenTelemetryExtensions.cs
public static IServiceCollection AddMicrotecOpenTelemetry(
    this IServiceCollection services,
    IConfiguration configuration)
{
    var settings = configuration.GetSection("OpenTelemetry").Get<OpenTelemetrySettings>();

    services.AddOpenTelemetry()
        .ConfigureResource(resource => resource
            .AddService(
                serviceName: settings.ServiceName,
                serviceVersion: settings.ServiceVersion)
            .AddAttributes(new Dictionary<string, object>
            {
                ["deployment.environment"] = configuration["ASPNETCORE_ENVIRONMENT"] ?? "unknown",
                ["service.instance.id"] = Environment.MachineName
            }))
        .WithTracing(tracing =>
        {
            if (!settings.EnableTracing) return;

            tracing
                .AddAspNetCoreInstrumentation(opts =>
                {
                    opts.RecordException = true;
                    opts.Filter = ctx =>
                        !ctx.Request.Path.StartsWithSegments("/health") &&
                        !ctx.Request.Path.StartsWithSegments("/metrics");
                })
                .AddHttpClientInstrumentation(opts =>
                {
                    opts.RecordException = true;
                })
                .AddEntityFrameworkCoreInstrumentation(opts =>
                {
                    // Only include SQL text in non-production environments
                    opts.SetDbStatementForText =
                        configuration["ASPNETCORE_ENVIRONMENT"] != "production";
                })
                .AddRedisInstrumentation()
                .AddAzureServiceBusInstrumentation()
                .SetSampler(new TraceIdRatioBasedSampler(settings.SamplingRatio))
                .AddOtlpExporter(opts =>
                {
                    opts.Endpoint = new Uri(settings.OtlpEndpoint);
                    opts.Protocol = settings.OtlpProtocol == "grpc"
                        ? OtlpExportProtocol.Grpc
                        : OtlpExportProtocol.HttpProtobuf;
                });

            // Cloud: also export to Application Insights
            if (IsCloudEnvironment(configuration))
            {
                tracing.AddAzureMonitorTraceExporter(opts =>
                {
                    opts.ConnectionString = configuration["ApplicationInsights:ConnectionString"];
                });
            }
        })
        .WithMetrics(metrics =>
        {
            if (!settings.EnableMetrics) return;

            metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddProcessInstrumentation()
                .AddOtlpExporter(opts =>
                {
                    opts.Endpoint = new Uri(settings.OtlpEndpoint);
                });

            if (IsCloudEnvironment(configuration))
            {
                metrics.AddAzureMonitorMetricExporter(opts =>
                {
                    opts.ConnectionString = configuration["ApplicationInsights:ConnectionString"];
                });
            }
        });

    return services;
}

---

Traces

What Is Traced

Instrumentation	Activity Name	Key Attributes
ASP.NET Core	Microsoft.AspNetCore.Hosting.HttpRequestIn	http.method, http.url, http.status_code
HttpClient	System.Net.Http.HttpRequestOut	http.url, http.method, http.status_code
EF Core	Microsoft.EntityFrameworkCore.Database.Command	db.statement (dev only), db.name
Redis	StackExchange.Redis	db.statement, net.peer.name
Service Bus	Azure.Messaging.ServiceBus	messaging.destination, messaging.operation

Trace Propagation Headers

All services read and write W3C Trace Context headers:

traceparent: 00-{traceId}-{spanId}-{flags}
tracestate:  (vendor-specific, may be empty)

Gateway.API generates the root traceparent for incoming external requests. Downstream services propagate it via HttpClient automatically (handled by AddHttpClientInstrumentation).

Viewing Traces Locally

Open Jaeger at http://localhost:16686:

1. Select a service from the dropdown (e.g., accounting-api)
2. Set time range and click Find Traces
3. Click any trace to see the full span waterfall

---

Metrics

Default Metrics (Auto-Instrumented)

Metric Name	Type	Description
http.server.request.duration	Histogram	Latency of handled HTTP requests (ms)
http.server.active_requests	UpDownCounter	In-flight request count
http.client.request.duration	Histogram	Latency of outbound HTTP calls
db.client.operation.duration	Histogram	Database query latency
process.cpu.time	Counter	Total CPU time used
process.memory.usage	ObservableGauge	Resident set size
dotnet.gc.collections	Counter	GC collection count by generation

Custom Business Metrics

Custom metrics are registered via IMeterFactory:

// In a service or handler
public class TenantMetrics
{
    private readonly Counter<long> _requestCounter;
    private readonly Histogram<double> _processingDuration;

    public TenantMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("Microtec.Accounting");

        _requestCounter = meter.CreateCounter<long>(
            "microtec.accounting.requests",
            unit: "{request}",
            description: "Total accounting requests processed");

        _processingDuration = meter.CreateHistogram<double>(
            "microtec.accounting.processing_duration",
            unit: "ms",
            description: "Time to process accounting transactions");
    }

    public void RecordRequest(string tenantId, string operation)
    {
        _requestCounter.Add(1,
            new KeyValuePair<string, object?>("tenant.id", tenantId),
            new KeyValuePair<string, object?>("operation", operation));
    }
}

Viewing Metrics Locally

• Prometheus: http://localhost:9090 — query using PromQL
• Grafana: http://localhost:3000 — pre-built dashboards for all services

Example PromQL queries:

# Request rate per service
rate(http_server_request_duration_seconds_count[5m])

# P95 latency
histogram_quantile(0.95, rate(http_server_request_duration_seconds_bucket[5m]))

# Error rate
rate(http_server_request_duration_seconds_count{http_status_code=~"5.."}[5m])
  /
rate(http_server_request_duration_seconds_count[5m])

---

OpenTelemetry Collector Configuration

The OTEL Collector in dev/ routes signals to local backends:

# dev/otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: microtec

  logging:
    loglevel: warn

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, logging]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, logging]

---

Production Configuration

In cloud environments, the OTLP Collector is not deployed. Services export directly to Azure Monitor:

{
  "OpenTelemetry": {
    "EnableTracing": true,
    "EnableMetrics": true,
    "OtlpEndpoint": "",
    "SamplingRatio": 0.1
  },
  "ApplicationInsights": {
    "ConnectionString": "keyvaultref:https://mic-erp-be-production-kv.vault.azure.net/secrets/ApplicationInsights--ConnectionString"
  }
}

The IsCloudEnvironment() check in the registration code enables the Azure Monitor exporter when ASPNETCORE_ENVIRONMENT is not Development.

---

Disabling Specific Instrumentation

To reduce noise or avoid sensitive data in traces, individual instrumentations can be disabled:

{
  "OpenTelemetry": {
    "EnableTracing": true,
    "DisableInstrumentation": ["EntityFrameworkCore", "Redis"]
  }
}

Note: Disabling EF Core instrumentation in production is recommended to avoid logging SQL queries that may contain sensitive data, even without the db.statement attribute.

---

Related Pages

• Observability Overview
• Seq Logging
• Local Dev Services (Docker Compose)