Import Package

Section: 16 — Packages
Last Updated: 2026-05-30
Scope: Microtec.Import — bulk data import via Excel and CSV

---

Overview

The Microtec.Import package provides a structured pipeline for importing large datasets from Excel (.xlsx) and CSV files into any ERP module. It handles validation, error reporting, and background processing so that import operations never block the HTTP request thread.

Microtec.Import.Domain          ← Domain models (IImportable, ImportJob)
Microtec.Import.Infrastructure  ← Excel/CSV engines, validation pipeline
Microtec.Import.Integration     ← Integration events, IImportPublicApi HTTP client

---

Package Structure

Microtec.Import.Domain

What's Included
IImportable — marker interface for entities that support bulk import
ImportJob — aggregate root tracking the lifecycle of one import operation
ImportRow — a single row from the uploaded file
ImportValidationResult — per-row validation outcome
ImportStatus enum: Pending, Processing, Completed, Failed, PartialSuccess

Microtec.Import.Infrastructure

Category	What's Included
Main service	IImportService, ImportService
Excel engine	ExcelImportEngine — uses ClosedXML; reads .xlsx
CSV engine	CsvImportEngine — uses CsvHelper; handles encoding, BOM
Validation	ImportValidationPipeline, IImportValidator<T>
Column mapping	ImportColumnAttribute — decorate DTO properties
Background	ImportBackgroundService — hosted service that dequeues and processes jobs
Storage	IImportFileStorage — saves uploaded files to Blob Storage

Microtec.Import.Integration

What's Included
ImportCompletedEvent — published when a job finishes (success or partial)
ImportFailedEvent — published when a job fails entirely
IImportPublicApi — typed HTTP client for services to check import status

---

How Import Works

The client polls GET /import/jobs/{jobId} or subscribes to ImportCompletedEvent to know when the job is done.

---

Setup

Service Registration

// Program.cs (Import.Apis service)
builder.Services.AddMicrotecImport(builder.Configuration);

This registers:

• IImportService (scoped)
• ImportBackgroundService (hosted service)
• ExcelImportEngine and CsvImportEngine
• IImportFileStorage → Azure Blob Storage adapter

Configuration

{
  "Import": {
    "StorageConnectionString": "keyvaultref:import--storage-connection",
    "ContainerName": "import-files",
    "MaxFileSizeMb": 50,
    "MaxRowsPerJob": 50000,
    "WorkerConcurrency": 3
  }
}

---

Implementing an Importable Entity

Step 1 — Create the import DTO

public class EmployeeImportDto : IImportable
{
    [ImportColumn("Employee Code", Required = true, MaxLength = 20)]
    public string Code { get; set; } = default!;

    [ImportColumn("Full Name (Arabic)", Required = true)]
    public string NameAr { get; set; } = default!;

    [ImportColumn("Full Name (English)", Required = true)]
    public string NameEn { get; set; } = default!;

    [ImportColumn("National ID", Required = true)]
    public string NationalId { get; set; } = default!;

    [ImportColumn("Department Code", Required = false)]
    public string? DepartmentCode { get; set; }

    [ImportColumn("Hire Date", Required = true, Format = "yyyy-MM-dd")]
    public DateTime HireDate { get; set; }

    [ImportColumn("Basic Salary", Required = true)]
    public decimal BasicSalary { get; set; }
}

Step 2 — Implement the validator

public class EmployeeImportValidator : IImportValidator<EmployeeImportDto>
{
    private readonly IRepository<Department> _departments;

    public EmployeeImportValidator(IRepository<Department> departments)
        => _departments = departments;

    public async Task<ImportValidationResult> ValidateAsync(
        EmployeeImportDto row, int rowNumber, CancellationToken ct)
    {
        var errors = new List<string>();

        if (row.BasicSalary <= 0)
            errors.Add("Basic salary must be greater than zero.");

        if (row.DepartmentCode is not null)
        {
            var exists = await _departments.AnyAsync(d => d.Code == row.DepartmentCode, ct);
            if (!exists)
                errors.Add($"Department '{row.DepartmentCode}' not found.");
        }

        return errors.Count > 0
            ? ImportValidationResult.Failure(rowNumber, errors)
            : ImportValidationResult.Success(rowNumber);
    }
}

Step 3 — Register the validator

// In your module's DI setup (Infrastructure layer)
services.AddScoped<IImportValidator<EmployeeImportDto>, EmployeeImportValidator>();

Step 4 — Implement the import handler

The ImportBackgroundService calls the registered handler after validation:

public class EmployeeImportHandler : IImportHandler<EmployeeImportDto>
{
    private readonly IUnitOfWork<HrDbContext> _unitOfWork;

    public async Task HandleAsync(
        IEnumerable<EmployeeImportDto> validRows,
        ImportJob job,
        CancellationToken ct)
    {
        var employees = validRows.Select(row => Employee.Create(
            row.Code, row.NameAr, row.NameEn, row.NationalId, row.HireDate, row.BasicSalary
        )).ToList();

        await _unitOfWork.Repository<Employee>().AddRangeAsync(employees, ct);
        await _unitOfWork.CompleteAsync(ct);
    }
}

---

API Endpoints (Import.Apis)

Method	Path	Description
POST	/import/{entityType}	Upload file, returns jobId
GET	/import/jobs/{jobId}	Get job status and progress
GET	/import/jobs/{jobId}/errors	Download error report (Excel)
GET	/import/templates/{entityType}	Download blank import template
DELETE	/import/jobs/{jobId}	Cancel a pending job

Upload example

curl -X POST https://gateway.microtecstage.com/import/employees \
  -H "Authorization: Bearer <token>" \
  -F "file=@employees.xlsx"
# Response: { "jobId": "3f1a7c2e-...", "status": "Pending" }

Poll for completion

curl https://gateway.microtecstage.com/import/jobs/3f1a7c2e-... \
  -H "Authorization: Bearer <token>"
# Response:
# {
#   "jobId": "3f1a7c2e-...",
#   "status": "PartialSuccess",
#   "totalRows": 1200,
#   "successRows": 1187,
#   "failedRows": 13,
#   "completedAt": "2026-05-30T09:45:12Z"
# }

---

Error Report

When a job completes with PartialSuccess or Failed, an Excel error report is generated automatically. Each failed row appears with:

• Row number (matches the original file)
• Original row data (all columns)
• Error messages (one per line in the last column)

Download via GET /import/jobs/{jobId}/errors.

---

ImportColumn Attribute Reference

Property	Type	Default	Description
Name	string	—	Exact column header in the Excel/CSV file
Required	bool	false	Fails validation if cell is empty
MaxLength	int?	null	String length validation
Format	string?	null	Date format string (e.g. "yyyy-MM-dd")
AllowedValues	string[]?	null	Validates against an enum-like fixed set
Trim	bool	true	Trim whitespace from string values

---

Integration Events

// Subscribe in your service's consumer registration
busConfig.AddConsumer<ImportCompletedEventHandler>();

public class ImportCompletedEventHandler : ConsumerBase<ImportCompletedEvent>
{
    public override async Task Consume(ConsumeContext<ImportCompletedEvent> context)
    {
        var ev = context.Message;
        // ev.JobId, ev.EntityType, ev.TenantId, ev.SuccessRows, ev.FailedRows
        // Notify the user via the Notification service if needed
    }
}

---

Troubleshooting

Symptom	Likely Cause	Fix
Job stuck in Processing	Worker crashed mid-job	Check Import service logs; re-enqueue via admin endpoint
File too large error	Exceeds MaxFileSizeMb	Compress or split the file; raise limit in config if justified
All rows fail validation	Column headers don't match	Download the import template; compare column names exactly
Encoding error on CSV	File is not UTF-8	Re-save CSV as UTF-8 (with BOM) from Excel
Department not found	Lookup data missing	Seed departments before importing employees

---

Related

• Package Catalog
• NuGet Setup
• Service Bus