MLOps Engineer: AI Observability and Monitoring

Role Purpose
The Pakistan Digital Authority (PDA) is seeking an MLOps Engineer to build and maintain deployment pipelines, observability and monitoring for production AI applications. The role enables teams to trace requests, detect failures and quality deterioration, investigate incidents and operate AI services reliably.

Key Responsibilities

• Build CI/CD pipelines for models, prompts and AI services, including automated checks, controlled releases and rollback procedures.
• Maintain registries and version records for models, prompts and datasets, linking deployed releases to their source artifacts.
• Instrument AI requests and agent steps with traces covering model version, latency, token usage, errors and cost.
• Capture inputs and outputs only where authorised, applying redaction, access controls and retention limits to sensitive data.
• Develop dashboards for latency, error rates, token consumption, GPU utilisation, service availability and cost.
• Configure alerts for operational failures, performance degradation and measured declines in output quality.
• Run scheduled and release-based evaluations; compare new versions against agreed baselines and acceptance thresholds.
• Implement checks for drift, hallucination and unsafe content, documenting detection limitations and escalating findings for review.
• Investigate incidents, identify root causes and prepare incident reports with corrective actions.
• Work with engineering and security teams to maintain reliable monitoring and ensure logs and traces follow applicable privacy requirements.

Required Qualifications & Experience

• Bachelor’s degree in Computer Science, Software Engineering, Information Technology, or a related discipline.
• At least 4 years of relevant experience in MLOps, DevOps, platform engineering or site reliability engineering, including at least 1 year supporting production AI/ML systems within that period.
• Hands-on experience with CI/CD, Git, Linux, containers, scripting and automated deployment workflows.
• Practical experience with metrics, logs, distributed tracing, dashboards, alerting and production incident investigation.
• Working knowledge of model serving, artifact versioning, LLM evaluation and token-based usage and cost tracking.
• Strong problem-solving, technical documentation and collaboration skills.

Preferred Experience

• Experience with tools such as OpenTelemetry, Prometheus, Grafana, MLflow or LLM observability platforms.
• Exposure to Kubernetes, GPU workloads, infrastructure as code and production agent tracing.
• Experience applying data minimisation, redaction and access controls in government or enterprise environments.