Infrastructure signals
Linux and Windows metrics, central logs, service probes, and existing monitoring integrations.
Case study / Observability
Grafana stack. From signals to operational decisions.
An observability platform I designed, built, and operate: metrics, central logging, custom collectors, and operational workflows managed as code.
The problem
Hybrid-cloud operations needed more than isolated dashboards. Engineers needed to connect infrastructure health, logs, recent changes, backup status, and cloud cost without rebuilding that context during every investigation.
My responsibility
As the primary engineer, I designed, built, and operate the observability and automation platform. My ownership includes integrations, troubleshooting, and documentation.
The decisions
Keep metrics in Prometheus and logs in VictoriaLogs, with Grafana as the shared investigation layer. Use pull collection where connectivity allows it and Alloy push collection for remote workloads. Version dashboards, data sources, alert rules, and deployment configuration.
The solution
A containerised stack combines exporters, Alloy, custom API collectors, and workflow integrations. Terraform manages the infrastructure layer; Ansible configures the stack through Azure DevOps. Deployment records connect infrastructure changes with telemetry, while scheduled reports bring availability and performance into routine operations.
The outcome
A working production platform brings millions of daily log lines together with metrics, deployment history, DNS findings, backup health, cost views, and incident context. Engineers can investigate infrastructure and operational events in a shared workspace.
The takeaway
A useful platform needs reliable collection, clear ownership, validation, and documentation as much as it needs dashboards. My work includes onboarding data sources, investigating failures, maintaining integrations, and documenting deployment and recovery.
07 / Inside the solution
A platform I built.
And continue to operate.
My role spans design, implementation, custom integrations, deployment, troubleshooting, and the documentation that keeps the platform operable.
The scale of this platform
- Dashboards as code
- 40+
Versioned and deployed with the platform.
- Log lines per day
- Millions
Central logging with operational context.
- Log sources
- 40+
Hosts and devices sending logs to the stack.
- DNS zones audited
- 100+
Automated checks across two DNS providers.
Rounded figures from an October 2026 snapshot of this stack. The wider team environment is described separately on the home page.
- 01 / Collect
Exporters · Alloy · APIs
Pull where reachable; push for remote workloads. Custom collectors add operational context.
- 02 / Store
Prometheus · VictoriaLogs
Metrics and logs have separate stores. Cost data uses a dedicated database.
- 03 / Investigate
Grafana
Dashboards, log exploration, change history, and alert evaluation in one workspace.
- 04 / Act
Reports · Alerts · Workflows
Teams, PagerDuty, Jira, and n8n connect visibility with daily operations.
Terraform · Ansible · Azure DevOpsVersioned infrastructure, configuration, dashboards, and alert rules.
Simplified conceptual architecture. Production identifiers and access details are omitted.
Custom collectors
DNS auditing, backup health, cloud cost, and credential expiry brought into a shared operational view.
Change context
Pipeline deployment records alongside telemetry, with incident history and automation workflow results.
Operational workflows
Alerts and reports through Teams and PagerDuty, with n8n connecting findings to Jira and recurring operational tasks.
Decisions and trade-offs
Git as the source of truth
Dashboards, data sources, alerts, and configuration are versioned and deployed through Azure DevOps. Dashboard validation runs before deployment. Lasting changes belong in code rather than only in the UI.
Separate infrastructure and configuration
Terraform owns cloud resources; Ansible owns stack configuration and deployment. This keeps responsibilities explicit when changing or rebuilding the platform.
Adapt collection to connectivity
Pull and push collection solve different connectivity constraints. Supporting both broadens coverage, but adds agent configuration and ingestion paths to maintain.
Keep the operating model in scope
A container-based runtime keeps the stack understandable. Availability design, retention, backup verification, and recovery procedures remain engineering responsibilities; containers alone do not provide high availability.
More from my work.
Explore the labCloud migration
From vCenter to the cloud, with purpose.
A production data centre migration to Azure and IONOS, with attention to continuity, sizing, and ongoing cost.
~20% lower IONOS costs through optimisation
Automation
Less repetition. More repeatability.
Infrastructure as code and connected workflows for provisioning, configuration, and recurring operational tasks.
~70% of provisioning automated
Let's connect
Something to build?
Let’s talk.
Hybrid cloud, observability, automation, or an interesting engineering challenge.
Email
LinkedIn