Central logs and a cheap event store with VictoriaLogs

VictoriaLogs keeps a year of logs from more than 40 machines and from the firewalls, all on one VM. Along the way it also became the event store for dashboards that have little to do with logs. Incidents from the on-call system, workflow runs, expiring credentials, deployments and DNS changes end up there as well. All it takes is JSON sent over HTTP.

It has become my favourite way to handle logs. Very good compression keeps the data small on disk. Live tailing and search work smoothly, and the speed is simply excellent. My HomeLab runs VictoriaLogs as well.

At work it handles the logs in the Grafana stack I built and operate. Below are the data paths, the pattern for API events and the LogsQL pitfalls collected while building the dashboards.

A log database in a single binary

VictoriaLogs is an open source log database from the people behind VictoriaMetrics. A single binary accepts logs over many protocols, including syslog, Loki push, JSON lines, Elasticsearch bulk and OpenTelemetry. Queries are written in LogsQL.

Every entry is a set of fields, and three of them are special. _time holds the timestamp, _msg the message and _stream the fields that identify the stream, for example a host and an application. All other fields are indexed automatically, without a schema. You can send any JSON and then filter or aggregate on any field.

POST /insert/jsonline
?_stream_fields=source,record{
  "_time": "2026-10-20T02:00:05Z",
  "_msg": "expires in 12 days",
  "source": "cert-check",
  "record": "item",
  "run_id": "r-1020",
  "name": "api.example.com",
  "days_left": 12
}

Retention comes down to two flags. -retentionPeriod sets how long data is kept, and -retention.maxDiskSpaceUsageBytes sets a disk limit, above which the oldest days are dropped. Here that means a year of history, with the disk limit as a safety net.

How the data gets in

There is no single way in, because not every machine can reach the same addresses. The diagram at the top shows five paths.

• Private cloud. Hosts send logs with rsyslog straight to the syslog receiver built into VictoriaLogs. Nothing sits in between.
• Public cloud. These hosts cannot reach the private address. Grafana Alloy sends their journal over HTTPS instead. It is the same agent that already sends their metrics.
• The stack host. Alloy reads files from /var/log, Docker container logs and the event file of the DNS auditor. VictoriaLogs offers a Loki-compatible endpoint, so the standard loki.write in Alloy works without changes.
• Firewalls. OPNsense sends packet filter logs and IDS alerts over syslog. This is by far the heaviest stream.
• API data. Workflows normalise data to NDJSON and send it in a POST request to the JSON lines endpoint. The write path for this data is deliberately narrow, and the server sets a fixed stream contract. It started with usage and cost data for AI tools. Today it feeds every dashboard built on API data.

Grafana reads all of it through the VictoriaLogs data source. Raw queries feed log panels and stats queries feed charts. Alert rules use LogsQL too, for example one that fires when a source stops sending logs.

Events that are not logs

Dashboards built on API data, for example from the on-call system, the workflow platform or Entra ID, need history. There are two usual ways to get it. A separate database means Postgres, a schema and an ETL job to maintain. Querying the API live means rate limits, slow panels and no history.

Here workflows collect and normalise the data, VictoriaLogs stores it as events with fields and Grafana calculates the statistics in LogsQL. Nothing needs a schema or a migration. A new field is simply a new field in the JSON.

Today VictoriaLogs holds a start and an end record for every pipeline run, with the commit and the changed, failed and unreachable counts. Every workflow execution is there, and so is the incident history from the on-call system, backfilled from its API with the alert labels. A daily snapshot of secrets and certificates with their expiry dates sits next to the DNS audit findings. It holds names and dates only, never the values. Together with the regular logs, that data feeds more than a dozen dashboards.

Snapshots instead of sums

Some data describes a state rather than an event. A list of certificates with their expiry dates is a good example. When a workflow writes the full list every night, a plain count adds up every night it has run.

The fix is a simple pattern. Each run first writes one run record with a run_id, then its data records with the same run_id. The dashboard uses a subquery to take only the newest run.

In LogsQL it looks like this.

source:="cert-check" record:="item"
  run_id:in(
    source:="cert-check" record:="run"
    | sort by (_time desc) | limit 1 | fields run_id
  )
| stats count() as certificates

LogsQL in practice

A LogsQL query reads like a pipeline. Filters come first, for example app_name:filterlog _time:24h for a day of firewall packet filter logs. Pipes then do the rest, such as | extract, | stats by (...) count(), | math, | sort or | join. In Grafana, the output of stats becomes a panel.

source:="workflows"
| stats by (status) count() as runs

Pitfalls that cost me time

These come from building the dashboards. Behaviour can change between versions, so treat them as notes from my setup.

• Counting unique values. | last 1 by (_time) partition by (id) | stats count() returned 1. A bare count() reads no fields, so the partition had no effect. count_uniq(id) answers the same question directly.
• A median that comes back empty. Missing fields count as empty values, so median(x) returns nothing when most rows have no x. Writing median(x) if (x:*) limits it to rows that have the field.
• Stricter panel queries. A stats query in a panel has to end with stats or math. Range queries for charts rejected last and join.
• Local dates and daylight saving. A fixed offset, as in stats by (_time:1d offset 2h), is wrong for part of the year. I calculate the local date from _time with math on nanoseconds, which also follows the clock change.
• 200 with nothing stored. A POST without Content-Type: application/stream+json got a 200 response and stored zero rows. VictoriaLogs also skips JSON lines it cannot parse and carries on with the rest. The status code alone does not prove that anything was written.
• Backfill tests on a local binary. Its default retention is seven days, and older entries are dropped at ingestion. The request still succeeds, and only a WARN line in the server log mentions it. That can be confusing.
• Windows events without _msg. Events from Alloy keep their text in a message field, so a plain word filter does not find them. A filter on that field, such as message:timeout, does.
• Long ranges on heavy streams. Queries over seven days or more of firewall logs need a cheap word filter first and a longer query timeout.

What I would do again

A log database without a schema is a handy store for any stream of events with fields. There is one condition, a fixed stream contract. That means the same _stream fields and the same time field every time.

If a write path has to face the internet, keep it narrow. It should have one way in, accept writes only and let the server set the contract, not the sender.

I test dashboards locally first, with the VictoriaLogs binary and a Grafana preview. That saves fixes in production.

What I like most is how little it takes to add a new source. A workflow sends JSON over HTTP, and the next dashboard already has its data. The wider stack is described in Grafana beyond CPU charts.