Breaking up Wazuh: a small collector, NATS, TimescaleDB and Sigma

A standard Wazuh install has four big parts: the manager, a worker, the indexer (OpenSearch) and the dashboard. The manager and worker are light. OpenSearch is not. It wants gigabytes of heap and fast disks, and it is the part that keeps falling over on a small cluster.

In Samma we only need the part of Wazuh that talks to agents. Storage, search, detection and dashboards we already have in our own SIEM pipeline. So we broke Wazuh up.

Before and after

Before:  agents ──► manager/worker ──► Filebeat ──► OpenSearch ──► Wazuh dashboard

After:   agents ──► manager/worker ──► Vector ──► NATS JetStream ──┬──► siem-storage ──► TimescaleDB
                    (collector mode)   sidecar   samma.logs.wazuh  │
                                                                   └──► detection-engine (Sigma) ──► alerts

What we kept: agent enrollment (1515), agent events (1514), the Wazuh decoders and rules. Wazuh still turns raw logs into alerts like "sshd: authentication failed" (rule 5760) or "agent enrolled" (rule 501).

What we removed: the indexer, the dashboard, Filebeat, Keycloak and vulnerability detection.

Collector mode in the Helm chart

Our Wazuh Helm chart (samma-io/wazuh-kubernetes) got a new mode. The default still renders the full stack, byte for byte. mode: collector renders only the manager and the worker:

mode: collector

collector:
  logallJson: false          # true also ships archives (every event, not only alerts)
  vector:
    enabled: true
    natsUrl: nats://nats.samma.svc:4222
    subject: samma.logs.wazuh

ingress:
  enabled: false
keycloak:
  enabled: false

services:
  external: false            # no Wazuh API exposed
  agentNodePorts:
    enabled: true            # only 1515 (enroll) and 1514 (events)

helm upgrade --install wazuh-collector chart/ -n samma \
  -f chart/values-collector-v2.yaml -f my-private-credentials.yaml

In collector mode the chart turns off the indexer and vulnerability detection in ossec.conf, so the manager doesn't sit trying to reach an OpenSearch that isn't there. It also creates queue/tasks, otherwise the master's task manager exits on start.

agentNodePorts opens just the two ports an agent needs, on fixed NodePorts 30515 and 30514. The Wazuh API (55000) stays inside the cluster.

Vector: from alerts.json to NATS

Each manager pod gets a Vector sidecar. It tails /var/ossec/logs/alerts/alerts.json, adds a small envelope, and publishes one alert per message to NATS JetStream:

sources:
  wazuh_alerts:
    type: file
    include: [/wazuh/alerts/alerts.json]
sinks:
  nats_logs_wazuh:
    type: nats
    inputs: [wazuh_envelope]
    url: ${NATS_URL}
    subject: ${WAZUH_LANE_SUBJECT}     # samma.logs.wazuh
    jetstream: true
    buffer:
      type: disk
      when_full: block

The envelope adds source: wazuh, feed: alerts, event_time (taken from the alert's own timestamp) and search_text (a copy of full_log). Everything else is the Wazuh alert as it was.

Two settings matter for not losing alerts:

• End-to-end acknowledgements + a disk buffer. Vector only moves its file position forward once the record is buffered for publish. The buffer and the checkpoints live on the manager's PVC, so a pod restart picks up where it left off.
• The sidecar runs as uid 999, the Wazuh user that owns alerts.json, with a read-only root filesystem and all capabilities dropped.

Why NATS in the middle

Putting NATS JetStream between Wazuh and the rest is what makes this design work.

• Many Wazuh clusters, one pipeline. Every collector publishes to the same subject. Two, five or twenty Wazuh managers in different networks or for different customers all land in one stream. Storage and detection don't care how many there are.
• Scale each part on its own. Add Wazuh workers when you have more agents. Add detection-engine replicas when you have more rules. They never talk to each other directly.
• Several readers, one copy. The stream has two independent durable consumers: storage-logs writes to TimescaleDB, detection-engine-wazuh runs the rules. A slow database never holds back detection.
• Back pressure instead of data loss. If storage is down, messages wait in JetStream. If NATS is down, Vector's disk buffer fills and then blocks.

TimescaleDB instead of OpenSearch

siem-storage reads the stream and writes every record into one TimescaleDB hypertable, siem_events:

time | lane | subject | stream_seq | line | record (jsonb)

time is the alert's own event time. A redelivered message keeps its stream sequence, and inserts are ON CONFLICT DO NOTHING, so a retry never makes a duplicate row. Retention is a TimescaleDB policy, set from one environment variable.

For our volume, Postgres with Timescale does everything we used OpenSearch for: time-range queries, JSON filters on record, and Grafana panels. It runs in a fraction of the memory, and it is a database we already run anyway.

Sigma rules instead of a custom engine

Detection used to be our own YAML rule format. We replaced it with Sigma, the open, vendor-neutral rule format the rest of the industry is moving to, Wazuh included. The same detection-engine runs every lane (AWS, GitHub, Kubernetes, Wazuh, osquery). On Kubernetes it reads from NATS with ENGINE_TRANSPORT=nats and publishes matches to samma.alerts.wazuh.

A Wazuh alert is plain JSON, so a Sigma rule can match on any field, nested ones included:

title: Wazuh Agent Connected For The First Time Or Removed
status: experimental
level: low
tags: [attack.defense-evasion, attack.t1562.001]
logsource:
  product: wazuh
  service: alerts
detection:
  selection:
    feed: alerts
    rule.id: ['501', '505']
  condition: selection
falsepositives:
  - Planned enrollment or decommissioning of a host.

The Wazuh lane ships with nine rules today: failed logins, SSH brute force, failed sudo, first-time sudo, user account changes, crontab changes, AppArmor denials, agent enrolled/removed and any alert at level 12 or above. Every rule has a .tests.yml next to it with records that must and must not match, and CI runs them.

Wazuh still does what it is good at: decoding hundreds of log formats into structured alerts. Sigma decides what is worth a human looking at. And the same Sigma rules work on alerts from every Wazuh cluster on the stream.

What you give up

Be honest about the trade-off. In collector mode there is:

• No Wazuh dashboard. You look at data in Grafana or the Samma dashboard.
• No vulnerability detection. It needs the indexer.
• No search inside Wazuh. Search is in TimescaleDB.

For us that is fine, because all of those live elsewhere in Samma. If you need the Wazuh UI, keep the default mode. The chart does both.

Result

The Wazuh part of our cluster went from four StatefulSets, with OpenSearch as the heaviest, to two small manager pods with a 20m CPU Vector sidecar each. Alerts from any number of Wazuh clusters now flow into one pipeline, get stored once, and are checked by the same Sigma rules as everything else.

Next posts: running the Wazuh agent in Docker and a new osquery lane into Samma.