A new osquery lane into Samma, and the osquery client in Docker

Samma has a new input. Next to scanners and Wazuh, hosts can now send osquery results straight into the pipeline on their own lane, samma.logs.osquery.

osquery lets you ask a machine questions in SQL: which users exist, who is in the sudo group, which SSH keys can log in, what is listening on which port. Run those queries on a schedule, send only what changed, and you have a very good picture of what is happening on a host.

How it fits

host: osquery-client (Docker)
   │ HTTPS POST, mutual TLS, client certificate per host
   ▼
cluster: osquery-gateway (Vector)  ── one record per result ──► NATS samma.logs.osquery
                                                                 ├──► siem-storage ──► TimescaleDB
                                                                 └──► detection-engine (Sigma) ──► alerts

• The client (samma-io/osquery-client) is osqueryd 5.24.1 with its TLS logger, in a container.
• The gateway is Vector with an http_server source and TLS. It only accepts clients whose certificate our CA signed. It splits each batch into one record per result, adds source, feed, event_time, search_text and gateway_peer, and publishes to NATS.
• After NATS it is the same pipeline as every other lane: one consumer stores to TimescaleDB, another runs Sigma rules.

Nothing gets lost on the way

The gateway uses end-to-end acknowledgements. It only answers the client's POST once NATS JetStream has stored the batch. If the gateway or NATS is down, osquery keeps the results in its on-disk buffer and retries. That buffer is a Docker volume, so it survives a container restart too.

What the client collects

Scheduled queries every 10 minutes to 1 hour:

• local users, members of root/sudo/wheel/adm/docker
• SSH authorized_keys, sudoers, system and user crontabs
• listening ports with the owning process
• logged-in users and login history
• kernel modules, packages, setuid binaries, OS version

Real-time events, sent once a minute:

• process events (eBPF): every program started, with parent, uid and path
• socket events (eBPF): outbound connects and inbound accepts
• file events (inotify): changes to account, sudo, SSH and cron files and the binary folders

Scheduled queries are differential. After the first run, osquery only sends rows that were added or removed. The first run on a new host sends everything with counter: 0, and the rules skip it, so a new host doesn't fire a wall of alerts.

Some things are left out on purpose. Command lines often carry passwords and tokens. Compilers, container runtimes and polling commands like ps are filtered from process events, and DNS lookups from socket events. On a busy host they are most of the volume and none of the signal. A developer machine running Docker and Kubernetes tools ends up at about 10 to 20 MiB per hour.

Connect a host

1. Once: the CA and the gateway certificate

pki/make-ca.sh
pki/issue-server.sh        # SAN DNS:osquery-gateway

The gateway's Kubernetes Secret is made from the server cert, key and ca.crt. In our cluster the gateway listens on NodePort 30516.

2. Per host: a client certificate

pki/issue-client.sh web-01
# -> pki/out/clients/web-01/{ca.crt,client.crt,client.key}

Copy that folder to the host as ./certs, next to compose.yaml. Nothing in pki/out/ is committed to git.

3. On the host: start the container

OSQUERY_CLIENT_TAG=<tag> \
OSQUERY_GATEWAY_IP=10.0.0.33 \
OSQUERY_HOST_IDENTIFIER=web-01 \
  docker compose up -d

<tag> is a published image tag (a short commit sha from main) of ghcr.io/samma-io/osquery-client. OSQUERY_HOST_IDENTIFIER should match the name on the certificate. The compose file maps the name osquery-gateway to the node IP, so TLS checks the name in the server certificate.

Within five minutes the samma_heartbeat query sends its first row, and you can see the host in TimescaleDB:

SELECT time, record->>'name' AS query, record->>'action' AS action
FROM siem_events
WHERE lane = 'osquery' AND record->>'hostIdentifier' = 'web-01'
ORDER BY time DESC LIMIT 20;

The compose file, and what it costs

services:
  osquery-client:
    image: ghcr.io/samma-io/osquery-client:${OSQUERY_CLIENT_TAG}
    restart: unless-stopped
    pid: host
    network_mode: host
    extra_hosts:
      - "osquery-gateway:${OSQUERY_GATEWAY_IP}"
    environment:
      OSQUERY_GATEWAY: osquery-gateway:30516
      OSQUERY_HOST_IDENTIFIER: ${OSQUERY_HOST_IDENTIFIER}
      OSQUERY_EVENTS: "true"
    cap_add: [SYS_ADMIN, BPF, PERFMON, SYS_PTRACE]
    security_opt: [apparmor=unconfined, seccomp=unconfined, label=disable]
    ulimits:
      memlock: -1
    volumes:
      - ./certs:/etc/osquery/certs:ro
      - osquery-client-db:/var/osquery
      - /sys/kernel/debug:/sys/kernel/debug:ro
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
      # ... sudoers, crontab, utmp/wtmp, dpkg, /home, /root
      - /:/host:ro
volumes:
  osquery-client-db:

Be clear about what this is. To see every process start and file change, the container traces the host kernel with eBPF and reads the whole host file system. Treat it like a root agent on the host.

If that is too much for a machine, drop the capabilities, the security options, memlock, /sys/kernel/debug and the / mount, and set OSQUERY_EVENTS=false. The scheduled queries still work. You lose the real-time events.

Detection: Sigma rules for osquery

The osquery lane ships with 11 Sigma rules in the detection-engine:

Rule	Fires when
user added	a new local account appears
admin group member added	someone joins root/sudo/wheel/adm/docker
authorized key added	a new SSH key can log in
sudoers changed / crontab changed	the file gets a new row
SUID binary added	a new setuid program shows up in /usr
kernel module loaded	a new module appears
new listening port	a new TCP listener (not Docker's DNS)
exec from /tmp	a program runs from /tmp, /var/tmp or /dev/shm
shell outbound connection	a shell process opens a connection out
sensitive file changed	passwd, shadow, group, sudoers, /etc/ssh/ or any .ssh folder

They match on osquery's own fields: name (the query), action and columns.*. For example "a new SSH key, but not on the first run":

detection:
  selection:
    name: samma_authorized_keys
    action: added
  filter_baseline:
    counter: 0
  condition: selection and not filter_baseline

Revoking a host

There is no revocation list yet. Client certificates are valid for one year. To cut a host off sooner, replace the CA, re-issue the gateway certificate and re-issue the certificates for the hosts you keep. It is blunt, but with a handful of hosts it takes a few minutes.

Wazuh or osquery?

Both, and they cover different things. The Wazuh agent is good at reading logs and decoding them into known events. osquery is good at answering "what is the state of this machine right now, and what changed". In Samma they land on two lanes of the same NATS pipeline, in the same TimescaleDB table, checked by the same Sigma engine.