Scan from the outside, see it on the inside: Samma results as Prometheus metrics

When you run security scans from inside a Kubernetes cluster you see your services the way the cluster sees them. That is useful, but an attacker sees them from the internet: through your load balancer, your CDN, your firewall. Those two views are not always the same.

Samma can do both. The operator in your cluster runs scanners locally, and samma.io runs scanners against the same hosts from the outside. Until now the outside results stayed on samma.io. You had to log in to the dashboard to see them.

The new collector brings them home. Results from samma.io end up in your own Prometheus, your own logs and your own Grafana, next to everything else you monitor.

The loop

 your cluster                                   samma.io
 ────────────                                   ────────
 Ingress  (annotated)
    │
    ▼
 samma operator ── POST /api/v1/targets ──────► target
                                                  │
                                                  ▼
                                               scanners run from the internet
                                                  │
                                                  ▼
 samma collector ◄─ GET /api/v1/targets/…/results ┘
    │
    ├──► /metrics   ──► Prometheus ──► Grafana
    └──► stdout     ──► Loki / your log shipper

1. You opt an Ingress in with one annotation. Nothing is scanned unless you ask for it.
2. The operator registers the host on samma.io as a target. This part already existed: the operator posts every opted-in host when it has an API token.
3. samma.io scans it from the outside, on its own schedule.
4. The collector pulls the results back and turns them into metrics and log lines inside your cluster.

Step 4 is new. Everything goes out from the cluster over HTTPS. Nothing on the internet has to reach into it.

Step 1: opt in an Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: samma-io
  annotations:
    samma-io.alpha.kubernetes.io/enable: "true"
spec:
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: shop
                port:
                  number: 80

The operator handles Ingresses in its own namespace, plus any namespaces you list in ingress.watchNamespaces. Setting the annotation to "false" or removing it opts the host out again.

Step 2: the collector

The collector is a small Python service in the operator's Helm chart. It uses only the standard library: no Kubernetes client, no prometheus_client. It is off by default. Every few minutes it does three things:

• Lists the Ingresses in the cluster and keeps the hosts the operator would scan. It uses the same rules as the operator: the annotation set and not "false", an allowed namespace, and no wildcard hosts.
• Reads the targets from the samma.io API and keeps only the ones whose host is in that list. A samma.io organisation can hold targets from many clusters, and each cluster exports only its own.
• Fetches the latest results for each of those targets: the newest run of every scanner, and its findings.

It needs a samma.io API token, and a read-only one is enough. It only reads. In the cluster its only right is to get and list Ingresses.

kubectl -n samma-io create secret generic samma-collector-token \
  --from-literal=token=<read-only token from samma.io>

helm upgrade samma-operator oci://ghcr.io/samma-io/charts/samma-operator --reuse-values \
  --set collector.enabled=true \
  --set collector.existingSecret=samma-collector-token

What you get: metrics

The collector serves Prometheus metrics on port 9100:

samma_findings{target="shop.example.com",scanner="tls-scanner",severity="HIGH"} 2
samma_findings{target="shop.example.com",scanner="tls-scanner",severity="CRITICAL"} 0
samma_scan_last_finished_timestamp_seconds{target="shop.example.com",scanner="tls-scanner"} 1794312000.0
samma_scan_outcome{target="shop.example.com",scanner="tls-scanner",outcome="succeeded"} 1
samma_collector_targets 1
samma_collector_unmatched_hosts 0
samma_collector_polls_total{result="ok"} 42

A few design choices:

• Bounded labels. A label is a host of one of your own Ingresses, a scanner name or a severity. The check ID and the finding title are not labels. They go to the logs.
• Zero-filled severities. A target with no critical findings shows 0, not a missing series. That way "no critical findings" and "no data" look different in a panel.
• A failed poll changes nothing. If samma.io is unreachable or the token is refused, the old values stay and samma_collector_polls_total{result="auth"} (or network, http) goes up. A short outage does not make every finding drop to zero and fire your alerts.
• Gone is gone. If you delete an Ingress, its series disappear on the next poll. They are not frozen at their last value.
• unmatched_hosts counts opted-in hosts that have no target on samma.io yet. It is the first thing to check when a new host does not show up.

The Service has prometheus.io/scrape annotations. If you run the Prometheus Operator, set collector.serviceMonitor.enabled=true instead.

A simple alert on top of it:

- alert: SammaCriticalFinding
  expr: sum by (target) (samma_findings{severity="CRITICAL"}) > 0
  for: 15m
  labels:
    severity: page
  annotations:
    summary: "samma.io found a critical issue on {{ $labels.target }}"

What you get: logs

Metrics tell you how many. Logs tell you what. For every new scan run, the collector writes one JSON line per finding to stdout:

{"event":"samma.finding","target":"shop.example.com","scanner":"tls-scanner","runId":"tls-scanner-…","severity":"HIGH","checkId":"TLS-OLD-PROTOCOL","title":"TLS 1.0 is enabled","host":"shop.example.com","port":"443","time":"…"}

Whatever already ships your pod logs (Loki, Vector, Fluent Bit, Filebeat) picks these up. Each run is logged once. The collector remembers which runs it has written, so a five-minute poll does not repeat the same finding 288 times a day.

The evidence a scanner collected is never logged on purpose. Evidence is what the scanner saw, and sometimes that is something you would not want in a log index, like a banner, a header or a token. Evidence stays on samma.io, behind your login.

The Grafana dashboard

The chart ships a dashboard (dashboards/samma-findings.json). Import it by hand, or set collector.grafanaDashboard.enabled=true to have the kube-prometheus-stack Grafana sidecar load it from a ConfigMap. It shows:

• Critical / high / medium totals, the number of matched targets, and how long ago the last successful poll was
• Findings by severity over time, so you can see whether a fix actually made the numbers go down
• Critical and high findings per target
• A table of every target and scanner with the outcome of its last run and how old it is. It turns orange after a week and red after two, so a scanner that has silently stopped is easy to spot.
• Collector health: polls by result

Why pull and not push?

samma.io could also push results into the cluster. We chose pull for three reasons:

• Nothing has to reach into your cluster. No webhook endpoint, no ingress for the results, no firewall hole. The collector only makes outgoing HTTPS calls, like the operator already does.
• The token is read-only. If it leaks, it can only read results. It cannot change targets or start scans.
• It works behind NAT. A home lab or an edge cluster without a public IP gets its results the same way as a cloud cluster.

The price is a delay of up to one poll interval (five minutes by default). For scan results that is nothing.

Status

The collector is in review now (samma-io/operator#47) and will be part of the next operator release. The operator README has a "Testing the collector" section that walks through the whole loop step by step, from the token to the Grafana panel. Until the release is out, install it with --set collector.tag=latest once the PR is merged.

Next up is testing it on our own clusters, against real Ingresses scanned from the outside.