Skip to content

Log store: centralized log

The log store is an optional service that centralizes NGX Ops logs: Nginx access, decisions from the challenge, the emulated WAF, the blocklists and CrowdSec. It enables search, visit statistics, security analysis and per-IP history.

It consists of an API (Go) and a PostgreSQL database with TimescaleDB. It is standalone: neither the dashboard nor the Hub embeds it, they query it over HTTPS.

analyzer / shipper ──gzip batches, HTTPS, write token──► log store (API) ──► PostgreSQL + TimescaleDB
dashboard / Hub ───────────── read token ─────────────►        (the database port is never published)

Without a log store, nothing changes for the rest of the suite.

Installation

The log store ships with an example docker-compose (API service and TimescaleDB database). Copy the example variables file, fill in the two required values, then start:

cp sample.env logstore.env
# fill in LOGSTORE_DB_PASSWORD and LOGSTORE_ADMIN_TOKEN (openssl rand -hex 32)
docker compose --env-file logstore.env -f docker-compose.example.yml up -d
curl -s localhost:8443/v1/health

The expected response is {"status":"ok"}. The admin token is only used to create access tokens: it is hashed at first start.

TLS

The log store requires TLS by default (LOGSTORE_TLS_CERT and LOGSTORE_TLS_KEY). LOGSTORE_INSECURE_HTTP=true is reserved for an internal Docker network or a trusted TLS reverse proxy.

Tokens

There are two roles:

Role Usage Scope
write Send logs (the shipper) Bound to one instance (instance_id)
read Browse With instance_id: a dashboard only sees its own logs. Without instance_id: global view, reserved for the Hub and administration

Creating a token (example for a server web1):

curl -s -H "Authorization: Bearer $LOGSTORE_ADMIN_TOKEN" \
  -d '{"label":"web1","role":"write","instance_id":"web1"}' \
  localhost:8443/v1/tokens

The token is shown only once. It is stored hashed (SHA-256), compared in constant time, and can be revoked.

The https://<log store>/ui/ interface lets you view, create and revoke tokens after entering the admin token (kept in memory in the tab, never stored). It is disabled with LOGSTORE_UI=false. If possible, restrict /ui/ by IP address at the reverse proxy level.

Personal data: IP mode

The mode is applied at ingestion, on the log store side: no producer can bypass it. It is not retroactive.

Mode Stored value Search by IP
clear The address Yes
truncated The /24 (IPv4) or /48 (IPv6) prefix Yes, reduced to the prefix
pseudonymized A truncated HMAC-SHA256 fingerprint, no address Yes, thanks to the key

In pseudonymized mode, LOGSTORE_PSEUDO_KEY (32 characters minimum) is required and the service refuses to start without it. Changing it breaks correlation between before and after.

No raw query string or full referer is stored: only a fingerprint and the host.

Retention

Setting Range
Raw logs 7 to 30 days
Rollups (hourly and daily aggregates) 30 to 800 days

The variables are only used for initialization. After that, retention and IP mode are changed on the fly with PUT /v1/settings (admin role).

PostgreSQL maintenance

TimescaleDB purge, compression and retention are automatic. In addition, the service runs a VACUUM (ANALYZE) on the tables every LOGSTORE_MAINTENANCE_HOURS (24 h by default, first run about ten minutes after startup). LOGSTORE_MAINTENANCE=false turns this task off, PostgreSQL autovacuum remaining active.

The maintenance status (dead rows, compression, retention respected) is visible in the "PostgreSQL maintenance" section of the /ui/ interface and via GET /v1/maintenance; POST /v1/maintenance runs it immediately.

Main variables

Variable Role
LOGSTORE_DB_PASSWORD PostgreSQL password (required)
LOGSTORE_ADMIN_TOKEN Admin token, 32 characters minimum (required)
LOGSTORE_IP_MODE Initial IP mode: clear, truncated, pseudonymized
LOGSTORE_PSEUDO_KEY Pseudonymization key
LOGSTORE_RETENTION_DAYS Initial raw log retention (7 to 30)
LOGSTORE_TLS_CERT / LOGSTORE_TLS_KEY TLS certificate mounted in the container
LOGSTORE_INSECURE_HTTP Plain HTTP, internal network only
LOGSTORE_REQUIRE_TIMESCALE true: refuses to start without TimescaleDB
LOGSTORE_UI /ui/ interface (true by default)
LOGSTORE_RATE_READ / LOGSTORE_RATE_WRITE Requests per second per token (20 / 10)
LOGSTORE_MAX_BATCH_EVENTS Maximum batch size (5000)
LOGSTORE_QUERY_TIMEOUT_SECONDS Maximum query time (15)

Two separate PostgreSQL roles (read-only and write-only) are possible with LOGSTORE_DATABASE_URL_READ and LOGSTORE_DATABASE_URL_WRITE.

Security

  • TLS required by default, PostgreSQL port never published, non-root container with a read-only file system.
  • Rate limiting per token and a brake on authentication failures.
  • Bounded sizes everywhere (body, batch, line, decompressed size), parameterized SQL, search window of 31 days at most.
  • Errors without internal detail; no token or password in the logs.

Known limits

  • Without TimescaleDB, the log store runs in "plain PostgreSQL" mode: no compression or long rollups. Reserved for development and testing.
  • The number of distinct visitors is only available over the period still stored raw.
  • Rollups are computed in UTC.

To send logs to the log store, see Shipper and Logs page. For orders of magnitude of volume and performance, see Capacity and performance.