DataSteamn · Riga

Field notes on running small streaming systems.

Retention is a product decision

How long you keep raw events decides what questions you can answer next quarter. Write the retention policy down next to the schema, not in someone's memory.

Partition by what you delete

If old data is removed by day, partition by day. Deleting a partition is instant; deleting rows from a hot table at three in the morning is not.

One dashboard per on-call

The person paged at night needs one screen with lag, error rate and disk headroom. Everything else can wait for daylight.