Retention is a product decision
How long you keep raw events decides what questions you can answer next quarter. Write the retention policy down next to the schema, not in someone's memory.
Field notes on running small streaming systems.
How long you keep raw events decides what questions you can answer next quarter. Write the retention policy down next to the schema, not in someone's memory.
If old data is removed by day, partition by day. Deleting a partition is instant; deleting rows from a hot table at three in the morning is not.
The person paged at night needs one screen with lag, error rate and disk headroom. Everything else can wait for daylight.