The Great Cardinality Disasters of Our Time

Abstract

Many Cloud Native tools generate Prometheus metrics; together they form a great combination to operate and monitor your infrastructure. But sometimes things go wrong: a quirk in the metric labels can make the volume of data explode, and, soon after, your Prometheus will explode too. Chris and Bryan will share their war-stories such as receiving 46,000 simultaneous alerts or squashing the source of 100kB label values. Then, they will provide top tips to avoid this happening to your tools in the future.

Date
Nov 20, 2019
Event
KubeCon NA
Avatar
Chris Marchbanks
Senior Software Engineer

Skiing, hiking, Prometheus, and all things observability