Robust Perception

Mimir needs 30TB of RAM for a billion of active time series. That's too big price for horizontal scalability comparing to the original Prometheus, which requires around 4TB of RAM for a billion of active time series according to https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-… It would be great comparing Mimir RAM usage to other horizontally scalable time series databases such as M3DB, TimescaleDB, InfluxDB and others.

These might be a good read if you are considering it: - https://www.robustperception.io/evaluating-performance-and-correctness - https://medium.com/@valyala/evaluating-performance-and-correctness-vic…

Yep, used in production here - https://www.robustperception.io

I run a consulting/support business around Prometheus. I've found that giving talks and weekly writing blog posts (https://www.robustperception.io/blog/) are great for acquiring customers. Beware though it's often at least 6-9 months before any of it directly bears fruit.

Honest question, how do you handle high cardinality data points? Reference to where my brain is at: https://www.robustperception.io/cardinality-is-key/ I feel like splunk’s business model favors a healthy system and gives major disadvantages to an unhealthy one. What I mean in an example: when the system is unhealthy, I know it because all my splunk queries get queued up because everyone is slamming it with queries. I hate it. But I’m stuck in knowing how to move some things to Prometheus. Like say we have a CustomerID and we want to track number of times something is done by user. If we have thousands of customers, cardinality breaks that solution. Is there a good solution for this?

I second the Thanos link, but if you want to only use Prometheus, I would look into federation: https://www.robustperception.io/scaling-and-federating-prometheus https://prometheus.io/docs/prometheus/latest/federation/ You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.

Ways to monitor for cert expiry in no particular order: 1) Prometheus + blackbox_exporter https://www.robustperception.io/get-alerted-before-your-ssl-certificat… 2) Sensu/Nagios https://github.com/sensu-plugins/sensu-plugins-http/blob/master/bin/ch… 3) Openssl in a crontab: echo | openssl s_client -connect ${DOMAIN}:443 -servername ${DOMAIN} -verify_hostname ${DOMAIN} 2>/dev/null | openssl x590 -noout -startdate -enddate

Usually you start by some functional sharding (giving each team of services their own Prometheus servers), but also having per-datacenter Prometheus servers and then some hierarchical federation layer ontop of that. There's no built-in automatic horizontal scaling though (which would be going against the design goal of not having a clustered system, for reliability). Some resources: - Scaling in general: http://www.robustperception.io/scaling-and-federating-prometheus/ - Federation: https://prometheus.io/docs/operating/federation/

It's interesting to me how Prometheus came to dominate the metrics ecosystem over the past decade. When I first encountered it, I had a visceral reaction against it, primary over: 1) The PromQL query language is really not intuitive. I've been using it for years and I still stick to really simple joins to keep myself out of hot water 2) Being forced to use rate() on counter metrics was a really big initial barrier against one of the presumably simpler things a metrics system should handle. See this popular long blog post explaining what Prometheus Counters are: https://www.robustperception.io/how-does-a-prometheus-counter-work/ But complicated rate() and counters were a symptom of the killer feature about Prometheus, that it was a pull-based rather than push-based model. This meant you could easily re-deploy subcomponents of your metrics system, getting rid of the hair-raising stop-the-world updates that we used to have to do with graphite.