I really hope upstream node_exporter is happy adding new exports!

Still slowly hacking on my homelab Prometheus bringup, but I've already got a start developing a battery level/power supply exporter. By far the most important Node information I want metrics and alarms on in my developer life! https://github.com/rektide/node_exporter/tree/export-battery

Almost went with a standalone Node exporter, but I decided I'd try some Go, and tour some of the Prometheus codebase. MustNewConstMetric is a very confusing thing to me, but hopefully I'm on the right track! Feeling double-inspired to get my homelab Prometheus up and going right now, between renewed excitement for these exporters and the 1.0! So close!

Funny coincidence, yesterday I did almost the same thing, though mine's based on lm-sensors and hddtemp:

https://github.com/ncabatoff/sensor-exporter

It's still focused on pull, while there is the Pushgateway (see https://prometheus.io/docs/instrumenting/pushing/ and https://prometheus.io/docs/practices/pushing/) for dealing with one-off situations where you cannot scrape something. Pull works great in most situations where people have their own private clouds or datacenters, but it's less suitable for very restrictive environments where you can't run Prometheus on the same network or behind the same firewall as the targets you want to monitor.

In the usual cases, pull has many benefits however:

- You can get high availability by simply running two identically configured, independent Prometheus servers. No clustering required.

- You can run a copy of production monitoring (or similar) on your laptop without changing production. This is great for experimentation and testing changes.

- You get free up-ness monitoring via scrapes and can use this for alerting.

- When there's an HTTP pull endpoint on service instances, you can also go there manually as a human and check out the current metrics state of any target, independent of the Prometheus server.

- Knowledge of service identities is inverted: instead of each service instance having to know its own identity (usually instance="hostname:port" and some job/service name), the monitoring system knows (usually via some form of service discovery) what instances should be there and how they are labeled, and proactively checks on them. Services have no knowledge of where the monitoring system lives anymore, enabling the above use cases.

- Debatable, but push-based monitoring systems can make it easier for someone to accidentally DDoS your monitoring. (still possible with pull, but you have one central place where you know what you pull from)

There is no first-class push option.

I don't see why processing pipelines would be special for push vs. pull, it's generally a wash.

That would be the correct issue about the staleness limit, yes. Note that Prometheus does not track individual events, but only numerical time series and their current and historical values. What's the exact use case you're stuck on? I guess you are using the pushgateway with client-side timestamps?

Prometheus range queries work a bit differently, so this is not 100% reproducible in a graph query, but a similar thing is possible. A given PromQL expression is evaluated at every resolution step along the graph and doesn't have context about what the "graph range" is. At every evaluation point, it can still look back over a given time window, but that's more of a sliding window approach then and independent of what the visible graph time range is.

For example, instead of highestCurrent, you could do something like:

topk(3, my_metric)

This would at every point along the graph select the current top 3 series that have the my_metric metric name.

Or if you want to average each series over the last 10 minutes at every point in the graph before selecting the top 3:

topk(3, avg_over_time(my_metric[10m]))

Note that due to the reasons mentioned above, topk() here does not select whatever line has the largest area under the entire visible graph range, but whatever is at the top at each given resolution step. So you may actually get more than 3 series in your graph, but only 3 at a time at any given X.

There's also an issue asking about this, but we're not sure if that is fundamentally compatible with Prometheus's query execution model without major changes: https://github.com/prometheus/prometheus/issues/586

We've a slightly different computational model, so that takes two passes.

You can calculate the highest value now (topk) or the highest averages (topk+avg_over_time) and now that you know which timeseries you want, graph those. I believe this is doable in Grafana.

We are! And unsurprising it's a great fit given k8s is inspired by Borg and Prometheus is inspired by Borgmon.

There's a few, here's a recent talk by Weave for example: http://www.slideshare.net/weaveworks/kubernetes-and-promethe...

Here is the CoreOS blog on using it: https://coreos.com/blog/prometheus-and-kubernetes-up-and-run...

Yes, it works wonderfully. I published a Grafana dashboard for this here - https://grafana.net/dashboards/162

There is an project we've just started: https://docs.google.com/document/d/1C7yhMnb1x2sfeoe45f4mnnKC...

Hopefully have something to show for promcon.

Usually you start by some functional sharding (giving each team of services their own Prometheus servers), but also having per-datacenter Prometheus servers and then some hierarchical federation layer ontop of that. There's no built-in automatic horizontal scaling though (which would be going against the design goal of not having a clustered system, for reliability).

Some resources:

- Scaling in general: http://www.robustperception.io/scaling-and-federating-promet...

- Federation: https://prometheus.io/docs/operating/federation/

http://www.robustperception.io/scaling-and-federating-promet... explains it. Unless you're absolutely massive, it's fairly easy.

Datadog and SignalFX are both far more scalable, easier to use, and have more features than Prometheus and are SaaS offerings. Prometheus is about 5 years behind them in terms of engineering effort, I believe.

Not yet, but one or more companies are working on this as part of their offerings in the future. Stay tuned!

Note that a hosted Prometheus service has different design tradeoffs. As Brian mentioned, Prometheus as it is is really meant to be run as close as possible to your monitored services for maximum reliability. Also, there's no clustered and long-term durable storage for a similar reason, which will likely be different in hosted versions.

There's noone presently offering that, it's designed more with a view to being on-prem for reliability (http://www.robustperception.io/monitoring-without-consensus/).

Not quite, though there is some overlap. Someone else asked me to compare Prometheus to other tools in the APM (application performance monitoring) space, and I'm going to share the summary I came up with:

The way I would describe Prometheus in relation to those other tools:

- Prometheus is open-source and self-hosted.

- Prometheus is about dimensional numeric time series metrics only (no log-based analysis, no per-request tracing, etc.).

- Prometheus has a strong focus on systems and service monitoring, not so much on business metrics.

- Prometheus is more of a Swiss army knife of monitoring rather than a ready-to-drop-in package that starts monitoring everything automatically.

- Prometheus is very much about whitebox monitoring and manually defining any metrics that could be useful for you (although we support blackbox exporting and bridging metrics from existing systems as well).

- We don't do machine-learning-style anomaly detection, but we do alerting based on manually defined rules.

- For a purely metrics-based solution, the insight we deliver is one of the best in the field (via the dimensional data model and the query language to go with it).

- Many open-source projects are starting to expose native Prometheus metrics (like k8s, etcd, ...), which gives Prometheus an advantage when being used together with those.

EDIT: Also try the "Getting Started" tutorial - that should only take a couple of minutes to try it out: https://prometheus.io/docs/introduction/getting_started/

I'm not sure there is a typical user. We've got everything from single-sysadmin small companies to Fortune 500s to companies looking to integrate it into their products.

https://prometheus.io/blog/2016/03/23/interview-with-life360... and https://prometheus.io/blog/2016/05/01/interview-with-showmax... look at two of our users.

Yeah, it goes from people putting Prometheus on their Raspberry PI all the way to companies like DigitalOcean monitoring millions of machines with Prometheus (https://promcon.io/talks/scaling_to_a_million_machines_with_...).

A lot of open-source projects (Kubernetes, Etcd, ...) are also exposing native Prometheus metrics now, making it easier to integrate with those.

EDIT: also check the PromCon schedule for other user companies giving talks: https://promcon.io/schedule/

And the sponsors are also users: https://promcon.io/#our-sponsors

Our focus is not logging only, it's the opposite: we don't support storing logs of individual events. What Prometheus does is store dimensional numerical time series. See the data model: https://prometheus.io/docs/concepts/data_model/

So it's a question of whether you can squeeze your data into that model and whether you need per-event details or whether aggregated time series are ok.