How to set up Prometheus for Kubernetes monitoring

Kubernetes helps IT teams manage distributed applications in containers, but it also introduces new challenges — such as monitoring. To properly monitor the dynamic components in a containerized ecosystem, teams need tools that deploy and scale natively in Kubernetes.To use Prometheus, administrators need to understand its client libraries, the components in the kube-prometheus project and other technologies, such as Operators in Kubernetes. Then, they can install and run Prometheus for Kubernetes monitoring with the steps presented here, relying on the Helm package manager and kubectl command line.

Prometheus overview

Prometheus is an open source event monitoring and alerting application. Originally developed at SoundCloud, Prometheus was donated to the Cloud Native Computing Foundation (CNCF) in 2016 and is now hosted on GitHub as a standalone independent project. It captures real-time metrics in a time-series database, and provides a multi-dimensional data model and a flexible query language. Prometheus uses the pull method to collect metrics over HTTP, though users can implement push-based monitoring for limited use cases.

IT administrators can use Prometheus for Kubernetes monitoring — but first, they must add the necessary instrumentation to their code.

To do this, use one of the Prometheus client libraries that implements various types of metrics. Choose the client library that matches the primary application development language. This enables developers to define metrics and expose them on the application instance through HTTP endpoints.

Some of the most commonly used client libraries to write and expose monitoring metrics include Go, Java or Scala, Python and Ruby.

In addition, unofficial third-party client libraries include Bash, C, C++, Elixir, PHP and Rust.

Prometheus integrates with other applications as well, such MySQL, Amazon CloudWatch and Kubernetes, through official and third-party exporters.

The Prometheus Operator and kube-prometheus project

An Operator in Kubernetes is a domain- and app-specific controller that acts as an extension of the Kubernetes API to manage application instances. The Prometheus Operator for Kubernetes provides a way to build, configure and manage Prometheus clusters on Kubernetes.

The kube-prometheus project is for cluster monitoring and is configured to gather metrics from Kubernetes components. It also has capabilities for dashboards and alerting rules. The project is open sourced on GitHub and combines Kubernetes manifest files, preconfigured Grafana dashboards, and Prometheus rules to deliver an abstraction layer and Kubernetes cluster monitoring capabilities via the Prometheus Operator.

The following components are included in the kube-prometheus package, as noted on GitHub: