Guides
Connect Prometheus
Let the agents read what your servers were doing, from Prometheus, Grafana Cloud, VictoriaMetrics or Mimir. They read CPU, memory, disk, the database, the web server and alerts, and can change nothing.
After this guide, the agents see what your servers were doing when they look at your numbers and your errors. A review can then name a server cause, with the number and the time. For example: MySQL reached its maximum of connections at 03:05, and the errors started one minute later.
Use this guide when your servers' numbers are in Prometheus, or in a store that answers like it: Grafana Cloud, VictoriaMetrics or Mimir. When your servers run on AWS, you can also connect AWS CloudWatch. The agents read whichever you connect, or both.
Who can do it:
- A key for every product: a workspace admin or owner.
- A key for one product: a workspace admin or owner, or a person who may change the product's own keys and tools.
See Roles.
What the agents read
Prometheus keeps the numbers that small programs on your servers report. These programs are called exporters. The agents read the numbers of these exporters, when they find them:
| Exporter | What they read |
|---|---|
| node_exporter, on each server | CPU, time spent waiting for the disk, CPU taken by other machines on the same host, load against the number of cores, memory available, swap, how full each disk is, files left on each disk, how busy each disk is, processes the system stopped because memory ran out |
| cAdvisor, beside your containers | CPU and memory of each container against its limit, containers stopped for memory, restarts |
| mysqld_exporter, beside MySQL or MariaDB | Connections against the maximum, slow queries, queries running |
| postgres_exporter, beside PostgreSQL | Connections against the maximum |
| A PHP-FPM exporter | Busy workers against all workers, and how often all of them were busy |
| The NGINX exporter | Connections and requests a second |
| Alerts | The alerts firing now |
For each server, container or database, the agents see the average and the peak over the week, and the time of the peak. They also see which exporters are missing, and which servers stopped reporting. With a question, they can read any other number your store has.
Three places use them:
- The Product analytics review compares the last week with the week before. It looks for a server near a limit, and checks a slow page or a drop in your numbers against the servers at the same hours.
- Error triage reads the servers' numbers around the first errors, when an error looks like the servers' own: a timeout, a lost connection, out of memory.
- In Chat, you can ask about them. For example: "How busy was the database last night?"
The connection is read-only, and the agents never see its token. The platform keeps none of these numbers. Your store keeps them, and the agents read them again each time.
1. Install the exporters
Install the exporters for what you run. Each one has its own guide:
- node_exporter on each server. Without it, the agents see no CPU, memory or disk.
- cAdvisor, when you run containers.
- mysqld_exporter or postgres_exporter, beside your database.
- A PHP-FPM exporter and the NGINX exporter, beside your web server.
Then let Prometheus, or Grafana's agent, collect their numbers. When a number is missing, a review says which exporter to install.
2. Give the agents an address and a token
The platform reads your store over the internet. So the store needs an https address that the internet can reach, with a token. The platform calls no address on a private network, and follows no redirect. Choose the case that is yours.
Your own Prometheus
Prometheus on your own server has no token by itself. Put it behind a web server that asks for one, such as nginx with https:
- Give Prometheus an https address, such as
https://prometheus.example.com, with a certificate that browsers trust. - In the web server, let a request through only with the header
Authorization: Bearer <your token>. Make the token long and random. - Let only reads through, if you can: the paths under
/api/v1/that start withquery,label,series,alertsandrules.
The address is the part before /api/v1. VictoriaMetrics and Mimir work the same way, at their own address for the query API.
Grafana Cloud
- In Grafana Cloud, open your stack, then Prometheus → Details. Copy the URL, such as
https://prometheus-prod-01-eu-west-0.grafana.net/api/prom, and the user, a number. - Open Access policies, and make a policy with the scope
metrics:readonly. Make a token for it.
Grafana Cloud signs in with a username and a password: the user is the number, and the password is the token.
3. Connect it
- In the app, open Settings → Integrations. In the group Your numbers and errors, press Connect on the row Prometheus.
- Paste the key into Address and token, in one of these forms:
{"address": "https://prometheus.example.com", "token": "..."}
{"address": "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom", "username": "123456", "password": "glc_..."}
- Choose the place under Use it for, and press Save and test.
The test asks the store for one number. It says whether the store answered, refused the token, or is not a Prometheus address.
If the test says that the address points to a private or local address, the platform cannot reach your store. Put it behind an address on the internet, as in step 2, or use a hosted store such as Grafana Cloud.
What you see when it worked
- The row Prometheus says Connected, and Working after its test.
- In Chat, ask: "What were the servers doing in the last week?" The answer names your servers and their numbers.
- The next report of the Product analytics review has a part about the servers.
More
- Connect AWS CloudWatch: your servers' numbers from AWS.
- Connect analytics and Sentry: the reviews that read these numbers.
- Ask in chat.
- Integrations: every integration and where its key can be.
- When something does not work.