Skip to content

In case of database lock or low performance prometheus-exporer accumulates connections to the database #1228

@pierresouchay

Description

@pierresouchay

What did you do?

  1. Have postgres-exporter running
  2. Have a kind of lock to the database -> some requests of postgres-exporter are locked (for instance, on a WAL saturated instance, a big DELETE locking the stats)
  3. Have a prometheus server fetching the data

What did you expect to see?

  1. one or a few connections connection opened to the server
  2. Some errors of scrapping maybe

What did you see instead? Under which circumstances?

  1. On the database, a huge perf issue arise (likely due to dbt)
  2. postgres-exporter accumulates postgresql queries stuck
  3. Since we did not cap enough the user, it accumulates more than 50% of available connections (in an example, 53 open connections (fair enough, I should have used a user with less privileges, will do), avoiding other clients to connect

I would expect the connections to be canceled at some point (for instance when the prometheus query times out.

Example:

Image

We have 53 queries in flight with the user...

Environment

  • System information:

In the container:

uname -srm
Linux 5.15.0-161-generic x86_64
  • postgres_exporter version:

Using docker image prometheuscommunity/postgres-exporter:v0.17.1:

postgres_exporter --version
postgres_exporter, version 0.17.1 (branch: HEAD, revision: 1e574cf4fd2a75a8a707d424eafcaa0b88cb7af4)
  build user:       root@b0ba132eddb0
  build date:       20250226-13:53:15
  go version:       go1.23.6
  platform:         linux/amd64
  tags:             unknown
  • postgres_exporter flags:

None set, using the default ones only

  • PostgreSQL version: 17.1

  • Logs:

insert logs relevant to the issue here

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions