Run Prometheus on a different port, optionally. by hawkowl · Pull Request #3274 · matrix-org/synapse

hawkowl · 2018-05-23T18:50:46Z

This should allow us to be resistant to Twisted melting down meaning we don't get metrics, in theory.

hawkowl · 2018-05-23T18:58:14Z

@erikjohnston this is based off the other PR, so will become much smaller when that one merges :)

erikjohnston · 2018-05-24T13:43:13Z

I think my preference would be to make it configurable via the listeners config rather than reusing the old metrics_port config options. Maybe something like:

listeners:
- port: 9000
  type: metrics

Also, we need to update each component rather than just the main one, i.e. everything else in synapse/app/* (yes, the duplication there makes me sad too)

turt2live · 2018-05-24T14:30:28Z

If the configuration changes, please yell loudly in the changelog too 😇

erikjohnston · 2018-05-24T14:33:12Z

@turt2live I imagine we'd still support hosting the metrics endpoint in the twisted reactor as well, so everything should be backwards compat :)

…port

hawkowl · 2018-05-28T13:45:58Z

@erikjohnston New docs, added the config to all the apps, listeners config added.

erikjohnston

Woo, looks good! Just a couple of comments.

erikjohnston · 2018-05-29T10:41:38Z

+
+    for host in bind_addresses:
+        reactor.callInThread(start_http_server, int(port),
+                             addr=host, registry=RegistryProxy)


start_http_server should spawn its own thread, so is it necessary to callInThread?

I had issues with it jamming Twisted, for some reason, and this made it work fine. ~~the mysteries of threading~~

erikjohnston · 2018-05-29T10:42:52Z

-                        "compress": False,
-                    },
-                ]
-            })


Do we not need to keep this for backwards compat for people using metrics_port?

…port

hawkowl · 2018-05-31T05:07:36Z

@erikjohnston I put back the old metrics_port, and raised a warning saying it's deprecated, with a link to the new docs. I couldn't find any better globally accessible link than just the file in the repo, though...

richvdh · 2018-06-04T16:25:27Z

+       - type: metrics
+         port: 9000
+         bind_addresses:
+           - '0.0.0.0'


why is this just IPv4 rather than an IPv6 address too? (I see f551ac3 changed this, but the commit comment is, ahem, unenlightening.)

SocketServer (from the stdlib that Prometheus Client uses) only supports IPv4 out of the box, so it refuses IPv6 addresses. :(

richvdh · 2018-06-04T16:28:34Z

+
+All time duration-based metrics have been changed to be seconds. This affects:
+
+================================


this table doesn't appear to be working. Please take a look at the rendered version: https://github.com/matrix-org/synapse/blob/develop/docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310

@thegcat

Changes in synapse v0.31.0 (2018-06-06) ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!

hawkowl added 3 commits May 23, 2018 13:42

serve metrics on a different thread

47ac7e3

kick off the metrics server, in another thread.

40c2d35

pep8

e627216

hawkowl requested review from erikjohnston and removed request for erikjohnston May 23, 2018 18:57

hawkowl assigned erikjohnston May 23, 2018

erikjohnston assigned hawkowl and unassigned erikjohnston May 24, 2018

hawkowl added 8 commits May 28, 2018 22:39

Merge branch '3218-official-prom' into hawkowl/prom-diff-port

6c5c7d3

add some docs on the metrics changes

81bc4ce

update docs

b1029fd

update listener config

183f186

add listening for metrics to all the apps

98c0c71

just have quad zero

f551ac3

Merge remote-tracking branch 'origin/develop' into hawkowl/prom-diff-…

8abceb3

…port

fix pep8 :)

c157c59

hawkowl assigned erikjohnston and unassigned hawkowl May 28, 2018

richvdh mentioned this pull request May 29, 2018

Switch to the Python Prometheus library #3256

Merged

erikjohnston reviewed May 29, 2018

View reviewed changes

erikjohnston assigned hawkowl and unassigned erikjohnston May 29, 2018

Merge remote-tracking branch 'origin/develop' into hawkowl/prom-diff-…

625110b

…port

hawkowl added 3 commits May 31, 2018 13:22

put back the old metrics port option for compatibility

644b62d

add a warning

357033c

make this 0.31.0

ce9b112

hawkowl assigned erikjohnston and unassigned hawkowl May 31, 2018

erikjohnston approved these changes May 31, 2018

View reviewed changes

hawkowl merged commit febe0ec into develop May 31, 2018

hawkowl deleted the hawkowl/prom-diff-port branch May 31, 2018 09:05

richvdh reviewed Jun 4, 2018

View reviewed changes

richvdh mentioned this pull request Jun 5, 2018

Fix metric documentation tables #3341

Merged

MadLittleMods mentioned this pull request Jul 15, 2025

Remove metrics listener type in favor of http listener with metrics resource element-hq/synapse#18584

Closed

4 tasks done


		All time duration-based metrics have been changed to be seconds. This affects:

		================================

Conversation

hawkowl commented May 23, 2018

Uh oh!

hawkowl commented May 23, 2018

Uh oh!

erikjohnston commented May 24, 2018

Uh oh!

turt2live commented May 24, 2018

Uh oh!

erikjohnston commented May 24, 2018

Uh oh!

hawkowl commented May 28, 2018

Uh oh!

erikjohnston left a comment

Choose a reason for hiding this comment

Uh oh!

erikjohnston May 29, 2018

Choose a reason for hiding this comment

Uh oh!

hawkowl May 31, 2018

Choose a reason for hiding this comment

Uh oh!

erikjohnston May 29, 2018

Choose a reason for hiding this comment

Uh oh!

hawkowl commented May 31, 2018

Uh oh!

richvdh Jun 4, 2018

Choose a reason for hiding this comment

Uh oh!

hawkowl Jun 5, 2018

Choose a reason for hiding this comment

Uh oh!

richvdh Jun 5, 2018

Choose a reason for hiding this comment

Uh oh!

richvdh Jun 4, 2018

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants