Repository navigation
openam agent chokes openAM with create session requests after openAM pod upgrade #303
Description
Describe the bug
Upgrade openAM/openDJ pods. After some time, agents configured on application pods start to flood openAM with create token requests.
Only way to stop this seems to restart application pod.
This problem seems not to affect those pods that are restarted after openAM/openDJ upgrade.
In activity.csv log hundreds of lines like the following related to the agent of the offending pod without any attempt of DELETE:
66442ec3ce5601""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"5f1666442ec3ce5601";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153366";"2020-08-19T21:41:28.643Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153362";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""44a4
6633afba238601""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"44a46633afba238601";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153388";"2020-08-19T21:41:29.019Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153384";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""19d6
4d01263615c201""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"19d64d01263615c201";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153410";"2020-08-19T21:41:29.415Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153406";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""e1b3
a72b4b1daaf101""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"e1b3a72b4b1daaf101";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153432";"2020-08-19T21:41:29.823Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153428";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""af3f
4d9a4616746001""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"af3f4d9a4616746001";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153454";"2020-08-19T21:41:30.316Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153450";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""dcab
079bcbcd3c2401""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"dcab079bcbcd3c2401";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153476";"2020-08-19T21:41:30.686Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153472";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""ee57
87498fd0026801""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"ee5787498fd0026801";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153498";"2020-08-19T21:41:31.102Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153494";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""24e7
306aa792ae4801""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"24e7306aa792ae4801";"CREATE";;;;;"Session";"/"
In openDJ situation reflects the log above (snippet is taken from another syste in a moment where other application were acting the same on activity.csv) - some of the offending pod were already been restarted:
root@ems-opendj-0:/opt# /opt/opendj/bin/ldapsearch -h localhost -p 1389 -D 'cn=Directory Manager' -w 'XXXXXXXXX' -b 'ou=tokens,dc=openam,dc=forgerock,dc=org' -s sub '(objectClass=*)' coreTokenUserId | grep coreTokenUserId | sort | uniq -c
1 coreTokenUserId: id=barb,ou=user,dc=openam,dc=forgerock,dc=org
26 coreTokenUserId: id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org
2 coreTokenUserId: id=esaportal55b4d89756ff69f,ou=agent,dc=openam,dc=forgerock,dc=org
1 coreTokenUserId: id=ewpauditf485f59949lhtj,ou=agent,dc=openam,dc=forgerock,dc=org
2 coreTokenUserId: id=ewpdashboards667f49f499sc55h,ou=agent,dc=openam,dc=forgerock,dc=org
3701 coreTokenUserId: id=ewpdashboards667f49f499spzkj,ou=agent,dc=openam,dc=forgerock,dc=org
1 coreTokenUserId: id=ewpdpa558b885c989s2zz,ou=agent,dc=openam,dc=forgerock,dc=org
1556 coreTokenUserId: id=ewpisrestapi74cc766dbfhs789,ou=agent,dc=openam,dc=forgerock,dc=org
2 coreTokenUserId: id=ewpisrestapi74cc766dbfvsgqn,ou=agent,dc=openam,dc=forgerock,dc=org
1 coreTokenUserId: id=ewpmanagement587dc77b6d2td75,ou=agent,dc=openam,dc=forgerock,dc=org
1 coreTokenUserId: id=ewpmdm9bcfdb7bcd888m,ou=agent,dc=openam,dc=forgerock,dc=org
2 coreTokenUserId: id=ewpmediationd7b8cf47fb5kpc,ou=agent,dc=openam,dc=forgerock,dc=org
4262 coreTokenUserId: id=ewpmediationd7b8cf47fngbnc,ou=agent,dc=openam,dc=forgerock,dc=org
1 coreTokenUserId: id=ewppcap67d98c648c6t6rz,ou=agent,dc=openam,dc=forgerock,dc=org
2 coreTokenUserId: id=ewptracec7998549626z2z,ou=agent,dc=openam,dc=forgerock,dc=org
A workaround that seems to work is to restart all application pods after openAM/openDJ upgrade.
To Reproduce
Steps to reproduce the behavior:
- Upgrade openAM/openDJ pods
- Access an application whose pod has not been restarted after openAM/openDJ. Its agent will start (sometimes after a few hours of usage) to flood openAM with CREATE-SESSION requests.
- Openam pod becomes unhealty showing "no healty upstreams". Liveness check on pod fails
Expected behavior
We expect that agents renews session properly without flooding openam without the need of application pod restart
Desktop (please complete the following information):
- this seems unrelated to browser/system used
Additional context
- dockers running on kubernetes environment in a single namespace
- openam agent version 4.1.4
Metadata
Metadata
Assignees
Labels
No labels
Activity
vharseko commentedon Dec 11, 2023
please provide
CVE-2026-112942 CVE-2026-112943 CVE-2026-112944 CVE-2026-112945 CVE-2…