Repository navigation
Conversation
BarthV
requested review from
Buzzglo,
TebogoTS,
ampie,
januarios,
k0da and
kuritka
as code owners
July 10, 2026 16:36
Author
|
Hi ! This PR is 2 of the 2 (the other is #67 ) required to mitigate/lower the risk of leaving orphaned A / TXT records on infoblox. The code is heavily claude reviewed. I've benchtested / stressed both of them in lab and I'm now running them in production. It works as expected, only a benign risk of lone TXT still persist, but cannot be solved because of the atomic WAPI call design (non transactional). Feel free to ask questions or raise any remark. |
Collaborator
|
@BarthV Hi could you please sign a commit before we can proceed with merge? |
kuritka
approved these changes
Jul 27, 2026
BarthV
force-pushed
the
fix/apply-changes-ownership-ordering
branch
from
September 22, 2026 21:41
6437555 to
b3ec7d2
Compare
external-dns pairs every managed record with a registry TXT carrying the owner label, and submitChanges applies the two as independent WAPI calls with no transaction. It ordered the data record first on create, so a batch that stopped part-way (pod restart, OOM, write timeout, or a failed call) could leave a live data record with no ownership TXT: one external-dns can neither adopt nor delete (logged forever as "missing owner label"), silently accumulating in the grid. Order each batch so the ownership TXT is created before its data record and deleted after it, so the half that survives a partial apply is at worst a harmless TXT-without-data record (no live resolution, reclaimable by the registry) instead of a live orphaned record. Make that guarantee hold end to end rather than only within a single zone on an interrupted run: - Apply the whole batch across all zones in one ordered pass. submitChanges previously applied per-zone buckets in Go map (random) order, so the ordering held only within a zone; a data record and its ownership TXT that resolve to different zones (the createPTR reverse zone, or an apex record of a delegated sub-zone whose type-prefixed TXT resolves to the parent) could still be applied in the wrong order. - Abort on the first error instead of collecting errors and continuing. Best-effort application would create a data record after its ownership TXT create failed (or delete a TXT after its data delete failed), reintroducing the orphan the ordering prevents. Unreached changes are retried on the next reconcile; changes already applied are current and are not redone. - Order by partitioning changes into ownership-TXT creates, data creates, updates, data deletes and ownership-TXT deletes instead of a numeric rank, keeping the rule explicit and independent of the incoming order. The one case left for external-dns to reconcile is a TXT source record, whose data TXT and ownership TXT are both TXT records and so cannot be ordered relative to each other. Add tests for the create/delete ordering, the cross-zone ordering, and the abort-on-failed-ownership-TXT path.
BarthV
force-pushed
the
fix/apply-changes-ownership-ordering
branch
from
September 22, 2026 21:42
b3ec7d2 to
9026616
Compare
Author
Author
|
rebased & fixed conflicts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
external-dns pairs every managed record with a registry TXT carrying the owner label, and submitChanges applies the two as independent WAPI calls with no transaction. It ordered the data record first on create, so a batch that stopped part-way (pod restart, OOM, write timeout, or a failed call) could leave a live data record with no ownership TXT: one external-dns can neither adopt nor delete (logged forever as "missing owner label"), silently accumulating in the grid.
Order each batch so the ownership TXT is created before its data record and deleted after it, so the half that survives a partial apply is at worst a harmless TXT-without-data record (no live resolution, reclaimable by the registry) instead of a live orphaned record.
Make that guarantee hold end to end rather than only within a single zone on an interrupted run:
Apply the whole batch across all zones in one ordered pass. submitChanges previously applied per-zone buckets in Go map (random) order, so the ordering held only within a zone; a data record and its ownership TXT that resolve to different zones (the createPTR reverse zone, or an apex record of a delegated sub-zone whose type-prefixed TXT resolves to the parent) could still be applied in the wrong order.
Abort on the first error instead of collecting errors and continuing. Best-effort application would create a data record after its ownership TXT create failed (or delete a TXT after its data delete failed), reintroducing the orphan the ordering prevents. Unreached changes are retried on the next reconcile; changes already applied are current and are not redone.
Order by partitioning changes into ownership-TXT creates, data creates, updates, data deletes and ownership-TXT deletes instead of a numeric rank, keeping the rule explicit and independent of the incoming order.
The one case left for external-dns to reconcile is a TXT source record, whose data TXT and ownership TXT are both TXT records and so cannot be ordered relative to each other.
Add tests for the create/delete ordering, the cross-zone ordering, and the abort-on-failed-ownership-TXT path.