Skip to content

fix: create ownership TXT before data record, delete it after - #66

Open
BarthV wants to merge 2 commits into
AbsaOSS:mainfrom
qube-rt:fix/apply-changes-ownership-ordering
Open

BarthV wants to merge 2 commits into
AbsaOSS:mainfrom
qube-rt:fix/apply-changes-ownership-ordering

Conversation

@BarthV

@BarthV BarthV commented Jul 10, 2026

Copy link
Copy Markdown

external-dns pairs every managed record with a registry TXT carrying the owner label, and submitChanges applies the two as independent WAPI calls with no transaction. It ordered the data record first on create, so a batch that stopped part-way (pod restart, OOM, write timeout, or a failed call) could leave a live data record with no ownership TXT: one external-dns can neither adopt nor delete (logged forever as "missing owner label"), silently accumulating in the grid.

Order each batch so the ownership TXT is created before its data record and deleted after it, so the half that survives a partial apply is at worst a harmless TXT-without-data record (no live resolution, reclaimable by the registry) instead of a live orphaned record.

Make that guarantee hold end to end rather than only within a single zone on an interrupted run:

  • Apply the whole batch across all zones in one ordered pass. submitChanges previously applied per-zone buckets in Go map (random) order, so the ordering held only within a zone; a data record and its ownership TXT that resolve to different zones (the createPTR reverse zone, or an apex record of a delegated sub-zone whose type-prefixed TXT resolves to the parent) could still be applied in the wrong order.

  • Abort on the first error instead of collecting errors and continuing. Best-effort application would create a data record after its ownership TXT create failed (or delete a TXT after its data delete failed), reintroducing the orphan the ordering prevents. Unreached changes are retried on the next reconcile; changes already applied are current and are not redone.

  • Order by partitioning changes into ownership-TXT creates, data creates, updates, data deletes and ownership-TXT deletes instead of a numeric rank, keeping the rule explicit and independent of the incoming order.

The one case left for external-dns to reconcile is a TXT source record, whose data TXT and ownership TXT are both TXT records and so cannot be ordered relative to each other.

Add tests for the create/delete ordering, the cross-zone ordering, and the abort-on-failed-ownership-TXT path.

@BarthV

BarthV commented Jul 10, 2026

Copy link
Copy Markdown
Author

Hi !

This PR is 2 of the 2 (the other is #67 ) required to mitigate/lower the risk of leaving orphaned A / TXT records on infoblox. The code is heavily claude reviewed.

I've benchtested / stressed both of them in lab and I'm now running them in production. It works as expected, only a benign risk of lone TXT still persist, but cannot be solved because of the atomic WAPI call design (non transactional).

Feel free to ask questions or raise any remark.
Thanks

@k0da k0da left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@k0da

k0da commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

@BarthV Hi could you please sign a commit before we can proceed with merge?

@BarthV
BarthV force-pushed the fix/apply-changes-ownership-ordering branch from 6437555 to b3ec7d2 Compare September 22, 2026 21:41
external-dns pairs every managed record with a registry TXT carrying the
owner label, and submitChanges applies the two as independent WAPI calls with
no transaction. It ordered the data record first on create, so a batch that
stopped part-way (pod restart, OOM, write timeout, or a failed call) could
leave a live data record with no ownership TXT: one external-dns can neither
adopt nor delete (logged forever as "missing owner label"), silently
accumulating in the grid.

Order each batch so the ownership TXT is created before its data record and
deleted after it, so the half that survives a partial apply is at worst a
harmless TXT-without-data record (no live resolution, reclaimable by the
registry) instead of a live orphaned record.

Make that guarantee hold end to end rather than only within a single zone on
an interrupted run:

- Apply the whole batch across all zones in one ordered pass. submitChanges
  previously applied per-zone buckets in Go map (random) order, so the
  ordering held only within a zone; a data record and its ownership TXT that
  resolve to different zones (the createPTR reverse zone, or an apex record of
  a delegated sub-zone whose type-prefixed TXT resolves to the parent) could
  still be applied in the wrong order.

- Abort on the first error instead of collecting errors and continuing.
  Best-effort application would create a data record after its ownership TXT
  create failed (or delete a TXT after its data delete failed), reintroducing
  the orphan the ordering prevents. Unreached changes are retried on the next
  reconcile; changes already applied are current and are not redone.

- Order by partitioning changes into ownership-TXT creates, data creates,
  updates, data deletes and ownership-TXT deletes instead of a numeric rank,
  keeping the rule explicit and independent of the incoming order.

The one case left for external-dns to reconcile is a TXT source record, whose
data TXT and ownership TXT are both TXT records and so cannot be ordered
relative to each other.

Add tests for the create/delete ordering, the cross-zone ordering, and the
abort-on-failed-ownership-TXT path.
@BarthV
BarthV force-pushed the fix/apply-changes-ownership-ordering branch from b3ec7d2 to 9026616 Compare September 22, 2026 21:42
@BarthV

BarthV commented Sep 22, 2026

Copy link
Copy Markdown
Author

@k0da @kuritka
Rebased & signed (sorry for the huge lag)

@BarthV

BarthV commented Oct 27, 2026

Copy link
Copy Markdown
Author

rebased & fixed conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants