Skip to content

Kafka Headers#3014

Merged
iancooper merged 6 commits into
masterfrom
guid_in_header
Feb 19, 2024
Merged

Kafka Headers#3014
iancooper merged 6 commits into
masterfrom
guid_in_header

Conversation

@iancooper

Copy link
Copy Markdown
Member

Our implementation of a header builder for Kafka lacked tests and had bugs around GUIDs. Adding tests revealed naive thinking about translating headers to and from Kafka.

  • A Kafka header a string key and a byte[] value
  • If it is a Brighter header, we can convert a type to a byte[] and back.
  • If it a user defined header (the bag) we can at best convert the type to a string and byte[] and then back to a string
    • We would need to know the serialized type to convert back into the same type
    • Adding type information has implications for security if implemented (see retirement of BinaryFormatter)
    • We can in this case expect the consumer to convert back from a string to a type they know

We used unix timestamps for our own Header timestamp. There is no need to do this, and it simplifies not tol

@holytshirt holytshirt changed the title Kakfa Headers Kafka Headers Feb 13, 2024
@iancooper iancooper merged commit afa2d92 into master Feb 19, 2024
@iancooper iancooper deleted the guid_in_header branch March 27, 2024 08:17
DevJonny pushed a commit to DevJonny/Brighter that referenced this pull request Feb 28, 2026
* create tests for kafka headers:

* Test serialization of Kafka header; fix issue with Dynamo GUID converter.

* Fix up how we deal with Kafka headers. Simplify timestamps. Treat bag value as a string.

* comment back in time test post change away from unix milliseconds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant