Repository navigation
Commit db2f8f0
fix(kafka): decode Confluent Protobuf message indexes as zigzag varints (#8670)
* fix(kafka): decode Confluent Protobuf message indexes as zigzag varints
Confluent's Protobuf serializer writes the message index array as zigzag
encoded varints (array length, then each index), with [0] as a single 0x00
byte. ProtobufDeserializer read it with _DecodeSignedVarint, so it read
twice the real length and consumed payload bytes as indexes. Records for
any message type other than the first in the schema either failed to
decode or decoded with fields silently missing.
Decode the array as zigzag varints. Tighten
test_confluent_schema_registry_complex_schema to assert userId and name,
which the 04 02 00 fixture record was losing, and add a parametrised test
for indexes [0], [1], [2], [1, 0] and a multi-byte index.
Closes #8665
* test(kafka): cover an invalid Confluent message index array length
---------
Co-authored-by: Leandro Damascena <lcdama@amazon.pt>1 parent 07f2a7a commit db2f8f0
2 files changed
Lines changed: 121 additions & 12 deletions
File tree
- aws_lambda_powertools/utilities/kafka/deserializer
- tests/functional/kafka_consumer/_protobuf
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
106 | 110 | | |
107 | 111 | | |
108 | | - | |
109 | 112 | | |
110 | 113 | | |
111 | 114 | | |
112 | | - | |
113 | | - | |
114 | | - | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
115 | 119 | | |
116 | 120 | | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
| 121 | + | |
| 122 | + | |
121 | 123 | | |
122 | 124 | | |
123 | 125 | | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
357 | 357 | | |
358 | 358 | | |
359 | 359 | | |
360 | | - | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
361 | 366 | | |
362 | 367 | | |
363 | 368 | | |
364 | 369 | | |
365 | 370 | | |
366 | 371 | | |
367 | 372 | | |
368 | | - | |
369 | | - | |
| 373 | + | |
| 374 | + | |
370 | 375 | | |
371 | 376 | | |
| 377 | + | |
372 | 378 | | |
373 | 379 | | |
374 | 380 | | |
| |||
429 | 435 | | |
430 | 436 | | |
431 | 437 | | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
0 commit comments