`dictlist` is using `list` instead of `dict`

[janstarke]

My Testcode:

import sys
from flow.record import RecordDescriptor, RecordStreamWriter

TestRecord1 = RecordDescriptor(
    "test/csv_test1",
    [
        ("string", "field11"),
        ("string", "field12"),
        ("string", "field13"),
        ("dictlist", "field14")
    ]
)

rec1 = TestRecord1(
    field11 = "AB",
    field12 = "CD",
    field13 = "EF",
    field14 = {
        "Name": "George",
        "Surname": "Washington"
    }
)

record_writer = RecordStreamWriter(sys.stdout.buffer)

record_writer.write(rec1)

How to run the code:

python3 tests/test-records.py | rdump

Expected output:

[reading from stdin]
<test/csv_test1 field11='AB' field12='CD' field13='EF' field14={'Name': 'George', 'Surname': 'Washington'}>

Actual output:

[reading from stdin]
<test/csv_test1 field11='AB' field12='CD' field13='EF' field14=['Name', 'Surname']>

Suggested Fix

Change to code in

flow.record/flow/record/fieldtypes/__init__.py

Line 195 in f2606d1

class dictlist(list, FieldType):

to

class dictlist(dict, FieldType):

[yunzheng]

Hi @janstarke,

The dictlist type was meant to be a list of dicts, eg: list[dict]. This is also seen in our tests:

def test_dictlist_packer() -> None:
    packer = RecordPacker()
    TestRecord = RecordDescriptor(
        "test/dictlist",
        [
            ("dictlist", "hits"),
        ],
    )

    r1 = TestRecord([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
    data = packer.pack(r1)
    r2 = packer.unpack(data)

    assert len(r1.hits) == 2
    assert r1.hits == [{"a": 1, "b": 2}, {"a": 3, "b": 4}]
    assert r1.hits[0]["a"] == 1
    assert r1.hits[0]["b"] == 2
    assert r1.hits[1]["a"] == 3
    assert r1.hits[1]["b"] == 4

Note that some types (including dictlist) were added as a quickfix for something we needed before we open-sourced flow.record, meaning that those types have not been widely adopted/tested.

We might consider deprecating and removing those types.