Add read-only support for Visual FoxPro compound index (.cdx) files

[chrisrichards]

Summary

Adds a DbfDataReader.Cdx namespace that opens Visual FoxPro compound index files, enumerates their named tag indexes, walks entries in key order, and searches for keys. Search results carry the DBF record number, and CdxKeyEntry.RecordIndex converts it to the zero-based index that DbfTable.Seek/DbfDataReader.Seek (#283) expect — so an index hit becomes a row in two lines:

using var dbfTable = new DbfTable(dbfPath);
using var cdxFile = new CdxFile(cdxPath, dbfTable.CurrentEncoding);

var index = cdxFile.GetIndex("CONTACT_ID");
foreach (var entry in index.Search("C0000000042"))
{
    dbfTable.Seek(entry.RecordIndex);
    dbfTable.Read(dbfRecord);
}

Credit

The CDX format handling is ported from the dev/async fork by Dai Rees (@daiplusplus) (MIT), who reverse-engineered the hard parts of this format in 2017–2018:

• the compound-file tag directory (a root node whose keys are tag names and whose record numbers are file offsets of the per-tag index headers);
• interior node entries — each key followed by two big-endian UInt32s (record number + child node pointer), where Microsoft's documentation incorrectly describes the old IDX "4 hex characters" layout;
• the bit-packed leaf format: per-node bit widths and masks for [record number][duplicate count][trailing count], with keys reconstructed via prefix compression (against the previous key) and suffix compression (trailing padding trimmed), and new key bytes growing backwards from the end of the 488-byte area;
• the << Int64 promotion fix for packed entries wider than 32 bits, and undocumented option/attribute flag values observed in real-world files.

The exhaustive packed-entry unit test vectors are ported directly from the fork.

What the port changes relative to the fork

• Rewritten onto this library's parsing style: fixed-size reads into buffers, ReadOnlySpan<byte> + BinaryPrimitives — no BinaryReader, no Windows-only FileStream overloads, no reflection. Works on both target frameworks and all platforms.
• Small public surface (CdxFile, CdxIndex, CdxKeyEntry, CdxIndexHeader, enums, CdxException); node types and traversal internals are internal.
• Format validation is always on (the fork gated it behind a compile-time flag), sibling-chain walks are guarded against cycles in corrupt files, and leaf-search recursion is converted to iteration.
• Fixes a comparer defect present in the fork: stored leaf keys are trimmed of trailing padding, and the fork compared only the stored prefix — so a stored key that was a strict prefix of the target compared as equal (searching "ABC" could also match "AB"). Trimmed bytes now compare as the pad byte (0x20), which also preserves correct B-tree ordering during descent.
• Encoding is configurable (pass the table's CurrentEncoding); Search(string) pads keys to the index key length.

Limitations (documented in the README)

Ascending, byte-wise (MACHINE collation) character keys only: descending indexes throw NotSupportedException on search, FoxPro's binary key transforms for numeric/date index keys are not decoded, key/FOR expressions are exposed as strings but not evaluated, and index entries include deleted records (check IsDeleted after seeking). .idx single-index files are not supported.

Test plan

44 new tests, all green (full suite: 163 passed, 1 pre-existing WIP skip):

• CdxKeyPackingTests — the fork's exhaustive packed-entry bit-unpacking vectors (lengths 0–8 at multiple offsets).
• CdxKeyComparerTests — full-length comparison plus the trimmed-prefix regression cases.
• CdxTests — integration over the test/fixtures/foxprodb files that have shipped (unused) in this repo since 2017: calls.CDX, contacts.CDX, setup.CDX, types.CDX, and FOXPRO-DB-TEST.DCX (database container indexes parse with the same reader). Per tag: entry count matches Count(), every adjacent entry pair is in ascending padded-key order, exact-key searches return the correct duplicate sets, missing-key probes return empty (regression for the fork's historical infinite-loop bug), and — end to end — index entries for every tag whose key expression is a plain character column are resolved via Seek to actual DBF rows whose column value equals the entry key.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[chrisrichards]

@daiplusplus — eight years on, your dev/async fork has finally made it home 🙂

This PR ports your CDX compound-index work upstream (credit in the PR description and commit message): the compound tag directory, the big-endian interior node entries — where you spotted that Microsoft's docs describe the wrong layout — and the bit-packed, prefix/suffix-compressed leaf keys, including your 64-bit shift promotion fix. Your exhaustive packed-entry test vectors passed unchanged against the port on the first run, and searches now feed a Seek API (#283) so index hits resolve straight to DBF rows. Async record reading landed alongside in #284, shaped in part by studying your fork's async experiments.

Thanks for the reverse-engineering effort — it held up remarkably well after all this time. If you ever feel like giving the port a once-over, I'd welcome your eyes on it.

🤖 Generated with Claude Code

[daiplusplus]

I'm flattered (honestly!).

Regrettably I don't think I can provide any real feedback as I barely remember writing this; and re-reading through it now today gives me a slight feeling of embarrassment from patterns I've since moved-past from - and the extra rigour needed in the async codepath (e.g. I didn't carry CancellationToken around; my use of ConfigureAsync in test-cases needs reviewing as it can break some test-runners; my comments should be converted to runtime assertions, etc).

I have some general recommendations I stick-to in all projects today though (in addition to the things I mentioned above)...

• Add nullable-reference-type annotations.
• Convert all .csproj files to SDK-style, with multitargeting, and more.
• readonly struct-based refinement types are tedious to implement, but help eliminate entire classes of bugs and make program code self-documenting.
• Trace and document all possible exceptions in /// <exception> comments, so no-one's guessing about where and what to catch.
• And run VS+.NET's own code-analysis as part of the build process - also tedious to deal with at first, but pays for itself in the long-term, especially when it's a library project like this that will have downstream consumers. I see you're using SonarQube and it isn't complaining about much, but I'm seeing things that FxCop/Roslyn would definitely raise, like the empty DbfFileFormatException class.

...stuff like that. Not that anyone needs to be told any of this, but you did ask ;)

[daiplusplus]

@chrisrichards May I ask how much of your posts are “you” as opposed to Claude acting autonomously through your account?

[chrisrichards]

@daiplusplus I'm not using Claude autonomously, I'm still reviewing everything myself. For the post above I asked Claude to create a draft as I genuinely wanted to thank you (especially for the work on supporting indexes), I reviewed, made some minor updates and posted it. Hope that's OK

[daiplusplus]

@chrisrichards I'm not using Claude autonomously, I'm still reviewing everything myself. For the post above I asked Claude to create a draft as I genuinely wanted to thank you (especially for the work on supporting indexes), I reviewed, made some minor updates and posted it. Hope that's OK

In these kinds of human-to-human messages I don't think it's a good idea to leave the "Generated with Claude Code" footer because it gives the impression that Claude wrote the entire comment by itself (and on its own volition) - which I imagine many people will interpret the opposite way as intended (i.e.: if you want to thank someone personally, don't ask AI to do it for you - because that's how it comes across, unfortunately).

[chrisrichards]

@daiplusplus Yep, point taken. Will remember for future posts