Streaming flatbuffers?

[bog-dan-ro]

With ProtocolBuffers is very easy to serialize, stream multiple "messages".
Basically you just need to write one after another, same when you read the data, you don't need to know the length of every message.
It will be nice is flatbuffers (or the verifier) will tell how much data it's using from the buffer, this way we know where the next message starts.

Yes, a FlatBuffer doesn't encode length, and we can't add that now.

What we can do is introduce the concept of a (uint32_t) length-prefixed buffer, intended to be chained. This is of course simple to do, but it be good if there was standard functionality for it in FlatBuffers, so people don't have to reinvent the wheel. It could be added as a flag to Finish, such that it can take care of ensuring the buffer is properly aligned.

Are you able to make a PR?

[bog-dan-ro]

I thought the Verifier knows (or at least can "guess") the size of the message...

I can try to implement and make a PR (PR stands for Pull Request, right), but probably it will take much more than for you :), I'm still not very familiar with flatbuffers internals.

Yes, the verifier could tell you the upper bound within which all data sits, bit it is slightly messy (it would need to know what the buffer is aligned to, which there is no way to tell). Without a guaranteed exact number, there's no way to reliably skip to the next message. And besides that it is an expensive way to compute the size, if you didn't need verification.

Yes, a PR. I typically suggest this, because me getting to have time to implement it may take a while, so if you're in a hurry..

Actually, alignment padding for the last element in a buffer only happens for strings, vectors with 16bit elements or less, so it IS possible to reliably figure out the buffer size this way. Basically it needs to account for all cases where flatbuffers.h calls PreAlign(). Even simpler, since a FlatBuffer is always at least 4-byte aligned, it can blindly increase the upper bound to be 4-byte aligned when it isn't, and this should be accurate. Any bigger elements (e.g. doubles, SIMD types) always produce a reliable upper bound.

I'll make a note of that, this would be nice functionality to add to the verifier. It can even be controlled with a #define such that it doesn't slow down people that don't need it.

[bog-dan-ro]

Well, in principle people will (should) use the verifier in such cases, because first and foremost you need to know if there is garbage in the stream which will crash your application, at the end of the verification if you'll get also the size of the stream, that is the bonus ;-).

I'll try to implement it myself, any more tips about how to do it will be highly appreciated ;-) !

[bog-dan-ro]

Can you please check if I got it right #3905 ? :)
It seems way too easy :)