Currently, in #119 bento.go we specifically note this and route all failed messages to dlq.unsafe_table with the table name in the NATS message with JSON wrapping, which works, but makes it harder to subscribe to DLQs or retry messages for specific tables, etc. This also all uses our custom query regex for safe table names to prevent SQL injections, but we want to let people use any table names ClickHouse would allow, not adding any additional constraints or reserved keywords etc to things on top of what ClickHouse already has.
To accomplish this, the solution should be two-part:
- Internally, we should convert table names to something known to be safe for all places we serialize and store it, potentially like Base64 (TBD).
- All messages sent to/from the server via the API layer should not enforce reserved fields, etc, but should instead nest the original data/content in a child JSON element, letting us attach metadata around it without adding reserved names/fields.
Currently, in #119 bento.go we specifically note this and route all failed messages to
dlq.unsafe_tablewith the table name in the NATS message with JSON wrapping, which works, but makes it harder to subscribe to DLQs or retry messages for specific tables, etc. This also all uses our custom query regex for safe table names to prevent SQL injections, but we want to let people use any table names ClickHouse would allow, not adding any additional constraints or reserved keywords etc to things on top of what ClickHouse already has.To accomplish this, the solution should be two-part: