CBOR parser doesn't skip tags

Using 456478b (3.7.3)

The following code should parse a CBOR value:

#include "json.hpp"

#include <sstream>

class NullSAXParser : public nlohmann::json_sax<nlohmann::json> {
public:
  bool null() override {
  	return true;
  }

  bool boolean(bool val) override {
    return true;
   }

  bool number_integer(number_integer_t val) override {
    return true;
  }

  bool number_unsigned(number_unsigned_t val) override {
    return true;
  }

  bool number_float(number_float_t val, const string_t &s) override {
    return true;
  }

  bool string(string_t &val) override {
    return true;
  }

  bool start_object(std::size_t elements) override {
    return true;
  }

  bool key(string_t &val) override {
    return true;
  }

  bool end_object() override {
    return true;
  }

  bool start_array(std::size_t elements) override {
    return true;
  }

  bool end_array() override {
    return true;
  }

  bool parse_error(std::size_t position, const std::string &last_token,
                   const nlohmann::detail::exception &ex) override {
    throw ex;
  }
};

int main() {
    std::stringstream in("\xd9\xd9\xf7");
    NullSAXParser parser;
	nlohmann::json::sax_parse(in, &parser, nlohmann::json::input_format_t::cbor);	
}

The value I'm parsing (0xd9d9f7) is simply the optimal "magic number" tag for CBOR documents. From the specification:

The serialization of this tag is 0xd9d9f7, which appears not to be in
use as a distinguishing mark for frequently used file types. In
particular, it is not a valid start of a Unicode text in any Unicode
encoding if followed by a valid CBOR data item.

Byte 0xd9 should be fine because it is equal to (6 << 5) | 25, in other words it has a major type of 6 (a tag), and lower 5 bits of 25, which for a tag means the actual tag value follows in a uint16, so it should just skip the following 2 bytes.

Some extra code needs to be added here. It doesn't understand tags at all.

Can you provide the error message?

It's this one here: invalid byte: 0xD9.

@nlohmann

I‘v read the CBOR standard document and its implementation in the source code, it does not support data of major-type 6(Optional Tagging of Items). One of the reasons is that the other major-type data has satisfy various data types of JSON.
but user can parse to any kind of data, like 0xd9d9f7...
55799 is one of tags, it serialized to 0xd9d9f7 by CBOR spec. As written in the specification：

Tag 55799 is defined for this purpose. It does not impart any special semantics on the data item that follows; that is, the semantics of a data item tagged with tag 55799 is exactly identical to the semantics of the data item itself.

so can we add a case to skip it to avoid parsing error? @nlohmann

The documentation states that the library does not support tags at this point, so I would not speak of a bug. I am hesitant of skipping tags, because this would, to my understanding, silently change the type or intention of the encoding. Or, to put it differently, how would you propose mapping such read bytes to JSON values?

I am hesitant of skipping tags, because this would, to my understanding, silently change the type or intention of the encoding. Or, to put it differently, how would you propose mapping such read bytes to JSON values?

Just ignore them. This is perfectly acceptable according to the standard:

A decoder that comes across a tag (Section 2.4) that it does not
recognize, such as a tag that was added to the IANA registry after
the decoder was deployed or a tag that the decoder chose not to
implement, might issue a warning, might stop processing altogether,
might handle the error and present the unknown tag value together
with the contained data item to the application (as is expected of
generic decoders), might ignore the tag and simply present the
contained data item only to the application, or take some other type
of action.

It's a bit odd that you have a bot to close issues that nobody replies to after 1 month. I don't think it's going to magically fix itself!

This is a one-man-show, and I do not want to be overwhelmed by inactive issues. Since I will never be able to work on all of them, I'd rather keep the active ones open.

Tags should be skipped to allow roundtripping after #2244 is merged.

@matthewbauer

CC @matthewbauer

Alright, I finally had time to work on this. My idea would be to pass the from_cbor function an enum parameter tag_handler that would have two values:

error: The current behavior of the library - in case a tag is read, a parse_error exception is thrown. This is the default to avoid breaking existing client code.
ignore: The tag is ignored and the following value is parsed as before. On case of tags 0xD8-0xDB, the following 1-8 bytes are read, but also discarded. This would treat any tagged value as a "standard" CBOR value.

Also possible (but more difficult to implement would be):

binary: Tagged values are parsed to binary values. The subtype is the tag value and the following bytes are just copied to the binary value as is. I am not sure whether this is helpful.

Any comments on this?

✨ add CBOR tag handler #1968

Sounds sensible to me.

Navigation Menu

Repository navigation

CBOR parser doesn't skip tags #1968

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

CBOR parser doesn't skip tags #1968

Description

Activity

nlohmann commented on Mar 5, 2020

Timmmm commented on Mar 5, 2020

dota17 commented on Mar 20, 2020

nlohmann commented on Mar 20, 2020

stale commented on Apr 19, 2020

Timmmm commented on Apr 19, 2020

stale commented on May 19, 2020

Timmmm commented on May 19, 2020

nlohmann commented on May 19, 2020

3 remaining items

nlohmann commented on Jul 10, 2020

Ericson2314 commented on Jul 10, 2020

nlohmann commented on Jul 12, 2020

Timmmm commented on Jul 15, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions