Issues with the arithmetic in iterator and reverse iterator

[HenryRLee]

Hi, I just found that the iterator class hasn't override the n + iterator operator, which is a requirement in the random access iterator type.

More specifically, this operator is missing:

operator+(difference_type, const iter_impl&);

The failing code is:

json j_object
auto it = j_object.cbegin();
1 + it; // error

[HenryRLee]

Actually I found some more issues during the fix up.

The iter_impl was defined as a random access iterator (https://github.com/nlohmann/json/blob/develop/src/json.hpp#L7994), but a couple of lines later, the iterator category is then changed to bidirectional (https://github.com/nlohmann/json/blob/develop/src/json.hpp#L8018). I tried to fix this, but it introduced some fail the test cases, e.g. distance(rbegin(), rend()) == size() failed.

So I dig it up and found that the json reverse iterator is overriding all the operators, while the implementation of the iter - iter operator is not correct (https://github.com/nlohmann/json/blob/develop/src/json.hpp#L8677). Instead of this->base() - other.base(), it should be other.base() - this->base().

Here is the implementation in llvm:

template <class _Iter1, class _Iter2>
inline _LIBCPP_INLINE_VISIBILITY
typename reverse_iterator<_Iter1>::difference_type
operator-(const reverse_iterator<_Iter1>& __x, const reverse_iterator<_Iter2>& __y)
{
    return __y.base() - __x.base();
}

In addition, the iter + n and iter - n operators are implemented in a reversed way.

Personally, I would recommend removing all the operator override in the reverse iterator, since the template class reverse_iterator<iterator> should give us all the correct implementations.

Reference: std::reverse_iterator

[HenryRLee]

Sorry, I made a mistake in the previous message.

As for the reverse iterator, I can see why the operators are override. But still three operators were implemented in an incorrect way, which are iter + n, iter - n and iter - iter.

[nlohmann]

It took me quite a while to get the iterators "working". I expect that there are better ways to implement them, and I am not surprised that there are issues. I am grateful for any hints how to improve the situation.

[HenryRLee]

I could make them compiled. But some of the cases are failing. I think it's the problem of the test code.

For example, I can see that iterators of object types are not able to compare, but some test cases were still using distance(j.begin(), j.end()) (unit-capacity.cpp +345). Shall we remove these cases?

And the other cases are related to the reverse iterator, could be fixed easily.

[HenryRLee]

Oh, just misunderstood your last message. I have an idea of implementing the reverse iterator, instead of inheriting the reverse_iterator<iter_impl>, we could inherit template <class T> reverse_iterator. In such a way, the return type of all the operators will still be consistent with the iter_impl classes, so no need to implement the operators again. But it's just an idea, I haven't tested it yet.

[nlohmann]

As I said: I was happy once the iterators worked, and the did not receive much love until they were refactored with #395. If you have any proposals how to improve them, I would be happy about a PR. Maybe as a bonus, you may have and idea how to fix #550. :-)

[HenryRLee]

Sure. I'll have these bugs related to arithmetic operators fixed today, then I'll see what I can do to improve the design, and also #550.

Pleasure to help.

[HenryRLee]

I did a code walk today, and wondering if I could improve the iterator code structure.

Currently the iterator has three internal iterators, which looked very redundant at the first glance. So I was trying to merge the three internal iterators into one type.

Another issue is that the iterator category, actually should be consistent with the iterator class in the internal container. For example, if the container is std::map, the iterator category tag should be bidirectional. While if the container is std::vector, the tag should be random accessible. I was wondering if I could define the category tag dynamically, according to the real container type.

It turned out that, none of these improvement is possible, because the basic_json class is using the value_type as a variable instead of a type name, unless we introduce polymorphism. I understand that there is no easy way to change value_type to a type name, so I guess we can only keep it that way.

My last recommendation is to change the iterator category to random accessible tag, because in this tag, running functions like std::advance and std::distance will be constant time complexity if the container is a vector-like container. As for the non random accessible containers, we can still re-implement the linear running time iter + n operators for these types, such that std::distance wouldn't throw an exception.

[nlohmann]

I'm sorry that I won't find the time to properly answer here or have a deeper look at the PRs. Sorry about this - I shall get back to this next week.

[HenryRLee]

No worries. Take your time.

[nlohmann]

Regarding #593 (comment):

I agree to paragraph 1-4. The idea is that all JSON values have the same type (basic_json) with the specific type encoded in variable m_type. We won't change this.

In the 5th paragraph, you propose to change the behavior of the iterator so iter + n would work for objects. I don't really like this idea, because it would change the behavior of JSON objects compared to std::map and would also suggest an ordering of the keys in an object. This would contradict the definition in RFC 7159 which states

An object is an unordered collection of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array.

[HenryRLee]

I understand.

How about the iterator category tag? Do you think we should change it to random access tag? Even if we don't change the implementation, we can still use the random accessible tag. All we need to do is removing the test cases that are using random access operations on object types. (i.e. https://github.com/nlohmann/json/blob/develop/test/src/unit-capacity.cpp#L327)

[nlohmann]

Your mentioned line is

CHECK(std::distance(j.rbegin(), j.rend()) == j.size());

for an object j. Would this mean std::distance would not work any more?

[HenryRLee]

Correct. Because std::distance would call iter - iter if it finds the iterator is a random accessible iterator. But iter - iter of an object type throws an exception in the current implementation.