HDDS-12742. Implement pool backed RDBStoreCodecBufferIterator for consistency#9561
HDDS-12742. Implement pool backed RDBStoreCodecBufferIterator for consistency#9561swamirishi wants to merge 2 commits intoapache:masterfrom
Conversation
swamirishi
force-pushed
the
HDDS-12742_Alt
branch
from
e444138 to
b5dfe25
Compare
…or prefix based iteration Change-Id: I3acf410885571d0b26da9b89a801d9f2f9070e33
…sistency Change-Id: Ieb7a356cde868208f197d55b21889a733575d1dd
swamirishi
force-pushed
the
HDDS-12742_Alt
branch
from
b5dfe25 to
c5743a9
Compare
|
@swamirishi , questions:
|
This is part of the spliterator implementation requirement wherein we would be using this iterator to just get the value from rocksdb one at a time and asynchronously use this CodecBuffer for deserialization parallely across threads.
RDBStoreAbstractIterator is an internal iterator currently deserialization ensures a pojo is always created from the CodecBuffer. However if we want to collect multiple CodecBuffer values we cannot do so currently since calling next updates the same CodecBuffer again. |
szetszwo
left a comment
szetszwo
left a comment
There was a problem hiding this comment.
This is part of the spliterator implementation requirement ...
For spliterator, we need a design first. Please on hold for this.
As mentioned, the previous design is inefficient.
- See #8203 (comment)
|
@vyalamar do you wanna take a look at this patch? |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
2 participants
What changes were proposed in this pull request?
Currently when RDBStoreCodecBufferIterator returns a keyValue to a caller the KeyValue may not be consistent and could have been modified when the next() value is invoked from the iterator. The codecBuffer returned from the first call may have been modified and thus also making this entire Iterator implementation not thread safe.
The proposal here is to have a pool of Buffers and return a closeable handle of KeyValue to the caller which would be only released when the handle to the codec buffers is released.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12742
How was this patch tested?
Added unit test for the new implementation