Skip to content

Fix utf8 read peformance #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mourner merged 2 commits into from
Oct 11, 2019
+18 −3
Merged

Fix utf8 read peformance #109

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
21bbaf2
Use TextDecoder API to read longer strings based on length heuristic
ahk Oct 11, 2019
a38b6e1
minor stylistic changes
mourner Oct 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ Pbf.Fixed32 = 5; // 32-bit: float, fixed32, sfixed32
var SHIFT_LEFT_32 = (1 << 16) * (1 << 16),
SHIFT_RIGHT_32 = 1 / SHIFT_LEFT_32;

// Threshold chosen based on both benchmarking and knowledge about browser string
// data structures (which currently switch structure types at 12 bytes or more)
const TEXT_DECODER_MIN_LENGTH = 12;
const utf8TextDecoder = typeof TextDecoder === 'undefined' ? null : new TextDecoder('utf8');

Pbf.prototype = {

destroy: function() {
Expand Down Expand Up @@ -112,10 +117,16 @@ Pbf.prototype = {
},

readString: function() {
var end = this.readVarint() + this.pos,
str = readUtf8(this.buf, this.pos, end);
const end = this.readVarint() + this.pos;
const pos = this.pos;
this.pos = end;
return str;

if (end - pos >= TEXT_DECODER_MIN_LENGTH && utf8TextDecoder) {
// longer strings are fast with the built-in browser TextDecoder API
return readUtf8TextDecoder(this.buf, pos, end);
}
// short strings are fast with our custom implementation
return readUtf8(this.buf, pos, end);
},

readBytes: function() {
Expand Down Expand Up @@ -573,6 +584,10 @@ function readUtf8(buf, pos, end) {
return str;
}

function readUtf8TextDecoder(buf, pos, end) {
return utf8TextDecoder.decode(buf.subarray(pos, end));
}

function writeUtf8(buf, str, pos) {
for (var i = 0, c, lead; i < str.length; i++) {
c = str.charCodeAt(i); // code point
Expand Down