Repository navigation
Conversation
- `scan_completion_date_context`: `#[allow(dead_code)]`, no callers. - A verbatim duplicate of the first line of the `ArchonShard` doc comment. - A truncated doc fragment above the full-account blob parser section header; it repeats the opening of the `capture_all_blobs` doc block, which still sits intact on that function further down. - `capture_inventory_blob`, its `#[tauri::command]` wrapper and the invoke_handler registration: no frontend call site, so the command was unreachable.
`scan_auth_credentials` and `scan_steam_id` ran byte-at-a-time loops that scanned for a single anchor byte (`"`, `a`, `s`) and then compared the following slice against the full needle. These functions run per memory region over multi-gigabyte address space scans, so every anchor-byte false positive paid for a full needle comparison. Replace each loop with `memchr::memmem` (SIMD substring search). Match positions and extraction bounds are unchanged. One deliberate narrowing: the old loops resumed at `next + 1` and so could match overlapping occurrences of a needle, while `find_iter` resumes past the end of a match. Of the five needles only `"id":"` has a self-border, and exercising the difference requires a buffer holding the literal `"id":"id":"` followed by a valid account id. The game's JSON cannot produce that, so the change is not observable. memchr is already in Cargo.lock as a transitive dependency of aho-corasick and regex, so depending on it directly leaves the dependency tree and the resolved version untouched.
find_blob_end, blob_seed_offsets, and extract_blob_json searched
multi-megabyte inventory blobs with scalar `windows().position()`
loops. Swap these for memchr::memmem::find, which is SIMD-accelerated,
and use memchr::memchr for the single-byte closing-brace scan in
find_blob_end.
extract_blob_json's start-marker re-scan now reuses the existing
START_MARKER constant instead of a duplicate local one, since both
searched for the same bytes.
enclosing_object_start stays a manual backward loop: it counts brace
depth to find the *enclosing* `{`, not the last occurrence of a byte,
so memrchr does not apply.
Two of the converted sites were unpinned: the `.or_else` marker
priority (the existing assertion held for either branch) and the
extract_blob_json path that reinstates a missing opening brace. Both
now have exact-offset assertions, so a drift in search order or offset
arithmetic fails the harness.
parse_full_account_blob aggregated Upgrades/RawUpgrades/FlavourItems/ WeaponSkins into hash maps via entry(it.to_string()), which allocates a String on every one of thousands of entries even though these sections are dominated by duplicate keys. Check get_mut first and only allocate on first insert.
Each ~10s scan cycle rebuilds BlobInventory's HashMaps/Vecs from a multi-megabyte JSON tree, even though inventory changes maybe once per mission, so almost every parse was wasted. Add LAST_BLOB_DIGEST beside the existing LAST_BLOB_REGION address cache, and blob_unchanged() to compare a candidate blob's bytes against the last one seen. Check-and-update in a single call keeps each of the three call sites (fast-path re-probe, full-walk completion, full-walk immediate hit) one early-exit instead of separate check and update steps. The digest is recorded before the parse outcome is known, so every parse-failure path drops the baseline via forget_blob_digest(); otherwise persistent unparseable bytes would read as a cached success and stall the walk until the PID changed. On a hit, the scan is dropped without parsing or sending to blob_tx. The UI already tolerates a cycle producing nothing and keeps its last-known state, so the "Inventory Loaded" badge now appears when the inventory actually changed rather than every 10s. The skip is gated on `!save`, matching the existing fast-path convention, so an explicit debug dump still writes the file even when the blob hasn't moved. reset_last_blob_region() also clears the digest so a PID change forces a re-parse.
extract_blob_json always copied the blob region into a fresh Vec,
even in the common case where the buffer still starts with the
original `{` and the bytes could be returned as a slice. For
multi-megabyte scan buffers this is a needless memcpy on every parse.
Add extract_blob_json_ref, returning Cow<[u8]>: Cow::Borrowed when
the opening brace survived, Cow::Owned only in the fallback path
where it was overwritten and has to be reinstated.
extract_blob_json is now a thin extract_blob_json_ref(..).into_owned()
wrapper, so its signature and behavior are unchanged for existing
callers.
parse_full_account_blob already computes end_pos via find_blob_end
for its size check, then extract_blob_json used to redo that scan
and re-locate the start marker. Factor the shared body into
extract_blob_json_at(raw, end_pos), taking the bound as a parameter,
and have parse_full_account_blob call it directly with the end_pos
it already has.
`blob_unchanged` hashed the whole buffer it was handed, and all three callers hand it a stitched one. The stitch stops at the region that closes the JSON but keeps everything after the closing brace, which is whatever heap shares that region: tens of megabytes of it, rewritten constantly by a running client. The digest therefore changed on every scan of a completely settled inventory, so the skip never fired once and every cycle reparsed. Hashing now stops at `find_blob_end`, which also means hashing a few MB rather than the whole read. Buffers with no blob end are hashed whole; they are not blob data and only need to compare equal to themselves. The existing fixtures could not catch this. Both carry no `DeathSquadable` key and no trailing bytes, so they exercise only the whole-buffer path. The digest tests also reset a process-global static and then assert against it while running in parallel, so they could already clobber each other. They share a mutex now.
The blob qualification runs six needles over every committed region, and the end-marker check runs over the whole pending scan buffer. Those were still scalar `windows().any()` passes while the rest of the scanner had moved to `memmem`, so they fell back to a byte-at-a-time compare on buffers measured in tens of megabytes. `memmem::find(..).is_some()` matches the old semantics on short slices too: a haystack shorter than the needle yields no windows and no match.
A freed copy of the FULL_ACCOUNT blob can sit ahead of the live one
with its start marker intact but its opening brace overwritten.
Matching braces backward from that copy walks into binary garbage and
seeds at a stray `{`, so the parse dies ("key must be a string at line
1 column 2") and the scan drops the inventory even though a healthy
copy is in the same buffer.
Accept a candidate brace only when a quote follows it, the shape of a
real JSON object head, and keep trying later marker occurrences until
one qualifies. When none does, seed at the first marker itself and let
the parser rebuild the object head, so a lone headless copy still
yields its inventory.
The marker walk uses `memmem::find` to match the scans earlier in this
branch.
The monitor walked every committed region of the game process every 10 seconds, reading gigabytes to find a blob that usually had not moved since the last pass. That walk costs the player frames for seconds at a time, and it ran whether or not the inventory had changed. Split the cached-address fast path out of capture_all_blobs into probe_tick, which reads the remembered address and nothing else, and never falls back to a walk of its own. The monitor polls it every 2 s and decides for itself when a result is worth escalating: a fresh parse needs no walk at all, a cache miss with a sync marker walks immediately, and everything else is rate-limited by walk_is_due. The marker itself comes from two places. Warframe formats its log lines in memory well before flushing them to EE.log, and sampling a live client showed the in-memory copy running 23 s ahead, so probe_tick finds the newest "OnInventoryResults completed in" line by content in the game's log buffers, and the existing EE.log tail sets the same flag whenever it gets there first. Neither is required: every case in walk_is_due has an interval that fires without a marker, so a missing EE.log or an unlocatable buffer costs latency only.
Contributor
Author
|
thank you for integrating the PRs into the main repo! verified it on/against the linux branch, and everything is working well! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf(monitor): probe the cached blob instead of walking every 10 s
Depends on #42. This branch is #42 plus one commit, so it will not apply cleanly
until that merges.
The problem. The monitor calls
capture_all_blobsevery 10 s. That functionwalks every committed region in the game process, reads gigabytes, and drops the
player's framerate for seconds at a time. It runs on that fixed cadence whether or
not the inventory changed, and the inventory changes maybe once per mission.
capture_all_blobsalready had a fast path for this: probe the address the lastsuccessful scan found the blob at, and if it is still there, skip the walk. But the
fast path lives inside the walk function, so a miss means the walk runs anyway.
At a 10 s cadence with a stale address that is a full walk every 10 s for as long
as it stays stale.
What lands. One commit, three pieces.
probe_tick, the fast path pulled out into its own function, with no walkbehind it. Returns
ScanOutcome::{Updated, Unchanged, CacheMiss}and neverescalates on its own. The monitor polls it every 2 s; a hit costs a few
megabytes of reads.
walk_is_due, the escalation policy, pure and unit-tested.Updatednever walks, since the probe already delivered the inventory.
CacheMisswith async marker walks immediately.
CacheMisson a known blob keeps the old 10 scadence. A client with no blob yet, meaning the login screen, waits 60 s. A
settled inventory answering
Unchangedwaits 900 s, which covers the one casethe probe genuinely cannot see: the game reallocated the blob but the old
address still holds a parseable copy of the old bytes.
OnInventoryResults completed in 339mswhen it finishes fetching inventory.That line is the signal that a walk is worth doing. The obvious source is the
EE.log tail you already run, and this adds that, one
buf.containsinstart_log_watcher. But the game buffers its log writes and flushes inbursts. Sampling a live client showed the newest in-memory log line running
23 s ahead of the newest line on disk, so the tail alone would put that
delay in front of every capture.
probe_ticktherefore also finds the line bycontent in the game's own log buffers, identified by
Sys [Info]:preceded bya seconds-since-launch timestamp. No pointer chain, no per-build offsets; the
buffer address caches like
LAST_BLOB_REGIONdoes, with a 30-probe backoff ifit cannot be found at all.
What this changes in behaviour. In the common case, playing with the inventory
settled, the walk stops happening and the 2 s probe replaces it. When the
inventory does change, capture is faster than before, because the marker
escalates immediately instead of waiting out the remainder of the 10 s slot.
The premise I could not verify on your platform. The 23 s figure and the claim
that the marker is findable by content in the Windows client's memory both come
from my Linux fork against a Proton-hosted client, sampled with
process_vm_readv.The Windows arm here is re-authored against
ReadProcessMemory/VirtualQueryExand has never been run. If the buffers are laid out differently on Windows and
the cold search never finds them,
LOG_SEARCH_BACKOFFsits it out and the EE.logtail carries the marker instead, so the degradation is latency, not breakage. But
you should expect to check that specifically.
Verification.
memory_scanner.rs: the scanner tests run on Linux (26 pass, 7 of them newaround the marker parsing and the steady-state notice) and the Windows arm
type-checks under
--target x86_64-pc-windows-gnu.walk_is_due: 7 tests, one per rule. They are inlib.rs, which does notcompile here, so I ran them by extracting the function and its tests into a
standalone file. They pass. The function is pure, four arguments in and a bool
out, so that is a real check of the policy, but it is not a check that the
surrounding
lib.rscompiles.lib.rsis not compiled. No MSVC linker, and--target x86_64-pc-windows-gnudies in
ring's build script for want of mingw. Expect to fix something.If you reimplement this rather than taking the commit, the load-bearing parts are:
splitting it out of
capture_all_blobs. A probe that walks on miss, polled at2 s, is strictly worse than the 10 s cadence it replaced.
capture_all_blobsmust lose its inline fast path. The monitor already ranthat scan and escalated on its result; running it again at the top of the walk
answers the same and skips the walk that was just asked for.
walk_is_dueneeds an interval that fires without a marker.The marker is an optimisation. If any branch depends on it for correctness, a
client with no EE.log and unlocatable log buffers stops updating entirely.
ScanOutcome::Unchangedplus a sync marker must still walk. "The clientfetched inventory and our copy did not move" is indistinguishable from "our
address is stale and still holds the old bytes". The probe cannot tell them
apart, so the marker has to win.
marker exactly once. A marker that arrives on a tick which cannot act on it
(rate limit, walk in flight) is lost unless it is folded into a sticky flag and
cleared only when acted on.
The timestamps are seconds since that client launched, so a baseline from the
previous process swallows everything the new one writes. And the login sync,
the one that populates the inventory, arrives moments after the PID change, so
spending the first observation on establishing a baseline drops it on every
restart.
so the newest line is not at the highest address.
OnInventoryResults completed in %dmslives in.rdataand matches the samebytes. Skipping executable pages catches most of it; the timestamp check catches
the rest.
region under 16 MB. Without a backoff, a client whose buffers cannot be located
pays that on the monitor thread every 2 s for the whole session.