Turn MudStats on, add three more directories, and record why the rest are unread - #10
Closed
HarryCordewener wants to merge 4 commits into
Closed
Turn MudStats on, add three more directories, and record why the rest are unread#10HarryCordewener wants to merge 4 commits into
HarryCordewener wants to merge 4 commits into
Conversation
The same crawler on the same site publishes a second page for MSDP, in the same box-drawing format, reaching a partly different population — the games that answer MSDP rather than MSSP. It is the same tier for the same reason (the site connects, and prints when it did), so what it needed was a reader, not an argument. The two pages differ in exactly one thing that matters and it is a trap: the label column is seventeen cells wide on the MSSP page and twenty-five on the MSDP one, because CONFIGURABLE_VARIABLES does not fit in seventeen. Read at the wrong width the page does not fail — it yields every label and value sliced eight columns early, which is the same quiet mis-parse the id-keyed reading in MudStatsSource exists to avoid. So the width is a required argument with no default, and a test reads the MSDP page at the MSSP width to prove what going wrong looks like. MsspCrawlerTable is now TinTinCrawlerTable and the record mapping moves to a shared TinTinCrawlerSource base, since it was already page-agnostic: an absent field simply never appears, so one whitelist covers both. TinTinMsspCrawlerSource keeps its public Parse and its etiquette exactly as they were. One hardening, which fixes a latent hole in the shipped MSSP source as well. Both pages interleave the crawler own frames with the CONNECT SCREENS of the games it dialled, and a connect screen is text a game controls completely. The reader opened a record on any line merely beginning with a box corner, so a game could draw a box in its own login banner, point it at another game HOSTNAME and PORT, and have a fabricated player count attached to somebody else listing. A frame must now be the full width of the page, corner to matching corner, which every genuine one is. The MSDP page also stamps itself in CET, which the zone table did not know; unknown abbreviations still cost the page its presence rows rather than being guessed at, because a guessed offset puts a reading in the wrong hour of the heatmap. Its copy was generated in January 2024 and its readings are imported at that instant, not at the instant we read the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…upant MudStats maintainer has been approached and the run authorised, so its ContactedMaintainer default is now true. The gate was satisfied, not removed, and the difference is the point: EtiquettePlanner is unchanged, and MudVerse arrives in this same commit sitting behind it — written, tested, credited on the about page, and refused at the moment of fetching with the reason printed. One short email is the whole of what stands between that file and 273 dated readings. A test retracts the MudStats fact and watches the gate bite, so what is pinned is the mechanism rather than a source that happens to be past it. Permission to run is not permission to hurry, and the rest of 7.6 is enforced in code rather than remembered: fifteen seconds between page fetches, a bound on how many world pages one run touches, a User-Agent naming us with an info URL, robots.txt read before the first content fetch, and an attribution the registry derives from itself so a credited source and a read source cannot diverge. Two new sources besides. MudVerse is imported_measured and is the strongest source in the table on every axis except permission. It crawls hourly and prints Connection Tested and Last Successful Connection to the minute in GMT, so a count read off it is dated by the source. Measured and asserted are separated in its markup and visibly disagree — the owner panel says Codebase: Other and Player Count: 0-5 where the crawler read DikuMUD/Merc/Envy and PLAYERS 1 — and only the crawler panel is read. The count is dated at the last SUCCESSFUL connection, because a failed test reads nothing, and only when the MSSP block own crawled-on date agrees; otherwise the block is older than the connection above it by an amount the page does not state. Its per-game daily series going back to 2022 is deliberately not imported: a day AVERAGE is a derived statistic and a PresenceSample is a count somebody read at an instant, so placing one at the midnight its bucket is keyed to would put a number nobody measured into that hour of the heatmap. The Mud Connector is imported_asserted, which is what 7.6 names it as. It does connect — its Connect Status column says Connected, Connect Refused or N/A, which no submission form produces — but the page states no time for that result, so it is read by nothing. Its only player figure is a submission-form bucket on listings whose headers can be fifteen years old. It is read as a bulk export on arithmetic rather than on a claim about the maintainer intent: 689 games for one GET, and the per-game pages are never fetched. One defect, found by running the importer against the live site rather than by reasoning about it. MudStats index links 144 worlds and one of them, /World/TheChattingZone, answers 404 No such world — and that one stale link ended the whole import at the hundredth page. IDirectoryFetcher grows TryGetStringAsync, which is null for 404 and 410 and throws for everything else: a 429 or a 503 means we are being asked to stop or the site is unwell, and swallowing those would walk a struggling server to the end of its index and report a catalogue full of holes. It is the same fetch otherwise — still behind robots.txt, the route and the rate limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he rest are unread live-tintin-import read one page and was named after it. It now takes --source, --dsn and --cache, and --list prints every registered source with its tier, its route and — for the ones that may not run — the reason. Running it with no --source reads everything the etiquette permits and names what it skipped, which is the shape a one-off backfill actually wants. --cache is the part worth explaining. It is a directory of raw fetched bodies, read before the network and written after it, so that a second look at the same data costs the site nothing: a MudStats run walks 145 pages fifteen seconds apart, and without it every parsing mistake costs somebody else server that walk again. A page the site does not have is cached as an empty file, so a resumed run does not ask twice for something it already knows is gone. It is SOMEBODY ELSE CATALOGUE and it does not belong here. The path is named by the operator and pointed outside the working tree; .gitignore carries a backstop and says so. Nothing in this repository holds a harvested dataset — the fixtures under tests/MUI.Import.Tests/Fixtures are five hand-trimmed records each, they are test inputs, and no test in this repository touches the network. The import itself is one command a human runs once against one deployment: there is no hosted service and no timer, and AddMuiImporters composes a reader rather than a schedule. docs/import-sources.md is the other half. Every directory looked at, its tier, and for the four not read, why — so that a source is investigated once. Grapevine measures but publishes no dated per-game count and its stats endpoints answer 500 on every game and series tried; its documented API needs a client id and secret issued PER REGISTERED GAME, which would mean registering a game that does not exist. bestmuds.com (was mudlistings) does timestamp its readings, but the check fires on a record write rather than on a schedule — of five samples the freshest was fifteen days old and two had never been checked — and it costs a thousand requests. Top Mud Sites is frozen and read-only and measures nothing at all; its two candidate fields are owner-entered and their own tooltips say so. muds.fandom.com answers 403 behind a bot challenge, and working around one is the opposite of what this component is for. Intermud gets the longest entry because the answer is the least obvious. There IS a live HTTP mudlist, at the *wpr router own server, listing 409 muds with a real up/down state across three routers — a largely LPMud population no MSSP source reaches. But the index carries no addresses (409 further requests), the site certificate is expired, and there are no player counts at all: the I3 mudlist packet has no count field, so getting one means speaking I3 to a router. That is a crawler feature, not an importer one, and the entry says so rather than leaving the next person to find out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 35 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (34)
Comment |
…g on nothing Review of the previous two commits, and four of the findings were real. The route gate compared paths and ignored the query, which nothing had noticed because every source until now had a route the path defined. The Mud Connector does not: its whole catalogue is one page at ?mode=mobile_biglist, and every other thing that site does is the SAME script under another mode — including mode=mud_listing, the 689 per-game pages the importer promises never to fetch, and mode=check_connect, which opens a live socket to a third party server on our behalf. Compared on path alone the permitted route was that script with any query at all, so both were authorised and the promise was kept only by the importer being written correctly — precisely what EtiquettePlanner exists so that nothing has to rely on. A root with a query now authorises that query and no other; a root without one still authorises its subtree, so paginated listings stay readable. MudVerse claimed its connection panel was what the site DIALLED, and used that to mint a TLS endpoint and to discard the game declared MSSP HOSTNAME. The page does not support it: it states one Connection Tested and one Last Successful Connection, both singular, and the panel ids share a family with detailsName, which is the owner submitted title. So the claim is withdrawn. The TLS port is still seeded — as a CANDIDATE, which 7.2 makes safe, and it mints no capability field — and HOSTNAME is kept as a field like every other source keeps it, because where the declared host and the published one disagree that disagreement is the interesting fact. The test for the most careful decision in that file could not fail. ReadPresence dates a count by LAST SUCCESSFUL CONNECTION rather than by the last test, because a failed test reads nothing — but both fixtures carried the same value in both stamps, so reading the wrong one passed. The fixture now differs by 85 minutes and the test asserts against the one it must not use, plus a new case where the last test failed the following day. And the tier assertions read a list written beside them rather than the registry AddMuiImporters composes — so a source registered with the wrong tier, or left out of DI entirely, was invisible to the one test whose comment says mis-tiering is worse than skipping a source. Both the suite and tools/live-import now resolve the registry from the extension method, and there is one list of directories instead of three. Smaller, all real: IdMatches used \b, which matches after a hyphen, so data-listing-id="671" — on the very page it parses — was an element id; two row regexes demanded attribute-free tags and would have read a panel as EMPTY rather than failing, which this importer reports as "no MSSP data" and is indistinguishable from a game that has none; the href preference applied to every cell rather than the four that are anchors, so the day the site links a CODEBASE that field silently becomes a URL; a trailing --dsn printed usage and exited 0, which for a command whose job is a one-off write to production is the worst available answer; a cached 404 on a required page reported an empty catalogue instead of erroring; and the console printed a sum of players across every game, which is the one number 15.7 says this project does not compute. The full width frame guard added in the first commit is also restated honestly: it refuses every casually shaped box and it does not close the hole, there is no second factor available, and what is actually load bearing downstream is that ImportIdentity attaches an imported record only by an address we already know. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HarryCordewener
marked this pull request as ready for review
July 31, 2026 04:51
Member
Author
|
Closing unmerged, by decision: the backfill importer is a one-time tool that primes the one deployment and then has no further job, so it is not carried on Every commit on this branch is preserved on the local What survives on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns MudStats on, adds three more directories, and writes down why the rest are not read.
What runs now
imported_measuredimported_measuredimported_assertedimported_measuredimported_measuredLive run
MudStats, once, against a real database — plus the three one-request sources, into the same
empty database for the combined figure. New is targets a source contributed that the ones
above it had not already, so all the overlap lands on the later rows.
MudStats took 17m30s at the fifteen-second floor for 143 pages. Two things worth reading off
the table: the MSDP page contributes 7 addresses in 44 — it is very nearly a subset of its MSSP
sibling, and it was added for those 7 and for its 41 readings rather than for volume — and The
Mud Connector, the one asserted source, is two thirds of the address haul while contributing
not one number. Coverage and measurement are different jobs, and the tier split is what lets one
source do each without either pretending to be the other.
No source emitted a single availability span, so no archive grace is granted by any of them.
The gate moved; it did not go away
MudStats' maintainer has been approached and the run authorised, so its
ContactedMaintainerdefault is nowtrue— for that source.EtiquettePlanneris untouched,and MudVerse arrives in the same commit sitting behind the gate: written, tested, credited on
the about page, and refused at the moment of fetching with the reason printed by
ImportRunner.One short email is the whole of what stands between that file and 273 dated readings.
A test retracts the MudStats fact and watches the gate bite, so what is pinned is the mechanism
rather than a source that happens to be past it. Another asserts that permission to run is not
permission to hurry: the fifteen-second floor, the bound on how many world pages one run touches,
the self-identifying
User-Agent,robots.txtbefore the first content fetch, and the derivedattribution are all still enforced in code.
Three defects the live run and the review found
A dead link ended the whole import. MudStats' index links 144 worlds and one of them,
/World/TheChattingZone, answers404 No such world.— and that killed the run at the hundredthpage.
IDirectoryFetchergrowsTryGetStringAsync: null for 404 and 410, throwing for everythingelse, because a 429 or a 503 means we are being asked to stop or the site is unwell, and
swallowing those would walk a struggling server to the end of its index and report a catalogue
full of holes. Same fetch otherwise — still behind
robots.txt, the route and the rate limit.The route gate ignored query strings, which nothing had noticed because every source until
now had a route its path defined. The Mud Connector does not: its whole catalogue is one page at
?mode=mobile_biglist, and every other thing that site does is the same script under anothermode— includingmode=mud_listing(the 689 per-game pages this importer promises never tofetch) and
mode=check_connect, which opens a live socket to a third party's server on ourbehalf. Both were authorised, and the promise was kept only by the importer being written
correctly, which is exactly what
EtiquettePlannerexists so nothing has to rely on.A game could forge a record in its own connect screen. Both TinTin pages interleave the
crawler's frames with the login banners of the games it dialled, and the reader opened a record on
any line merely beginning with
┌. A frame must now be the full width of the page, corner tomatching corner. Stated honestly in the code: that refuses every casually-shaped box and does not
close the hole, there is no second factor available, and what is actually load-bearing downstream
is that
ImportIdentityattaches an imported record only by an address we already know.Tiering, which is the part worth arguing about
column is 25 cells against the MSSP page's 17, and read at the wrong width the page does not
fail, it yields everything sliced eight columns early — so the width is a required argument
with no default and a test reads it wrong on purpose.
Connect StatussaysConnected/Connect Refused/N/A, which no submission form produces) but states notime for the result, so it is read by nothing. Its only player figure is a submission-form
bucket on listings whose headers can be fifteen years old.
to the minute in GMT. The count is dated at the last successful connection (a failed test
reads nothing) and only when the MSSP block's own "Crawled on" date agrees. Its per-game daily
series back to 2022 is deliberately not imported — a day's average is a derived statistic
and a
PresenceSampleis a count somebody read at an instant, so placing one at the midnightits bucket is keyed to would put a number nobody measured into that hour of the heatmap.
The tier assertions now read the registry
AddMuiImporterscomposes rather than a list writtenbeside them, because a hand-written list keeps passing while a source is registered with the wrong
tier or left out of DI entirely — and mis-tiering is the one failure here worse than skipping a
source.
Investigated and not read —
docs/import-sources.mdGrapevine, bestmuds.com (was MUDListings), Top Mud Sites, muds.fandom.com and Intermud, each with
its reason, so none of them is investigated twice. Grapevine measures but publishes no dated
per-game count, its stats endpoints answer 500 on every game and series tried, and its documented
API needs credentials issued per registered game. bestmuds timestamps its readings but the
check fires on a record write rather than a schedule (freshest of five samples: fifteen days old;
two never checked) and it costs a thousand requests. Top Mud Sites is frozen, read-only and
measures nothing — its two candidate fields are owner-entered and their own tooltips say so.
Fandom answers 403 behind a bot challenge.
Intermud gets the longest entry because the answer is the least obvious: there is a live HTTP
mudlist reaching a largely LPMud population no MSSP source touches, but it carries no addresses
without 409 further requests, its certificate is expired, and the I3 mudlist packet has no
player-count field at all, so a count means speaking I3 to a router. That is a crawler feature,
not an importer one.
The tool, and where the data does not go
live-tintin-importread one page and was named after it; it is nowtools/live-importwith--source,--dsn,--cacheand--list.--cacheis a directory of raw fetched bodies, readbefore the network and written after it, so a second look at the same data costs the site nothing
— a MudStats run walks 145 pages fifteen seconds apart, and without it every parsing mistake costs
somebody else's server that walk again.
It is somebody else's catalogue and it is not in this repository. The path is named by the
operator and points outside the working tree;
.gitignorecarries a backstop and says why. Thefixtures under
tests/MUI.Import.Tests/Fixturesare around five hand-trimmed records each — testinputs, and no test here touches the network. The import is one command a human runs once against
one deployment: no hosted service, no timer, and
AddMuiImporterscomposes a reader rather than aschedule.
Verification
dotnet build MUIndex.slnx -c Releaseclean (warnings are errors), and all six suites green:Catalog 133, Crawl 125, Crawler 76, Discovery 182, Import 142, Web 107 — 765 tests, none
skipped.
🤖 Generated with Claude Code