Reconstruct the history of the package heidrun_notebook_ctrl as an interactive web page #7

Activity

1. jeromeetienne

   changed the title [-]Reconstruct the history of the package data_layer_coding_agents as an interactive web page[/-] [+]Reconstruct the history of the package heidrun_notebook_ctrl as an interactive web page[/+] on Oct 25, 2026
2. jeromeetienne

   closed this as completedin 18d4d70on Oct 25, 2026
3. jeromeetienne commented on Oct 25, 2026

   

   jeromeetienne

   on Oct 25, 2026

   Last edited by jeromeetienne

   ContributorAuthor

   More actions

   Report: how the module history page was built

   This report logs the work, so another person or another coding agent can repeat it and use the experience.

   Result

   • The page is the folder experiment_heidrun_notebook_ctrl_module_history, made by the commit 18d4d70.
   • It has seven sections of the documented module history report, plus a section "Data and method".
   • The timeline draws 140 records (27 commits, 23 merge commits, 20 pull requests, 33 issues, and 37 chat sessions). The reader can filter by kind, by day, by text, and by file, and can click a record to see its linked records.
   • The decisions are cards with a decision, a reason, alternatives, and unknowns. Every statement shows a status: recorded fact, unknown, or interpretation.
   • The check script read 70 sources again and found 58 of 58 quotes in the original records, with 0 failures.
   • The folder holds about 3000 lines in 12 files written by hand (7 script files for the page, 2 Python scripts, 1 style sheet, 1 page, and the file of statements), plus 2 data files that the scripts write.

   How I built it, in order

   1. Choose the module. I counted the records for each package of Heidrun Notebook with heidrun_data queries. I chose heidrun_notebook_ctrl because it had the most issues (31) and the most commits that name it (10) of the three packages, and a decision with a stated reason (issue 23).
   2. Check that the data exists before writing any page. I read the issues 1 to 4, 23, 26, 35, 36, 87, and 96, and the pull request 13, with heidrun_data read. I searched 15 chat sessions for the word "Commander".
   3. Collect the records. The script collect_module_history.py runs heidrun_data and writes module_history_data.json:
      • It reads every commit of the repository with one query, then reads the files of each of the 146 commits with heidrun_data read commit <hash> --no-diff, started in the folder of the repository.
      • It reads every issue and every pull request with one query each.
      • It finds the chat sessions with the link tables chat_session_issues and chat_session_pull_requests, and with the table chat_session_files.
   4. Write the statements by hand. I read each record and wrote module_history_statements.json. Every statement has a status and at least one source. Most sources hold an exact quote.
   5. Check the statements by script. verify_module_history.py reads each source again with heidrun_data and tests that the quote is in the record. I tested the script on a false quote and on a false field value: it failed both times, as it must.
   6. Build the page. The page is one web page file, one style sheet, and script files that each hold one class. The timeline is an image drawn in the page, with one row for each kind of record. The selected record draws red curves to its linked records. No library is used.
   7. Test in a real browser. I opened the page through the local server and ran the filters, the day buttons, the search, and the file filter. I read the console: no error.
   8. Update the navigation and commit. I added the experiment to the workspace page and made one commit with "fixes Reconstruct the history of the package heidrun_notebook_ctrl as an interactive web page #7".

   How it went: problems and fixes

   • The installed heidrun_data does not work. The link ~/.local/bin/heidrun_data points to a file that does not exist, and heidrun_code_data does not know the option --database. I used the build target/debug/heidrun_data of this repository.
   • read chat_session fails in testbed_folder/. It looks for the chat session files of the folder where it starts. The fix is to start it in the folder of the repository Heidrun Notebook. The script does this for every read command.
   • A shell mistake. In zsh, an unquoted variable that holds --database <path> is one word, so the option was "not known". I wrote the option and its value as two words.
   • read issue does not accept --no-diff. read pull_request and read commit do.
   • My first count was too low. The first data check found 10 commits by searching the message for the package name. Reading the files of all 146 commits found 50 commits that change the package: 27 commits and 23 merge commits. The page uses 50. The body of this issue still says 10, because I did not edit it.
   • Rename notation in the file lists. read commit writes a moved file as {src => end_to_end}/file.ts, and writes a path with special characters in quotes. The first table showed these raw texts, and a fake area named "data_fixture. I added a small reader for this notation in the collector, and the 78 paths became 68.
   • A silent failure. I chained a sed command that does not run on macOS with the command that wrote the file of the timeline, so the file was never written. I saw it when I listed the files, and I wrote the file again. The lesson: do not chain a file write after a command that can fail.
   • Python option. python3 -I blocks the import of a script of the same folder, so the check script ran without -I.
   • Found in the browser, then fixed. The commit check box said 50 while the row said 27 (the merge commits were counted). The axis title hid the first time label. One statement said "four problems" and listed three.
   • Style rules. I wrote the scripts with 4 spaces, then converted the indentation to tabs with a small program. I wrapped 33 lines longer than 120 characters, and checked the syntax of each file with node --check.

   Choices I made

   • What belongs to the module. A commit belongs when it changes a file under packages/heidrun_notebook_ctrl/. A pull request belongs when its merge commit changes the package or its text names the package. An issue belongs when a commit or pull request of the package says "fixes" or "part of" for it, or when its title or body names the package. A chat session belongs when the data links it to those records or when it wrote a file of the package. A record is "direct" when its link is strong, and "related" when it only mentions the package. Related records are faded, not hidden.
   • Only heidrun_data supplies data. This is the rule of testbed_folder/AGENTS.md. So the page does not read the current source files, and the Summary cannot say what the package does today. I wrote this as an unknown statement.
   • No invented relationships. The data has no static code analysis. The section "Current relationships" shows only the areas that changed in the same commits, with one interpretation that says this is not proof of a dependency.
   • Missing reasons stay missing. For the library Commander, no issue, no commit message, and no chat session gives a reason. The only 3 mentions (in 2 chat sessions) are summaries of the technology stack. The page says "Unknown" and shows where I searched. The search is a keyword search, and the page says so.
   • Statements by hand, checks by script. A person reads and judges. A script only proves that the source exists and holds the quote. The script cannot prove that my reading is right.
   • Links between records only from stated facts. A number in a message ("fixes #23", "Part of Move the package data_layer_user_activity back to the repository heidrun_code #3"), the merge commit of a pull request, or a link table. A non-merge commit is not linked to its pull request, because the data does not give that link.
   • Times in Coordinated Universal Time (UTC). The data holds UTC, and the page does not convert it.
   • Chat sessions have no address. The page shows the command heidrun_data read chat_session <id> with a copy button.
   • The data loads with fetch. A page opened as a local file cannot load its data, so the page shows a message that explains how to open it through the local server.

   How to repeat it

   1. Start the local server in testbed_folder/: python3 outputs/serve_outputs.py.
   2. In the experiment folder, run python3 collect_module_history.py, then python3 verify_module_history.py.
   3. Open http://127.0.0.1:8765/workspace_heidrun_data_example_database/experiment_heidrun_notebook_ctrl_module_history/index.html.

   The collector expects the build target/debug/heidrun_data (or the variable HEIDRUN_DATA_COMMAND) and the repository Heidrun Notebook in ~/webwork/heidrun_notebook. For another module, change the constants at the top of collect_module_history.py and write a new file of statements by hand.

   What I would do differently, and what Heidrun Data lacks

   • Start the data check by reading the files of every commit, not by searching the messages.
   • Give the table commits the names of the changed files, with renames resolved. Today one read commit call is needed for each commit (146 calls).
   • Give a way to read the current source file, and a static code analysis, so the Summary and the relationships can be checked against the code.
   • Give a text search in the chat sessions. Today I read the 15 chat sessions one by one.
   • Link a non-merge commit to its pull request.
   • Open points: the answers to the two open questions of issue 96, whether a limit on wrong codes shipped with the sign in, and the corrected count of commits in the body of this issue.