Skip to content

fix(arxiv-researcher): optimize token usage and raise max-ai-credits - #49965

Merged
pelikhan merged 4 commits into
mainfrom
copilot/increase-max-ai-credits
Aug 3, 2026
Merged

pelikhan merged 4 commits into
mainfrom
copilot/increase-max-ai-credits

Conversation

Copilot AI commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

The daily arXiv researcher workflow consistently hits the 250 AIC budget (253.7 used on last run), triggering an AI-credits guardrail failure.

Token optimizations

  • Smaller input per screener call: abstract truncation reduced from 1200 → 800 chars; each paper-screener (model: small) gets a smaller context
  • Fewer opportunity-extractor calls: max relevant papers capped 8 → 5; this is the most expensive step (model: large), so this is the highest-leverage reduction (~37% worst-case cost)
  • Fewer arXiv papers fetched: max_results reduced 40 → 25; directly reduces screener fan-out
  • Delayed discussion template loading: moved the discussion body structure from the main prompt body into an inline ## skill: discussion-template block — the runtime only injects it when the agent explicitly retrieves the skill, keeping early turns (Steps 1–4) lean

Budget

max-ai-credits raised 250 → 300 (20% headroom after the optimizations above).

Copilot AI and others added 2 commits August 3, 2026 12:16
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…ts limit

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Update max AI credits for workflow fix(arxiv-researcher): optimize token usage and raise max-ai-credits Aug 3, 2026
Copilot AI requested a review from pelikhan August 3, 2026 12:23
@pelikhan

pelikhan commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

@copilot use sub agents with smaller models as much as possible to filter

…ore large-model extractor

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@copilot use sub agents with smaller models as much as possible to filter

Added a relevance-ranker sub-agent (model: small) between the screener and extractor in commit fix(arxiv-researcher): add small-model relevance-ranker.

The updated pipeline is now:

  1. paper-screener (model: small) — binary pass/fail
  2. relevance-ranker (model: small) — scores 1–5 by gh-aw actionability, keeps top 3
  3. opportunity-extractor (model: large) — runs on at most 3 papers instead of up to 5

Both cheap small-model stages run before any large-model call, so the expensive extractor is only invoked for the highest-value papers.

@pelikhan
pelikhan marked this pull request as ready for review August 3, 2026 12:43
Copilot AI review requested due to automatic review settings August 3, 2026 12:43
@pelikhan
pelikhan merged commit 0b50946 into main Aug 3, 2026
@pelikhan
pelikhan deleted the copilot/increase-max-ai-credits branch August 3, 2026 12:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Optimizes the daily arXiv workflow’s AI-credit usage and increases its budget ceiling.

Changes:

  • Reduces fetched papers and abstract size.
  • Adds small-model ranking before extracting the top three opportunities.
  • Lazily loads the discussion template and raises the budget to 300 AIC.
Show a summary per file
File Description
.github/workflows/daily-arxiv-researcher.md Updates limits, ranking, template loading, and budget.
.github/workflows/daily-arxiv-researcher.lock.yml Regenerates the compiled workflow.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Suppressed comments (2)

.github/workflows/daily-arxiv-researcher.md:164

  • The PR description says the extractor cap changes from 8 to 5, but this implementation adds an undocumented ranking stage and ultimately caps extraction at 3. That materially changes both output coverage and the cost estimate; either implement the documented five-paper cap or update the PR description and estimates to describe the ranker/top-three design.
For each relevant paper, invoke the `relevance-ranker` sub-agent with:

.github/workflows/daily-arxiv-researcher.md:169

  • Keeping only three ranked papers leaves the remaining relevant papers without opportunity or area values, but Step 4 still requires those fields for every paper marked relevant (lines 188–197). Runs with more than three relevant papers therefore cannot satisfy the ledger instructions without inventing data. Track “selected for extraction” separately and define how unselected relevant papers are recorded, or update the ledger schema accordingly.
Sort ranked papers by `score` descending. Keep only the top 3.
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

fs.mkdirSync(BASE_DIR, { recursive: true });

const ARXIV_URL = 'https://export.arxiv.org/api/query?search_query=(cat:cs.AI+OR+cat:cs.SE+OR+cat:cs.LG)+AND+(agentic+OR+%22multi-agent%22+OR+%22llm+agent%22+OR+%22workflow+automation%22+OR+%22code+generation%22+OR+%22ai+agent%22)&max_results=40&sortBy=submittedDate&sortOrder=descending';
const ARXIV_URL = 'https://export.arxiv.org/api/query?search_query=(cat:cs.AI+OR+cat:cs.SE+OR+cat:cs.LG)+AND+(agentic+OR+%22multi-agent%22+OR+%22llm+agent%22+OR+%22workflow+automation%22+OR+%22code+generation%22+OR+%22ai+agent%22)&max_results=25&sortBy=submittedDate&sortOrder=descending';
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.84.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw] arXiv Paper Researcher: GitHub Agentic Workflows exceeded max AI credits

3 participants