GSoC Behavioral evals, Quality, and the OSS Community

[gundermanc]

This item is tracking a Google Summer of Code project during which a participant will work closely with the Agent Intelligence team at Google on Gemini CLI to improve the OSS community's ability to contribute to quality impacting areas of the Gemini CLI, including the prompt, tools, behavioral evals, and various pre-release guardrails.

The participant will do some or all of the following, depending on time:

• Onboard to the quality area and learn how to iterate and validate changes.
• Write and stabilize behavioral eval tests and corresponding prompt and tool changes, to refine the product's prompt driven features.
• Write skills or subagents for helping 1st and 3rd party contributors work more effectively with evals, prompt changes, subagents, and skills.
• Identify and fix processes, documentation, logging, and tooling gaps that impede ones ability to diagnose and fix bugs in the agent, subagent, skills, prompts, and tools.
• Implement tooling for external contributors to inventory, assess gaps in, generate, validate, and improve behavioral evals tests.
• Implement tooling for generating evals or unit tests from chat logs

Finally, and most importantly -- dogfood and improve the community contribution scenario for the agent intelligence area and help us to build a more vibrant quality ecosystem with the community.

[deadsmash07]

I'd love to work on this. I'm a fourth-year CS student at IIT Delhi (dual degree) and this aligns closely with my research and contributions so far.

I've already contributed to the repo -- got my environment sanitization fix merged in #20514, and opened #20552 identifying missing test coverage in grep-utils. On the research side, I have a solo-authored paper accepted at ICLR 2026 on probing how reasoning models encode hierarchical structure across layers (hyperbolic vs euclidean probing classifiers), and a paper accepted at ACL (ARR October) where I built an end-to-end framework for generating multi-hop QA benchmarks from knowledge graphs to evaluate long-context reasoning across 15 LLMs.

I've also built an LLM code agent evaluation framework from scratch that measures multi-step coding capabilities with iterative self-correction, so the eval tooling and quality iteration workflow described here is something I have direct hands-on experience with. Happy to put together a detailed proposal and start on a proof of concept.

[HVVSATHWIK]

Hi @gundermanc, I would love to submit a proposal for this project.

The requirement for "generating evals from chat logs" aligns perfectly with my recent work building AI-assisted monitoring pipelines that parse unstructured logs into structured test assertions.

Looking at the architecture, a clean path forward is intercepting the conversation JSON from chatRecordingService.ts (or activityLogger.ts) and orchestrating a sub-agent to translate those interactions into replayable Vitest scenarios modeled after model_steering.eval.ts.

I’m currently drafting a technical proposal for this chat-log-to-eval pipeline. Let me know if you have specific constraints for the sub-agent tooling you'd like to see included!

[ayazhankadessova]

Hello @gundermanc,

I’m a CS MSc/PhD student at KAUST🇸🇦, my graduate research is on Agentize an open-source AI agent SDK, where I’ve worked on eval harnesses (swe-bench, Nginx), cost and token tracking, an FSM-based orchestrator, and multi-agent code review flows, similar to the eval and quality infrastructure described in this issue. I’m especially interested in the “tooling for external contributors to inventory, assess gaps in, generate, validate, and improve behavioral eval tests” goal, as well as the related long-context and complex reasoning eval work.

I've tried contributing to gemini-cli’s eval infra and core, including #23361, #23360, #23358, #23359, and #23354.

I’d be happy to share a more detailed proposal and some concrete milestone ideas if that would be helpful.

[yugjindal22]

Hey @gundermanc, I'm Yug, a pre-final year computer science undergrad.

I am really interested in this one! The angle of generating evals from chat logs and building tooling for external contributors to validate prompt-driven behavior is something I'd love to work on.

I've gone through the evals README... is there a known gap in eval coverage that would make a good starter contribution before the proposal deadline?

[PoleacLiviuStefan]

/assign

[gemini-cli]

👋 @PoleacLiviuStefan, thanks for your interest in this issue! We're reserving self-assignment for issues that have been marked with the help wanted label. Feel free to check out our list of issues that need attention.