[CI Failure Doctor] Pre-existing JavaScript test failures in CI (47 failures across 4 test files)

[github-actions]

Summary

The JavaScript test suite has 47 pre-existing test failures across 4 test files that are causing CI runs to fail. These failures are not related to recent code changes - they occur consistently on the main branch.

Failure Details

• Run: 21225238947
• Commit: bc08279
• Trigger: push
• Job: js (ID: 61070350153)
• Failed Step: "Run tests"

Root Cause Analysis

Verification

✅ Confirmed: These test failures are pre-existing and unrelated to the commit that triggered the failure:

• Tested the failing commit (bc08279): 47 failures
• Tested current main branch (b786b66): 47 failures (same failures)
• The failing commit only modified Go files and campaign templates, not JavaScript test files

Failure Categories

1. collect_ndjson_output.test.cjs - 36 failures

Missing safe output type validators:

• add-labels type validation not recognizing the type
• create-pull-request type validation failing
• create-discussion type validation failing
• create-pull-request-review-comment type validation failing
• update_release type validation failing
• assign_milestone type validation failing
• assign_to_agent type validation failing
• link_sub_issue type validation failing

JSON repair functionality issues:

• Array syntax repair not working
• Wrong bracket type repair failing
• Mixed bracket types not handled
• Simple missing closing brackets not repaired

Code scanning alert validation (7 failures):

• Valid entries not being recognized
• Invalid entries not being rejected properly
• Field type validation not working

Min validation tests (5 failures):

• Minimum requirement checks not enforcing limits
• Multiple types with different mins not handled

Example error:

AssertionError: expected [] to have a length of 1 but got +0
 ❯ collect_ndjson_output.test.cjs:205:35

2. safe_outputs_mcp_server_defaults.test.cjs - 2 failures (timeouts)

Test timeouts (10 seconds):

• should have optional branch parameter for create_pull_request - times out waiting for MCP server response
• should have optional branch parameter for push_to_pull_request_branch - times out waiting for MCP server response

Uncaught exceptions:

AssertionError: expected undefined to deeply equal [ 'title', 'body' ]
 ❯ Timeout._onTimeout safe_outputs_mcp_server_defaults.test.cjs:182:57

3. markdown_code_region_balancer.test.cjs - 5 failures

Fence balancing logic issues:

• Multiple language specifiers: adds 4 backticks instead of 3
• AI-generated code with nested markdown: adds extra backtick
• Documentation with multiple code examples: adds extra backtick
• Deeply nested examples: adds extra backtick
• Consecutive code blocks: adds extra backtick

Example error:

AssertionError: expected '````javascript\njs code\n```' to be '```javascript\njs code\n```'
 ❯ markdown_code_region_balancer.test.cjs:262:52

Impact Assessment

Severity: 🔴 Critical - CI is failing on every push to main

Scope:

• Affects all CI runs
• Blocks merges that rely on CI passing
• Creates noise making it hard to identify real failures

Test Health Metrics:

• Total tests: 2,939
• Passing: 2,875 (97.8%)
• Failing: 47 (1.6%)
• Skipped: 17 (0.6%)

Recommended Actions

Immediate (Fix CI)

1. Investigate collect_ndjson_output.test.cjs validator issues

   • Check if safe output type definitions have changed
   • Verify validator registry includes all expected types
   • Review recent changes to safe output validation logic

2. Fix MCP server timeout issues

   • Increase test timeout or mock MCP server responses
   • Check if MCP server is starting properly in tests
   • Review test setup for safe outputs MCP server

3. Fix markdown code region balancer

   • Review recent changes to fence balancing logic
   • Check if balancer is being too aggressive with nested fences
   • Consider the fix already applied in commit d8df6ac

Prevention Strategies

1. Make CI tests a gate: Configure branch protection to require passing tests
2. Add pre-commit hooks: Run npm test before allowing commits
3. Test stability monitoring: Track flaky tests over time
4. Regular test maintenance: Review and fix failing tests weekly

AI Team Self-Improvement

Add to AGENTS.md for AI coding agents:

## JavaScript Testing Requirements

**CRITICAL**: Always run JavaScript tests before committing:

```bash
cd actions/setup/js && npm ci && npm test

Before every commit with JS changes:

1. Run full test suite: npm test
2. Fix any failing tests - do NOT commit with test failures
3. If tests are flaky, investigate and fix or skip with proper documentation
4. Never merge PRs with failing tests

Common test failure patterns:

• Safe output type validators: Ensure all types are registered
• MCP server tests: Check for timeouts and proper mocking
• Markdown balancer: Verify fence handling logic changes

## Historical Context

- Last known passing commit: Unknown (needs investigation)
- Similar failures: None found in issue history
- Related commits:
  - b786b668: "Improve error messages for invalid target configuration in safe outputs (#11066)"
  - 7c863f68: "Move JavaScript tests and production files to actions/setup/js (#7764)"

## Investigation Files

- Investigation report: `/tmp/gh-aw/cache-memory/investigations/2026-01-21-run-21225238947.json`
- Full test output: `/tmp/gh-aw/agent/logs/npm-test-output.txt`

---

**Labels**: `bug`, `ci`, `tests`, `javascript`, `critical`
**Priority**: P0 - Blocks all CI runs




> AI generated by [CI Failure Doctor](https://github.com/githubnext/gh-aw/actions/runs/21225348457)
>
> To add this workflow in your repository, run `gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d`. See [usage guide](https://githubnext.github.io/gh-aw/guides/packaging-imports/).

<!-- gh-aw-agentic-workflow: CI Failure Doctor, engine: copilot, run: https://github.com/githubnext/gh-aw/actions/runs/21225348457 -->

[github-actions]

Additional CI Failure - Run #31015

Another CI run has failed with the exact same 47 test failures reported in this issue:

• Run: 21225360006
• Commit: 0c3a4e0
• PR: Add missing get_repository tool to repos toolset #11067 "Add missing get_repository tool to repos toolset"
• Failed Job: js (ID: 61070763482)
• Trigger: push to main

Failure Summary

Same 47 test failures across 4 test files:

• collect_ndjson_output.test.cjs: 36 failures
• markdown_code_region_balancer.test.cjs: 5 failures
• safe_outputs_mcp_server_defaults.test.cjs: 2 failures
• mcp_handler_go.test.cjs: 4 failures

Key Observations

✅ Confirmed: This is a pre-existing issue unrelated to the commit:

• The failing commit only modified:
  • pkg/workflow/github_toolsets_permissions.json (added get_repository to repos toolset)
  • .github/workflows/test-github-repos-mcp-toolset.lock.yml (recompiled workflow)
• No JavaScript test files were modified
• The same test failures have been occurring since before this commit

Impact

This is the second documented occurrence of these test failures blocking CI:

1. Run #21225238947 (reported in this issue)
2. Run #21225360006 (this comment)

CI is currently broken and will continue failing on every push to main until these tests are fixed.

Investigation Data

Full investigation report saved to: /tmp/memory/investigations/2026-01-21-run-21225360006.json

---

AI generated by CI Failure Doctor - Run 21225535567

AI generated by CI Failure Doctor