chore: automatic repository groups parsing for openstack (CM-1100)

[joanagmaia]

This pull request introduces a new scheduled job to the cron_service that syncs OpenStack repository groups from an upstream YAML source. It also adds the necessary dependencies for YAML parsing. The main changes are:

New OpenStack repository groups sync job:

• Added openstackRepositoryGroups.job.ts which fetches OpenStack governance YAML, parses project/repo info, and upserts repository groups in the database. The job is scheduled to run weekly and only includes repositories present in the database.

Dependency updates:

• Added js-yaml and its type definitions (@types/js-yaml) to package.json and pnpm-lock.yaml for YAML parsing support. [1] [2] [3] [4] [5] [6]

These changes enable automatic, reliable syncing of OpenStack repository groups from their official governance source, ensuring the database remains up-to-date with the latest project structure.

---

Note

Medium Risk
Adds a new scheduled job that fetches and parses a remote YAML file and then upserts repository-group records in the write DB; failures or YAML format changes could lead to missed/incorrect grouping updates. Dependency changes are low risk, but the automated DB-writing sync increases operational risk.

Overview
Adds a new weekly cron_service job (openstackRepositoryGroups.job.ts) that fetches OpenStack’s upstream governance projects.yaml, parses project→repo mappings, and creates/updates repository groups for the OpenStack insights project based on repos that already exist in the DB.

Updates cron_service dependencies to include js-yaml (and @types/js-yaml) to support YAML parsing, with corresponding pnpm-lock.yaml changes.

^{Reviewed by Cursor Bugbot for commit ca8c62d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[Copilot]

Pull request overview

Adds a new weekly cron job in cron_service to sync OpenStack repository groups from the OpenDev governance YAML into the database, and introduces YAML parsing dependencies to support it.

Changes:

• Added openstackRepositoryGroups.job.ts to fetch/parse OpenStack governance YAML and upsert repository groups for an insights project.
• Added js-yaml (+ @types/js-yaml) to enable YAML parsing in the cron service.
• Updated pnpm-lock.yaml accordingly.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 4 comments.

File	Description
services/apps/cron_service/src/jobs/openstackRepositoryGroups.job.ts	New scheduled job that fetches OpenStack governance YAML, maps projects to repo URLs present in DB, and upserts repository groups.
services/apps/cron_service/package.json	Adds js-yaml runtime dependency and @types/js-yaml for TypeScript.
pnpm-lock.yaml	Locks new YAML parsing dependencies and updates dependency graph.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Inconsistent slug generation risks duplicate repository groups

Low Severity

The new toSlug function generates slugs differently than the existing getCleanString(name).replace(/\s+/g, '-') used in collectionService.ts for repository group slugs. For names with special characters like periods, toSlug converts them to hyphens while getCleanString strips them entirely (e.g., "oslo.messaging" → "oslo-messaging" vs "oslomessaging"). Since both the API and cron job create and match repository groups by slug, this inconsistency could produce duplicates. @crowd/common (which exports getCleanString) is already a dependency of the cron service.

Additional Locations (1)

• services/apps/cron_service/src/jobs/openstackRepositoryGroups.job.ts#L165-L168

 

^{Reviewed by Cursor Bugbot for commit 4e024be. Configure here.}

[ulemons]

Looks good to me overall ✅
Maybe we could move some of these logs to info? With everything at debug the job runs completely silently in production

[joanagmaia]

Looks good to me overall ✅ Maybe we could move some of these logs to info? With everything at debug the job runs completely silently in production

Very good point. Will update, thank you for the review!

[cursor]

Slug lookup may silently fail due to casing

Medium Severity

insightsProjectSlug is set to 'OpenStack' (mixed case), but every slug generation path in the codebase produces lowercase output — generate_slug in migrations applies lower(), getCleanString calls .toLowerCase(), and even this file's own toSlug lowercases. PostgreSQL TEXT comparison is case-sensitive by default, so WHERE slug = 'OpenStack' won't match a stored slug of 'openstack'. This would cause the job to silently skip the entire sync, logging only a warning.

Additional Locations (1)

• services/apps/cron_service/src/jobs/openstackRepositoryGroups.job.ts#L107-L109

 

^{Reviewed by Cursor Bugbot for commit f272d0b. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ca8c62d. Configure here.}

[cursor]

Wrong URL base causes zero repository matches

High Severity

The repoUrlBase is set to https://review.opendev.org/, but the git integration service (services/apps/git_integration/src/crowdgit/services/utils.py) explicitly converts review.opendev.org URLs to opendev.org URLs before storing them. This means the database stores repo URLs with https://opendev.org/ as the base. Since getRepositoriesByUrl does an exact URL match, every candidate URL generated by this job will fail to match, causing the job to silently skip all projects every week while reporting success.

 

^{Reviewed by Cursor Bugbot for commit ca8c62d. Configure here.}