Skip to content

feat(snowflake-connectors): meetings implementation [CM-1034]#3998

Open
mbani01 wants to merge 13 commits intomainfrom
feat/meetings_implementation
Open

feat(snowflake-connectors): meetings implementation [CM-1034]#3998
mbani01 wants to merge 13 commits intomainfrom
feat/meetings_implementation

Conversation

@mbani01
Copy link
Copy Markdown
Contributor

@mbani01 mbani01 commented Apr 3, 2026

This pull request adds support for processing and tracking meeting attendance activities from the "meetings" platform. It introduces new activity types for meeting invitations and attendance, integrates the meetings data source into the Snowflake connectors pipeline, and updates relevant enums and configurations to recognize meetings as a valid platform and organization source.

Meetings platform support:

  • Added MEETINGS as a new PlatformType, OrganizationSource, and OrganizationAttributeSource, and updated the source priority list to include meetings. [1] [2] [3] [4]
  • Introduced new meetings activity types (invited-meeting, attended-meeting) and their scoring grid, and registered them in the database. [1] [2]

Snowflake connectors integration:

  • Implemented the meetings attendance data source: added MeetingAttendanceTransformer and its source query, and registered the source in the Snowflake connectors integration index. [1] [2] [3] [4] [5]

Transformer pipeline improvements:

  • Updated the transformer pipeline to support multiple activities per row: transformRow and safeTransformRow now handle arrays of activities, and the consumer processes all returned activities. [1] [2] [3] [4]

Integrations exports:

  • Exported meetings types from the integrations library for use throughout the codebase.

Note

Medium Risk
Adds a new Snowflake connector source plus new enums/migrations, and changes the transformation pipeline to process multiple activities per input row, which can affect downstream activity volume/dedup behavior if any transformer starts returning arrays unexpectedly.

Overview
Adds a new meetings Snowflake connector source for meeting attendance, including buildSourceQuery (segment matching, org lookup, incremental + dedup logic) and a MeetingAttendanceTransformer that can emit invited and/or attended meeting activities from a single exported row.

Extends the Snowflake connectors framework to support multiple activities per row by updating TransformerBase.transformRow/safeTransformRow to allow arrays and updating TransformerConsumer to iterate and emit each result.

Registers the new platform and activity types across the stack: PlatformType.MEETINGS, meetings activity types + scoring grid (exported from @crowd/integrations), Flyway migration to insert new activityTypes, and organization source/attribute source updates (including org attribute source priority) to recognize MEETINGS.

Reviewed by Cursor Bugbot for commit e7b8669. Bugbot is set up for automated code reviews on this repo. Configure here.

mbani01 added 2 commits April 3, 2026 13:41
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
@mbani01 mbani01 self-assigned this Apr 3, 2026
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

mbani01 added 2 commits April 3, 2026 15:27
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
@mbani01 mbani01 marked this pull request as ready for review April 3, 2026 14:53
@mbani01 mbani01 requested review from Copilot and joanagmaia April 3, 2026 14:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds “meetings” platform support to the Snowflake connectors pipeline, including new activity types for meeting invitations and attendance, plus the necessary enum/config updates so meetings is recognized across the codebase.

Changes:

  • Introduces MEETINGS as a supported platform/org source and wires it into source priority.
  • Adds meetings activity types (invited-meeting, attended-meeting) with scoring grid + DB registration.
  • Extends the transformer pipeline to allow a single row to produce multiple activities and updates the consumer to emit them all.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
services/libs/types/src/enums/platforms.ts Adds MEETINGS to PlatformType.
services/libs/types/src/enums/organizations.ts Adds MEETINGS to organization source enums.
services/libs/integrations/src/integrations/meetings/types.ts Defines meetings activity types + scoring grid.
services/libs/integrations/src/integrations/index.ts Exports meetings integration types.
services/libs/data-access-layer/src/organizations/attributesConfig.ts Adds meetings to org attribute source priority list.
services/apps/snowflake_connectors/src/integrations/types.ts Registers meetings meeting-attendance as a data source name.
services/apps/snowflake_connectors/src/integrations/meetings/meeting-attendance/transformer.ts Implements meetings attendance transformer producing invited/attended activities.
services/apps/snowflake_connectors/src/integrations/meetings/meeting-attendance/buildSourceQuery.ts Adds Snowflake query for exporting meeting attendance rows + incremental logic.
services/apps/snowflake_connectors/src/integrations/index.ts Registers meetings platform and its source in the connectors registry.
services/apps/snowflake_connectors/src/core/transformerBase.ts Updates transformer API to allow multiple activities per row; normalizes safeTransformRow to arrays.
services/apps/snowflake_connectors/src/consumer/transformerConsumer.ts Updates consumer to handle multiple transformed activities per input row.
backend/src/database/migrations/V1775219382__addMeetingsActivityTypes.sql Inserts new meetings activity types into activityTypes.
backend/src/database/migrations/U1775219382__addMeetingsActivityTypes.sql (Present but empty) rollback migration file for the above insert.
.claude/skills/scaffold-snowflake-connector/SKILL.md Updates internal scaffolding guidance related to org mapping discovery.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +98 to 111
for (const result of results) {
const resolved = await this.integrationResolver.resolve(platform, result.segment)
if (!resolved) {
resolveSkippedCount++
continue
}

await this.emitter.createAndProcessActivityResult(
resolved.segmentId,
resolved.integrationId,
result.activity,
)
transformedCount++
}
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that transformers may return multiple activities per row, this loop resolves the same segment repeatedly when multiple activities share it (common when a row yields multiple activity types). Consider de-duplicating integrationResolver.resolve(...) calls per row (e.g., cache by slug+sourceId) to avoid extra Redis/DB lookups and reduce latency.

Copilot uses AI. Check for mistakes.
platform: PlatformType.MEETINGS,
timestamp,
score: MEETINGS_GRID[MeetingsActivityType.INVITED_MEETING].score,
sourceId: `${primaryKey}_invited`,
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joanagmaia, this one is different from specs, as meeting_id isn't unique (similar to committees). Used primaryKey as it seems to be unique

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
: null
if (!date || isNaN(date.getTime())) return null
if (typeof rawTime === 'number') {
date.setTime(date.getTime() + rawTime)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parquet TIME value unit assumption may produce wrong timestamps

High Severity

toISOTimestamp adds rawTime directly to the date via date.setTime(date.getTime() + rawTime), assuming the Parquet TIME column value is in milliseconds. Snowflake typically exports TIME columns to Parquet using the TIME_MICROS logical type (microseconds since midnight). If @dsnp/parquetjs returns the raw microsecond value as a number, every meeting timestamp would be off by a factor of 1000 — e.g., a 10 AM meeting would appear ~416 days in the future. This is the first transformer in the codebase to handle a separate TIME column, so there's no prior precedent to validate the assumption.

Fix in Cursor Fix in Web

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

if (!accountName && !website && !domainAliases) {
return undefined
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Organizations created without identities when only account name exists

Medium Severity

The buildOrganizations guard (!accountName && !website && !domainAliases) differs from the established pattern in Cvent/TNC, which only create orgs when domain data exists (!website && !domainAliases). When only ACCOUNT_NAME from the main meeting table is present but the org LEFT JOIN produces no match, an organization is created with a displayName but an empty identities array, no domain, no logo, and no industry. These identity-less org records cannot be merged with other org records for the same company, leading to duplicate organization entries.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 188c6c5. Configure here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mbani01 also valid feedback if we are attempting to create organizations without identities -> this will fail

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot this one, will get back to it tomorrow

mbani01 and others added 5 commits April 9, 2026 15:50
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mbani@contractor.linuxfoundation.org>
@mbani01 mbani01 requested a review from joanagmaia April 9, 2026 17:25
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e7b8669. Configure here.

{
name: DataSourceName.MEETINGS_MEETING_ATTENDANCE,
buildSourceQuery: meetingAttendanceBuildQuery,
transformer: new MeetingAttendanceTransformer(),
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEETINGS entry missing closing brackets in platform registry

High Severity

The [PlatformType.MEETINGS] entry in the supported object is missing its closing }, ], and }, after the transformer line. Line 32 (transformer: new MeetingAttendanceTransformer(),) is immediately followed by [PlatformType.COMMITTEES] on line 33 without any closing brackets. This causes all subsequent platform definitions (COMMITTEES, CVENT, TNC) to be parsed as nested computed properties inside the MEETINGS source object rather than as top-level entries in supported. This breaks the entire platform registry — no platform will resolve correctly at runtime.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e7b8669. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants