You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add a Spark skill: how to write Spark Connect code in IceGraph #137
Do this after #132 (the CLAUDE.md -> AGENTS.md migration), so the skill hangs off AGENTS.md and not CLAUDE.md.
Write an agent skill that captures how Spark code is actually written in this repo, so agents stop producing round-trip-heavy or accidentally-eager Spark.
Must cover
Spark is lazy. Transformations only build a plan; nothing executes until an action.
Spark Connect is extra lazy. Even metadata resolution is deferred: spark.read... / spark.table(...) does not hit the catalog or files until something forces it. Call out the peripheral triggers (Metadata information: .schema, .columns, .printSchema(), or an Action .count(), .collect(), .toPandas()).
Actions and collect(). When collecting is correct, what it costs (every row lands on the driver), and the caps IceGraph already enforces (MAX_DATA_FILES_TO_COLLECT, MAX_SNAPSHOTS_TO_COMPUTE).
One collect, many things. Prefer building a single DataFrame (union / aggregate / join) and collecting once over collecting per snapshot, per manifest, or per file.
Repo conventions. Where Spark calls belong (backend/collectors/, backend/extractors/, backend/base_classes/), the shared session, and the read-only + Iceberg v2 constraints.
Activity