feat(ingest/superset): add domains and platform_instance support by rinzool · Pull Request #9647 · datahub-project/datahub

rinzool · 2024-01-17T16:08:21Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Description

Hi, in my company we really need a simple way to add a domain to chart/dashboard entities coming from Superset.
So I copied what is done for Glue ingestion and add a domain argument in the config

I tried locally and it works as I expected 👌

Furthermore I add a platform_instance argument and deprecate env argument that commit b088512 (It may be done in another PR if necessary)

Example output when running ingestion:

Config:

source:
  type: superset
  config:
    # Coordinates
    connect_uri: ***
    env: "STG"
    platform_instance: "my_awesome_platform"
    domain: 
      my-domain:
        allow: [".*"]

    # Credentials
    username: admin
    password: ***
    provider: db

Output:

ConfigurationWarning: env is deprecated and will be removed in a future release. Please use platform_instance instead.
...
{'events_produced': 455,
 'events_produced_per_sec': 44,
 'entities': {'dashboard': ['urn:li:dashboard:(superset,my_awesome_platform.623)',
                            'urn:li:dashboard:(superset,my_awesome_platform.433)',
                            'urn:li:dashboard:(superset,my_awesome_platform.171)',
                            'urn:li:dashboard:(superset,my_awesome_platform.476)',
                            'urn:li:dashboard:(superset,my_awesome_platform.385)',
                            'urn:li:dashboard:(superset,my_awesome_platform.239)',
                            'urn:li:dashboard:(superset,my_awesome_platform.219)',
                            'urn:li:dashboard:(superset,my_awesome_platform.188)',
                            'urn:li:dashboard:(superset,my_awesome_platform.155)',
                            'urn:li:dashboard:(superset,my_awesome_platform.19)',

…nzool/datahub into feat/add-domain-argument-for-superset

hsheth2 · 2024-01-18T19:53:20Z


    def construct_dashboard_from_api_data(self, dashboard_data):
-        dashboard_urn = f"urn:li:dashboard:({self.platform},{dashboard_data['id']})"
+        dashboard_urn = f"urn:li:dashboard:({self.platform},{self.platform_instance}{dashboard_data['id']})"


let's use our make_chart_urn and make_dashboard_urn helpers - they also have a proper platform_instance kwargs, which should also let us avoid needing self.platform_instance

Thank you for the tip 👍
It works fine perfectly 👌

Example of logs:

'entities': {'dashboard': ['urn:li:dashboard:(superset,prod-bedrock.508)', 'urn:li:dashboard:(superset,prod-bedrock.543)', 'urn:li:dashboard:(superset,prod-bedrock.242)', 'urn:li:dashboard:(superset,prod-bedrock.337)', 'urn:li:dashboard:(superset,prod-bedrock.509)', 'urn:li:dashboard:(superset,prod-bedrock.567)', 'urn:li:dashboard:(superset,prod-bedrock.118)', 'urn:li:dashboard:(superset,prod-bedrock.428)', 'urn:li:dashboard:(superset,prod-bedrock.340)', 'urn:li:dashboard:(superset,prod-bedrock.183)', '... sampled of 212 total elements'], 'chart': ['urn:li:chart:(superset,prod-bedrock.3143)', 'urn:li:chart:(superset,prod-bedrock.9784)', 'urn:li:chart:(superset,prod-bedrock.11400)', 'urn:li:chart:(superset,prod-bedrock.11399)', 'urn:li:chart:(superset,prod-bedrock.11398)', 'urn:li:chart:(superset,prod-bedrock.10060)', 'urn:li:chart:(superset,prod-bedrock.10052)', 'urn:li:chart:(superset,prod-bedrock.10050)', 'urn:li:chart:(superset,prod-bedrock.10046)', 'urn:li:chart:(superset,prod-bedrock.11396)',

hsheth2 · 2024-01-18T19:58:24Z

    def get_report(self) -> StaleEntityRemovalSourceReport:
        return self.report
+
+    def _gen_domain_urn(self, title: str) -> Optional[str]:


we have a "DomainRegistry" helper, which lets us convert domain names into urns using a lookup against DataHub - let's see if we can reuse the same pattern

A good example to copy would be the kafka integration's domain support:

datahub/metadata-ingestion/src/datahub/ingestion/source/kafka.py

Line 445 in 1741c07

for domain, pattern in self.source_config.domain.items():

Ok thanks, it's done 👍 Let me know if it is what you expected

Although, I removed the _get_domain_urn to have only one function _get_domain_wu (I don't think it's relevant to have two distinct functions here)

hsheth2

LGTM

anshbansal · 2024-01-20T09:27:07Z

@rinzool We merged in your other superset PRs. Seems they caused a conflict. If you could resolve that and ensure lint is passing we can get this merged in

rinzool · 2024-01-20T13:03:47Z

@rinzool We merged in your other superset PRs. Seems they caused a conflict. If you could resolve that and ensure lint is passing we can get this merged in

Yep with 3 PRs I expected some conflicts 😅 I resolved them, CI should be successful (it's ok locally)

Note that I can edit unit tests to add a test for domain and platform instance too

…ument-for-superset

… mappings (#9647) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…harts Superset has accepted platform_instance since datahub-project#9647 but never emitted the dataPlatformInstance aspect, so the Navigate panel files every dashboard and chart under "Default" no matter how many deployments are ingested. Add a get_data_platform_instance helper, modelled on Tableau's except that it returns nothing when platform_instance is unset, so existing deployments and the five existing goldens are untouched. Append the aspect at the dashboard and chart snapshot sites and declare the PLATFORM_INSTANCE capability. Preset builds on this source and inherits both with no code of its own; a unit test pins that its aspect resolves to the preset platform. Datasets are deliberately left out. Their URNs are built through get_datasource_urn_from_id with plain make_dataset_urn and carry no instance (the instance-aware gen_dataset_urn helper exists but is unused), so an aspect on them would name an instance the URN does not have, and two deployments exposing the same table would overwrite each other's aspect on the one entity. Making the dataset URN instance-aware is a breaking change that also moves chart-to-dataset lineage, so it belongs with the containers follow-up. A unit test pins the omission and the TODO says why. Tests: unit present/absent cases for dashboards and charts, an empty-string case, the Preset inheritance cases, and one integration test with a new golden that sets platform_instance. Docs: a Platform instance section on both connector pages. The connector registry JSON is regenerated. Fixes datahub-project#11153

feat: add a new argument domani to superset source

a551ced

github-actions Bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Jan 17, 2024

rinzool changed the title ~~feat: add a new argument domani to superset source~~ feat(superset): add a new argument domani to superset source Jan 17, 2024

vercel Bot deployed to Preview January 17, 2024 16:28 View deployment

rinzool added 2 commits January 17, 2024 17:29

style: reformat file using black

dc9ca65

Merge branch 'master' into feat/add-domain-argument-for-superset

b492f3b

vercel Bot deployed to Preview January 17, 2024 17:28 View deployment

rinzool added 4 commits January 18, 2024 09:46

Merge branch 'master' into feat/add-domain-argument-for-superset

2a7d708

feat(superset): add platform_instance argument and deprecate env

b088512

Merge branch 'feat/add-domain-argument-for-superset' of github.com:ri…

5307344

…nzool/datahub into feat/add-domain-argument-for-superset

Merge branch 'master' into feat/add-domain-argument-for-superset

c063538

rinzool changed the title ~~feat(superset): add a new argument domani to superset source~~ feat(superset): add a new argument domain and platform_instance to superset source Jan 18, 2024

rinzool added 2 commits January 18, 2024 11:03

feat: remove env argument since it's already inherited

1056a47

Merge branch 'feat/add-domain-argument-for-superset' of github.com:ri…

740dc69

…nzool/datahub into feat/add-domain-argument-for-superset

vercel Bot deployed to Preview January 18, 2024 10:48 View deployment

rinzool added 2 commits January 18, 2024 15:50

fix: remove unused import

946bc60

Merge branch 'master' into feat/add-domain-argument-for-superset

1badae6

vercel Bot deployed to Preview January 18, 2024 16:22 View deployment

hsheth2 reviewed Jan 18, 2024

View reviewed changes

ref: use make_chart_urn and make_dashboard_urn functions

075ded0

rinzool force-pushed the feat/add-domain-argument-for-superset branch 2 times, most recently from 480a18e to bd91e40 Compare January 19, 2024 08:47

ref: use domain registry

c19bd28

rinzool force-pushed the feat/add-domain-argument-for-superset branch from bd91e40 to c19bd28 Compare January 19, 2024 08:49

vercel Bot deployed to Preview January 19, 2024 09:25 View deployment

hsheth2 approved these changes Jan 20, 2024

View reviewed changes

hsheth2 added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Jan 20, 2024

vercel Bot deployed to Preview January 20, 2024 01:51 View deployment

rinzool force-pushed the feat/add-domain-argument-for-superset branch 2 times, most recently from 7b4a3f9 to baa0565 Compare January 20, 2024 13:02

vercel Bot deployed to Preview January 20, 2024 13:38 View deployment

Merge remote-tracking branch 'origin/master' into feat/add-domain-arg…

0d8fd42

…ument-for-superset

rinzool force-pushed the feat/add-domain-argument-for-superset branch from baa0565 to 0d8fd42 Compare January 20, 2024 16:37

vercel Bot deployed to Preview January 20, 2024 17:01 View deployment

hsheth2 changed the title ~~feat(superset): add a new argument domain and platform_instance to superset source~~ feat(ingest/superset): add domains and platform_instance support Jan 21, 2024

hsheth2 merged commit 344eeae into datahub-project:master Jan 21, 2024

rinzool deleted the feat/add-domain-argument-for-superset branch January 22, 2024 08:42

joshuaclausen mentioned this pull request Aug 12, 2024

platform_instance is not shown for Superset platform assets #11153

Open

david-leifker pushed a commit that referenced this pull request May 27, 2026

feat(slack): allow registered bots to ask DataHub via service account…

7781b45

… mappings (#9647) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Danishlynx mentioned this pull request Sep 20, 2026

feat(ingest/superset): emit dataPlatformInstance for dashboards and charts #19879

Open

4 tasks

Navigation Menu

feat(ingest/superset): add domains and platform_instance support - #9647

feat(ingest/superset): add domains and platform_instance support#9647
hsheth2 merged 14 commits into
datahub-project:masterfrom
rinzool:feat/add-domain-argument-for-superset

rinzool commented Jan 17, 2024 •

edited

Loading

Uh oh!

hsheth2 Jan 18, 2024

Uh oh!

rinzool Jan 19, 2024

Uh oh!

hsheth2 Jan 18, 2024

Uh oh!

rinzool Jan 19, 2024

Uh oh!

rinzool Jan 19, 2024

Uh oh!

hsheth2 left a comment

Uh oh!

anshbansal commented Jan 20, 2024

Uh oh!

rinzool commented Jan 20, 2024 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

rinzool commented Jan 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Description

Uh oh!

hsheth2 Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

rinzool Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

hsheth2 Jan 18, 2024

Choose a reason for hiding this comment

Uh oh!

rinzool Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

rinzool Jan 19, 2024

Choose a reason for hiding this comment

Uh oh!

hsheth2 left a comment

Choose a reason for hiding this comment

Uh oh!

anshbansal commented Jan 20, 2024

Uh oh!

rinzool commented Jan 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rinzool commented Jan 17, 2024 •

edited

Loading

rinzool commented Jan 20, 2024 •

edited

Loading