Skip to content

Druid Cannot Pushdown Offset #17957

Closed
Closed

Description

@carltal

Affected Version

29.0.1 and below (have not tested newer versions)

The Druid version where the problem was encountered.
29.0.1

Description

For the following SQL query, Druid throws an error: cannot pushdown offset.
SELECT SUM(sale_quantity) AS sale_quantity, SUM(sale_total_amount) AS sale_total_amount, SUM(sale_revenue) AS sale_revenue, SUM(sale_revenue) AS sale_percent_of_revenue, SUM(refund_quantity) AS refund_quantity, SUM(refund_total_amount) AS refund_total_amount, SUM(refund_revenue) AS refund_revenue, SUM(refund_revenue) AS refund_percent_of_revenue, SUM(sale_quantity) - SUM(refund_quantity) AS netsale_quantity, SUM(sale_revenue) - SUM(refund_revenue) AS netsale_revenue, SUM(sale_revenue) - SUM(refund_revenue) AS netsale_percent_of_revenue, FLOOR(TIME_PARSE (event_time, NULL) to MINUTE) AS event_time_group, tag_name FROM ( SELECT DISTINCT __time, CASE WHEN (NVL (base_product_id, '0') = '0') THEN variant_product_id ELSE base_product_id END as tag_product_id, od_parent_id, m1.od_id, event_time, tag_name, sale_quantity, sale_total_amount, sale_subtotal_amount, sale_item_cost_amount, sale_discount_amount, sale_taxable_revenue, sale_non_taxable_revenue, sale_tax_amount, sale_revenue, sale_gross_revenue, sale_profitability, refund_quantity, refund_total_amount, refund_subtotal_amount, refund_item_cost_amount, refund_discount_amount, refund_taxable_revenue, refund_non_taxable_revenue, refund_tax_amount, refund_revenue, refund_gross_revenue, refund_profitability FROM master_order_detail_primary AS m1 LEFT JOIN ( SELECT DISTINCT od_id, CASE WHEN (NVL (base_product_id, '0') = '0') THEN product_id ELSE base_product_id END as variant_product_id FROM master_order_detail_primary WHERE __time >= xxx AND __time < xxx AND event_time >= xxx AND event_time < xxx AND merchant_id = xxx AND od_type = xxx ) AS m2 ON m1.od_parent_id = m2.od_id JOIN ( SELECT t.tag_id, t.tag_name, t.tag_type, NVL ( ( CASE WHEN (NVL (base_product_id, '0') = '0') THEN product_id ELSE base_product_id END ), t.entity_id ) as combo_product_id FROM tag_map t LEFT JOIN ( SELECT distinct category_id, product_id, base_product_id FROM master_order_detail_primary WHERE __time >= xxx AND __time < xxx AND event_time >= xxx AND event_time < xxx AND merchant_id = xxx AND od_type = xxx ) p ON t.entity_id = p.category_id WHERE NOT ( t.tag_type = xxx AND p.product_id IS NOT NULL ) AND t.merchant_id = xxx ) AS tg ON CASE WHEN (NVL (variant_product_id, '0') = '0') THEN ( CASE WHEN (NVL (base_product_id, '0') = '0') THEN product_id ELSE base_product_id END ) ELSE variant_product_id END = tg.combo_product_id WHERE __time >= xxx AND __time < xxx AND event_time >= xxx AND event_time < xxx AND merchant_id = xxx AND od_type = xxx ) GROUP BY FLOOR(TIME_PARSE (event_time, NULL) to MINUTE), tag_name limit 9 offset 1

However, if I remove the outer-most layer (the SUM and the GROUP BY), then Druid can successfully paginate (use offset)

Activity

  1. gianm commented on May 8, 2025

    @gianm
    Contributor

    I think this might be fixed by #17837. It is the same error message, at least. Could you please try this on Druid 33, which includes the patch, and see if that fixes it?

  2. carltal commented on May 8, 2025

    @carltal
    Author

    @gianm Thank you for letting me know! I will try Druid 33 when we are able to upgrade to it on a lower env and report back.

  3. carltal commented on May 8, 2025

    @carltal
    Author

    @gianm btw, we see that the bug ticket you mentioned says: an offset specified on a groupBy that itself has an inner groupBy would lead to an error like "Cannot push down offsets". For our query, there is no groupBy inside groupBy; we just have 1 layer of groupBy. Could you let me know if this still applies? Thanks.

  4. gianm commented on May 8, 2025

    @gianm
    Contributor

    It's possible the native query generated from that SQL did have two stacked groupBy. The SQL is fairly complex and some of the DISTINCT are likely being planned into groupBy queries. I would still try with Druid 33 first, then if you still have problems, let us know what the native query is. (Use EXPLAIN PLAN FOR to get it.)

  5. carltal commented on May 8, 2025

    @carltal
    Author

    @gianm Oh I see, that makes sense! We will try with version 33 then. Thanks a lot!

  6. carltal commented on Jun 3, 2025

    @carltal
    Author

    OK it seems that Druid v33 has resolved the issue. Thanks. @gianm

  7. gianm commented on Jun 3, 2025

    @gianm
    Contributor

    Good to know, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions