[Topi, Relay] Add cumprod by AndrewZhaoLuo · Pull Request #7722 · apache/tvm

AndrewZhaoLuo · 2021-03-22T21:35:44Z

Extends PR #7334 to also have a numpy style cumprod operation.

Creates the notion of a cumulative binary operation. This does what it sounds and assumes the binary operation is associative and commutative. By passing in an identity value we can also extend the notion to exclusive cumulative sums/products. Examples of other operations which would fit this bill include binary xor as well as binary and.

@masahi @mbrookhart please select people who would be suitable to review.

masahi · 2021-03-23T23:58:22Z

You've got a merge issue, need to rebase.

AndrewZhaoLuo · 2021-03-24T01:00:53Z

@masahi PTAL

masahi

Looks nice. I suggest the following changes

combineop -> scanop
merge topi/cumsum.py and topi/cumprod.py into topi/scan.py (like the cuda version)
merge test_topi_cumsum.py and test_topi_cumprod.py to remove duplication. Something like what you have in relay/test_op_level3.py

AndrewZhaoLuo · 2021-03-24T18:52:05Z

@masahi PTAL. I've implemented all your suggested changes.

add cumprod and tests reinstate tests rethink

typos fix sum -> prod more typos more typo fixes more typos add doc strings

make exclusive a bool up and down stack fix x fix bool err it is a bool now fix fix thing formatting to pass linter lint python cumprod pylint fix attribute fix ordering add exclusivity tests for end to end fix things cuda identity_value

simplify construction clang-format more tests undo simpler construction due to function passing stuff fix docs more exclusive doc changes more fixins"

fix stuff

mbrookhart

Supporting exclusive for these ops adds quite a bit of complexity to the implementation and the testing, and it doesn't seem to exist in the numpy op. Are we sure we need it?

Otherwise LGTM.

mbrookhart · 2021-03-24T19:54:26Z

+    exclusive : bool, optional
+        If True, will return exclusive product in which the first element is not
+        included. In other terms, if True, the j-th output element would be
+        the product of the first (j-1) elements. Otherwise, it would be the product of
+        the first j elements.


I don't think numpy has an exclusive option? Do we need this? https://numpy.org/doc/stable/reference/generated/numpy.cumprod.html

I'll defer to @masahi on this. I do agree that it adds a lot of complexity for not much gain.

For what it's worth, thrust has inclusive and exclusive scan operations which is what may have motivated the current interface.

Hehehe, I tagged the wrong line. I know why cuda scan needs to be inclusive or exclusive (it's for other ops), but I'm not sure why cumprod needs the option?

Yes, ONNX cumsum has an exclusive option. There is a good use case for exclusive cumsum but probably not for cumprod. We can always do inclusive scan for cumprod.

Well we might as well keep cumprod's exclusive option since it has very little implementation overhead. The only thing that is there to support it is the identity_value fields which are pretty simple. I say we might as well keep it at this point.

masahi

LGTM

jwfromm

LGTM, very nicely written code. Thanks @AndrewZhaoLuo

masahi · 2021-03-25T04:25:33Z

thanks @AndrewZhaoLuo @mbrookhart @jwfromm

* make cumbinop, refactor cumsum, add cumprod * cumsum exclusive test * Add cumprod + flesh out cumsum tests add cumprod and tests reinstate tests rethink * add rudimentary scan implementation * add attributes of cumprod node * add cumprod strategy * add cuda strategy * python relay node construction * change attrs to be reusuable * add cumprod nodes * complete tests * Fix some typos about sum --> prod typos fix sum -> prod more typos more typo fixes more typos add doc strings * Use Bool instead of int to represent exclusive make exclusive a bool up and down stack fix x fix bool err it is a bool now fix fix thing formatting to pass linter lint python cumprod pylint fix attribute fix ordering add exclusivity tests for end to end fix things cuda identity_value * Overall improve formatting, add doc message corrections simplify construction clang-format more tests undo simpler construction due to function passing stuff fix docs more exclusive doc changes more fixins" * merge cumsum and cumprod to scan, merge tests fix stuff * remove other mentions of cumbinop -> scanop * lint formatting Co-authored-by: Andrew Zhao Luo <andrewzhaoluo@Andrews-MacBook-Pro.local>

AndrewZhaoLuo changed the title ~~[Topi, Relay] Add cumprod~~ WIP: [Topi, Relay] Add cumprod Mar 22, 2021

masahi self-assigned this Mar 22, 2021

AndrewZhaoLuo changed the title ~~WIP: [Topi, Relay] Add cumprod~~ [Topi, Relay] Add cumprod Mar 23, 2021

AndrewZhaoLuo force-pushed the andrewzhaoluo-add-cumprod branch 4 times, most recently from da75c66 to 4ead969 Compare March 24, 2021 00:55

masahi reviewed Mar 24, 2021

View reviewed changes

AndrewZhaoLuo force-pushed the andrewzhaoluo-add-cumprod branch from a37657d to 23d4325 Compare March 24, 2021 18:38

Andrew Zhao Luo added 16 commits March 24, 2021 11:52

make cumbinop, refactor cumsum, add cumprod

f2ef782

cumsum exclusive test

2dc85a0

Add cumprod + flesh out cumsum tests

9406150

add cumprod and tests reinstate tests rethink

add rudimentary scan implementation

b908bdf

add attributes of cumprod node

cdedbb3

add cumprod strategy

0de8e21

add cuda strategy

4b48ec9

python relay node construction

7b3e372

change attrs to be reusuable

5062a3c

add cumprod nodes

67d9441

complete tests

f75d64c

Fix some typos about sum --> prod

b3290bd

typos fix sum -> prod more typos more typo fixes more typos add doc strings

Use Bool instead of int to represent exclusive

6f944d1

make exclusive a bool up and down stack fix x fix bool err it is a bool now fix fix thing formatting to pass linter lint python cumprod pylint fix attribute fix ordering add exclusivity tests for end to end fix things cuda identity_value

Overall improve formatting, add doc message corrections

43df67e

simplify construction clang-format more tests undo simpler construction due to function passing stuff fix docs more exclusive doc changes more fixins"

merge cumsum and cumprod to scan, merge tests

8ffff27

fix stuff

remove other mentions of cumbinop -> scanop

b9fb779

AndrewZhaoLuo force-pushed the andrewzhaoluo-add-cumprod branch from 4137c15 to b9fb779 Compare March 24, 2021 18:52

lint formatting

dbea713

AndrewZhaoLuo force-pushed the andrewzhaoluo-add-cumprod branch from e04e03d to dbea713 Compare March 24, 2021 19:44

mbrookhart reviewed Mar 24, 2021

View reviewed changes

masahi reviewed Mar 24, 2021

View reviewed changes

Comment thread python/tvm/relay/op/_transform.py

masahi approved these changes Mar 24, 2021

View reviewed changes

mbrookhart approved these changes Mar 24, 2021

View reviewed changes

jwfromm approved these changes Mar 24, 2021

View reviewed changes

masahi merged commit 8e23806 into apache:main Mar 25, 2021

AndrewZhaoLuo mentioned this pull request Apr 20, 2021

[Docs] Update Dev. Doc. on how to add a new relay operator #7893

Merged

junrushao mentioned this pull request Nov 1, 2021

Apache TVM v0.8 Release Note Candidate #9416

Closed

Uh oh!

Conversation

AndrewZhaoLuo commented Mar 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

masahi commented Mar 23, 2021

Uh oh!

AndrewZhaoLuo commented Mar 24, 2021

Uh oh!

masahi left a comment

Choose a reason for hiding this comment

Uh oh!

AndrewZhaoLuo commented Mar 24, 2021

Uh oh!

mbrookhart left a comment

Choose a reason for hiding this comment

Uh oh!

mbrookhart Mar 24, 2021

Choose a reason for hiding this comment

Uh oh!

AndrewZhaoLuo Mar 24, 2021

Choose a reason for hiding this comment

Uh oh!

mbrookhart Mar 24, 2021

Choose a reason for hiding this comment

Uh oh!

masahi Mar 24, 2021

Choose a reason for hiding this comment

Uh oh!

AndrewZhaoLuo Mar 24, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

masahi left a comment

Choose a reason for hiding this comment

Uh oh!

jwfromm left a comment

Choose a reason for hiding this comment

Uh oh!

masahi commented Mar 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndrewZhaoLuo commented Mar 22, 2021 •

edited

Loading