Skip to content

PyTorch frontend: make type inference incremental#6900

Merged
masahi merged 1 commit into
apache:mainfrom
t-vi:incremental_type_inference
Dec 9, 2020
Merged

PyTorch frontend: make type inference incremental#6900
masahi merged 1 commit into
apache:mainfrom
t-vi:incremental_type_inference

Conversation

@t-vi

@t-vi t-vi commented Nov 11, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor

Currently, the PyTorch frontend will use the vanilla TVM type inference pass to get the types.
Combined with the incremental nature of translating the graph, this makes for quadratic (in the graph size) runtime.
This patch runs type inference on a subgraph (starting from the things that has types) instead. It is a bit hacky though because it essentially tries to implement in-place modification where TVM does not foresee it.

For converting Huggingface BERT, this patch gives a ~10x increase in the speed of conversion (from 31 seconds to just below 3), so it is solving a very real and quite bad problem.

@t-vi t-vi force-pushed the incremental_type_inference branch from 9117082 to a9badc1 Compare November 11, 2020 14:52
@t-vi

t-vi commented Nov 11, 2020

Copy link
Copy Markdown
Contributor Author

The other question is whether we would want to make the translator a use a class interface so that the type inference can directly access the outputs' types.

@t-vi

t-vi commented Nov 11, 2020

Copy link
Copy Markdown
Contributor Author

https://discuss.tvm.apache.org/t/incremental-type-propagation/7446/10

@masahi

masahi commented Nov 12, 2020

Copy link
Copy Markdown
Member

The other question is whether we would want to make the translator a use a class interface so that the type inference can directly access the outputs' types.

Yes we can consider this approach. Initially I thought function-only approach would be cleaner (e.g. for recursively converting blocks in If and Loop) but this resulted in passing various constants such as convert_map, prelude, default_dtype etc around each function.

We can introduce a global object (like a class) to hold these constants.

@t-vi t-vi mentioned this pull request Dec 2, 2020
Closed
t-vi added a commit to t-vi/tvm that referenced this pull request Dec 3, 2020
@t-vi
Save PyTorch frontend state in object
38a66a1 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
@t-vi t-vi mentioned this pull request Dec 3, 2020
Merged
t-vi added a commit to t-vi/tvm that referenced this pull request Dec 4, 2020
@t-vi
Save PyTorch frontend state in object
4643967 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
t-vi added a commit to t-vi/tvm that referenced this pull request Dec 4, 2020
@t-vi
Save PyTorch frontend state in object
e0df5b1 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
t-vi added a commit to t-vi/tvm that referenced this pull request Dec 4, 2020
@t-vi
Save PyTorch frontend state in object
5a590a3 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
masahi pushed a commit that referenced this pull request Dec 4, 2020
@t-vi
Save PyTorch frontend state in object (#7023)
f278c42 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see #6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
trevor-m pushed a commit to trevor-m/tvm that referenced this pull request Dec 4, 2020
@t-vi
Save PyTorch frontend state in object (apache#7023)
21df600 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
trevor-m pushed a commit to neo-ai/tvm that referenced this pull request Dec 4, 2020
@t-vi @trevor-m
Save PyTorch frontend state in object (apache#7023)
4ef134e 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
@t-vi t-vi force-pushed the incremental_type_inference branch from a9badc1 to 5e07318 Compare December 8, 2020 15:38
@t-vi

t-vi commented Dec 8, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@masahi @siju-samuel I think this is ready for review now. In BERT conversion, I get a 10x speedup for from_python. In addition to removing the N² problem, it prunes the module before type inference, which seems essential for prelude.

@masahi

masahi commented Dec 9, 2020

Copy link
Copy Markdown
Member

@t-vi Please have a look at the CI issue. It is due to the recent change I made.

@t-vi t-vi force-pushed the incremental_type_inference branch from 5e07318 to 1046b65 Compare December 9, 2020 12:15
@t-vi

t-vi commented Dec 9, 2020

Copy link
Copy Markdown
Contributor Author

Oh, right, an undetected merge confict. Fixed. Thank you @masahi.

@masahi masahi merged commit db0215e into apache:main Dec 9, 2020
@masahi

masahi commented Dec 9, 2020

Copy link
Copy Markdown
Member

Thanks @t-vi

@t-vi

t-vi commented Dec 9, 2020

Copy link
Copy Markdown
Contributor Author

Thank you @masahi, for the guidance and discussion, and review!

TusharKanekiDey pushed a commit to TusharKanekiDey/tvm that referenced this pull request Jan 20, 2021
@t-vi
Incremental type inference (apache#6900)
3a1f5ea
trevor-m pushed a commit to neo-ai/tvm that referenced this pull request Jan 21, 2021
@t-vi @trevor-m
Incremental type inference (apache#6900)
e90ab87
electriclilies pushed a commit to electriclilies/tvm that referenced this pull request Feb 18, 2021
@t-vi @electriclilies
Save PyTorch frontend state in object (apache#7023)
8c6e13f 
While the functional approach is pretty neat, we ended up having
global state (default frontend, dtype) and it'll be more soon
(caching of inferred types, see apache#6900). To not have to pass around
the state, this moves the op conversion into a class with instances
having the state.
electriclilies pushed a commit to electriclilies/tvm that referenced this pull request Feb 18, 2021
@t-vi @electriclilies
Incremental type inference (apache#6900)
cb86310
@masahi masahi mentioned this pull request Aug 27, 2021
Closed
@junrushao junrushao mentioned this pull request Nov 1, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@masahi masahi masahi approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@t-vi @masahi