[microNPU] Add the infrastructure for lookup table and TANH by ekalda · Pull Request #9547 · apache/tvm

ekalda · 2021-11-22T16:44:19Z

Some activation functions like TANH and SIGMOID are implemented
by calculating the values based on the QNN parameters and
recording the values into a lookup table (LUT).

This patch adds the LUT functionality alongside with the TANH
activation function and the tests.

ekalda · 2021-11-22T16:46:09Z

@manupa-arm @mbaret @lhutton1 @NicolaLancellotti @dchauhan-arm

lhutton1

Thanks @ekalda, mostly minor things

lhutton1 · 2021-11-22T21:54:23Z

+    def create_op_with_lut(self, call):
+        """Extract the parameters and attributes from the NPU operator and create
+        a new operator with LUT.
+        ----------


Nit: missing Parameters and I think we need spacing for these doc strings?

lhutton1 · 2021-11-22T21:56:49Z

+        lut_activations = ["TANH", "LUT"]
+
+        if (
+            call.op.name == "contrib.ethosu.identity"


Should there be a check here to make sure call is an op?

From my digging of the code, it looks like tvm.relay.expr.Call always has an op attribute, so the equality check shouldn't fail with an error (but correct me if I'm wrong there)

I think that while it will always have an op attribute, op may not be a tvm.ir.Op and could, for example, be a Function. See here:

tvm/include/tvm/relay/expr.h

Line 269 in 52edc9a

* - It can be tvm::Op which corresponds to the primitive operators.

Ok yes, I added the op type check :)

lhutton1 · 2021-11-22T22:05:25Z

+
+@relay.transform.function_pass(opt_level=1, name="LutOptimizer")
+class LUTsOptimizer(Pass):
+    """Register LutOptimizer as a relay pass."""


Nit: LUTsOptimizer and the same for name=LUTsOptimizer above.

lhutton1 · 2021-11-22T22:08:06Z

+        Parameters
+        ----------
+        func : tvm.relay.function.Function
+            The function to apply the layout optimization pass to.


Suggested change

The function to apply the layout optimization pass to.

The function to apply the optimization pass for multiple LUTs to.

lhutton1 · 2021-11-22T22:11:17Z

+        Returns
+        -------
+        mod : tvm.IRModule
+            New module with augmented layouts.


Suggested change

New module with augmented layouts.

New module with optimized LUTs.

lhutton1 · 2021-11-22T22:17:24Z

        pass


+def round_away_zero(f):


Could we reuse round_away_zero from util.py?

Ah nice, I didn't know it was there, thanks for pointing this out! Changed it to use the one from util.py

lhutton1 · 2021-11-22T22:25:22Z

+        input_zp = int(dequantize_args[2].data.asnumpy())
+
+        lut_values = find_tanh_values(input_scale, input_zp, output_scale, output_zp)
+        lut = relay.const(lut_values, dtype="uint8")


Just curious, can LUT be any other dtype?

Currently, the values are calculated for an int8 activation and then cast into uint8 due to that can of worms dtype problem in the tir_to_cs_translator.py

Nice, glad to see a fix for that issue will improve something else :)

lhutton1 · 2021-11-22T22:38:39Z

+    ifm_shape = (1, 55, 55, 3)
+
+    def create_tflite_graph():
+        tf.config.run_functions_eagerly(True)


Do we need to enable eager execution?

No we don't :D Removed it!

lhutton1 · 2021-11-23T09:56:45Z

+        new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]
+        new_args.append(lut)


Suggested change

new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]

new_args.append(lut)

new_args = ethosu_op.args[:-1] + [lut]

NicolaLancellotti · 2021-11-24T16:24:56Z

+        a new NPU operator.
+        Parameters
+        ----------
+        call : tvm.relay.expr.Call
+            The current call node being visited.
+        Returns
+        -------


Suggested change

a new NPU operator.

Parameters

----------

call : tvm.relay.expr.Call

The current call node being visited.

Returns

-------

a new NPU operator.

Parameters

----------

call : tvm.relay.expr.Call

The current call node being visited.

Returns

-------

NicolaLancellotti · 2021-11-24T16:29:29Z

+        """Visit relay nodes in the given module.
+        Parameters


Suggested change

"""Visit relay nodes in the given module.

Parameters

"""Visit relay nodes in the given module.

Parameters

ekalda · 2021-11-26T11:48:09Z

Thanks @lhutton1 and @NicolaLancellotti for the reviews! :)

NicolaLancellotti

LGTM!

lhutton1

LGTM!

Some activation functions like TANH and SIGMOID are implemented by calculating the values based on the QNN parameters and recording the values into a lookup table (LUT). This patch adds the LUT functionality alongside with the TANH activation function and the tests. Change-Id: Ibe49759fd02724af869826663ff0babd352e5894

manupak

LGTM!

I was hoping we could merge #9597 before this as it get conflicted :) -- but it got conflicted anyway. Therefore getting this in. Lets prioritize getting a version of #9597 otherwise it would be perma-conflicted.
cc : @leandron @mbaret

manupak · 2021-12-01T07:31:35Z

Thanks all!

) Some activation functions like TANH and SIGMOID are implemented by calculating the values based on the QNN parameters and recording the values into a lookup table (LUT). This patch adds the LUT functionality alongside with the TANH activation function and the tests.

ekalda requested review from MarisaKirisame, ZihengJiang, anijain2305, areusch, comaniac, icemelon, jroesch, junrushao, jwfromm, mbrookhart, merrymercy, slyubomirsky, tqchen, vinx13, wweic, yzhliu and zhiics as code owners November 22, 2021 16:44

lhutton1 reviewed Nov 23, 2021

View reviewed changes

ekalda mentioned this pull request Nov 23, 2021

[Tracking Issue] Arm(R) Ethos(TM)-U NPU operator support #9410

Closed

15 tasks

NicolaLancellotti reviewed Nov 24, 2021

View reviewed changes

ekalda force-pushed the lut_upstream branch from 56b9d84 to d6cad0c Compare November 26, 2021 11:38

NicolaLancellotti approved these changes Nov 26, 2021

View reviewed changes

lhutton1 approved these changes Nov 26, 2021

View reviewed changes

ekalda force-pushed the lut_upstream branch from c10f7ca to 27743a7 Compare November 29, 2021 10:13

ekalda added 3 commits November 29, 2021 10:39

Responding to the reviews

46b4602

Checking that tvm.relay.expr.Call.op exists

2218102

ekalda force-pushed the lut_upstream branch from 27743a7 to 2218102 Compare November 29, 2021 10:43

manupak approved these changes Dec 1, 2021

View reviewed changes

manupak merged commit 9ada371 into apache:main Dec 1, 2021

driazati mentioned this pull request Jul 14, 2022

TVM v0.9.0.rc0 Release Candidate Notes #12102

Closed

	The function to apply the layout optimization pass to.
	The function to apply the optimization pass for multiple LUTs to.

	New module with augmented layouts.
	New module with optimized LUTs.

		new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]
		new_args.append(lut)

	new_args = [ethosu_op.args[n] for n in range(len(ethosu_op.args) - 1)]
	new_args.append(lut)
	new_args = ethosu_op.args[:-1] + [lut]

Uh oh!

Conversation

ekalda commented Nov 22, 2021

Uh oh!

ekalda commented Nov 22, 2021

Uh oh!

lhutton1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ekalda commented Nov 26, 2021

Uh oh!

NicolaLancellotti left a comment

Choose a reason for hiding this comment

Uh oh!

lhutton1 left a comment

Choose a reason for hiding this comment

Uh oh!

manupak left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manupak commented Dec 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

manupak left a comment •

edited

Loading