relay_mod = relay.transform.ToMixedPrecision('bfloat16')(relay_mod)
pass, we are supposed to get a well-transformed model which contains different kinds of operators with auto mix precision.
However, we have seen some error messages.
The Relay type checker is unable to show the following types match.
In particular `Tensor[(64), float32]` does not match `Tensor[(64), bfloat16]`
The Relay type checker is unable to show the following types match.
In particular `Tensor[(64), float32]` does not match `Tensor[(64), bfloat16]`
After we print out the relay graph, we can see this error occurs because nn.layer_norm belongs to DEFAULT_NEVER_LIST but there is not cast operators which promoted its input from bfloat16 to float32.
def test_layernorm(run_module, dtype="float32"):
x = relay.var("x", shape=(4, 3, 224, 224), dtype="float32")
k_shape = (64, 3, 4, 4)
kernel = relay.const(np.random.randint(0, 1, k_shape).astype(dtype))
out = relay.nn.conv2d(
x,
kernel,
channels=64,
kernel_size=k_shape[2:4],
strides=[4, 4],
padding=[0, 0, 0, 0]
)
# bias = relay.var("bias", shape=(64), dtype=dtype)
bias = relay.const(np.random.randint(0, 1, (64,)).astype(dtype))
out = relay.nn.bias_add(out, bias)
out = relay.reshape(out, newshape=[4, 64, -1])
out = relay.transpose(out, axes=[0, 2, 1])
beta = relay.const(np.zeros((64)).astype("float32"))
gamma = relay.const(np.ones((64)).astype("float32"))
out = relay.nn.layer_norm(out, gamma=gamma, beta=beta)
dic = {"x": (4, 3, 224, 224), "kernel": k_shape, "bias": (4, 64, 56, 56)}
param_lst = ["kernel", "bias"]
out = tvm.IRModule.from_expr(out)
config = out, dic, param_lst
run_and_verify_func(config, run_module=run_module, dtype=dtype)
When we invoke
pass, we are supposed to get a well-transformed model which contains different kinds of operators with auto mix precision.
However, we have seen some error messages.
After we print out the relay graph, we can see this error occurs because nn.layer_norm belongs to DEFAULT_NEVER_LIST but there is not cast operators which promoted its input from bfloat16 to float32.
There is a minimal script which can reproduce this issue:
I posted the code into my personal repo:
fa1fb3b#diff-8b8dff568d8ee66e899bdc2b00d56db2a75e050c42b771fe38df17c4ca1ccdfb