TF_ADDONS_PY_OPS does not appear to work with beam_search_decoder.py unit test

[jimthompson5802]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.15.6 running Docker for Mac 2.3.0.4
• TensorFlow version and how it was installed (source or binary): TF 2.3.0, from TF docker image tensorflow/tensorflow:2.3.0-jupyter
• TensorFlow-Addons version and how it was installed (source or binary): In docker build, pip install tfa-nightly==0.12.0.dev20200809083253
• Python version: 3.6.9
• Is GPU used? (yes/no): No

Describe the bug
Several failures in unit test tensorflow_addons/seq2seq/test/beam_search_decoder_test.py. All appear to be this error

tensorflow.python.framework.errors_impl.NotFoundError: /opt/project/tensorflow_addons/custom_ops/seq2seq/_beam_search_ops.so: cannot open shared object file: No such file or directory

This error occurs even though environment variable TF_ADDONS_PY_OPS=1 is set.

Based on TFA documention I expected the unit test to complete successfully.

Code to reproduce the issue

• git clone the TFA github
• build docker image with TF 2.3.0 and TFA from nightly build. build command: docker build --progress auto -t addons:tf23 -f Dockerfile .
  Dockerfile
• cd position into root directory of the TFA local repo
• start container docker run -it -v ${PWD}:/opt/project addons:tf23 /bin/bash
• within the container execute this command:

TF_ADDONS_PY_OPS=1  pytest -v /opt/project/tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py

Other info / logs

Log file
containing all the output.

[WindQAQ]

I try the same commands with yours but do not fail any tests. tensorflow.python.framework.errors_impl.NotFoundError only shows as warning. Can you check it again?

[jimthompson5802]

I tried again....same failures.

I rebuilt docker image using the most current tfa-nightly install. Here is the Dockerfile

FROM tensorflow/tensorflow:2.3.0-jupyter
RUN pip install tfa-nightly \
	pytest 
WORKDIR /opt/project

Within the docker container, I use this command to run pytest

TF_ADDONS_PY_OPS=1 pytest /opt/project/tensorflow_addons/seq2seq/

Excerpt from log

=========================================================== test session starts ============================================================
platform linux -- Python 3.6.9, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /opt/project, configfile: pytest.ini
plugins: typeguard-2.9.1
collected 128 items

tensorflow_addons/seq2seq/tests/attention_wrapper_test.py .............................s................s                            [ 36%]
tensorflow_addons/seq2seq/tests/basic_decoder_test.py .................s                                                             [ 50%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py FFFFFFs..s.s..s.FFFs                                                     [ 66%]
tensorflow_addons/seq2seq/tests/beam_search_ops_test.py FF.sF                                                                        [ 70%]
tensorflow_addons/seq2seq/tests/decoder_test.py ......s                                                                              [ 75%]
tensorflow_addons/seq2seq/tests/loss_test.py ...............................                                                         [100%]


========================================================= short test summary info ==========================================================
SKIPPED [10] ../../usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/test_util.py:2261: Not a test.
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree - tensorflow.python.framework.errors_impl.NotFoundEr...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_scalar - tensorflow.python.framework.erro...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d - tensorflow.python.framework.errors_i...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d_with_merged_batch_beam - tensorflow.py...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_2d - tensorflow.python.framework.errors_i...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestGatherTree::test_gather_tree_from_array_complex_trajectory - tens...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::testDynamicDecodeRNNBatchMajorNoAttention - te...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::testDynamicDecodeRNNBatchMajorYesAttention - t...
FAILED tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::testDynamicDecodeRNNBatchMajorYesAttentionWithAlignmentHistory
FAILED tensorflow_addons/seq2seq/tests/beam_search_ops_test.py::test_gather_tree_one - tensorflow.python.framework.errors_impl.NotFoundEr...
FAILED tensorflow_addons/seq2seq/tests/beam_search_ops_test.py::GatherTreeTest::testBadParentValuesOnCPU - AssertionError: Exception of t...
FAILED tensorflow_addons/seq2seq/tests/beam_search_ops_test.py::test_gather_tree_batch - tensorflow.python.framework.errors_impl.NotFound...
========================================= 12 failed, 106 passed, 10 skipped, 5 warnings in 42.98s ==========================================

Here is the full log file.
seq2seq_unit_test_log.txt

If needed, here is pip list from my running container:

root@fdcaf6785ef4:/opt/project# pip list
Package                Version
---------------------- ------------------------
absl-py                0.9.0
asn1crypto             0.24.0
astunparse             1.6.3
attrs                  19.3.0
backcall               0.2.0
bleach                 3.1.5
cachetools             4.1.1
certifi                2020.6.20
chardet                3.0.4
cryptography           2.1.4
cycler                 0.10.0
decorator              4.4.2
defusedxml             0.6.0
entrypoints            0.3
gast                   0.3.3
google-auth            1.19.2
google-auth-oauthlib   0.4.1
google-pasta           0.2.0
grpcio                 1.30.0
h5py                   2.10.0
idna                   2.6
importlib-metadata     1.7.0
iniconfig              1.0.1
ipykernel              5.1.1
ipython                7.16.1
ipython-genutils       0.2.0
ipywidgets             7.5.1
jedi                   0.17.2
Jinja2                 2.11.2
jsonschema             3.2.0
jupyter                1.0.0
jupyter-client         6.1.6
jupyter-console        6.1.0
jupyter-core           4.6.3
jupyter-http-over-ws   0.0.8
Keras-Preprocessing    1.1.2
keyring                10.6.0
keyrings.alt           3.0
kiwisolver             1.2.0
Markdown               3.2.2
MarkupSafe             1.1.1
matplotlib             3.3.0
mistune                0.8.4
more-itertools         8.4.0
nbconvert              5.6.1
nbformat               4.4.0
notebook               6.0.3
numpy                  1.18.5
oauthlib               3.1.0
opt-einsum             3.3.0
packaging              20.4
pandocfilters          1.4.2
parso                  0.7.1
pexpect                4.8.0
pickleshare            0.7.5
Pillow                 7.2.0
pip                    20.1.1
pluggy                 0.13.1
prometheus-client      0.8.0
prompt-toolkit         3.0.5
protobuf               3.12.2
ptyprocess             0.6.0
py                     1.9.0
pyasn1                 0.4.8
pyasn1-modules         0.2.8
pycrypto               2.6.1
Pygments               2.6.1
pygobject              3.26.1
pyparsing              2.4.7
pyrsistent             0.16.0
pytest                 6.0.1
python-dateutil        2.8.1
pyxdg                  0.25
pyzmq                  19.0.1
qtconsole              4.7.5
QtPy                   1.9.0
requests               2.24.0
requests-oauthlib      1.3.0
rsa                    4.6
scipy                  1.4.1
SecretStorage          2.3.1
Send2Trash             1.5.0
setuptools             49.2.0
six                    1.15.0
tensorboard            2.3.0
tensorboard-plugin-wit 1.7.0
tensorflow             2.3.0
tensorflow-estimator   2.3.0
termcolor              1.1.0
terminado              0.8.3
testpath               0.4.4
tfa-nightly            0.12.0.dev20200817145846
toml                   0.10.1
tornado                6.0.4
traitlets              4.3.3
typeguard              2.9.1
urllib3                1.25.10
wcwidth                0.2.5
webencodings           0.5.1
Werkzeug               1.0.1
wheel                  0.30.0
widgetsnbextension     3.5.1
wrapt                  1.12.1
zipp                   3.1.0
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@fdcaf6785ef4:/opt/project#

[bhack]

Yes loading so could be a little bit confusing currently.
It is why I've prepared #1909 some months ago.

[jimthompson5802]

@WindQAQ I'm just checking in. Was my response to your request, "Can you check it again?" adequate? Do you need any additional information from me?

[guillaumekln]

Can you remove the tfa-nightlty package and install the package in editable mode from /opt/project? See the documentation.

If you have issues with custom ops, you could also skip these tests with pytest --skip-custom-ops tensorflow_addons/seq2seq.

[jimthompson5802]

@guillaumekln

Here is updated Dockerfile with the change to install in editable form

FROM tensorflow/tensorflow:2.3.0-jupyter

# setup to install tfa
WORKDIR /app
COPY . /app

RUN pip install -r tools/install_deps/pytest.txt
RUN pip install	-e ./

WORKDIR /opt/project

pip list in the docker container

root@f6701a1747f6:/opt/project# pip list | grep ^tensor
tensorboard            2.3.0
tensorboard-plugin-wit 1.7.0
tensorflow             2.3.0
tensorflow-addons      0.10.0.dev0 /app
tensorflow-estimator   2.3.0

Note: even if the test works, my need is for later release of TFA.

• BeamSearchDecoder with non LSTM cells raises ValueError exception #1900
• AttentionWrapper with non LSTM cells returns ValueError #1856

Result of running with --skip-custom-ops parameter

root@f6701a1747f6:/opt/project# pytest --skip-custom-ops tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py
2020-08-22 10:37:00.944196: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-08-22 10:37:00.944251: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/opt/project/tensorflow_addons/utils/ensure_tf_install.py:68: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.1.0 and strictly below 2.3.0 (nightly versions are not supported).
 The versions of TensorFlow you are currently using is 2.3.0 and is not supported.
Some things might work, some things might not.
If you were to encounter a bug, do not file an issue.
If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version.
You can find the compatibility matrix in TensorFlow Addon's readme:
https://github.com/tensorflow/addons
  UserWarning,
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --skip-custom-ops
  inifile: /opt/project/pytest.ini
  rootdir: /opt/project

My fundamental question, does the environment variable TF_ADDONS_PY_OPS=1 influence the running of unit tests? My understanding of this environment variable tells the TFA run-time to not use the custom ops but instead use Python equivalent. If my understanding is correct, I expected this test tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree to pass with the use of the Python equivalent for custom ops.

root@f6701a1747f6:/opt/project# TF_ADDONS_PY_OPS=1 pytest -v /opt/project/tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py
=========================================================== test session starts ============================================================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /opt/project, inifile: pytest.ini
plugins: typeguard-2.9.1, forked-1.3.0, xdist-1.34.0
collected 20 items

tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree FAILED                                                 [  5%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_scalar FAILED                               [ 10%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d FAILED                                   [ 15%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d_with_merged_batch_beam FAILED            [ 20%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_2d FAILED                                   [ 25%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestGatherTree::test_gather_tree_from_array_complex_trajectory FAILED   [ 30%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestGatherTree::test_session SKIPPED                                    [ 35%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestArrayShapeChecks::test_array_shape_dynamic_checks PASSED            [ 40%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestArrayShapeChecks::test_array_shape_static_checks PASSED             [ 45%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestArrayShapeChecks::test_session SKIPPED                              [ 50%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_eos_masking PASSED                                                 [ 55%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestBeamStep::test_session SKIPPED                                      [ 60%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestBeamStep::test_step PASSED                                          [ 65%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestBeamStep::test_step_with_eos PASSED                                 [ 70%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestLargeBeamStep::test_session SKIPPED                                 [ 75%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::TestLargeBeamStep::test_step PASSED                                     [ 80%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::testDynamicDecodeRNNBatchMajorNoAttention FAILED [ 85%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::testDynamicDecodeRNNBatchMajorYesAttention FAILED [ 90%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::testDynamicDecodeRNNBatchMajorYesAttentionWithAlignmentHistory FAILED [ 95%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::BeamSearchDecoderTest::test_session SKIPPED                             [100%]

Full log file
beam_search_decoder_unit_test_log2.txt

[guillaumekln]

tensorflow-addons 0.10.0.dev0 /app

Can you first update your TensorFlow Addons version? (with git pull)

[jimthompson5802]

Unit tests that failed before are now working. See below for details.

@guillaumekln Thank you very much for helping me with this issue. To help me learn, what fixed the problem? From what I can tell, I've been working off the master branch since I reported the problem couple of days ago. I did not capture the output from the git pull so I can't go back to review what changes came down. Knowing how the problem was resolved will help me be more self-reliant on future problems. At the surface, it appears that a change that was implemented between 0.10 and 0.12 fixes the issue(version indicator on master now says 0.12)

Performed git pull, re-built image using this Dockerfile

FROM tensorflow/tensorflow:2.3.0-jupyter

# setup to install tfa
WORKDIR /app
COPY . /app

RUN pip install -r tools/install_deps/pytest.txt
RUN pip install	-e ./

WORKDIR /opt/project

pip list in docker container

root@617a1365ec45:/opt/project# pip list | grep "^tensor\|^tfa"
tensorboard            2.3.0
tensorboard-plugin-wit 1.7.0
tensorflow             2.3.0
tensorflow-addons      0.12.0.dev0 /app
tensorflow-estimator   2.3.0

Re-run of tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py

pytest -v /opt/project/tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py
=========================================================== test session starts ============================================================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /opt/project, inifile: pytest.ini
plugins: typeguard-2.9.1, extra-durations-0.1.3, forked-1.3.0, xdist-1.34.0
collected 148 items

tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree[custom_ops-cpu] PASSED                                 [  0%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree[py_ops-cpu] PASSED                                     [  1%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_scalar[custom_ops-cpu] PASSED               [  2%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_scalar[py_ops-cpu] PASSED                   [  2%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d[custom_ops-cpu] PASSED                   [  3%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d[py_ops-cpu] PASSED                       [  4%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d_with_merged_batch_beam[custom_ops-cpu] PASSED [  4%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_1d_with_merged_batch_beam[py_ops-cpu] PASSED [  5%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_2d[custom_ops-cpu] PASSED                   [  6%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_2d[py_ops-cpu] PASSED                       [  6%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_complex_trajectory[custom_ops-cpu] PASSED   [  7%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_gather_tree_from_array_complex_trajectory[py_ops-cpu] PASSED       [  8%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_array_shape_dynamic_checks[eager_mode-cpu] PASSED                  [  8%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_array_shape_dynamic_checks[tf_function-cpu] PASSED                 [  9%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_array_shape_static_checks[cpu] PASSED                              [ 10%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_eos_masking[cpu] PASSED                                            [ 10%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_missing_embedding_fn[cpu] PASSED                                   [ 11%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_step[cpu] PASSED                                              [ 12%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_step_with_eos[cpu] PASSED                                          [ 12%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_large_beam_step[cpu] PASSED

<<<<<< lines deleted>>>>>

tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-has_attention-with_alignment_history-output_all_scores] PASSED [ 95%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-has_attention-with_alignment_history-no_output_all_scores] PASSED [ 95%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-has_attention-no_with_alignment_history-output_all_scores] PASSED [ 96%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-has_attention-no_with_alignment_history-no_output_all_scores] PASSED [ 97%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-no_has_attention-with_alignment_history-output_all_scores] PASSED [ 97%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-no_has_attention-with_alignment_history-no_output_all_scores] PASSED [ 98%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-no_has_attention-no_with_alignment_history-output_all_scores] PASSED [ 99%]
tensorflow_addons/seq2seq/tests/beam_search_decoder_test.py::test_beam_search_decoder[py_ops-tf_function-cpu-GRUCell-no_time_major-no_has_attention-no_with_alignment_history-no_output_all_scores] PASSED [100%]

<<<<<<<lines deleted>>>>>>


-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================================================== sum of all tests durations ========================================================
89.04s
=============================================== 148 passed, 74 warnings in 89.93s (0:01:29) ================================================

[guillaumekln]

This commit c946906 is needed to make the beam search decoder tests pass without compiling the custom ops beforehand. The version you were using was older than this commit and the tests failed because the custom op was not found.

[bhack]

In the thread there was also a problem about loading the custom ops .so from your source code checkout /opt/project/tensorflow_addons/custom_ops/seq2seq/. It is required to run a manual script. See #1909 (comment)