From 08cf53c968c213856ac1b5e942ffa67713123fad Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Fri, 19 Jun 2020 15:01:58 -0700 Subject: [PATCH 01/11] Minor error improvements --- subset/cloud/test_udmi | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/subset/cloud/test_udmi b/subset/cloud/test_udmi index 4b0cfb2a32..fde175a26d 100755 --- a/subset/cloud/test_udmi +++ b/subset/cloud/test_udmi @@ -1,4 +1,5 @@ #!/bin/bash -e + source reporting.sh REPORT=/tmp/report.txt @@ -87,3 +88,7 @@ function message_report { for message_type in $message_types; do message_report $message_type done + +fgrep RESULT $REPORT + +echo Done with test_udmi From b7325075bccfd4efe59858b7a86428b4794f133b Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Fri, 19 Jun 2020 16:32:12 -0700 Subject: [PATCH 02/11] Initial doc migrate --- docs/{integration_testing.md => cloud_tests.md} | 4 ++++ 1 file changed, 4 insertions(+) rename docs/{integration_testing.md => cloud_tests.md} (97%) diff --git a/docs/integration_testing.md b/docs/cloud_tests.md similarity index 97% rename from docs/integration_testing.md rename to docs/cloud_tests.md index 1c6435a9b0..9a8164e43c 100644 --- a/docs/integration_testing.md +++ b/docs/cloud_tests.md @@ -2,6 +2,10 @@ DAQ currently uses Travis CI for integration testing: https://travis-ci.org/ +* gcp_gred for cloud project +* site_path pointing to devices +* device_id in device module config for MAC address + ## Configuration The `test_udmi` test module uses the Registrar and Validator to check that a device is From 4e772250e8729dbbbd07c599b08bf4e372ad4dbd Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 22 Jun 2020 08:42:01 -0700 Subject: [PATCH 03/11] Updating test docs --- cmd/build | 5 +++-- docs/cloud_tests.md | 42 +++++++++++++++++++++++++++--------------- 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/cmd/build b/cmd/build index dc428ba90d..f4f09dcf32 100755 --- a/cmd/build +++ b/cmd/build @@ -26,11 +26,12 @@ DOCKER_IMAGE_VER=docker_images.ver cd $ROOT source etc/config_base.sh -host_tests=$host_tests bin/docker_build_files +echo host_tests=$host_tests +test_targets=$host_tests bin/docker_build_files function pull_images { TAG=$1 declare -A test_set - for target in $(host_tests=$host_tests bin/docker_build_files); do + for target in $test_targets; do target=$(echo $target | sed 's|^.*/Dockerfile.||' | echo daqf/$( -## Travis CI Testing +### Travis CI Testing * Run the [registrar tool](registrar.md) to properly configure the cloud project. * `gcp_topic` config to `local/system.conf` as described in this doc. From 494ff64ca7504681c7283ba0d4defdcc7560d603 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 22 Jun 2020 08:53:06 -0700 Subject: [PATCH 04/11] Updating docs --- docs/cloud_tests.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index 07d53da737..be4837fa33 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -6,21 +6,29 @@ module included in the standard DAQ distro. ## Base Local Test Setup -* _enabled in build_: When running `cmd/build` there should be a line like -`subset/cloud/Dockerfile.test_udmi`. This is enabled through the `host_tests` config parameter, -which can be set to `config/modules/all.conf` if necessary. -* _gcp service account_: `gcp_gred` setup as described in +* The `udmi` module needs to be enabled in build. When running `cmd/build` there should be a line +like `subset/cloud/Dockerfile.test_udmi`. This is enabled through the `host_tests` config parameter, +which can be set to `config/modules/all.conf` if necessary. On startup, there should be a log +message that includes `udmi`: +``` +Jun 22 08:32:52 runner INFO Configured with tests pass, fail, ping, bacnet, mudgee, nmap, discover, switch, macoui, bacext, tls, password, udmi, manual +``` +* A testing gcp service account `gcp_cred` needs to be setup as described in [service account setup instructions](service.md). -* The system's default `module_config` needs to enable the `udmi` test, as per -in `./resources/setups/baseline/module_config.json`: +* The system's default `module_config` needs to enable the `udmi` test, e.g. as per +`resources/setups/baseline/module_config.json`. This can be validated by (runtime) checking +`inst/run-port-01/nodes/udmi01/tmp/module_config.json` to see if it has something like the following: ``` "udmi": { "enabled": true } ``` 4. `site_path` config needs to point to a site definition directory, or defaults to `local/site`. +This contains all the site-specific information about devices needed for testing. 5. `{site_path}/mac_addrs/{mac_addr}/module_config.json` needs to have a `device_id` defined, e.g. as in `resources/test_site/mac_addrs/3c5ab41e8f0b/module_config.json`. +6. The GCP IoT Core setup needs to have a proper registry and device configred. This can either +be done manually or using the [registrar tool](registrar.md) tool. ## Integration Testing From 168e55ce72a984acd0cec7717f9ba3e08a07bdde Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 22 Jun 2020 08:53:32 -0700 Subject: [PATCH 05/11] Consistent bullets --- docs/cloud_tests.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index be4837fa33..ee740b7103 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -23,11 +23,11 @@ Jun 22 08:32:52 runner INFO Configured with tests pass, fail, ping, bacnet, "enabled": true } ``` -4. `site_path` config needs to point to a site definition directory, or defaults to `local/site`. +* `site_path` config needs to point to a site definition directory, or defaults to `local/site`. This contains all the site-specific information about devices needed for testing. -5. `{site_path}/mac_addrs/{mac_addr}/module_config.json` needs to have a `device_id` defined, e.g. +* `{site_path}/mac_addrs/{mac_addr}/module_config.json` needs to have a `device_id` defined, e.g. as in `resources/test_site/mac_addrs/3c5ab41e8f0b/module_config.json`. -6. The GCP IoT Core setup needs to have a proper registry and device configred. This can either +* The GCP IoT Core setup needs to have a proper registry and device configred. This can either be done manually or using the [registrar tool](registrar.md) tool. ## Integration Testing From a1fc40985b4d6c12dc47f177e822e22d28f40ee2 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 22 Jun 2020 09:29:36 -0700 Subject: [PATCH 06/11] Adding more docs --- docs/cloud_tests.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index ee740b7103..558a1eb62a 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -30,6 +30,27 @@ as in `resources/test_site/mac_addrs/3c5ab41e8f0b/module_config.json`. * The GCP IoT Core setup needs to have a proper registry and device configred. This can either be done manually or using the [registrar tool](registrar.md) tool. +## Manual Test Pipeline + +If anything goes wrong with the automatic DAQ setup, the ability to manually test everything will +help debug the problem to see what's going wrong. The overal pipeline looks something like the +following: + +* Device sends data to the cloud. There's two kinds of devices: + * A faux _reference design_ device called [pubber](pubber.md), which is a completely contained + software device. + * An actual physical device. The setup and configuration of that device will be manufacturer + dependent and so is out of scope for this (DAQ) documentation. +* A configured GCP IoT Core project, registry, and device entry. The +[GCP docs for IoT Core](https://cloud.google.com/iot/docs/how-tos/devices) describe the basics. The +key part is the _authentication key_ (hahaha) that needs to be setup between the local device and +cloud device entry. +* The `gcloud` command line can be used to validate that data is being sent from the device to the +cloud. Something like `gcloud pubsub subscriptions pull --auto-ack projects/{project}/subscriptions/{sub_id}`. +(Complete documentation for how to use `gcloud` commands is out of scope of this documentation.) +* The [validator tool](validator.md) is what programmatically validates a device data stream, and +is what is ultimately used by `test_udmi` to validate device-cloud communication. + ## Integration Testing If developing cloud-tests, then the CI build system also needs to have a service account configured From acffaf3e345a177b2d4964c3f9595aeff50b92d9 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 22 Jun 2020 09:36:56 -0700 Subject: [PATCH 07/11] Updating --- docs/cloud_tests.md | 48 ++++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 22 deletions(-) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index 558a1eb62a..d38da346dd 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -2,7 +2,32 @@ A number of additional setup steps are required for enabling testing against "smart devices" that communicate with the cloud. The tests themselves are part of the `subset/cloud/test_udmi` -module included in the standard DAQ distro. +module included in the standard DAQ distro. The same basic device-to-cloud validation test +pipeline can be done manually and automatically (through DAQ); it's instructure to fully +understand the manual test pipeline before engaging with the automated setup. + +## Manual Test Pipeline + +The overal device-to-cloud pipeline looks something like the following: + +* Device sends data to the cloud. There's two kinds of devices: + * A faux _reference design_ device called [pubber](pubber.md), which is a completely contained + software device. + * An actual physical device. The setup and configuration of that device will be manufacturer + dependent and so is out of scope for this (DAQ) documentation. +* A configured GCP IoT Core project, registry, and device entry. The +[GCP docs for IoT Core](https://cloud.google.com/iot/docs/how-tos/devices) describe the basics. The +key part is the _authentication key_ (hahaha) that needs to be setup between the local device and +cloud device entry. +* The IoT Core registry is configured with a _PubSub topic_ (not to be confused with an _MQTT topic_), +that provides the bridge between incoming data and consumers of that data. See the GCP documentation +on PubSub for more details. +* (optional) The `gcloud` command line can be used to validate that data is being sent from the +device to the cloud. Something like +`gcloud pubsub subscriptions pull --auto-ack projects/{project}/subscriptions/{sub_id}`. +(Complete documentation for how to use `gcloud` commands is out of scope of this documentation.) +* The [validator tool](validator.md) is what programmatically validates a device data stream, and +is what is ultimately used by `test_udmi` to validate device-cloud communication. ## Base Local Test Setup @@ -30,27 +55,6 @@ as in `resources/test_site/mac_addrs/3c5ab41e8f0b/module_config.json`. * The GCP IoT Core setup needs to have a proper registry and device configred. This can either be done manually or using the [registrar tool](registrar.md) tool. -## Manual Test Pipeline - -If anything goes wrong with the automatic DAQ setup, the ability to manually test everything will -help debug the problem to see what's going wrong. The overal pipeline looks something like the -following: - -* Device sends data to the cloud. There's two kinds of devices: - * A faux _reference design_ device called [pubber](pubber.md), which is a completely contained - software device. - * An actual physical device. The setup and configuration of that device will be manufacturer - dependent and so is out of scope for this (DAQ) documentation. -* A configured GCP IoT Core project, registry, and device entry. The -[GCP docs for IoT Core](https://cloud.google.com/iot/docs/how-tos/devices) describe the basics. The -key part is the _authentication key_ (hahaha) that needs to be setup between the local device and -cloud device entry. -* The `gcloud` command line can be used to validate that data is being sent from the device to the -cloud. Something like `gcloud pubsub subscriptions pull --auto-ack projects/{project}/subscriptions/{sub_id}`. -(Complete documentation for how to use `gcloud` commands is out of scope of this documentation.) -* The [validator tool](validator.md) is what programmatically validates a device data stream, and -is what is ultimately used by `test_udmi` to validate device-cloud communication. - ## Integration Testing If developing cloud-tests, then the CI build system also needs to have a service account configured From fb34e626b76de3d91781f5d184174ee47712e9cc Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 23 Jun 2020 11:37:20 -0700 Subject: [PATCH 08/11] Instructure typo --- docs/cloud_tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index d38da346dd..ad2875d377 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -3,7 +3,7 @@ A number of additional setup steps are required for enabling testing against "smart devices" that communicate with the cloud. The tests themselves are part of the `subset/cloud/test_udmi` module included in the standard DAQ distro. The same basic device-to-cloud validation test -pipeline can be done manually and automatically (through DAQ); it's instructure to fully +pipeline can be done manually and automatically (through DAQ); it's instructive to fully understand the manual test pipeline before engaging with the automated setup. ## Manual Test Pipeline From 0895ff907ea373083e7aae673eca7838ba2f0b3e Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 23 Jun 2020 11:40:39 -0700 Subject: [PATCH 09/11] Fix overall typo --- docs/add_test.md | 2 +- docs/cloud_tests.md | 2 +- docs/orchestration.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/add_test.md b/docs/add_test.md index f279359570..e460137584 100644 --- a/docs/add_test.md +++ b/docs/add_test.md @@ -34,7 +34,7 @@ A setup for the `pass` test, as an example, woud be configured as follows * `echo host_tests=local/local_tests.conf >> local/system.conf` -- Set tests configuration. This, of course, only works for local development when using the `local_tests.conf` config. To -formalize a test and include it in the overal system build it should be included in +formalize a test and include it in the overall system build it should be included in `config/modules/all.conf`. ## Component Build diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index ad2875d377..f45b6229a5 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -8,7 +8,7 @@ understand the manual test pipeline before engaging with the automated setup. ## Manual Test Pipeline -The overal device-to-cloud pipeline looks something like the following: +The overall device-to-cloud pipeline looks something like the following: * Device sends data to the cloud. There's two kinds of devices: * A faux _reference design_ device called [pubber](pubber.md), which is a completely contained diff --git a/docs/orchestration.md b/docs/orchestration.md index 4804ec356f..703c5235a0 100644 --- a/docs/orchestration.md +++ b/docs/orchestration.md @@ -11,7 +11,7 @@ to change. ## Data Rouces -The overal orchestration capability relies on several simple data sources: +The overall orchestration capability relies on several simple data sources: 1. [Overall network topology](topologies.md), which indicates how the network hardware is configured. 2. [Device MUD files](../mud_files), which provide an [IETF Standard MUD descriptor](https://datatracker.ietf.org/doc/draft-ietf-opsawg-mud/) that describes From 10d5891862fed5e5aa4ea0799c78196ca75f4e79 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 23 Jun 2020 11:41:27 -0700 Subject: [PATCH 10/11] Typo --- docs/cloud_tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index f45b6229a5..28857aa720 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -128,5 +128,5 @@ $ export DAQ_TEST=aux Take note the URL in your browser's address bar when running Travis. You might be on either travis-ci.com or travis-ci.org. Any particular setup -may end up across both sites for undertermined reasons. Please consult with your browser's +may end up across both sites for undetermined reasons. Please consult with your browser's exact URL for more clarity. From 226646ab25298847c7f2036a20d4a8e3ba547359 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Wed, 24 Jun 2020 10:41:36 -0700 Subject: [PATCH 11/11] Adressing typos --- docs/cloud_tests.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md index 28857aa720..6ed8a9efed 100644 --- a/docs/cloud_tests.md +++ b/docs/cloud_tests.md @@ -32,7 +32,8 @@ is what is ultimately used by `test_udmi` to validate device-cloud communication ## Base Local Test Setup * The `udmi` module needs to be enabled in build. When running `cmd/build` there should be a line -like `subset/cloud/Dockerfile.test_udmi`. This is enabled through the `host_tests` config parameter, +like `subset/cloud/Dockerfile.test_udmi` in the startup logs. +This is enabled through the `host_tests` config parameter, which can be set to `config/modules/all.conf` if necessary. On startup, there should be a log message that includes `udmi`: ``` @@ -58,12 +59,12 @@ be done manually or using the [registrar tool](registrar.md) tool. ## Integration Testing If developing cloud-tests, then the CI build system also needs to have a service account configured -pointing at a suitable GCP proejct. To run cloud-based tests, setup the Travis `GCP_BASE64_CRED` +pointing at a suitable GCP project. To run cloud-based tests, setup the Travis `GCP_BASE64_CRED` env variable with a `base64` encoded service account key for your project. It's recommended to use a dedicated key with a nice name like `daq-travis`, but not required. Encode the key value as per below, and cut/paste the resulting string into a [Travis environment variable](https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings) -for a `GCP_BASE64_CRED` varaible. Note the `-w 0` option is required for proper parsing/formatting, +for a `GCP_BASE64_CRED` variable. Note the `-w 0` option is required for proper parsing/formatting, as there can't be any newlines in the copied string. @@ -79,7 +80,7 @@ iOiAiaHR0cHM6Ly93LWRhcS10ZXN0aW5nLmlhbS5nc2VydmljZWFjY291bnQuY29tIgp9Cg== * `gcp_topic` config to `local/system.conf` as described in this doc. * Configure test subsystem with proper cloud endpoint in `{test_site}/cloud_iot_config.json`. * Configure the DUT with the proper cloud device credentials (device specific). For _faux_ devices, this means copying -the assocatied `rsa_private.pkcs8` file to someting like `inst/faux/daq-faux-2/local/` (exact path depends on which faux). +the associated `rsa_private.pkcs8` file to something like `inst/faux/daq-faux-2/local/` (exact path depends on which faux). * Test with `bin/registrar`, `pubber/bin/run`, and `bin/validate` manually, before integrated testing through DAQ. ### Is my Travis set up correctly?