diff --git a/cmd/build b/cmd/build index dc428ba90d..f4f09dcf32 100755 --- a/cmd/build +++ b/cmd/build @@ -26,11 +26,12 @@ DOCKER_IMAGE_VER=docker_images.ver cd $ROOT source etc/config_base.sh -host_tests=$host_tests bin/docker_build_files +echo host_tests=$host_tests +test_targets=$host_tests bin/docker_build_files function pull_images { TAG=$1 declare -A test_set - for target in $(host_tests=$host_tests bin/docker_build_files); do + for target in $test_targets; do target=$(echo $target | sed 's|^.*/Dockerfile.||' | echo daqf/$(> local/system.conf` -- Set tests configuration. This, of course, only works for local development when using the `local_tests.conf` config. To -formalize a test and include it in the overal system build it should be included in +formalize a test and include it in the overall system build it should be included in `config/modules/all.conf`. ## Component Build diff --git a/docs/cloud_tests.md b/docs/cloud_tests.md new file mode 100644 index 0000000000..6ed8a9efed --- /dev/null +++ b/docs/cloud_tests.md @@ -0,0 +1,133 @@ +# Cloud Connection Testing + +A number of additional setup steps are required for enabling testing against "smart devices" +that communicate with the cloud. The tests themselves are part of the `subset/cloud/test_udmi` +module included in the standard DAQ distro. The same basic device-to-cloud validation test +pipeline can be done manually and automatically (through DAQ); it's instructive to fully +understand the manual test pipeline before engaging with the automated setup. + +## Manual Test Pipeline + +The overall device-to-cloud pipeline looks something like the following: + +* Device sends data to the cloud. There's two kinds of devices: + * A faux _reference design_ device called [pubber](pubber.md), which is a completely contained + software device. + * An actual physical device. The setup and configuration of that device will be manufacturer + dependent and so is out of scope for this (DAQ) documentation. +* A configured GCP IoT Core project, registry, and device entry. The +[GCP docs for IoT Core](https://cloud.google.com/iot/docs/how-tos/devices) describe the basics. The +key part is the _authentication key_ (hahaha) that needs to be setup between the local device and +cloud device entry. +* The IoT Core registry is configured with a _PubSub topic_ (not to be confused with an _MQTT topic_), +that provides the bridge between incoming data and consumers of that data. See the GCP documentation +on PubSub for more details. +* (optional) The `gcloud` command line can be used to validate that data is being sent from the +device to the cloud. Something like +`gcloud pubsub subscriptions pull --auto-ack projects/{project}/subscriptions/{sub_id}`. +(Complete documentation for how to use `gcloud` commands is out of scope of this documentation.) +* The [validator tool](validator.md) is what programmatically validates a device data stream, and +is what is ultimately used by `test_udmi` to validate device-cloud communication. + +## Base Local Test Setup + +* The `udmi` module needs to be enabled in build. When running `cmd/build` there should be a line +like `subset/cloud/Dockerfile.test_udmi` in the startup logs. +This is enabled through the `host_tests` config parameter, +which can be set to `config/modules/all.conf` if necessary. On startup, there should be a log +message that includes `udmi`: +``` +Jun 22 08:32:52 runner INFO Configured with tests pass, fail, ping, bacnet, mudgee, nmap, discover, switch, macoui, bacext, tls, password, udmi, manual +``` +* A testing gcp service account `gcp_cred` needs to be setup as described in +[service account setup instructions](service.md). +* The system's default `module_config` needs to enable the `udmi` test, e.g. as per +`resources/setups/baseline/module_config.json`. This can be validated by (runtime) checking +`inst/run-port-01/nodes/udmi01/tmp/module_config.json` to see if it has something like the following: +``` + "udmi": { + "enabled": true + } +``` +* `site_path` config needs to point to a site definition directory, or defaults to `local/site`. +This contains all the site-specific information about devices needed for testing. +* `{site_path}/mac_addrs/{mac_addr}/module_config.json` needs to have a `device_id` defined, e.g. +as in `resources/test_site/mac_addrs/3c5ab41e8f0b/module_config.json`. +* The GCP IoT Core setup needs to have a proper registry and device configred. This can either +be done manually or using the [registrar tool](registrar.md) tool. + +## Integration Testing + +If developing cloud-tests, then the CI build system also needs to have a service account configured +pointing at a suitable GCP project. To run cloud-based tests, setup the Travis `GCP_BASE64_CRED` +env variable with a `base64` encoded service account key for your project. It's recommended to +use a dedicated key with a nice name like `daq-travis`, but not required. Encode the key value +as per below, and cut/paste the resulting string into a +[Travis environment variable](https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings) +for a `GCP_BASE64_CRED` variable. Note the `-w 0` option is required for proper parsing/formatting, +as there can't be any newlines in the copied string. + + +$ base64 -w 0 local/gcp_service_account.json +ewoICJ1eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAiYm9zLWRhcS10ZXN0aW5nIiwKICAicHJpd +… +iOiAiaHR0cHM6Ly93LWRhcS10ZXN0aW5nLmlhbS5nc2VydmljZWFjY291bnQuY29tIgp9Cg== + + +### Travis CI Testing + +* Run the [registrar tool](registrar.md) to properly configure the cloud project. +* `gcp_topic` config to `local/system.conf` as described in this doc. +* Configure test subsystem with proper cloud endpoint in `{test_site}/cloud_iot_config.json`. +* Configure the DUT with the proper cloud device credentials (device specific). For _faux_ devices, this means copying +the associated `rsa_private.pkcs8` file to something like `inst/faux/daq-faux-2/local/` (exact path depends on which faux). +* Test with `bin/registrar`, `pubber/bin/run`, and `bin/validate` manually, before integrated testing through DAQ. + +### Is my Travis set up correctly? + +If Travis is set up correctly, you should see messages at the beginning of the log file: +``` +Setting environment variables from repository settings +$ export DOCKER_USERNAME=[secure] +$ export DOCKER_PASSWORD=[secure] +$ export GCP_BASE64_CRED=[secure] +``` + +Further down there would be more details about the cred itself: +``` +Running test script testing/test_aux.sh +Writing test results to inst/test_aux.out and inst/test_aux.gcp +Decoding GCP_BASE64_CRED to inst/config/gcp_service_account.json +base64 wc: 1 1 3097 +GCP service account is "daq-travis@daq-testing.iam.gserviceaccount.com" +``` + +If the `3097` character count is wildly off, then likely something went wrong with the newlines. + +### Travis Build For "External" Pull Requests + +Travis will not use encrypted environment variables when testing against pull requests +from foreign github repositories, even if you've forked from another repository that you +have full control of via Github. Travis authorization != Github authorization, even if +you sign into Travis using Github! This is as it should be b/c security. see the following +for more info: + +- https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings +- https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions + +If your test is failing from a PR, you'll see something like in a similar log location: + +``` +Encrypted environment variables have been removed for security reasons. +See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions +Setting environment variables from .travis.yml +$ export DOCKER_STARTUP_TIMEOUT_MS=60000 +$ export DAQ_TEST=aux +``` + +### Other Travis Caveats + +Take note the URL in your browser's address bar when running Travis. You might be on either +travis-ci.com or travis-ci.org. Any particular setup +may end up across both sites for undetermined reasons. Please consult with your browser's +exact URL for more clarity. diff --git a/docs/integration_testing.md b/docs/integration_testing.md deleted file mode 100644 index 1c6435a9b0..0000000000 --- a/docs/integration_testing.md +++ /dev/null @@ -1,83 +0,0 @@ -# Integration Testing - -DAQ currently uses Travis CI for integration testing: https://travis-ci.org/ - -## Configuration - -The `test_udmi` test module uses the Registrar and Validator to check that a device is -properly communicating through Cloud IoT, automated through DAQ. - -### GCP Credential - -To run cloud-based tests, setup the Travis `GCP_BASE64_CRED` env variable with a `base64` encoded -service account key for your project. It's recommended to use a dedicated key with a nice name -like `daq-travis`, but not required. Encode the key value as per below, and cut/paste the -resulting string into a -[Travis environment variable](https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings) -for a `GCP_BASE64_CRED` varaible. Note the `-w 0` option is required for proper parsing/formatting, -as there can't be any newlines in the copied string. - - -$ base64 -w 0 local/gcp_service_account.json -ewoICJ1eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAiYm9zLWRhcS10ZXN0aW5nIiwKICAicHJpd -… -iOiAiaHR0cHM6Ly93LWRhcS10ZXN0aW5nLmlhbS5nc2VydmljZWFjY291bnQuY29tIgp9Cg== - - -## Travis CI Testing - -* Run the [registrar tool](registrar.md) to properly configure the cloud project. -* `gcp_topic` config to `local/system.conf` as described in this doc. -* Configure test subsystem with proper cloud endpoint in `{test_site}/cloud_iot_config.json`. -* Configure the DUT with the proper cloud device credentials (device specific). For _faux_ devices, this means copying -the assocatied `rsa_private.pkcs8` file to someting like `inst/faux/daq-faux-2/local/` (exact path depends on which faux). -* Test with `bin/registrar`, `pubber/bin/run`, and `bin/validate` manually, before integrated testing through DAQ. - -### Is my Travis set up correctly? - -If Travis is set up correctly, you should see messages at the beginning of the log file: -``` -Setting environment variables from repository settings -$ export DOCKER_USERNAME=[secure] -$ export DOCKER_PASSWORD=[secure] -$ export GCP_BASE64_CRED=[secure] -``` - -Further down there would be more details about the cred itself: -``` -Running test script testing/test_aux.sh -Writing test results to inst/test_aux.out and inst/test_aux.gcp -Decoding GCP_BASE64_CRED to inst/config/gcp_service_account.json -base64 wc: 1 1 3097 -GCP service account is "daq-travis@daq-testing.iam.gserviceaccount.com" -``` - -If the `3097` character count is wildly off, then likely something went wrong with the newlines. - -### Travis Build For "External" Pull Requests - -Travis will not use encrypted environment variables when testing against pull requests -from foreign github repositories, even if you've forked from another repository that you -have full control of via Github. Travis authorization != Github authorization, even if -you sign into Travis using Github! This is as it should be b/c security. see the following -for more info: - -- https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings -- https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions - -If your test is failing from a PR, you'll see something like in a similar log location: - -``` -Encrypted environment variables have been removed for security reasons. -See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions -Setting environment variables from .travis.yml -$ export DOCKER_STARTUP_TIMEOUT_MS=60000 -$ export DAQ_TEST=aux -``` - -### Other Travis Caveats - -Take note the URL in your browser's address bar when running Travis. You might be on either -travis-ci.com or travis-ci.org. Any particular setup -may end up across both sites for undertermined reasons. Please consult with your browser's -exact URL for more clarity. diff --git a/docs/orchestration.md b/docs/orchestration.md index 4804ec356f..703c5235a0 100644 --- a/docs/orchestration.md +++ b/docs/orchestration.md @@ -11,7 +11,7 @@ to change. ## Data Rouces -The overal orchestration capability relies on several simple data sources: +The overall orchestration capability relies on several simple data sources: 1. [Overall network topology](topologies.md), which indicates how the network hardware is configured. 2. [Device MUD files](../mud_files), which provide an [IETF Standard MUD descriptor](https://datatracker.ietf.org/doc/draft-ietf-opsawg-mud/) that describes diff --git a/subset/cloud/test_udmi b/subset/cloud/test_udmi index 4b0cfb2a32..fde175a26d 100755 --- a/subset/cloud/test_udmi +++ b/subset/cloud/test_udmi @@ -1,4 +1,5 @@ #!/bin/bash -e + source reporting.sh REPORT=/tmp/report.txt @@ -87,3 +88,7 @@ function message_report { for message_type in $message_types; do message_report $message_type done + +fgrep RESULT $REPORT + +echo Done with test_udmi