Support ad-hoc CWL workflows

[Nazim-crim]

https://crim-ca.atlassian.net/browse/NRCAN06-88

• Add support for ad-hoc CWL workflow execution through the POST /jobs endpoint using multipart/mixed
  or multipart/related content types. Users can now deploy and execute a Process in a single request by
  submitting both the CWL workflow definition (with Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description)
  and execution parameters (with Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/execute)
  in a multipart request body. The workflow is automatically deployed (with temporary ad-hoc tagging) before
  execution, eliminating the need for separate deployment and execution steps for one-time workflow runs

• resolves Support ad-hoc CWL workflows #834

[codecov]

Codecov Report

❌ Patch coverage is 90.00000% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.45%. Comparing base (4925f5f) to head (394d2a5).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
weaver/processes/utils.py	84.74%	2 Missing and 7 partials ⚠️
weaver/wps_restapi/jobs/utils.py	93.75%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #997   +/-   ##
=======================================
  Coverage   88.44%   88.45%           
=======================================
  Files          88       88           
  Lines       20874    20970   +96     
  Branches     2760     2788   +28     
=======================================
+ Hits        18462    18549   +87     
- Misses       1726     1727    +1     
- Partials      686      694    +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[fmigneault]

Great PR! Thanks for the updates. Some semantic adjustment needed for the process description profile URI that is misused.

[Copilot]

Pull request overview

This PR adds support for submitting and executing ad-hoc CWL workflows in a single POST /jobs request using multipart/mixed or multipart/related, aligning Weaver with OGC API - Processes Part 3 workflow submission patterns.

Changes:

• Add a new multipart-capable POST /jobs route and OpenAPI schema for multipart job submission.
• Implement multipart parsing to separate CWL deployment parts from the execute payload, then deploy-and-run inline.
• Add functional/unit tests plus documentation, examples, and changelog entries describing ad-hoc multipart execution.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
weaver/wps_restapi/swagger_definitions.py	Adds OpenAPI schemas/headers for multipart ad-hoc job execution.
weaver/wps_restapi/jobs/utils.py	Adds helper to parse multipart, deploy CWL payload, and return (process_id, execution_body).
weaver/wps_restapi/jobs/jobs.py	Registers multipart POST /jobs handler and wires deploy-then-submit flow with ad-hoc tagging.
weaver/wps_restapi/examples/job_execute_adhoc.http	Adds an HTTP example showing multipart ad-hoc workflow execution.
weaver/processes/utils.py	Adds multipart part profile extraction and a parser to split execution request vs CWL packages.
tests/wps_restapi/test_jobs.py	Adds functional tests covering successful multipart ad-hoc execution and failure scenarios.
tests/processes/test_utils.py	Adds unit tests for multipart profile extraction and multipart job execution parsing logic.
docs/source/references.rst	Adds a reference link for OGC Part 3 CWL ad-hoc workflow execution section.
docs/source/processes.rst	Documents how to submit ad-hoc workflows via multipart to POST /jobs with an included example.
CHANGES.rst	Adds release note entry for multipart ad-hoc CWL workflow execution support.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

weaver/processes/utils.py:789

• The _interpret_multipart_part type comment indicates the extracted profile element is always a str, but _extract_multipart_profile can return None. This makes the type annotation inaccurate and can trip static type checking / IDE assumptions.

def _interpret_multipart_part(part, request=None):
    # type: (Message, Optional[AnyRequestType]) -> Optional[Tuple[str, str, str, str, JSON]]
    """

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

docs/source/processes.rst:1724

• The docs claim the deployed workflow/tools are tagged with ad-hoc, but the implementation only tags the job (see create_job tags list). This should be corrected to match actual tagging behavior.

- The workflow (and any dependent tools) are automatically deployed with ``ad-hoc`` tagging
- The job is immediately submitted for execution using the provided inputs
- The workflow remains deployed after execution and can be reused or cleaned up later

docs/source/processes.rst:1739

• The note says the deployed workflow/tools are tagged with ad-hoc for identification, but the current implementation only tags the job. Either implement process tagging or update this note to avoid implying deployed processes are tagged.

    The deployed workflow and tools remain available after ad-hoc execution and are tagged with ``ad-hoc`` for
    identification. They can be cleaned up using the :ref:`Undeploy <proc_op_undeploy>` operation if no longer needed.

weaver/wps_restapi/swagger_definitions.py:7811

• The OpenAPI schema advertises an optional process_meta multipart part, but the current ad-hoc execution implementation never applies that metadata when deploying the process (the parser only returns CWL packages + execute body, and the deploy path ignores the metadata part). This is misleading for API consumers.

    process_meta = ProcessDeployment(
        missing=drop,
        description=(
            "Optional process description metadata. "
            f"Part with Content-Profile: {OGC_API_PROC_PROFILE_PROC_DESC_URI}"

docs/source/processes.rst:1714

• This section states that an optional process-description metadata part can be provided for ad-hoc execution, but the current implementation does not apply that metadata during deployment (it is effectively ignored). The docs should reflect current behavior to avoid confusion.

This issue also appears in the following locations of the same file:

• line 1722
• line 1738

   Optionally, additional :term:`Process` metadata parts can be provided with
   ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description``
   (see `crim-ca/weaver#990 <https://github.com/crim-ca/weaver/issues/990>`_).
   The example below demonstrates this optional metadata part.

CHANGES.rst:20

• The changelog entry mentions "temporary ad-hoc tagging" for the deployed workflow, but the code currently applies the ad-hoc tag to the job (not the deployed process). The wording should match actual behavior.

  in a multipart request body. The workflow is automatically deployed (with temporary ``ad-hoc`` tagging) before
  execution, eliminating the need for separate deployment and execution steps for one-time workflow runs

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (6)

weaver/wps_restapi/jobs/utils.py:165

• When multiple CWL parts are present, cwl_to_deploy becomes a Python list while container=request still has a multipart Content-Type header. This triggers parse_process_deploy_content to treat the list as raw multipart bytes and fails to deploy multi-part ad-hoc workflows.

    # Deploy the CWL packages
    cwl_to_deploy = cwl_packages[0] if len(cwl_packages) == 1 else cwl_packages
    deploy_response = deploy_process_from_payload(
        payload=cwl_to_deploy,
        container=request,
        overwrite=False
    )

docs/source/processes.rst:1716

• The documentation uses Content-Profile URIs under .../def/ogcapi-processes/2.0/..., but the multipart parser matches the Weaver-defined OGC profile URIs (e.g. https://www.opengis.net/def/profile/OGC/0/ogc-execute-request). As written, the documented Content-Profile values will not be recognized and clients will fall back to heuristic detection.

   Optionally, additional :term:`Process` metadata parts can be provided with
   ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description``
   (see `crim-ca/weaver#990 <https://github.com/crim-ca/weaver/issues/990>`_).
   The example below demonstrates this optional metadata part.
2. One execution request part with ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute``
   containing the job inputs and execution parameters.

weaver/wps_restapi/examples/job_execute_adhoc.http:81

• Example request uses Content-Profile values under .../def/ogcapi-processes/2.0/..., but the implementation matches Weaver's OGC profile URIs (e.g. https://www.opengis.net/def/profile/OGC/0/...). Update the example to the actual accepted profile URIs to avoid clients relying on fallback heuristics.

--adhoc-boundary
Content-Type: application/json
Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description

{
  "id": "adhoc-workflow",
  "title": "Ad-hoc Echo Workflow",
  "description": "Example ad-hoc workflow deployed and executed inline."
}

--adhoc-boundary
Content-Type: application/json
Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute

weaver/wps_restapi/examples/job_execute_adhoc_body.http:77

• Example body uses Content-Profile values under .../def/ogcapi-processes/2.0/..., but the multipart parser matches Weaver's OGC profile URIs (e.g. https://www.opengis.net/def/profile/OGC/0/...). Update the example to the actual accepted profile URIs.

--adhoc-boundary
Content-Type: application/json
Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description

{
  "id": "adhoc-workflow",
  "title": "Ad-hoc Echo Workflow",
  "description": "Example ad-hoc workflow deployed and executed inline."
}

--adhoc-boundary
Content-Type: application/json
Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute

{
  "inputs": {
    "message": "Hello from ad-hoc workflow!"
  },
  "outputs": {
    "result": {
      "transmissionMode": "reference"
    }
  },
  "mode": "async",
  "response": "document"
}

CHANGES.rst:20

• CHANGES entry documents Content-Profile URIs under .../def/ogcapi-processes/2.0/..., but the implementation matches Weaver's OGC profile URIs (e.g. https://www.opengis.net/def/profile/OGC/0/ogc-execute-request). Update the changelog to reflect the actual header values.

- Add support for ad-hoc `CWL` workflow execution through the ``POST /jobs`` endpoint using ``multipart/mixed``
  or ``multipart/related`` content types. Users can now deploy and execute a `Process` in a single request by
  submitting both the `CWL` workflow definition (with ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description``)
  and execution parameters (with ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute``)
  in a multipart request body. The workflow is automatically deployed (with temporary ``ad-hoc`` tagging) before
  execution, eliminating the need for separate deployment and execution steps for one-time workflow runs

weaver/processes/utils.py:889

• ExecuteBodyMultipart and the docs/examples describe an optional process-metadata part (Content-Profile: process-description), but parse_multipart_job_execution / _organize_job_execution_parts currently discard that JSON and only return CWL packages + execution request. As a result, submitted process metadata (title/description/id overrides, etc.) has no effect on the deployed ad-hoc process.

        # Check if this is a deployment part based on profile or content
        elif ((profile and sd.OGC_API_PROC_PROFILE_PROC_DESC_URI in profile) or (
            isinstance(part_data, dict) and part_data.get("class") in ["CommandLineTool", "Workflow", "ExpressionTool"]
        ) or (
            isinstance(part_data, dict) and "cwlVersion" in part_data and "$graph" in part_data
        )):
            deployment_parts.append((content_type_part, content_id, content_location, profile, part_data))
        # JSON/YAML parts without profile could be execution request (fallback)

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (8)

weaver/wps_restapi/examples/job_execute_adhoc.http:68

• The example uses Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute, but the multipart parser checks for https://www.opengis.net/def/profile/OGC/0/ogc-execute-request. Update the example to match what the server recognizes (or update the parser to accept both).

Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute

weaver/wps_restapi/examples/job_execute_adhoc_body.http:64

• The example uses Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute, but the multipart parser checks for https://www.opengis.net/def/profile/OGC/0/ogc-execute-request. Update the example to match what the server recognizes (or update the parser to accept both).

Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute

weaver/processes/utils.py:1012

• parse_multipart_job_execution can raise HTTPBadRequest("Content is not multipart format") from _parse_multipart_message, which results in a non-JSON error body even though this endpoint otherwise returns structured JSON errors. Wrapping the parse call to re-raise a JSON HTTPBadRequest would keep error responses consistent for clients.

    msg = _parse_multipart_message(content, content_type, request)
    root_workflow_cid = _extract_multipart_start_parameter(content_type)
    interpreted_parts = []

weaver/wps_restapi/swagger_definitions.py:7812

• ExecuteBodyMultipart documents/supports an optional process_meta part, but the multipart execution implementation only extracts (execution_body, cwl_packages) and never applies this metadata to the deployed ad-hoc process. This makes the OpenAPI/docs misleading and prevents clients from setting process title/description during ad-hoc execution. Either parse/forward the process metadata into deploy_process_from_payload (or update metadata after deployment) or remove process_meta from the schema/docs until supported.

    process_meta = ProcessDeployment(
        missing=drop,
        description=(
            "Optional process description metadata. "
            f"Part with Content-Profile: {OGC_API_PROC_PROFILE_PROC_DESC_URI}"
        )
    )
    execute_body = Execute(

docs/source/processes.rst:1716

• This section instructs clients to use Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/..., but the server-side multipart parser checks for Weaver's profile URIs (https://www.opengis.net/def/profile/OGC/0/ogc-process-description and .../ogc-execute-request). As written, the documented profiles won't be recognized and will force fallback heuristics (and error messages reference different URIs). Update the documentation (or expand the parser to accept both URI variants).

1. One or more :term:`CWL` parts (workflow and any dependent tools) with ``Content-Type: application/cwl[+json|+yaml]``.
   Optionally, additional :term:`Process` metadata parts can be provided with
   ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description``
   (see `crim-ca/weaver#990 <https://github.com/crim-ca/weaver/issues/990>`_).
   The example below demonstrates this optional metadata part.
2. One execution request part with ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute``
   containing the job inputs and execution parameters.

weaver/wps_restapi/examples/job_execute_adhoc.http:58

• The example uses Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description, but the multipart parser checks for https://www.opengis.net/def/profile/OGC/0/ogc-process-description. Update the example to match what the server recognizes (or update the parser to accept both).

This issue also appears on line 68 of the same file.

Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description

weaver/wps_restapi/examples/job_execute_adhoc_body.http:54

• The example uses Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description, but the multipart parser checks for https://www.opengis.net/def/profile/OGC/0/ogc-process-description. Update the example to match what the server recognizes (or update the parser to accept both).

This issue also appears on line 64 of the same file.

Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description

CHANGES.rst:18

• The changelog entry references Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/..., but the implementation checks for profile URIs under https://www.opengis.net/def/profile/OGC/0/.... Align the URIs so the changelog matches actual behavior (or expand the parser to accept both).

  submitting both the `CWL` workflow definition (with ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/process-description``)
  and execution parameters (with ``Content-Profile: https://www.opengis.net/def/ogcapi-processes/2.0/execute``)