Skip to content

feat(frontend): Support for UDF Ui Parameter#4268

Closed
carloea2 wants to merge 57 commits into
apache:mainfrom
carloea2:feat/ui-parameter
Closed

feat(frontend): Support for UDF Ui Parameter#4268
carloea2 wants to merge 57 commits into
apache:mainfrom
carloea2:feat/ui-parameter

Conversation

@carloea2
Copy link
Copy Markdown
Contributor

@carloea2 carloea2 commented Mar 7, 2026

What changes were proposed in this PR?

Add Python UDF Parameter support:
image

What the user writes (Python)

Users declare UI parameters once in open(), and then use the typed value directly:

from pytexera import *
from typing import Iterator, Optional

class ProcessTupleOperator(UDFOperatorV2):

    @overrides
    def open(self):
        # declare UiParameters once, store the parsed runtime values
        self.value1 = self.UiParameter(name="param1", type=AttributeType.DOUBLE).value
        self.value2 = self.UiParameter(name="param2", type=AttributeType.INT).value
        self.value3 = self.UiParameter(name="param3", type=AttributeType.STRING).value
        self.value4 = self.UiParameter(name="param4", type=AttributeType.TIMESTAMP).value

    @overrides
    def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
        print(self.value1)
        print(self.value2)
        print(self.value3)
        print(self.value4)
        yield tuple_

What shows up in the UI

From those self.UiParameter(...) lines, the property panel automatically generates a Parameters section with one row per parameter:

  • Name + Type are read-only (so they stay consistent with the code)
  • Value is editable (so users can change runtime values without touching the script)

How the values get into Python

When the workflow runs, we inject the UI values into the UDF and the base class applies them right before open() executes. That way, when the user calls UiParameter(...).value, they get the current value from the UI.

Overall:

sequenceDiagram
    autonumber

    participant FE as Frontend UI
    participant Parser as Parser Service
    participant Store as Operator Properties
    participant BE as Scala Backend
    participant Inj as Python Code Injector
    participant Py as Python Runtime
    participant UDF as User UDF

    FE->>Parser: Send Python code
    Parser->>Parser: Extract self.UiParameter declarations
    Parser-->>FE: Return parameter schema

    FE->>Store: Save uiParameters<br/>name, type, value
    Store->>BE: Submit operator descriptor

    BE->>Inj: Pass original code + uiParameters
    Inj->>Inj: Validate names and types
    Inj->>Inj: Generate injected hook
    Inj-->>BE: Return modified Python code

    BE->>Py: Execute injected code
    Py->>UDF: Load UDF class
    UDF->>Py: open() is wrapped

    Py->>UDF: Call injected hook
    UDF-->>Py: Return parameter values

    Py->>UDF: Provide typed UiParameter values
    UDF->>Py: Execute workflow logic
Loading

Detailed:

sequenceDiagram
    autonumber

    actor User

    participant CodeEditor as CodeEditorComponent<br/>Monaco editor
    participant YCode as Yjs shared code
    participant Sync as UiUdfParametersSyncService
    participant Parser as UiUdfParametersParserService
    participant ParamUI as UiUdfParametersComponent
    participant Formly as Formly / Operator Properties
    participant OpDesc as Python UDF Operator Descriptor<br/>Scala backend
    participant Injector as PythonUdfUiParameterInjector
    participant Runtime as Python Runtime
    participant Support as _UiParameterSupport
    participant UDF as User UDF Class

    User->>CodeEditor: Writes Python UDF code
    CodeEditor->>YCode: Updates shared code text
    YCode->>Sync: Emits code-change event

    Sync->>Parser: Parse latest Python code
    Parser->>Parser: Find supported UDF class
    Parser->>Parser: Scan self.UiParameter(...)
    Parser-->>Sync: Return inferred parameters<br/>name + AttributeType

    Sync->>Formly: Update operator uiParameters structure
    Formly->>ParamUI: Render parameter rows
    ParamUI-->>User: Shows inferred params<br/>name/type locked, value editable

    User->>ParamUI: Enters parameter values
    ParamUI->>Formly: Store edited values
    Formly->>OpDesc: Persist uiParameters in operator properties

    User->>OpDesc: Runs workflow

    OpDesc->>Injector: inject(code, uiParameters)
    Injector->>Injector: Validate uiParameters
    Injector->>Injector: Build parameter map
    Injector->>Injector: Generate reserved hook method
    Injector->>Injector: Insert hook into user UDF class
    Injector-->>OpDesc: Return injected Python code

    OpDesc->>Runtime: Execute OpExecWithCode(injectedCode, "python")
    Runtime->>UDF: Load user UDF class
    UDF->>Support: __init_subclass__ wraps open()

    Runtime->>UDF: Start operator execution
    UDF->>Support: wrapped open() runs first
    Support->>UDF: Call _texera_injected_ui_parameters()
    UDF-->>Support: Return UI parameter values
    Support->>Support: Store injected values

    Support->>UDF: Continue original user open()
    UDF->>Support: Create/read self.UiParameter(...)
    Support->>Support: Look up injected value by name
    Support->>Support: Convert value using AttributeType
    Support-->>UDF: Return typed Python value

    UDF->>Runtime: Process tuples / batches / source output
Loading

Any related issues, documentation, discussions?

Closes #4154

How was this PR tested?

Testing added to backend and frontend

Was this PR authored or co-authored using generative AI tooling?

Co-generated with GPT

@github-actions github-actions Bot added feature engine python frontend Changes related to the frontend GUI common labels Mar 7, 2026
@carloea2 carloea2 changed the title Feat/UI parameter feat(frontend): Support for UDF Ui Parameter Mar 7, 2026
@chenlica chenlica requested a review from aglinxinyuan March 7, 2026 16:06
@chenlica
Copy link
Copy Markdown
Contributor

chenlica commented Mar 7, 2026

@aglinxinyuan Please review it.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds UDF UI Parameter support, allowing users to declare typed parameters in their Python UDF open() method via self.UiParameter(name=..., type=AttributeType....). The frontend parses these from the code, displays them as an editable properties panel, and the backend injects the UI-supplied values into the Python code before execution.

Changes:

  • Frontend: New UiUdfParametersParserService and UiUdfParametersSyncService that parse self.UiParameter(...) calls from Python code and sync parsed parameters to the operator property store; new UiUdfParametersComponent renders the parameters table in the property panel.
  • Backend (Scala): New UiUDFParameter model and PythonUdfUiParameterInjector that injects a _texera_injected_ui_parameters hook method into the UDF class with the UI-supplied values encoded via the pybuilder mechanism; all three Python UDF operator descriptors updated to run the injector.
  • Backend (Python): New _UiParameterSupport mixin class added to all UDF base classes; it wraps open() via __init_subclass__ to apply injected values before user code runs.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 15 comments.

Show a summary per file
File Description
workflow-compiling.interface.ts Adds large_binary to AttributeType, JAVA/Python attribute type name constants, and derived union types for cross-language type token handling
ui-udf-parameters-parser.service.ts New service that parses self.UiParameter(...) calls from Python code and normalizes type tokens to canonical schema names
ui-udf-parameters-parser.service.spec.ts Tests for the parser service; contains incorrect expected type values
ui-udf-parameters-sync.service.ts New service that attaches to YText changes and syncs parsed parameter structure to operator properties
ui-udf-parameters.component.* New Angular Formly custom type component rendering the parameters table with read-only name/type and editable value columns
operator-property-edit-frame.component.ts/.scss Subscribes to param changes and maps the uiParameters field key to the custom Formly type; adds styling
code-editor.component.ts Attaches/detaches the YText listener when the Monaco editor is initialized/destroyed
formly-config.ts / app.module.ts Registers the new ui-udf-parameters Formly type and declares the component
PythonUdfUiParameterInjector.scala New Scala object that injects the _texera_injected_ui_parameters hook method into UDF classes
PythonUdfUiParameterInjectorSpec.scala Tests for the Scala injector; contains failing test assertions
UiUDFParameter.scala New Scala model class for a UI parameter (attribute + string value)
PythonUDFOpDescV2.scala / DualInputPortsPythonUDFOpDescV2.scala / PythonUDFSourceOpDescV2.scala Adds uiParameters field and wires the injector call
Attribute.java Adds @EncodableStringAnnotation to getName() for safe encoding in pybuilder templates; introduces unused imports
udf_operator.py Adds _UiParameterSupport mixin with UiParameter inner class and wrapping mechanism
attribute_type.py Adds FROM_STRING_PARSER_MAPPING for string-to-type conversion
pytexera/__init__.py / pyamber/__init__.py Exports AttributeType
collab-wrapper.component.css Minor whitespace change to an already-invalid CSS property

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread frontend/src/app/workspace/component/code-editor-dialog/code-editor.component.ts Outdated
Comment thread amber/src/main/python/pytexera/udf/udf_operator.py Outdated
Comment thread frontend/src/app/workspace/service/code-editor/ui-udf-parameters-sync.service.ts Outdated
Comment thread amber/src/main/python/core/models/schema/attribute_type.py Outdated
Comment thread amber/src/main/python/pytexera/udf/udf_operator.py
Copy link
Copy Markdown
Contributor

@aglinxinyuan aglinxinyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the test cases, as the frontend currently fails to compile. Also address the comments left by Copilot. The PR still feels quite draft-like, so please take some time to polish it before requesting for review.

Copy link
Copy Markdown
Contributor

@kunwp1 kunwp1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments. I tested the PR and the features look good. We need a architecture diagram do the review because the change is very big and I don't follow the details of the implementation. Can you add a diagram in the PR?

Comment thread amber/src/main/python/core/models/schema/attribute_type.py Outdated
Comment thread amber/src/main/python/pyamber/__init__.py
"Iterator",
"Optional",
"Union",
"Dict",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove Dict and Any

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this change is needed for the feature to work. The reason is the definitions of UiParameter use those as the return value types, so since the code comes from the User, I make them available from the import.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

Copy link
Copy Markdown
Contributor Author

@carloea2 carloea2 Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kunwp1 NameError: name 'Dict' is not defined
2026-04-18 12:17:21.586 | ERROR | core.architecture.rpc.async_rpc_server:receive:102 - name 'Dict' is not defined
File "C:\Users\carlo\AppData\Local\Temp\tmpkqz8h60kfsTempFS\udf-v1.py", line 12, in
class ProcessTupleOperator(UDFOperatorV2):
-> <class 'pytexera.udf.udf_operator.UDFOperatorV2'>

File "C:\Users\carlo\AppData\Local\Temp\tmpkqz8h60kfsTempFS\udf-v1.py", line 30, in ProcessTupleOperator
def _texera_injected_ui_parameters(self) -> Dict[str, Any]:

NameError: name 'Dict' is not defined

Comment thread frontend/src/app/workspace/types/workflow-compiling.interface.ts Outdated
@carloea2
Copy link
Copy Markdown
Contributor Author

@kunwp1 thanks for the review, I will let you know when the diagram is ready, though I think it may not reflect the changes, since the architecture is the same, just changed the way the UDFs contents are generated.

@carloea2
Copy link
Copy Markdown
Contributor Author

@kunwp1 are these diagrams ok?:

Overall:

sequenceDiagram
    autonumber

    participant FE as Frontend UI
    participant Parser as Parser Service
    participant Store as Operator Properties
    participant BE as Scala Backend
    participant Inj as Python Code Injector
    participant Py as Python Runtime
    participant UDF as User UDF

    FE->>Parser: Send Python code
    Parser->>Parser: Extract self.UiParameter declarations
    Parser-->>FE: Return parameter schema

    FE->>Store: Save uiParameters<br/>name, type, value
    Store->>BE: Submit operator descriptor

    BE->>Inj: Pass original code + uiParameters
    Inj->>Inj: Validate names and types
    Inj->>Inj: Generate injected hook
    Inj-->>BE: Return modified Python code

    BE->>Py: Execute injected code
    Py->>UDF: Load UDF class
    UDF->>Py: open() is wrapped

    Py->>UDF: Call injected hook
    UDF-->>Py: Return parameter values

    Py->>UDF: Provide typed UiParameter values
    UDF->>Py: Execute workflow logic
Loading

Detailed:

sequenceDiagram
    autonumber

    actor User

    participant CodeEditor as CodeEditorComponent<br/>Monaco editor
    participant YCode as Yjs shared code
    participant Sync as UiUdfParametersSyncService
    participant Parser as UiUdfParametersParserService
    participant ParamUI as UiUdfParametersComponent
    participant Formly as Formly / Operator Properties
    participant OpDesc as Python UDF Operator Descriptor<br/>Scala backend
    participant Injector as PythonUdfUiParameterInjector
    participant Runtime as Python Runtime
    participant Support as _UiParameterSupport
    participant UDF as User UDF Class

    User->>CodeEditor: Writes Python UDF code
    CodeEditor->>YCode: Updates shared code text
    YCode->>Sync: Emits code-change event

    Sync->>Parser: Parse latest Python code
    Parser->>Parser: Find supported UDF class
    Parser->>Parser: Scan self.UiParameter(...)
    Parser-->>Sync: Return inferred parameters<br/>name + AttributeType

    Sync->>Formly: Update operator uiParameters structure
    Formly->>ParamUI: Render parameter rows
    ParamUI-->>User: Shows inferred params<br/>name/type locked, value editable

    User->>ParamUI: Enters parameter values
    ParamUI->>Formly: Store edited values
    Formly->>OpDesc: Persist uiParameters in operator properties

    User->>OpDesc: Runs workflow

    OpDesc->>Injector: inject(code, uiParameters)
    Injector->>Injector: Validate uiParameters
    Injector->>Injector: Build parameter map
    Injector->>Injector: Generate reserved hook method
    Injector->>Injector: Insert hook into user UDF class
    Injector-->>OpDesc: Return injected Python code

    OpDesc->>Runtime: Execute OpExecWithCode(injectedCode, "python")
    Runtime->>UDF: Load user UDF class
    UDF->>Support: __init_subclass__ wraps open()

    Runtime->>UDF: Start operator execution
    UDF->>Support: wrapped open() runs first
    Support->>UDF: Call _texera_injected_ui_parameters()
    UDF-->>Support: Return UI parameter values
    Support->>Support: Store injected values

    Support->>UDF: Continue original user open()
    UDF->>Support: Create/read self.UiParameter(...)
    Support->>Support: Look up injected value by name
    Support->>Support: Convert value using AttributeType
    Support-->>UDF: Return typed Python value

    UDF->>Runtime: Process tuples / batches / source output
Loading

@kunwp1
Copy link
Copy Markdown
Contributor

kunwp1 commented Apr 18, 2026

I haven't read the details of the diagram yet but will do it later after all the comments are addressed. Please move the diagrams to the PR description.

@github-actions github-actions Bot added the dependencies Pull requests that update a dependency file label Apr 18, 2026
@carloea2
Copy link
Copy Markdown
Contributor Author

@kunwp1
Thanks. I have resolved your comments. The diagram is also now in the PR description.

@carloea2
Copy link
Copy Markdown
Contributor Author

@kunwp1 Hi do you have any update about the PR?

@chenlica
Copy link
Copy Markdown
Contributor

@kunwp1 ?

@kunwp1
Copy link
Copy Markdown
Contributor

kunwp1 commented Apr 23, 2026

We had an offline discussion and decided to split this PR.

@chenlica
Copy link
Copy Markdown
Contributor

For our records, please also include the reason of this decision.

@kunwp1
Copy link
Copy Markdown
Contributor

kunwp1 commented Apr 23, 2026

The PR is too big that reviewers couldn't finish the review.

@chenlica
Copy link
Copy Markdown
Contributor

It's another example to show the importance to have small PRs.

@carloea2 carloea2 closed this Apr 23, 2026
@carloea2
Copy link
Copy Markdown
Contributor Author

I think someone from outside wanting to contribute will feel somewhat frustrated after so much time and so many messages, only to have to close the PR. Do the Texera guides have any recommendations for the number of lines of code?

@chenlica
Copy link
Copy Markdown
Contributor

chenlica commented Apr 23, 2026

I feel the code can be reused as multiple PRs. So the effort is not wasted.

Quality control is important for the project. We don't have a fixed cap for lines of code. I hope other people can chime in. @kunwp1 @Yicong-Huang @aglinxinyuan ?

@aglinxinyuan
Copy link
Copy Markdown
Contributor

We should aim to keep PRs as small as possible unless you can clearly prove that a PR cannot be split. I don’t think that’s the case here.

Breaking a larger prototype into smaller PRs can also help you think more carefully about how to modularize your code.

While we don’t have a strict limit on PR size, in my experience, anything with more than ~500 lines of additions is usually too large—unless it’s primarily refactoring (e.g., moving existing code without significant changes) or includes generated files.

@Yicong-Huang
Copy link
Copy Markdown
Contributor

It's all about communication: in Pr context, it is communication between PR author and the reviewer. Larger PRs usually get fewer and shallower reviews. A huge PR is almost equivalent to being unreviewable.

I agree with Xinyuan. A reasonable rule of thumb is that the source-code of a PR should ideally be around 200 LoC, with 500 LoC as a rough upper bound, and should usually touch fewer than 10 files. Test files can be a bit bigger and usually reviewers pay less effort to review tests.

Of course, this also depends on the language and the nature of the change.

@kunwp1
Copy link
Copy Markdown
Contributor

kunwp1 commented Apr 24, 2026

I don't have a specific limit for LoC, but 1,500+ lines across 11 new files is a lot to review at once, especially since it changes both the frontend and backend. Reviewing new files is naturally harder because we lack existing context. To make this smoother, I'd suggest making the PR description more robust, including an architecture diagram, a breakdown of new modules, and an explanation of the new APIs. Also, for features of this scale, it might be helpful to have a quick walkthrough with reviewers before the formal request. This gives everyone the necessary context to review efficiently.

@carloea2
Copy link
Copy Markdown
Contributor Author

I want to share some feedback about the review process for apache/texera. This is not meant to reopen the PR or attack anyone personally, but I do think the experience raised some process concerns that I want to share.

This PR was +1,737 / -19. Around the same time, larger PRs were merged, for example:

  1. feat(agent-service): enable Texera Agent to do workflow editing and execution #4540: +2,476 / -10
  2. feat(gui): add full computing unit tab #4331: +1,888 / -384
  3. feat(agent-service): add agent-service that manages LLM agents #4495: +9,473 / -2,774

Because of that, I found it difficult to understand/trust how PR size and review expectations are being applied consistently.

I also do not think Copilot reviews should carry much weight without human validation. In this PR, Copilot hallucinated at least twice. AI review can be useful as an extra signal, but it should not become a substitute for timely human review, especially when a PR is being delayed or blocked.

More generally, my experience contributing has become increasingly time-consuming and less pleasant. Review cycles feel longer, there are idle periods, and it is not always clear what is expected from the contributor. That makes the process frustrating, especially when feedback comes slowly or mainly from automated tools.

I also want to point out that test-related PRs should be reviewed with the same seriousness as other PRs. I do not agree with the idea that test PRs are lower-risk or that weaker review is acceptable because they are “only tests.” Poor tests can still create maintenance burden, false confidence, and long-term quality issues.

If reviewers have the right to reject a PR for quality reasons, then contributors should also be able to expect timely, consistent, and meaningful human review. Otherwise, the process becomes unbalanced: contributors are expected to respond carefully to every comment, while automated review can generate unreliable or redundant feedback without the same level of accountability.

I am sharing this as my personal perspective based on this PR and nearby examples. I am not asking everyone to agree with me and I do not intend to defend this perspective, but I do think the review process could be more consistent, transparent, and respectful of contributor time.

@carloea2 carloea2 removed their assignment May 13, 2026
@chenlica
Copy link
Copy Markdown
Contributor

@carloea2 Thanks for sharing your thoughts. We are all busy with a coming hackathon, and will get back to you after that.

@chenlica
Copy link
Copy Markdown
Contributor

@carloea2 Sorry for this very late reply. As you know, over the past few days we have been very busy with the dkNet-AI · Apache Texera Agent Hackathon event (https://texera.io/hackathon/) and post-event tasks.

Thanks a lot for sharing your feedback, which is very useful to help us manage this project and grow the community. I notice your post mentioned several topics. To have efficient communication, I suggest you create multiple discussion posts on GitHub and add the URLs in this issue (similar to our practice of dividing a big PR into smaller PRs). Then we can continue the discussion in those posts.

Thank you.

@carloea2
Copy link
Copy Markdown
Contributor Author

@chenlica

After communicating with other team members, I can say that my concerns have been addressed. I appreciate your time and the team’s efforts despite the busy schedule.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file engine feature frontend Changes related to the frontend GUI python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants