Configuration Storage Trait for Connectors Runtime

[mmodzelewski]

Overview

I'd like to propose extending the connectors runtime to support pluggable configuration storage through a trait-based approach. This would align the connectors runtime configuration management with the patterns already established in the Iggy server.

Proposed Changes

1. Configuration Storage Trait

Introduce a trait that abstracts configuration storage operations. This would enable:

• Default file-based implementation: Maintaining backward compatibility with the current file-based configuration system
• API-backed implementation: Supporting dynamic configuration management through a remote configuration API
• Future extensibility: Allowing additional storage backends (e.g., database, etcd, consul) without modifying core logic

2. Extended Connectors API

Enhance the connectors API layer to support:

• PUT endpoints: Enable updating connector configurations at runtime
• RESTful configuration management: Provide a consistent API surface for configuration CRUD operations

Benefits

1. Consistency: Aligns connectors runtime with Iggy server's configuration management approach
2. Flexibility: Enables dynamic configuration updates without restarts
3. Operational improvements: Supports centralized configuration management in distributed deployments
4. Backward compatibility: File-based configuration remains the default, ensuring existing setups continue to work

Use Cases

• Dynamic connector reconfiguration in production environments
• Centralized configuration management across multiple connector instances
• Integration with configuration management systems and GitOps workflows

Questions for Discussion

1. Should configuration updates trigger automatic connector restarts, or should this be configurable?
2. What validation and rollback mechanisms should be in place for configuration changes?
3. Should we support configuration versioning and audit logging?
4. Are there specific configuration API backends we should prioritize beyond the default file-based implementation?

Looking forward to hearing your thoughts!

[spetz]

Hey,
This is a good idea, since currently, the connectors store the configuration in the local files, making it unusable with distributed computing (scaling out the instances). IMHO, to begin with, we would need a simple trait allowing us to upsert (save), get, delete, and list versions (since each connector has an independent config, the easy way to detect the changes would be by storing and comparing config versions).

And actually, the get and delete functions could take an optional version parameter (if not provided, fallback to the current one) so that it'd be easy to store/rollback to previous configurations if needed. Additional extensions like audit logs, etc., could be implemented in the future.

The connectors API (which are now read-only endpoints for sink and source plugins) should use the particular config storage trait to perform all the actions, and on top of this, there could be a basic background job with a configurable interval, polling for changes (e.g. by comparing the mentioned config versions) to automatically update the configs stored in memory.