Support for `IConsumerDefinition` ?

[wokket]

G'day again,

Another request/discussion rather than bug... We currently make heavy use of MassTransit's IConsumerDefinition type to keep all the per-consumer configuration (retry/redelivery logic, concurrent processing limits, whether to throw Fault, support skipped etc) nice and close to the consumer, rather than having to touch our mass transit installer helper every time we add a consumer (which also reduces merge conflicts)...

public sealed class ElidedConsumerDefinition : ConsumerDefinition<ElidedConsumer>
{
	protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator, IConsumerConfigurator<ElidedConsumer> consumerConfigurator, IRegistrationContext context)
	{
		endpointConfigurator.PrefetchCount = 100; // no point prefetching less than a full batch size or we'll just timeout.
		endpointConfigurator.ConcurrentMessageLimit = 100;

		consumerConfigurator.Options<BatchOptions>(options => options
			.SetMessageLimit(100)
			.SetTimeLimit(TimeSpan.FromSeconds(1))
			.SetConcurrencyLimit(1)); // don't add unnecessary database load by running concurrently
	}
}

public sealed class ElidedConsumer : IConsumer<MyMessageType> 
{ ... }

Is support for something like this something you'd consider, either as a separate type, or an optional static method on the IConsumer or something??

In a perfect world we'd be able to do all the topology/bus configuration for this from the one place given we don't do any custom endpoint naming etc in MT, it's strictly one endpoint per consumer, with queues/exchanges all created/named by MT.

If you're open to the idea I was thinking about prototyping something similar to:

public class ElidedConsumerDefinition : MTKebabCasedConsumerDefinition<ElidedConsumer>  // MT to indicate we want back-compat defaults
{
   protected override void ConfigureConsumer(Action<IQueueConfigurator> ??,  ...) {
      ...
  }

   protected override void ConfigureEndpoint(Action<IEndpointConfigurator> ??,  ...) {
      ...
  }

}

That would:

• declare the my-namespace-elided-consumer Exchange and Queue in the topology bound together (topology.DeclareExchange, topology.DeclareQueue and topology.BindExchangeToQueue)
• declare the MT compatible exchanges for MyMessageType
• create the receive endpoint for the consumer, and call e.Consumer<ElidedConsumer, MyMessageType>
  • As an aside, it feels awkward to have to pass the message type here when the IConsumer already declares the message types it supports, is there a perf-related reason for this?)
• apply the endpoint configuration re: concurrency, deadletters etc

Any thoughts? This is pretty rough and a. have no idea how plausible it is, and b. could no doubt be improved with smarter neurons.

[asawicki]

G'day, and thanks for another thoughtful one. Good discussion to have.

Short version: yes to the core of it, with a couple of deliberate divergences from the MassTransit shape.

Co-locating a consumer's config next to the consumer, instead of touching a central installer for every new one, is a real ergonomic win. It also lines up with where our per-consumer configuration is already heading; we've got a per-consumer configurator seam already (routing, envelope opt-in, and so on). A ConsumerDefinition<TConsumer> base type that groups the existing knobs (prefetch, concurrent message limit, retry/redelivery, dead-letter, routing) would be a mostly-additive extension moreover. We'd happily take a prototype focused on exactly this.

Topology is where we'd diverge. In a perfect world, you'd declare the exchange/queue/binding straight from the definition with automatic kebab-case naming, but we'd want that strictly opt-in rather than the default. Manual, explicit topology is a deliberate design choice for us (deterministic, no surprise entity creation on a shared broker), not an oversight, so the default stays "you declare it." We're very open to an opt-in helper on the definition that declares the standard exchange+queue+binding for a consumer in one call, as long as it's something you switch on rather than something that happens implicitly. One subtlety: we deliberately keep consumer routing keys as a client-side dispatch filter, not a broker binding, so an "endpoint+binding per consumer" helper would need to be explicit about which of the two it's doing.

A few things on your list we'd rather not pull in wholesale. Throwing Fault<T>, "support skipped", second-level redelivery, batch consumers / BatchOptions don't exist in BareWire yet, and each is a non-trivial subsystem rather than a flag we can flip. I'd sooner weigh those individually as their own features than bring them in as a side effect of "definition support." We borrow MassTransit's naming purposely, so the API feels familiar, but not its whole feature surface, keeping the core small and predictable is sort of the point of the library. Definition auto-discovery is the same story: consumers aren't auto-registered today, so a scan/registration mechanism is a bigger design decision we'd want to make on its own.

On the explicit message type, good question, and there's a reason. The consumer invoker gets built once at startup into a fully closed-generic delegate, so there's zero reflection on the hot path. The message type has to be known statically at that point for deserialization, the ConsumeContext<T>, and the IConsumer<T> cast, and it also disambiguates a consumer that implements several IConsumer<T>. So it's a zero-copy and disambiguation thing, not a per-message cost. For the common single-IConsumer<T> case we could easily add an overload that infers the type for you; that part really is just API sugar, happy to add it.

So: very open to a ConsumerDefinition<T> that groups the existing per-consumer/endpoint config (plus that inferred-message-type overload) and an opt-in topology helper, and we'll treat the fault/batch/redelivery/skipped bits as separate proposals. To be concrete about it: that first piece, the definition that just groups the config we already have, is additive enough that it doesn't need its own design round, so we're planning to add it in the near future either way. If you want to prototype, that's the lowest-friction slice and the one we can merge fastest, so it's a great place to start. Keen to see what you come up with.

[wokket]

I'm totally onboard with not trying to add the entire MT system in this PR, the features you don't have yet are obviously not going to be configurable yet 🤣

I also agree with your "no magic topology by default" standpoint, that's where I'd hope a base ConsumerDefinition would do no magic topology (but would expose the IEndpointConfigurator so the consumer definitioncould explicitly configure the endpoint). The subtypes that make it obvious they provide a base topology configuration (obviously named as MassTransitKebabCaseConfigurator or similar, and in the Interop.MassTransit namespace) would then save a heap of repeat work for people like us that almost entirely have a very simple 1:1 endpoint:consumer configuration with default naming.