Feature Request: Dynamic Inference Context Support for T5 #3016

Summary

The current inference implementation for T5 models only supports static batching via StaticInferenceEngine. It would be valuable to extend DynamicInferenceContext to support encoder-decoder architectures like T5 similar to how it currently works for GPT inference via DynamicInferenceEngine.

Current Behavior

The T5 inference wrapper does not support KV caching or dynamic batching

Desired Behavior

Enable KV caching (encoder, decoder self-attention, and cross-attention) with in-flight dynamic batching for T5.

Related Code References

• Dynamic context implementation: megatron/core/inference/contexts/dynamic_context.py#L193
• Dynamic inference engine:
  megatron/core/inference/engines/dynamic_engine.py