[BUG] torchrl uses tensordict.utils.Buffer as torch.nn.Buffer, but it is an empty class on torch>=2.5 #4535

Describe the bug

torchrl imports Buffer from tensordict.utils in three modules:

• torchrl/collectors/utils.py, in _cast and _map_weight
• torchrl/weight_update/weight_sync_schemes.py, in _to_module_compatible_tensor
• torchrl/objectives/common.py, in _make_target_param

This code expects Buffer to be torch.nn.Buffer. It is not. On torch 2.5 and later, tensordict.utils.Buffer is an empty placeholder class. On torch older than 2.5, it is a copy of torch.nn.Buffer.

On torch 2.5 and later, this has these results:

1. isinstance(x, Buffer) is always False. The branches that keep a torch.nn.Buffer as a buffer never run, so the three helpers return a plain tensor.
2. Buffer(x) raises TypeError: Buffer() takes no arguments. No code path reaches these calls now, because of result 1.
3. _cast raises RuntimeError: Cannot cast tensor ... with gradients for a torch.nn.Buffer that requires grad. When Buffer is torch.nn.Buffer, _cast returns a buffer instead.

I did not find a user-facing failure. In WeightStrategy.apply_weights and in the policy device mapping in collectors/_base.py, the buffers stay registered. The reason is that TensorDict.to_module keeps an existing buffer as a buffer. But the buffer handling in these helpers does not run.

To Reproduce

Run this script with torch 2.5 or later:

import torch
from torch import nn
from tensordict.utils import Buffer
from torchrl.collectors.utils import _cast, _map_weight
from torchrl.weight_update.weight_sync_schemes import _to_module_compatible_tensor

# 1. tensordict.utils.Buffer is not torch.nn.Buffer.
print(Buffer is nn.Buffer)  # False
print(isinstance(nn.Buffer(torch.zeros(3)), Buffer))  # False

# 2. The torchrl helpers return a plain tensor for a torch.nn.Buffer.
buf = nn.Buffer(torch.zeros(3))
print(isinstance(_cast(buf.data, buf), nn.Buffer))  # False
print(isinstance(_map_weight(buf, torch.device("cpu")), nn.Buffer))  # False
print(isinstance(_to_module_compatible_tensor(torch.zeros(3), buf), nn.Buffer))  # False

# 3. Calling tensordict.utils.Buffer fails.
Buffer(torch.zeros(3))

Output:

False
False
False
False
False
Traceback (most recent call last):
  File "repro.py", line 18, in <module>
    Buffer(torch.zeros(3))
TypeError: Buffer() takes no arguments

Expected behavior

On torch 2.5 and later, torchrl uses torch.nn.Buffer, and the calls in step 2 print True. I tested this by setting Buffer = torch.nn.Buffer in the three torchrl modules.

System info

• torch 2.14.1+cpu, from the PyTorch CPU index with pip
• tensordict 0.14.2 and torchrl 0.14.0, from PyPI with pip
• Python 3.12.12 on Linux
• The same code is on torchrl main at 648f50c.

>>> import torchrl, numpy, sys
>>> print(torchrl.__version__, numpy.__version__, sys.version, sys.platform)
0.14.0 2.5.3 3.12.12 (main, Oct 28 2025, 12:10:49) [Clang 20.1.4 ] linux

Additional context

• [Versioning] Require torch 2.13 or later tensordict#1830 (open) makes tensordict require torch 2.13 or later. That PR keeps the placeholder, so that tensordict does not change the behavior of torchrl. After torchrl stops importing tensordict.utils.Buffer, tensordict can remove the placeholder.
• _make_target_param has a second problem. It calls x.data before isinstance(x, nn.Parameter), and x.data is never a Parameter. So Buffer(x) does not run there, even with torch.nn.Buffer.

Reason and Possible fixes

Import Buffer from torch when it exists. torchrl supports torch 2.1 and later, so keep the tensordict copy for torch older than 2.5:

try:
    from torch.nn import Buffer  # torch>=2.5
except ImportError:  # torch<2.5, where tensordict provides a copy
    from tensordict.utils import Buffer

This change makes the helpers keep torch.nn.Buffer inputs as buffers, so their output changes. The script above can become a regression test.

The code that uses Buffer:

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have read the documentation (required)
• I have provided a minimal working example to reproduce the bug (required)