Skip to content

Fix scale slicing for negative indices on blocked dims - #4958

Open
Kaif10 wants to merge 1 commit into
pytorch:mainfrom
Kaif10:fix/slice-scale-negative-indices
Open

Kaif10 wants to merge 1 commit into
pytorch:mainfrom
Kaif10:fix/slice-scale-negative-indices

Conversation

@Kaif10

@Kaif10 Kaif10 commented Sep 28, 2026

Copy link
Copy Markdown

_slice_scale_for_dimension maps a data-tensor slice onto the scale with floor/ceil division by the block size, but never resolves negative indices first. A negative end with |end| < block_size ceil-divides to 0, so the scale is sliced to length zero.

For per-row quantization the block spans the whole row, so any negative end on the last dim triggers it:

m = torch.nn.Sequential(torch.nn.Linear(128, 64, bias=False, dtype=torch.bfloat16))
quantize_(m, Int8WeightOnlyConfig(version=2, granularity=PerRow()))
w = m[0].weight          # scale (64, 1)

w[:, :-32].scale.shape   # torch.Size([64, 0])   <- no error at slice time
w[:, :96].scale.shape    # torch.Size([64, 1])
w[:, :-32].dequantize()  # RuntimeError: shape '[64, 1]' is invalid for input of size 0

Same with Float8WeightOnlyConfig() / Float8DynamicActivationFloat8WeightConfig(granularity=PerRow()), which fails later with ZeroDivisionError. The slice itself succeeds and returns a tensor with a plausible shape, so the error surfaces far from its cause.

Negative start happens to give the right block because floor division lands on the same block when the dim is block-aligned, and dim=0 is unaffected because a block size of 1 delegates to aten.slice, which handles negatives natively. Only the ceil-divided end breaks.

Fix

Resolve negative start/end against data_shape[dim] (clamped at 0, same as Python slice semantics) before the block arithmetic. Since Float8Tensor, Int8Tensor (scale and zero_point) and the prototype static float8 tensor all call this helper, they are all fixed.

Tests

  • test_int8_tensor_cpu.py::test_slice_negative_indices: CPU. PerRow weight-only, PerGroup(32) weight-only and PerRow dynamic, over (None, -32), (-32, None), (-96, -32), (-1000, None) on both dims. Each negative slice is compared against the equivalent non-negative slice on qdata, scale and dequantize(). The PerRow + negative-end cases fail on main and pass with this change; the rest pass on both and guard the unaffected paths.
  • test_float8_tensor.py::test_slice_negative_indices: the same check for Float8Tensor with PerRow, next to the existing test_slice (GPU-gated like the rest of that class). I validated its body on CPU by substituting Float8WeightOnlyConfig: it fails on main with a (64, 0) scale and passes with this change.

ruff check and ruff format are clean. test_int8_tensor_cpu.py gains 12 passing tests. Its existing compile=True cases fail identically before and after this change on my machine (no C++ compiler for Inductor on Windows), so they're unrelated.

Suggested labels: module: inference, topic: bug fix.

_slice_scale_for_dimension maps a data slice onto the scale with
floor/ceil division by the block size, but never resolved negative
indices first. A negative `end` with |end| < block size ceil-divides to
0, so the scale is sliced to length zero. For per-row quantization the
block spans the whole row, so any negative `end` on the last dim hits
this: w[:, :-32] returns a Float8Tensor/Int8Tensor with a (N, 0) scale
and no error, and it only fails later in dequantize (shape error for
Int8, ZeroDivisionError for Float8).

Resolve negative start/end against the data size before the block
arithmetic. Covers Float8Tensor, Int8Tensor (scale and zero_point) and
the prototype static float8 tensor, which all share this helper.
@pytorch-bot

pytorch-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/4958

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 28, 2026
@Kaif10

Kaif10 commented Oct 1, 2026

Copy link
Copy Markdown
Author

Could a maintainer approve the CI workflows for this first-time-contributor PR? Suggested label: module: inference.

@Kaif10

Kaif10 commented Oct 4, 2026

Copy link
Copy Markdown
Author

@andrewor14 friendly ping. This is a 4-line fix to _slice_scale_for_dimension for negative indices, with tests that fail on main. CI is still waiting on workflow approval.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant