md: don't wait for q->limits_lock while md holds back I/O by blktests-ci-kpd[bot] · Pull Request #1270 · linux-blktests/linux-block

blktests-ci-kpd · 2026-09-08T15:46:35Z

Pull request for series with
subject: md: don't wait for q->limits_lock while md holds back I/O
version: 1
url: https://patchwork.kernel.org/series/1159795/

blktests-ci-kpd · 2026-09-08T15:46:36Z

Upstream branch: 28924df
series: https://patchwork.kernel.org/series/1159795/
version: 1

blktests-ci-kpd · 2026-09-09T08:07:11Z

Upstream branch: 893e117
series: https://patchwork.kernel.org/series/1159795/
version: 1

blktests-ci-kpd · 2026-09-10T08:39:39Z

Upstream branch: 893e117
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-10T15:30:41Z

Upstream branch: 50d05c7
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-12T02:27:39Z

Upstream branch: 5225b8e
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-13T08:56:06Z

Upstream branch: 2f0c1cf
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-13T08:58:35Z

Upstream branch: 2f0c1cf
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-13T09:10:01Z

Upstream branch: 2f0c1cf
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-13T22:57:10Z

Upstream branch: 2f0c1cf
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-09-15T09:30:50Z

Upstream branch: 5878583
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-01T23:55:48Z

Upstream branch: ce1e022
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-03T10:18:45Z

Upstream branch: e767a4e
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-03T22:27:34Z

Upstream branch: a74306e
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-05T02:34:53Z

Upstream branch: None
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-06T13:19:32Z

Upstream branch: 22430ae
series: https://patchwork.kernel.org/series/1161858/
version: 2

Adding a leg stacks its queue limits, which mddev_stack_new_rdev() does by taking q->limits_lock itself. Callers holding reconfig_mutex or a suspended array cannot allow that, and must own the update instead. Give ->hot_add_disk(), remove_and_add_spares() and md_choose_sync_action() a struct queue_limits argument with three states: an update to stack into, NULL to let the personality take the lock as before, or MDDEV_STACK_SKIP to add the leg without touching the limits, for callers that can do neither. mddev_stack_rdev_into() stacks into a caller-owned update without the lock. Every caller still passes NULL and nothing passes the sentinel yet, so there is no functional change; the users follow. Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

check_sb_changes() activates a spare another node added, reached from md_reload_sb() -> process_metadata_update() with reconfig_mutex held. Stacking the device's limits there waits for q->limits_lock under that mutex, which deadlocks: the lock's holder waits for the queue to drain, and that I/O can be waiting for a superblock update needing reconfig_mutex. The device is already a member, so its limits are stacked. Add it with MDDEV_STACK_SKIP and leave them alone. Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

state_store() and slot_store() can add a leg back to the array, which stacks its limits, and q->limits_lock has to be taken before the array is locked and suspended. Give the rdev sysfs store callback a struct queue_limits argument. rdev_attr_store() passes NULL, so there is no functional change; the user follows. Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

mddev_update_io_opt() runs from end_reshape() in the sync thread, and md_reap_sync_thread() waits for that thread with reconfig_mutex held. Taking q->limits_lock there hangs a finishing reshape whenever the lock's holder waits for I/O that only md_check_recovery() can let complete, and no ordering avoids it: the sync thread is what lets that I/O finish. Hand the update to a work item, which holds neither reconfig_mutex nor the suspend and so takes q->limits_lock in the order the rest of md uses, before suspending. __md_stop() flushes it, as it suspends the array. Also give the function a queue_limits argument, so a caller that already owns an update has it changed in place; the users of that path follow. Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

Writing a queue limits attribute while a spare is re-added deadlocks the array: udev-worker queue_attr_store() holds q->limits_lock, waits in blk_mq_freeze_queue() for q_usage_counter to drain fio holds a q_usage_counter reference, parked in md_handle_request()'s is_suspended() loop mdadm suspended the array, waits for reconfig_mutex md_start_sync holds reconfig_mutex, waits for q->limits_lock Blocking on q->limits_lock while holding reconfig_mutex, or with the array suspended, is waiting for normal I/O, which mddev_suspend() already warns about with lockdep_assert_not_held(). So q->limits_lock has to nest outside both. Take the update before the array is locked and suspended, and pass it down so the personality stacks into it: - md_start_sync(), at both suspend points - md_ioctl() for ADD_NEW_DISK and HOT_REMOVE_DISK - rdev_attr_store(), for slot and for state "remove"/"re-add" - raid5 skip_copy_store(), which took the lock while suspended They are converted together because a mix of the two orders is an ABBA. All of them commit while the array is still quiesced. Two callers still take the lock inside reconfig_mutex with the array suspended: ->start_reshape() from action_store(), which suspends before flushing sync_work so the update cannot be held across it, and raid*_run() from level_store(), which a later patch converts. Verified with a raid1 of two ram devices, fio in flight and a loop writing queue/max_sectors_kb: 20 fail/remove/add cycles complete, where the same test wedges the array before the change. Fixes: c99f66e ("block: fix queue freeze vs limits lock order in sysfs store methods") Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

raid*_run() -> queue_limits_set() takes q->limits_lock with reconfig_mutex held, the order the previous patches inverted. With lockdep on, creating an array and then adding a leg reports it: -> #1 (&q->limits_lock): -> #0 (&mddev->reconfig_mutex): queue_limits_set md_ioctl <- ADD_NEW_DISK raid1_run do_md_run md_ioctl <- RUN_ARRAY The earlier patch left this for level_store() alone; RUN_ARRAY reaches it too, so every array creation records the wrong order. Give ->run() a queue_limits argument and take the update at the entry points that start an array: md_ioctl() for RUN_ARRAY, level_store(), autorun_devices(), md_setup_drive(), and array_state_store() for readonly, read_auto and active -- but only while mddev->pers is NULL, as with the array running those states go to md_set_readonly(), which waits in stop_sync_thread() for the work that takes the same lock. dm-raid passes NULL: with no gendisk the personalities return before touching any limits. Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

Opening a leg takes disk->open_mutex, and the scsi disk probe path nests q->limits_lock inside it (sd_open() -> sd_revalidate_disk()). md opens legs under reconfig_mutex, which this series makes q->limits_lock nest outside, closing a cycle. Booting with lockdep on an md root reports it during assembly: -> #2 (&q->limits_lock): sd_revalidate_disk / sd_open -> #1 (&disk->open_mutex): md_import_device md_add_new_disk md_ioctl <- ADD_NEW_DISK -> #0 (&mddev->reconfig_mutex): md_ioctl <- RUN_ARRAY Move every open out from under the lock. md_import_new_disk() mirrors md_add_new_disk()'s branch selection so all three of its branches take a pre-opened leg, and hot_add_disk(), new_dev_store() and md_setup_drive() open before they lock as well. The mddev fields the open depends on are read without reconfig_mutex, so each caller rechecks them once the array is locked and rejects the add with -EBUSY if the branch or the superblock format would have changed. md_autostart_arrays() needs no change: it opens under detected_devices_mutex. Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

bd_link_disk_holder() takes the leg's disk->open_mutex, and bind_rdev_to_array() calls it with reconfig_mutex held, so the dependency the previous patch removed from md_import_device() is still there by another route: -> #2 (&q->limits_lock): sd_revalidate_disk / sd_open -> #1 (&disk->open_mutex): bd_link_disk_holder bind_rdev_to_array md_add_new_disk md_ioctl <- ADD_NEW_DISK -> #0 (&mddev->reconfig_mutex): md_ioctl <- RUN_ARRAY bd_unlink_disk_holder() only takes blk_holder_mutex, which is why the release side needs no change and made the link side easy to miss. Link the holder where the leg is opened, before the array is locked, and record it in a new HolderLinked flag so the release side knows whether there is a link to drop. A failed link is not fatal, as before. A leg that is linked but not yet bound is released through md_export_rdev(), which drops the link first. A leg is now linked before it is known to be acceptable, so a leg the array goes on to reject shows up in its slaves directory until the error path releases it. md then no longer takes disk->open_mutex under reconfig_mutex: of the functions that take it, md reaches bdev_open() and bd_link_disk_holder() from the paths above, bdev_release() and bdev_fput() only through fput(), which defers to task work, del_gendisk() only from mddev teardown, and never sync_bdevs(). Assisted-by: LLM Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>

blktests-ci-kpd · 2026-10-06T17:32:51Z

Upstream branch: 69f80fe
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-07T14:13:34Z

Upstream branch: 69f80fe
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-07T14:38:08Z

Upstream branch: 69f80fe
series: https://patchwork.kernel.org/series/1161858/
version: 2

blktests-ci-kpd · 2026-10-07T14:41:02Z

Github failed to update this PR after force push. Close it.

blktests-ci-kpd Bot added new V1 linus-master labels Sep 8, 2026

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 4ddd216 to 00cc4ca Compare September 9, 2026 07:40

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from fbe586b to 496c3db Compare September 9, 2026 08:07

blktests-ci-kpd Bot added V2 and removed V1 labels Sep 10, 2026

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from 496c3db to ea8d4ab Compare September 10, 2026 08:39

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 00cc4ca to 7efd8cd Compare September 10, 2026 15:29

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from ea8d4ab to ed8a026 Compare September 10, 2026 15:30

blktests-ci-kpd Bot added the V2-ci-pass label Sep 11, 2026

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 7efd8cd to a0aeca9 Compare September 12, 2026 02:26

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from ed8a026 to c77a8c5 Compare September 12, 2026 02:27

blktests-ci-kpd Bot force-pushed the linus-master_base branch from a0aeca9 to 772381e Compare September 13, 2026 08:50

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from c77a8c5 to 572ea3c Compare September 13, 2026 09:10

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 772381e to 0224dee Compare September 13, 2026 22:56

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from 572ea3c to 01edfda Compare September 13, 2026 22:57

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 0224dee to f14340f Compare September 15, 2026 09:28

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from 01edfda to b4f03e4 Compare September 15, 2026 09:30

blktests-ci-kpd Bot force-pushed the linus-master_base branch from f14340f to 0e174bc Compare September 17, 2026 19:27

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from bfd3ea7 to aad14bb Compare October 1, 2026 23:55

blktests-ci-kpd Bot force-pushed the linus-master_base branch from bbd3af0 to a9c0b46 Compare October 3, 2026 09:37

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from aad14bb to 0b54051 Compare October 3, 2026 10:18

blktests-ci-kpd Bot force-pushed the linus-master_base branch from a9c0b46 to 72d0f5e Compare October 3, 2026 21:50

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from 0b54051 to e9e4927 Compare October 3, 2026 22:27

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 72d0f5e to d980ad5 Compare October 5, 2026 02:03

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from e9e4927 to b4ee610 Compare October 5, 2026 02:34

blktests-ci-kpd Bot force-pushed the linus-master_base branch from d980ad5 to 83b99cc Compare October 6, 2026 12:35

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from b4ee610 to 6d0eaab Compare October 6, 2026 13:19

blktests-ci-kpd Bot force-pushed the linus-master_base branch from 83b99cc to 9081535 Compare October 6, 2026 16:58

jinpuwang added 8 commits October 6, 2026 17:32

blktests-ci-kpd Bot force-pushed the series/1159795=>linus-master branch from 6d0eaab to b36c904 Compare October 6, 2026 17:32

blktests-ci-kpd Bot closed this Oct 7, 2026

blktests-ci-kpd Bot deleted the series/1159795=>linus-master branch October 9, 2026 14:44

Conversation

blktests-ci-kpd Bot commented Sep 8, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 8, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 9, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 10, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 10, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 12, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 13, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 13, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 13, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 13, 2026

Uh oh!

blktests-ci-kpd Bot commented Sep 15, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 1, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 3, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 3, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 5, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 6, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 6, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 7, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 7, 2026

Uh oh!

blktests-ci-kpd Bot commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant