Skip to content

Rescan of volumes required on release-ibm-aix72-ppc64_be-1 #4425

Closed
Closed
Rescan of volumes required on release-ibm-aix72-ppc64_be-1#4425
Assignees
sxa
Labels
platform:aix

Description

@sxa
sxa
opened on Aug 12, 2026

Due to decommissioning of a storage controller in the datacenter where release-ibm-aix72-ppc64_be-1 is hosted we need to do a rescan before the 24th August to ensure we don't lose connectivity to the volume. It appears that is the only machine that will be affected.

Details at https://cloud.ibm.com/notifications?selected=1786979410275

Host Volume ID cron enabled bootlist updated
release-ibm-aix72-ppc64_be-1 217ce526-806d-49d4-8656-cfa05d9a5ca2 ✅ ✅
release-ibm-aix72-ppc64_be-1 228a0e33-18ff-432f-8802-6d3273f5e9d2
test-ibm-aix72-ppc64_b3-5 bef233fe-cd7c-4c7c-9d5b-32d570ae248a ✅(169.54.115.58) Not needed
test-ibm-aix72-ppc64_b3-5 cf22b10b-f123-4019-a0f3-78643be628c2
test-ibm-aix72-ppc64_be-1 03603e09-9699-4d00-8eca-285033776fa1 ✅ ✅
test-ibm-aix72-ppc64_be-1 1489cd1d-4e57-4be6-ad81-1206f5f9c50b
test-ibm-aix72-ppc64_be-2 401d8e52-5728-4eda-aaf6-09875d6d204c ✅ ✅
test-ibm-aix72-ppc64_be-2 77d0fa64-96bd-4eed-ba5c-66e1f871ba1b
test-ibm-aix72-ppc64_be-4 b59192da-182c-4763-b86a-57c548add27e ✅ (*) ✅
test-ibm-aix72-ppc64_be-4 f27417ff-b5d8-46c3-b05a-e1c81feed237
test-ibm-aix73-ppc64_be-2 1b50edab-6122-48c4-99d8-0d16ae401558 ✅ (disk full) ✅
test-ibm-aix73-ppc64_be-2 1d4a9fbe-15fe-4aa2-8934-95ca2323ea88
test-ibm-aix73-ppc64_be-5 86dfa90f-7057-4bcd-baa3-ccc965aceefb ✅ Not needed

(*) - 169.54.113.166 - has its hostname set to-3.

Process:

  • Add new-device-scan.sh to /usr/bin
  • Create crontab entry for root with 0,15,30,45 * * * * /usr/bin/new-device-scan.sh >> /var/log/rescan.log 2>&1

Activity

  1. richardlau
    added
    ci-release
    platform:aix
    and removed
    ci-release
    on Aug 12, 2026
  2. sxa
    self-assigned this
    on Aug 20, 2026

  3. sxa commented on Aug 20, 2026

    @sxa
    sxa
    on Aug 20, 2026
    MemberAuthor

    I believe this issue should remain open until the maintenance is complete on August 24th at which time old paths can be removed with lspath and rmpath commands and the bootlist updated.

  4. sxa commented on Aug 28, 2026

    @sxa
    sxa
    on Aug 28, 2026
    MemberAuthor

    Noting for reference (since I've done this once in the past) that if you do manage to restart a machine without updating the bootlist, and there isn't a valid path then you can still boot manually via the IBM Cloud console.

    Details of using the boot console to boot manually

    If the machine won't boot you'll get a loop telling you press CTL-C to interrupt the boot. Then you'll get a screen like this:

     Version FW950.H3 (VL950_230)
 SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Main Menu
 1.   Select Language
 2.   Setup Remote IPL (Initial Program Load)
 3.   I/O Device Information
 4.   Select Console
 5.   Select Boot Options

    Select option 5

     Version FW950.H3 (VL950_230)
 SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Multiboot
 1.   Select Install/Boot Device
 2.   Configure Boot Device Order
 3.   Multiboot Startup <OFF>
 4.   SAN Zoning Support

    Select option 1

    Version FW950.H3 (VL950_230)
 SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Select Device Type
 1.   Tape
 2.   CD/DVD
 3.   Hard Drive
 4.   Network
 5.   List all Devices

    You could select option 5 here, but for the purposes of demonstration I'll select 3

    Version FW950.H3 (VL950_230)
 SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Select Media Type
 1.   SCSI
 2.   SAN 
 3.   SAS 
 4.   SATA
 5.   USB 
 6.   NVMe
 7.   List All Devices

    Again we could select option 7, but we know this is SAN attached so I'll select option 2

     Version FW950.H3 (VL950_230)
 SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Select Fabric Type
 1.   FCP    
 2.   FC NVMe over Fabric
 3.   List All Devices

    At this point I will select option 3. The console will show a PLEASE WAIT message while various check and scan lines are listed. It could take some minutes to complete. Once done it will let you select one. In most cases it is unlikely to make a difference which one you select as they are all paths to the same disk. If you know better then feel free to select something specific:

     Version FW950.H3 (VL950_230)
 SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Select Device
 Device  Current  Device
 Number  Position  Name
 1.        -      26 GB   FC Harddisk, part=2 (AIX 7.2.0)
        ( loc=U9009.22A.7891070-V13-C4-T1-W500507681375432d-L0001000000000000 )
 2.        -      26 GB   FC Harddisk, part=2 (AIX 7.2.0)
        ( loc=U9009.22A.7891070-V13-C4-T1-W500507681375432e-L0001000000000000 )
 3.        -      26 GB   FC Harddisk, part=2 (AIX 7.2.0)
        ( loc=U9009.22A.7891070-V13-C6-T1-W500507681378432d-L0001000000000000 )
 4.        -      26 GB   FC Harddisk, part=2 (AIX 7.2.0)
        ( loc=U9009.22A.7891070-V13-C6-T1-W500507681378432e-L0001000000000000 )
 5.        -      26 GB   FC Harddisk, part=2 (AIX 7.2.0)
        ( loc=U9009.22A.7891070-V13-C7-T1-W500507681377432d-L0001000000000000 )
 6.        -      26 GB   FC Harddisk, part=2 (AIX 7.2.0)
        ( loc=U9009.22A.7891070-V13-C7-T1-W500507681377432e-L0001000000000000 )
 -------------------------------------------------------------------------------
 Navigation keys:
 M = return to Main Menu      N = Next page of list
 ESC key = return to previous screen         X = eXit System Management Services
 -------------------------------------------------------------------------------
 Type menu item number and press Enter or select Navigation key:

    I selected 1 and it asks how to boot:

     SMS (c) Copyright IBM Corp. 2000,2026 All rights reserved.
-------------------------------------------------------------------------------
 Select Task

26 GB   FC Harddisk, part=2 (AIX 7.2.0)
    ( loc=U9009.22A.7891070-V13-C4-T1-W500507681375432d-L0001000000000000 )

 1.   Information
 2.   Normal Mode Boot
 3.   Service Mode Boot

    If you select to view information from the next menu information you'll see the disk type - fcp in my case so I could have selected that explicitly from the earlier Select fabric type screen:

     Device Information
                     : /vdevice/vfc-client@30000004/disk@500507681375432d,0001000000000000
                     : (Bootable)       
DEVICE               : 26 GB   FC Harddisk, part=2 (AIX 7.2.0)
                        ( loc=U9009.22A.7891070-V13-C4-T1-W500507681375432d-L0001000000000000 )
NAME                 : disk
DEVICE-TYPE          : block

 Parent Information 
IBM,FW-ADAPTER-NAME  : Virtual Fibre Channel
NAME                 : vfc-client
DEVICE-TYPE          : fcp

    We now want to boot so select option 2. it will confirm that you want to exist SMS, so select the Yes option. You'll get an IBM splash screen as follows and then the system will boot:

    IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM                         IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM    STARTING SOFTWARE    IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM       PLEASE WAIT...    IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM                         IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
Elapsed time since release of system processors: 2928025 mins 29 secs
[...]
-------------------------------------------------------------------------------
                                Welcome to AIX.
                   boot image timestamp: 01:58:55 11/19/2021
                 The current time and date: 12:38:10 08/28/2026
       processor count: 4;  memory size: 32768MB;  kernel size: 51877410
boot device: /vdevice/vfc-client@30000004/disk@500507681375432d,0001000000000000
-------------------------------------------------------------------------------
[...]
AIX Version 7
Copyright IBM Corporation, 1982, 2021.
Console login:

    Course of action to rescan disks and update the bootlist:

    1. Run bootlist -m normal -o and note the output. This will be important shortly - specifically the hdiskX value (it may not be hdisk0 which is more normal and instructions from IBM may just tell you to use hdisk0 which won't work)

    2. Run lspath -F "name path_id parent status", then for anything that says Failed or Missing run rmpath -i <x> -d (where <x> is the second field (path_id) of the lspath output. If you're feeling brave you could use: lspath -F "name path_id parent status" | awk '/Failed|Missing/{print$2}' | sort | uniq | xargs -n 1 rmpath -d -i

    3. If bootlist -m normal -o now shows nothing but dashes then your machine definitely won't boot at this stage. You should clean things up regardless using the hdiskX value from the bootlist output from step 1: Run bootlist -m normal hdisk1 and bootlist -m service hdisk1 (Replace with hdisk0 or something else if needed but most of ours seem to boot from hdisk1).

  5. sxa commented on Aug 28, 2026

    @sxa
    sxa
    on Aug 28, 2026
    MemberAuthor

    All required rescans done - closing

  6. sxa
    closed this as completedon Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

platform:aix

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions