online resize of lvm filesystem on fibre channel or iscsi or similar when using multipath

By thomas, 27 May, 2010

I had a problem with a filesystem that was full on a vm. The vm's hard drive is just a lun on a fibre channel. So I resized the lun on the raid controller. The devices (/dev/sdx) for the drive noticed the new size after I did the usual scan and partprobe, but the multipath device didn't see the new size and was working off the old size. After some digging I found this page on how to get multipathd to notice the new size. I'll summarise here.

Assuming our hard drive in question is called vm1, /dev/mapper/vm1

[root@server0 ~]# multipath -ll vm1 vm1 (1ACNCorp_FF01000033100008) dm-15 DUMMY,R_dummy_root [size=56G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 3:0:0:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 3:0:1:0 sdu 65:64 [active][ready]

This shows that the drives in question are sdc and sdu. When we start, we have 70GB LUNs for the vm, we'll resize to 80GB. After the resize, we don't see a change on our hypervisor, we have to use blockdev to update the partition table in memory.

[root@server0 ~]# fdisk -l /dev/sdc Disk /dev/sdc: 70.0 GB, 70002409472 bytes 255 heads, 63 sectors/track, 8510 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 19 152586 83 Linux /dev/sdc2 20 8510 68203957+ 8e Linux LVM [root@server0 ~]# blockdev --rereadpt /dev/sdc [root@server0 ~]# fdisk -l /dev/sdc Disk /dev/sdc: 80.0 GB, 80003851264 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 19 152586 83 Linux /dev/sdc2 20 8510 68203957+ 8e Linux LVM [root@server0 ~]#

Repeat this for the other member of the group then notice that the multipath doesn't have the new size yet.

[root@server0 ~]# fdisk -l /dev/sdu Disk /dev/sdu: 80.0 GB, 80003851264 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdu1 * 1 19 152586 83 Linux /dev/sdu2 20 8510 68203957+ 8e Linux LVM [root@server0 ~]# fdisk -l /dev/mapper/vm [root@server0 ~]# fdisk -l /dev/mapper/vm1 Disk /dev/mapper/vm1: 70.0 GB, 70002409472 bytes 255 heads, 63 sectors/track, 8510 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/mapper/vm1p1 * 1 19 152586 83 Linux /dev/mapper/vm1p2 20 7294 58436437+ 8e Linux LVM

Now remove the drives from the multipath and add them, then resize

[root@server0 ~]# multipathd -k multipathd> del path sdc ok multipathd> add path sdc ok multipathd> del path sdu ok multipathd> add path sdu ok multipathd> resize map vm1 ok multipathd> [root@server0 ~]# fdisk -l /dev/mapper/vm1 Disk /dev/mapper/vm1: 80.0 GB, 80003851264 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/mapper/vm1p1 * 1 19 152586 83 Linux /dev/mapper/vm1p2 20 7294 58436437+ 8e Linux LVM

Next, resize the actual lvm. I got most of this from this page.

reboot the vm
use fdisk -u, delete the partition, make a new partition that starts on the same sector but extends to the end of the disk
partprobe to reread the partition table, or reboot
pvresize /dev/hda2
pvdisplay to see how much is now free
lvextend -l +[number free] /dev/vg/lv
lvdisplay (see new size)
resize2fs /dev/vg/lv

[root@vm1 ~]# fdisk -l /dev/hda Disk /dev/hda: 70.0 GB, 70002409472 bytes 255 heads, 63 sectors/track, 8510 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 19 152586 83 Linux /dev/hda2 20 8510 68203957+ 8e Linux LVM [root@vm1 ~]# blockdev --rereadpt /dev/hda BLKRRPART: Device or resource busy [root@vm1 ~]# poweroff Broadcast message from root (pts/0) (Thu May 27 16:43:09 2010): The system is going down for system halt NOW!

I believe a reboot is enough to make the vm reread the hard drive file...but I went for poweroff anyway

[root@vm1 ~]# fdisk -l /dev/hda Disk /dev/hda: 80.0 GB, 80003851264 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 19 152586 83 Linux /dev/hda2 20 8510 68203957+ 8e Linux LVM [root@vm1 ~]# fdisk -u /dev/hda The number of cylinders for this disk is set to 9726. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/hda: 80.0 GB, 80003851264 bytes 255 heads, 63 sectors/track, 9726 cylinders, total 156257522 sectors Units = sectors of 1 * 512 = 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 63 305234 152586 83 Linux /dev/hda2 305235 136713149 68203957+ 8e Linux LVM Command (m for help): d Partition number (1-4): 2 Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First sector (305235-156257521, default 305235): Using default value 305235 Last sector or +size or +sizeM or +sizeK (305235-156257521, default 156257521): Using default value 156257521 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot. Syncing disks. [root@vm1 ~]# reboot Broadcast message from root (pts/0) (Thu May 27 16:56:29 2010): The system is going down for reboot NOW!

Now the straightforward lvm stuff

[root@vm1 ~]# pvdisplay /dev/hda2 --- Physical volume --- PV Name /dev/hda2 VG Name Example PV Size 65.04 GB / not usable 13.24 MB Allocatable yes (but full) PE Size (KByte) 32768 Total PE 2081 Free PE 0 Allocated PE 2081 PV UUID jUn8V8-Ca29-HlTK-8Rnn-yq1k-2jae-aCfGnH [root@vm1 ~]# pvresize /dev/hda2 Physical volume "/dev/hda2" changed 1 physical volume(s) resized / 0 physical volume(s) not resized [root@vm1 ~]# pvdisplay /dev/hda2 --- Physical volume --- PV Name /dev/hda2 VG Name Example PV Size 74.36 GB / not usable 20.39 MB Allocatable yes PE Size (KByte) 32768 Total PE 2379 Free PE 298 Allocated PE 2081 PV UUID jUn8V8-Ca29-HlTK-8Rnn-yq1k-2jae-aCfGnH

Free PE is 298, so I can add that to my logical volume.

[root@vm1 ~]# lvdisplay /dev/Example/RootVol --- Logical volume --- LV Name /dev/Example/RootVol VG Name Example LV UUID eawcxY-KVZG-rcdO-Bf5E-Vck3-JOnJ-8KN2lm LV Write Access read/write LV Status available # open 1 LV Size 57.03 GB Current LE 1825 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 [root@vm1 ~]# lvextend -l +298 /dev/Example/RootVol Extending logical volume RootVol to 66.34 GB Logical volume RootVol successfully resized [root@vm1 ~]# lvdisplay /dev/Example/RootVol --- Logical volume --- LV Name /dev/Example/RootVol VG Name Example LV UUID eawcxY-KVZG-rcdO-Bf5E-Vck3-JOnJ-8KN2lm LV Write Access read/write LV Status available # open 1 LV Size 66.34 GB Current LE 2123 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 [root@vm1 ~]#

The last step is to resize the ext4 on the logical volume

[root@vm1 ~]# resize2fs /dev/Example/RootVol resize2fs 1.39 (29-May-2006) Filesystem at /dev/Example/RootVol is mounted on /; on-line resizing required Performing an on-line resize of /dev/Example/RootVol to 17391616 (4k) blocks. The filesystem on /dev/Example/RootVol is now 17391616 blocks long. [root@vm1 ~]# df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/Example-RootVol 65G 1.6G 60G 3% /

Done.