online resize of lvm filesystem on fibre channel or iscsi or similar when using multipath

I had a problem with a filesystem that was full on a vm. The vm's hard drive is just a lun on a fibre channel. So I resized the lun on the raid controller. The devices (/dev/sdx) for the drive noticed the new size after I did the usual scan and partprobe, but the multipath device didn't see the new size and was working off the old size. After some digging I found this page on how to get multipathd to notice the new size. I'll summarise here.

Assuming our hard drive in question is called vm1, /dev/mapper/vm1

[root@server0 ~]# multipath -ll vm1
vm1 (1ACNCorp_FF01000033100008) dm-15 DUMMY,R_dummy_root
[size=56G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 3:0:0:0  sdc  8:32   [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:1:0  sdu  65:64  [active][ready]
This shows that the drives in question are sdc and sdu. When we start, we have 70GB LUNs for the vm, we'll resize to 80GB. After the resize, we don't see a change on our hypervisor, we have to use blockdev to update the partition table in memory.
[root@server0 ~]# fdisk -l /dev/sdc
 
Disk /dev/sdc: 70.0 GB, 70002409472 bytes
255 heads, 63 sectors/track, 8510 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *           1          19      152586   83  Linux
/dev/sdc2              20        8510    68203957+  8e  Linux LVM
[root@server0 ~]# blockdev --rereadpt /dev/sdc
[root@server0 ~]# fdisk -l /dev/sdc
 
Disk /dev/sdc: 80.0 GB, 80003851264 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *           1          19      152586   83  Linux
/dev/sdc2              20        8510    68203957+  8e  Linux LVM
[root@server0 ~]# 
Repeat this for the other member of the group then notice that the multipath doesn't have the new size yet.
[root@server0 ~]# fdisk -l /dev/sdu
 
Disk /dev/sdu: 80.0 GB, 80003851264 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdu1   *           1          19      152586   83  Linux
/dev/sdu2              20        8510    68203957+  8e  Linux LVM
[root@server0 ~]# fdisk -l /dev/mapper/vm
[root@server0 ~]# fdisk -l /dev/mapper/vm1
 
Disk /dev/mapper/vm1: 70.0 GB, 70002409472 bytes
255 heads, 63 sectors/track, 8510 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
           Device Boot      Start         End      Blocks   Id  System
/dev/mapper/vm1p1   *           1          19      152586   83  Linux
/dev/mapper/vm1p2              20        7294    58436437+  8e  Linux LVM
Now remove the drives from the multipath and add them, then resize
[root@server0 ~]# multipathd -k
multipathd> del path sdc
ok
multipathd> add path sdc
ok
multipathd> del path sdu
ok
multipathd> add path sdu
ok
multipathd> resize map vm1
ok
multipathd> 
[root@server0 ~]# fdisk -l /dev/mapper/vm1
 
Disk /dev/mapper/vm1: 80.0 GB, 80003851264 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
           Device Boot      Start         End      Blocks   Id  System
/dev/mapper/vm1p1   *           1          19      152586   83  Linux
/dev/mapper/vm1p2              20        7294    58436437+  8e  Linux LVM

Next, resize the actual lvm. I got most of this from this page.

  • reboot the vm
  • use fdisk -u, delete the partition, make a new partition that starts on the same sector but extends to the end of the disk
  • partprobe to reread the partition table, or reboot
  • pvresize /dev/hda2
  • pvdisplay to see how much is now free
  • lvextend -l +[number free] /dev/vg/lv
  • lvdisplay (see new size)
  • resize2fs /dev/vg/lv
[root@vm1 ~]# fdisk -l /dev/hda
 
Disk /dev/hda: 70.0 GB, 70002409472 bytes
255 heads, 63 sectors/track, 8510 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          19      152586   83  Linux
/dev/hda2              20        8510    68203957+  8e  Linux LVM
[root@vm1 ~]# blockdev --rereadpt /dev/hda
BLKRRPART: Device or resource busy
[root@vm1 ~]# poweroff
 
Broadcast message from root (pts/0) (Thu May 27 16:43:09 2010):
 
The system is going down for system halt NOW!
I believe a reboot is enough to make the vm reread the hard drive file...but I went for poweroff anyway
[root@vm1 ~]# fdisk -l /dev/hda
 
Disk /dev/hda: 80.0 GB, 80003851264 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          19      152586   83  Linux
/dev/hda2              20        8510    68203957+  8e  Linux LVM
[root@vm1 ~]# fdisk -u /dev/hda
 
The number of cylinders for this disk is set to 9726.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
 
Command (m for help): p
 
Disk /dev/hda: 80.0 GB, 80003851264 bytes
255 heads, 63 sectors/track, 9726 cylinders, total 156257522 sectors
Units = sectors of 1 * 512 = 512 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *          63      305234      152586   83  Linux
/dev/hda2          305235   136713149    68203957+  8e  Linux LVM
 
Command (m for help): d
Partition number (1-4): 2
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First sector (305235-156257521, default 305235): 
Using default value 305235
Last sector or +size or +sizeM or +sizeK (305235-156257521, default 156257521): 
Using default value 156257521
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
 
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@vm1 ~]# reboot
 
Broadcast message from root (pts/0) (Thu May 27 16:56:29 2010):
 
The system is going down for reboot NOW!
Now the straightforward lvm stuff
[root@vm1 ~]# pvdisplay /dev/hda2
  --- Physical volume ---
  PV Name               /dev/hda2
  VG Name               Example
  PV Size               65.04 GB / not usable 13.24 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              2081
  Free PE               0
  Allocated PE          2081
  PV UUID               jUn8V8-Ca29-HlTK-8Rnn-yq1k-2jae-aCfGnH
 
[root@vm1 ~]# pvresize /dev/hda2
  Physical volume "/dev/hda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
[root@vm1 ~]# pvdisplay /dev/hda2
  --- Physical volume ---
  PV Name               /dev/hda2
  VG Name               Example
  PV Size               74.36 GB / not usable 20.39 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              2379
  Free PE               298
  Allocated PE          2081
  PV UUID               jUn8V8-Ca29-HlTK-8Rnn-yq1k-2jae-aCfGnH
Free PE is 298, so I can add that to my logical volume.
[root@vm1 ~]# lvdisplay /dev/Example/RootVol 
  --- Logical volume ---
  LV Name                /dev/Example/RootVol
  VG Name                Example
  LV UUID                eawcxY-KVZG-rcdO-Bf5E-Vck3-JOnJ-8KN2lm
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                57.03 GB
  Current LE             1825
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
[root@vm1 ~]# lvextend -l +298 /dev/Example/RootVol 
  Extending logical volume RootVol to 66.34 GB
  Logical volume RootVol successfully resized
[root@vm1 ~]# lvdisplay /dev/Example/RootVol 
  --- Logical volume ---
  LV Name                /dev/Example/RootVol
  VG Name                Example
  LV UUID                eawcxY-KVZG-rcdO-Bf5E-Vck3-JOnJ-8KN2lm
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                66.34 GB
  Current LE             2123
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
[root@vm1 ~]#
The last step is to resize the ext4 on the logical volume
[root@vm1 ~]# resize2fs /dev/Example/RootVol 
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/Example/RootVol is mounted on /; on-line resizing required
Performing an on-line resize of /dev/Example/RootVol to 17391616 (4k) blocks.
The filesystem on /dev/Example/RootVol is now 17391616 blocks long.
[root@vm1 ~]# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/Example-RootVol
                       65G  1.6G   60G   3% /
Done.

Comments

thomas

Wed, 2012.05.30 - 12:22

In reply to by Anonymous (not verified)

Add new comment

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

About the Author...

Slides from LISA 2019 Linux systems troubleshooting #LISA2019 https://t.co/D4dMKflK6R Tue Oct 29 05:59:30 +0000 2019

https://t.co/AGeihMALAv configuring grub2 with EFI Fri Sep 13 05:20:01 +0000 2019

I published a Thing on @thingiverse! https://t.co/IYpRyEb7Hz #thingalert Tue Jul 23 19:27:57 +0000 2019

Nokogiri install on MacOSX https://t.co/v3An0miW9L Fri Jul 12 15:06:49 +0000 2019

HTML email with plain mailer plugin on Jenkins https://t.co/Z6FSDMDjy8 Thu Jul 11 21:07:25 +0000 2019