Hey,
Had some problems last week with expanding the RAID6 I have on a storage with Openfiler. This is what was needed to be done:
I have a storage with 16 hard drives for storage+2 for the OS. When I built it almost 3 years ago the idea was to have 1 RAID6 with 14 drives and 1 RAID1 with 2 drives obviously. Don’t ask me why - it was stupid to begin with and thus was never really used (the RAID1 that is). But came the time when we needed that additional space of 2 drives and the decision was to cancel the RAID1 and add the spare drives to the main RAID6, expanding the existing space.
The RAID controller is 3ware 9650se and after reading the manual it seemed pretty simple: delete the RAID1, have the spare drives and do the RAID6 migration to include them. After the initialization(is this the right word?) I was supposed to have the 25-26TB of space instead of the existing 22.
And so it was. Well, almost. After everything was finished I saw the new empty space in the Openfiler’s web gui: 3775GB or something like that. But the thing was that when trying to create the new volume it would create a ridiculous 4MB thingy, although I entered the max size. After fiddling with this for a bit I realised I can’t risk the data (all the company’s docs and files) any further and it was decided to make a clean start. So I transferred them all to the backup storage and started playing.
By the way, one other thing that made me want to reinstall the OS was the fact that there was some problem with the last updates which were sometime in 2008. I know this because I already tried updating the backup storage before that (a month ago) and completely messed it up. I ended up installing fresh OS and updating it - it went smooth. And since I needed some of the updates badly it was in my interest anyway to do this.
If you care about what updates I needed so badly, it was the samba update. Apparently, win7 has some upgraded mechanisms of authentication, so if you try accessing the openfiler’s shares from win7 by name - all goes well, but if you do this by IP - you’ll end up entering the credentials over and over again - loop that never ends. There is a fix - you just need to uncheck a couple of boxes on the windows machine, but when you have more than one of those - it just makes sense to upgrade the Openfiler.
The solution for win7 is:
Go to: Control Panel -> Administrative Tools -> Local Security Policy
Select: Local Policies -> Security Options
There are two settings to change.
“Network security: LAN Manager authentication level” -> Send LM & NTLM responses
“Minimum session security for NTLM SSP” -> uncheck: Require 128-bit encryption
taken from here
Anyway, back to me. Formatted the OS drive, installed, updated and configured. And all that only to find out that nothing’s changed partition-wise. I still saw all the data partition: 22TB of previous partitions and 3.7TB of the new empty space. Trying to create a partition ends in 4MB. WTF?!
So it’s not the OS. Re-read the user guide from 3ware - not that many options to make mistakes. What then? The Openfiler’s forums never mentioned anything like this. So I decided that since I have the backup I should just delete the RAID and start the data partition from scratch. It was pretty sucky because I knew that I’ll have to restore the data afterwards - at least 6TB for now. That’s another week working on the backup. Oh, well…
So here we go, 3ware web gui, remove the unit, create new unit, initialize. Meanwhile access the Openfiler’s gui, see the 26TB space, create the 26TB partition. Wait, WHAT?! 22TB?!
This is where I was about to loose it. Mostly because it was so unexpected: I was supposed to be done twice by then.
Now, as you might know, if you want to have a more than 2TB partition on Linux you have to label it GPT. Otherwise it defaults to MSDOS label. I knew that because I learned it the first time I created this storage 3 years ago. You have to ssh into the machine and run
parted /dev/sdX mklabel gpt quit
replace the X with the right letter, of course.
Now, here’s the thing: I didn’t do it this time and yet it did recognize the 20+TB. How’s that? Maybe the partition wasn’t really deleted although the RAID was removed and recreated.
fdisk -l
showed me that the partition was in fact GPT.
So I ran parted and saw this (syntax):
mklabel LABEL-TYPE create a new disklabel (partition table)
This is when it hit me - the partition table didn’t change despite the fact that RAID was destroyed. Or was it…?
Anyway, a simple new
mklabel gpt
on this drive solved all of my problems. Of course, by then all of the data was destroyed, but I was ready for it and hopefully by the end of this week our office will be back on the main storage.
Hope it’ll help someone not to loose it when dealing with this.
Ozzik.
I recently had the same problem. I built a 32TB chassis system and filled it initially with 5 2TB drives. I then added another drive, migrated the array to get the additional drive added and of course discovered that the partition was not changing. In my case, I require a strategy for making this happen in a way that doesn’t require me to migrate my data and rebuild the array.
This time, that is exactly what I wound up doing as my file systems will ultimately be larger than 16TB xfs is my only option. The problem with using openfiler is that it is not possible to use openfiler with xfs and a hardware raid solution without also involving lvm.
Initially I thought, no problem…every time I add a new drive I wills imply create a new gpt partition with a new lvm and basically append all the new lvms onto the same volume but before adopting that strategy I started investigating how many lvms you can actually do that with before it starts choking, sputtering, and puking all over itself. Apparently the limit is something like 15 and in a dynamically growing array of more than 15 disks over time, that breaks as well so back to the drawing board.
My friend who deals with arrays of this size all the time recommended simply using a regular gpt partition with xfs and skipping lvm as though that somehow solves the problem because the xfs tools will know how to resize the partition as well?? Anyway that turned out to be a problem if openfiler is involved because as soon as openfiler sees that it did not create the partition it refuses to manage it or do anything with it at all.
I finally found a program called gptfdisk (or gdisk for short, but not to be confused with the windows/dos gdisk utility) that can resize gpt partitions and change the metadata associated with partition so that no data is actually destroyed. After that, resizing the lvm is straight forward. I am currently in the process of validating this procedure of adding the drive, modifying the array, changing the partition size, the resizing the lvm and then lastly of course the file system. It is taking many days, a couple of weeks even to get through it but if it works well at least I will know how to overcome all these issues.
Correction, the 15 limit I think might have been on partitions and not LVMs. Regardless it still presents a problem as each new LVM would require a new partition as the drives are added.