The CTERA P1200 appliance has 12 drive bays. It is helpful to understand how RAID arrays and storage pools operate under ZFS, in order to obtain the highest amount of usable storage from the available disks and the best redundancy.
Generally speaking, adding the maximum number of disks to the unit from the start, will result in more available capacity than adding smaller groups of disks over time. This is because under ZFS you cannot grow RAID arrays by adding more disks to an existing array. Adding disks to a RAID array to expand the capacity of the array is not supported. You can, however, combine multiple RAID arrays into a single storage pool to grow the storage pool size. The failure of any one RAID array in a storage pool may compromise the entire storage pool, so it is important to consider appropriate RAID levels for each array that is added to the storage pool.
It is not recommended to include different drive types, capacities, or models in the same RAID array, as this can lead to RAID instability and failures. Furthermore, it is not recommended to include different RAID array types in the same storage pool. Arrays added to a storage pool should be of the same size and RAID type, so that performance across the storage pool is consistent.
The following RAID array types are supported under ZFS:
Equivalent to RAID 5. raidz combines striping with distributed parity to protect against data loss. It uses one drive worth of capacity for parity data, meaning up to one drive in the array can fail without compromising the array data. (The array will be marked as “Degraded”, but data will not be lost.) If a second drive fails while the first failed drive is rebuilding, the array data is lost.
The array capacity is: (n-1)*s
Where n = number of drives, and s = size of smallest disk.
A minimum of three drives is required.
Equivalent to RAID 6. Two drives worth of capacity are used for parity data, meaning up to two drives in the array can fail simultaneously before the array data is compromised.
The array capacity is: (n-2)*s
Where n = number of drives, and s = size of smallest disk.
A minimum of four drives is required.
Triple Parity RAID. Unique to ZFS, raidz3 offers a triple parity RAID level. Three drives worth of capacity are used for parity data, meaning up to three drives in the array can be lost simultaneously.
The array capacity is: (n-3)*s
Where n = number of drives, and s = size of smallest disk.
A minimum of five drives is required.
RAID arrays can be mirrored, for optimal data protection. For example, two raidz (RAID5) arrays of six drives each can be mirrored, and data will be replicated in an identical fashion across all disks of the mirror.
One of the major benefits of ZFS is that mirrored arrays are "self-healing" and prevent silent disk corruption. When arrays are mirrored, it is possible to detect read/write errors and automatically correct those errors. Therefore, if a disk fails, the array's performance will be reduced (the array will be marked as “Degraded”), but data will not be lost, so long as there is at least one good disk. Data will only be lost if all the disks in the array fail.
This is one of the major benefits of ZFS, but it comes at the expense of usable capacity. Since the exact same data must be written on each disk in the array, the array's capacity is limited to that of the smallest disk.
A minimum of two drives is required.