NetApp FAS

All-Flash FAS, also known as AFF A-series. Usually, AFF systems based on the same hardware as FAS but first one optimized and works only with SSD drives on the back end while the second one can use HDD and SSD as a cache: for example, AFF A700 & FAS9000, A300 & FAS8200, A200 & FAS2600, A220 & FAS2700 use the same hardware, but AFF systems do not include Flash Cache cards. Also, AFF systems do not support FlexArray with third-party storage array virtualization functionality. AFF is a Unified system and can provide SAN & NAS data protocol connectivity, and in addition to traditional SAN & NAS protocols in FAS systems, AFF has block-based NVMe/FC protocol for systems with 32Gbit/s FC ports. AFF & FAS use the same firmware image, and nearly all noticeable functionality for the end-user is the same for both storage systems. However, internally data is processed and handled differently in ONTAP. AFF systems, for example, use different Write Allocation algorithms as compared to FAS systems. Because AFF systems have faster underlying SSD drives, Inline data deduplication in ONTAP systems is nearly not noticeable (~2% performance impact on low-end systems).[2]

All SAN Array running ONTAP, and based on AFF platform thus inherits its features & functionalities, and data internally processed and handled the same as in AFF systems. All other ONTAP-based hardware and software platforms can be referred to as Unified ONTAP meaning they can provide unified access with SAN & NAS data protocols. ONTAP architecture in ASA systems is the same as in FAS & AFF, with no changes. ASA systems using the same firmware image as AFF & FAS systems. ASA is the same as AFF, and the only difference is in the access to the storage over the network with SAN protocols: ASA provides symmetric active/active access to the block devices (LUN or NVMe namespaces), while Unified ONTAP systems continue to use ALUA and ANA for the block protocols. The original ASA platform was first announced in 2019 and re-launched in May 2023.[3]

In February 2023, NetApp introduced a new AFF product line code named the C-Series. This platform uses quad-level cell (QLC) NAND flash and is aimed at competing with the QLC-based products already in the market from Pure Storage (specifically FlashArray//C).[4] The C-Series has higher latency than typical AFF systems, at around 2-4 milliseconds compared to 500 microseconds on AFF using triple-level cell (TLC) media. However, the aim of the platform is to provide a lower price point for customers that might otherwise not choose all-flash systems.

NetApp uses either SATA, Fibre Channel, SAS or SSD disk drives, which it groups into RAID (Redundant Array of Inexpensive Disks or Redundant Array of Independent Disks) groups of up to 28 (26 data disks plus 2 parity disks). NetApp FAS storage systems which contain only SSD drives with installed SSD-optimized ONTAP OS called All-Flash FAS (AFF).

FAS, ASA, and AFF systems are using enterprise-level HDD and SSD (i.e. NVMe SSD) drives with two ports, each port connected to each controller in an HA pair. HDD and SSD drives can only be bought from NetApp and installed in NetApp's Disk Shelves for FAS/AFF platform. Physical HDD and SSD drives, partitions on them, and LUNs imported from third-party arrays with FlexArray functionality considered in ONTAP as a Disk. In SDS systems like ONTAP Select & ONTAP Cloud, logical block storage like virtual disk or RDM inside ONTAP also considered as a Disk. Do not confuse the general term "disk drive" and "disk drive term used in ONTAP system" because, with ONTAP, it could be an entire physical HDD or SSD drive, a LUN, or a partition on a physical HDD or SSD drive. LUNs imported from third-party arrays with FlexArray functionality in HA pair configuration must be accessible from both nodes of the HA pair. Each disk has ownership on it to show which controller owns and serves the disk. An aggregate can include only disks owned by a single node, therefore each aggregate owned by a node and any objects on top of it, as FlexVol volumes, LUNs, File Shares are served with a single controller. Each controller can have its own disks and aggregates them where both nodes can be utilized simultaneously even though they not serving the same data.

Advanced Drive Partitioning (ADP) can be used in ONTAP-based systems depending on the platform and use-case. ADP can be used only with native disk drives from NetApp Disk shelves, FlexArray technology does not support ADP. ADP also supported with third-party drives in ONTAP Select. This technique mainly used to overcome some architectural requirements and reduce the number of disk drives in ONTAP-based systems. There are three types of ADP: Root-Data partitioning; Root-Data-Data partitioning (RD2 also known as ADPv2); Storage Pool. Root-Data partitioning can be used in FAS & AFF systems to create small root partitions on drives to use them to create system root aggregates and, therefore, not to spend entire three disk drives for that purpose. In contrast, the bigger portion of the disk drive will be used for data aggregate. Root-Data-Data partitioning is used in AFF systems only for the same reason as Root-Data partitioning with the only difference that bigger portion of the drive left after root partitioning divided equally by two additional partitions, usually, each partition assigned to one of the two controllers, therefore reducing the minimum number of drives required for an AFF system and reducing waste for expensive SSD space. Storage Pool partitioning technology used in FAS systems to equally divide each SSD drive by four pieces which later can be used for FlashPool cache acceleration, with Storage Pool only a few SSD drives can be divided by up to 4 data aggregates which will benefit from FlashCache caching technology reducing minimally required SSD drives for that technology.

ONTAP Storage layout: Aggregate, Plex, RAID

In NetApp ONTAP systems, RAID and WAFL are tightly integrated. There are several RAID types available within ONTAP-based systems:

• RAID 4 with 1 dedicated parity disk allowing any 1 drive to fail in a RAID group.
• RAID-DP with 2 dedicated parity disks allowing any 2 drives to fail simultaneously in a RAID group.[5]
• RAID-TEC US patent 7640484 with 3 dedicated parity drives, allows any 3 drives to fail simultaneously in a RAID group.[6]

RAID-DP's double parity leads to a disk loss resiliency similar to that of RAID 6. NetApp overcomes the write performance penalty of traditional RAID-4 style dedicated parity disks via WAFL and a novel use of its nonvolatile memory (NVRAM) within each storage system.[7] Each aggregate consist of one or two plexes, a plex consists of one or more RAID groups. Typical ONTAP-based storage system have only 1 plex in each aggregate, two plexes used in local SyncMirror or MetroCluster configurations. Each RAID group usually consists of disk drives of same type, speed, geometry and capacity. Though NetApp Support could allow a user to install a drive to a RAID group with same or bigger size and different type, speed and geometry for temporary basis. Ordinary data aggregates if containing more than one RAID group must have same RAID groups across the aggregate, same RAID group size is recommended, but NetApp allows to have exception in last RAID group and configure it as small as half of the RAID group size across aggregate. For example, such an aggregate might consists of 3 RAID groups: RG0:16+2, RG1:16+2, RG2:7+2. Within aggregates, ONTAP sets up flexible volumes (FlexVol) to store data that users can access.

Aggregates enabled as FlshPool and with both HDD and SSD drives called hybrid aggregates. In Flash Pool hybrid aggregates same rules applied to the hybrid aggregate as to ordinary aggregates but separately to HDD and SSD drives, thus it is allowed to have two different RAID types: only one RAID type for all HDD drives and only one RAID type for all SSD drives in a single hybrid aggregate. For example, SAS HDD with RAID-TEC (RG0:18+3, RG1:18+3) and SSD with RAID-DP (RG3:6+2). NetApp storage systems running ONTAP combine underlying RAID groups similarly to RAID 0. Also, in NetApp FAS systems with FlexArray feature, the third-party LUNs could be combined in a Plex similarly to RAID 0. NetApp storage systems running ONTAP can be deployed in MetroCluster and SyncMirror configurations, which are using technique comparably to RAID 1 with mirroring data between two plexes in an aggregate.

NetApp Flash Pool is a feature on hybrid NetApp FAS systems that allows creating hybrid aggregate with HDD drives and SSD drives in a single data aggregate. Both HDD and SSD drives form separate RAID groups. Since SSD also used to write operations, it requires RAID redundancy contrary to Flash Cache but allows to use of different RAID types for HDD and SSD; for example, it is possible to have 20 HDD 8TB in RAID-TEC while 4 SSD in RAID-DP 960GB in a single aggregate. SSD RAID used as cache and improved performance for read-write operations for FlexVol volumes on the aggregate where SSD added as the cache. Flash Pool cache similarly to Flash Cache has policies for reading operations but also includes write operations that could apply separately for each FlexVol volume located on the aggregate; therefore, it could be disabled on some volumes while others could benefit from SSD cache. Both FlashCache & FlashPool can be used simultaneously to cache data from a single FlexVol to enable an aggregate with Flash Pool technology minimum 4 SSD disks required (2 data, 1 parity, and 1 hot spare), it is also possible to use ADP technology to partition SSD into 4 pieces (Storage Pool) and distribute those pieces between two controllers so each controller will benefit from SSD cache when there is a small amount of SSD. Flash Pool is not available with FlexArray and is possible only with NetApp FAS native disk drives in NetApp's disk shelves.

FlexArray is NetApp FAS functionality allows to visualize third party storage systems and other NetApp storage systems over SAN protocols and use them instead of NetApp's disk shelves. With FlexArray functionality RAID protection must be done with third party storage array thus NetApp's RAID 4, RAID-DP and RAID-TEC not used in such configurations. One or many LUNs from third party arrays could be added to a single aggregate similarly to RAID 0. FlexArray is licensed feature.

NetApp Storage Encryption (NSE) is using specialized purpose build disks with low level Hardware-based full disk encryption (FDE/SED) feature and also supports FIPS-certified self-encrypted drives, compatible nearly with all NetApp ONTAP features and protocols but does not offer MetroCluster. NSE feature does overall nearly zero performance impact on the storage system. NSE feature similarly to NetApp Volume Encryption (NVE) in storage systems running ONTAP can store encryption key locally in Onboard Key Manager or on dedicated key manager systems using KMIP protocol like IBM Security Key Lifecycle Manager and SafeNet KeySecure. NSE is data at rest encryption which means it protects only from physical disks theft and does not give an additional level of data security protection in a normal operational and running system. NetApp has passed NIST Cryptographic Module Validation Program for its NetApp CryptoMod (TPM) with ONTAP 9.2.[8]

SyncMirror replication using plexes

MetroCluster (MC) is free functionality for FAS and AFF systems for metro high availability with synchronous replication between two sites, this configuration requires additional equipment. Available in both modes: 7-mode (old OS) and Cluster-Mode (or cDOT - a newer version of ONTAP OS). MetroCluster in Cluster-Mode known as MCC. MetroCluster uses RAID SyncMirror (RSM) and plex technique where on one site number of disks form one or more RAID groups aggregated in a plex, while on the second site have the same number of disks with the same type and RAID configuration along with Configuration Replication Service (CRS) and NVLog replication. One plex synchronously replicates to another in a compound with non-volatile memory. Two plexes form an aggregate where data stored and in case of disaster on one site second site provide read-write access to data. MetroCluster Support FlexArray technology. MetroCluster configurations are possible only with mid-range and high-end models which provide the ability to install additional network cards required to MC to function.

MetroCluster local and DR pare memory replication in NetApp FAS/AFF systems configured as MCC

With MetroCluster it is possible to have one or more storage node per site to form a cluster or Clustered MetroCluster (MCC). Remote and local HA perter node must be the same model. MCC consists of two clusters each located on one of two sites. There may be only two sites. In MCC configuration each one remote and one local storage node form Metro HA or Disaster Recovery Pare (DR Pare) across two sites while two local nodes (if there is partner) form local HA pare, thus each node synchronously replicates data in non-volatile memory two nodes: one remote and one local (if there is one). It is possible to utilize only one storage node on each site (two single node clusters) configured as MCC. 8 node MCC consists of two clusters - 4 nodes each (2 HA pair), each storage node have only one remote partner and only one local HA partner, in such a configuration each site clusters can consist out of two different storage node models. For small distances, MetroCluster requires at least one FC-VI or newer iWARP card per node. FAS and AFF systems with ONTAP software versions 9.2 and older utilize FC-VI cards and for long distances require 4 dedicated Fibre Channel switches (2 on each site) and 2 FC-SAS bridges per each disk shelf stack, thus minimum 4 total for 2 sites and minimum 2 dark fiber ISL links with optional DWDMs for long distances. Data volumes, LUNs and LIFs could online migrate across storage nodes in the cluster only within a single site where data originated from: it is not possible to migrate individual volumes, LUNs or LIFs using cluster capabilities across sites unless MetroCluster switchover operation is used which disable entire half of the cluster on a site and transparently to its clients and applications switch access to all of the data to another site.

NetApp MetroCluster over IP with ADPv2 configuration

Starting with ONTAP 9.3 MetroCluster over IP (MCC-IP) was introduced with no need for a dedicated back-end Fibre Channel switches, FC-SAS bridges and dedicated dark fiber ISL which previously were needed for a MetroCluster configuration. Initially, only A700 & FAS9000 systems supported with MCC-IP. MCC-IP available only in 4-node configurations: 2-node Highly Available system on each site with two sites total. With ONTAP 9.4, MCC-IP supports A800 system and Advanced Drive Partitioning in form of Rood-Data-Data (RD2) partitioning, also known as ADPv2. ADPv2 supported only on all-flash systems. MCC-IP configurations support single disk shelf where SSD drives partitioned in ADPv2. MetroCluster over IP require Ethernet cluster switches with installed ISL and utilize iWARP cards in each storage controller for synchronous replication. Starting with ONTAP 9.5 MCC-IP supports distance up to 700 km and starts to support SVM-DR feature, AFF A300, and FAS8200 systems.

WAFL, as a robust versioning filesystem in NetApp's proprietary OS ONTAP, it provides snapshots, which allow end-users to see earlier versions of files in the file system. Snapshots appear in a hidden directory: ~snapshot for Windows (SMB) or .snapshot for Unix (NFS). Up to 1024 snapshots can be made of any traditional or flexible volume. Snapshots are read-only, although ONTAP provides additional ability to make writable "virtual clones", based at "WAFL snapshots" technique, as "FlexClones".

ONTAP implements snapshots by tracking changes to disk-blocks between snapshot operations. It can set up snapshots in seconds because it only needs to take a copy of the root inode in the filesystem. This differs from the snapshots provided by some other storage vendors in which every block of storage has to be copied, which can take many hours.

Each NetApp FAS system running Data ONTAP 8 could switch between modes either 7-Mode or Cluster mode. In reality each mode was a separate OS with its own version of WAFL, both 7-mode and Cluster mode where shipped on a single firmware image for a FAS system till 8.3 where 7-mode was deprecated. SnapLock migration from 7-Mode to ONTAP 9 now supported with Transition Tool. It is possible to switch between modes on a FAS system but all the data on disks must be destroyed first since WAFL is not compatible and server-based application called 7MTT tool was introduced to migrate data from old 7-mode FAS system to new Cluster-Mode:

• With SnapMirror based replication called Copy-based transition which helped to migrate all the data with planned downtime using only storage vendor capabilities. Copy-based transition require new controllers and disks with space no less than on source system if all the data to be migrated. Both SAN and NAS data are possible.
• Starting with 7-mode 8.2.1 and Cluster-Mode 8.3.2 WAFL compatibility where introduced and new feature in 7MTT tool called Copy-free transition to replace old controllers running 7-mode with new controllers running Cluster-Mode and planned downtime, while new system require additional system disks with root aggregates for new controllers (it could be as less as 6 disks). Since with Copy-free transition no data copying required 7MTT tool helping only for new controllers reconfiguration. Both SAN and NAS data conversion supported.

Additional to 7MTT there are two other paths to migrate data based on protocol type:

• SAN data could be copied with foreign LUN import (FLI) functionality integrated in NetApp FAS system which can copy data over SAN protocol while new storage system running ONTAP placed as SAN proxy between hosts and old storage system which require host reconfiguration and minimum downtime. FLI available as for old 7-mode systems and for some models of storage systems of competitors.
• NAS data could be copied with NetApp XCP free host-based utility thus host-based copy process processed with the utility from any copying data from source server with SMB or NFS protocols to ONTAP system with minimal downtime for client systems reconfiguration for new NAS server.