Storage area network

The Fibre Channel SAN connects servers to storage via Fibre Channel switches.

Storage area networks (SANs) are sometimes referred to as network behind the servers[2]^: 11 and historically developed out of a centralized data storage model, but with its own data network. A SAN is, at its simplest, a dedicated network for data storage. In addition to storing data, SANs allow for the automatic backup of data, and the monitoring of the storage as well as the backup process.[3]^: 16–17 A SAN is a combination of hardware and software.[3]^: 9 It grew out of data-centric mainframe architectures, where clients in a network can connect to several servers that store different types of data.[3]^: 11 To scale storage capacities as the volumes of data grew, direct-attached storage (DAS) was developed, where disk arrays or just a bunch of disks (JBODs) were attached to servers. In this architecture, storage devices can be added to increase storage capacity. However, the server through which the storage devices are accessed is a single point of failure, and a large part of the LAN network bandwidth is used for accessing, storing and backing up data. To solve the single point of failure issue, a direct-attached shared storage architecture was implemented, where several servers could access the same storage device.[3]^: 16–17

DAS was the first network storage system and is still widely used where data storage requirements are not very high. Out of it developed the network-attached storage (NAS) architecture, where one or more dedicated file server or storage devices are made available in a LAN.[3]^: 18 Therefore, the transfer of data, particularly for backup, still takes place over the existing LAN. If more than a terabyte of data was stored at any one time, LAN bandwidth became a bottleneck.[3]^: 21–22 Therefore, SANs were developed, where a dedicated storage network was attached to the LAN, and terabytes of data are transferred over a dedicated high speed and bandwidth network. Within the SAN, storage devices are interconnected. Transfer of data between storage devices, such as for backup, happens behind the servers and is meant to be transparent.[3]^: 22 In a NAS architecture data is transferred using the TCP and IP protocols over Ethernet. Distinct protocols were developed for SANs, such as Fibre Channel, iSCSI, Infiniband. Therefore, SANs often have their own network and storage devices, which have to be bought, installed, and configured. This makes SANs inherently more expensive than NAS architectures.[3]^: 29

Dual port 8 Gb FC host bus adapter card

SANs have their own networking devices, such as SAN switches. To access the SAN, so-called SAN servers are used, which in turn connect to SAN host adapters. Within the SAN, a range of data storage devices may be interconnected, such as SAN-capable disk arrays, JBODS and tape libraries.[3]^{: 32, 35–36}

Fabric layer

Qlogic SAN-switch with optical Fibre Channel connectors installed

The fabric layer consists of SAN networking devices that include SAN switches, routers, protocol bridges, gateway devices, and cables. SAN network devices move data within the SAN, or between an initiator, such as an HBA port of a server, and a target, such as the port of a storage device.

When SANs were first built, hubs were the only devices that were Fibre Channel capable, but Fibre Channel switches were developed and hubs are now rarely found in SANs. Switches have the advantage over hubs that they allow all attached devices to communicate simultaneously, as a switch provides a dedicated link to connect all its ports with one another.[4]^: 34 When SANs were first built, Fibre Channel had to be implemented over copper cables, these days multimode optical fibre cables are used in SANs.[4]^: 40

SANs are usually built with redundancy, so SAN switches are connected with redundant links. SAN switches connect the servers with the storage devices and are typically non-blocking allowing transmission of data across all attached wires at the same time.[4]^: 29 SAN switches are for redundancy purposes set up in a meshed topology. A single SAN switch can have as few as 8 ports and up to 32 ports with modular extensions.[4]^: 35 So-called director-class switches can have as many as 128 ports.[4]^: 36

In switched SANs, the Fibre Channel switched fabric protocol FC-SW-6 is used under which every device in the SAN has a hardcoded World Wide Name (WWN) address in the host bus adapter (HBA). If a device is connected to the SAN its WWN is registered in the SAN switch name server.[4]^: 47 In place of a WWN, or worldwide port name (WWPN), SAN Fibre Channel storage device vendors may also hardcode a worldwide node name (WWNN). The ports of storage devices often have a WWN starting with 5, while the bus adapters of servers start with 10 or 21.[4]^: 47

Storage layer

Fibre Channel is a layered technology that starts at the physical layer and progresses through the protocols to the upper-level protocols like SCSI and SBCCS.

The serialized Small Computer Systems Interface (SCSI) protocol is often used on top of the Fibre Channel switched fabric protocol in servers and SAN storage devices. The Internet Small Computer Systems Interface (iSCSI) over Ethernet and the Infiniband protocols may also be found implemented in SANs, but are often bridged into the Fibre Channel SAN. However, Infiniband and iSCSI storage devices, in particular, disk arrays, are available.[4]^: 47–48

The various storage devices in a SAN are said to form the storage layer. It can include a variety of hard disk and magnetic tape devices that store data. In SANs, disk arrays are joined through a RAID which makes a lot of hard disks look and perform like one big storage device.[4]^: 48 Every storage device, or even partition on that storage device, has a logical unit number (LUN) assigned to it. This is a unique number within the SAN. Every node in the SAN, be it a server or another storage device, can access the storage by referencing the LUN. The LUNs allow for the storage capacity of a SAN to be segmented and for the implementation of access controls. A particular server, or a group of servers, may, for example, be only given access to a particular part of the SAN storage layer, in the form of LUNs. When a storage device receives a request to read or write data, it will check its access list to establish whether the node, identified by its LUN, is allowed to access the storage area, also identified by a LUN.[4]^{: 148–149} LUN masking is a technique whereby the host bus adapter and the SAN software of a server restrict the LUNs for which commands are accepted. In doing so LUNs that should never be accessed by the server are masked.[4]^: 354 Another method to restrict server access to particular SAN storage devices is fabric-based access control, or zoning, which is enforced by the SAN networking devices and servers. Under zoning, server access is restricted to storage devices that are in a particular SAN zone.[5]

A mapping layer to other protocols is used to form a network:

• ATA over Ethernet (AoE), mapping of AT Attachment (ATA) over Ethernet
• Fibre Channel Protocol (FCP), a mapping of SCSI over Fibre Channel
• Fibre Channel over Ethernet (FCoE)
• ESCON over Fibre Channel (FICON), used by mainframe computers
• HyperSCSI, mapping of SCSI over Ethernet
• iFCP[6] or SANoIP[7] mapping of FCP over IP
• iSCSI, mapping of SCSI over TCP/IP
• iSCSI Extensions for RDMA (iSER), mapping of iSCSI over InfiniBand
• Network block device, mapping device node requests on UNIX-like systems over stream sockets like TCP/IP
• SCSI RDMA Protocol (SRP), another SCSI implementation for remote direct memory access (RDMA) transports

Storage networks may also be built using Serial Attached SCSI (SAS) and Serial ATA (SATA) technologies. SAS evolved from SCSI direct-attached storage. SATA evolved from Parallel ATA direct-attached storage. SAS and SATA devices can be networked using SAS Expanders.

The Storage Networking Industry Association (SNIA) defines a SAN as "a network whose primary purpose is the transfer of data between computer systems and storage elements". But a SAN does not just consist of a communication infrastructure, it also has a software management layer. This software organizes the servers, storage devices, and the network so that data can be transferred and stored. Because a SAN does not use direct attached storage (DAS), the storage devices in the SAN are not owned and managed by a server.[2]^: 11 A SAN allows a server to access a large data storage capacity and this storage capacity may also be accessible by other servers.[2]^: 12 Moreover, SAN software must ensure that data is directly moved between storage devices within the SAN, with minimal server intervention.[2]^: 13

SAN management software is installed on one or more servers and management clients on the storage devices. Two approaches have developed in SAN management software: in-band and out-of band management. In-band means that management data between server and storage devices is transmitted on the same network as the storage data. While out-of-band means that management data is transmitted over dedicated links.[2]^: 174 SAN management software will collect management data from all storage devices in the storage layer. This includes info on read and write failures, storage capacity bottlenecks and failure of storage devices. SAN management software may integrate with the Simple Network Management Protocol (SNMP).[2]^: 176

In 1999 Common Information Model (CIM), an open standard, was introduced for managing storage devices and to provide interoperability, The web-based version of CIM is called Web-Based Enterprise Management (WBEM) and defines SAN storage device objects and process transactions. Use of these protocols involves a CIM object manager (CIMOM), to manage objects and interactions, and allows for the central management of SAN storage devices. Basic device management for SANs can also be achieved through the Storage Management Interface Specification (SMI-S), were CIM objects and processes are registered in a directory. Software applications and subsystems can then draw on this directory.[2]^: 177 Management software applications are also available to configure SAN storage devices, allowing, for example, the configuration of zones and LUNs.[2]^: 178

Ultimately SAN networking and storage devices are available from many vendors and every SAN vendor has its own management and configuration software. Common management in SANs that include devices from different vendors is only possible if vendors make the application programming interface (API) for their devices available to other vendors. In such cases, upper-level SAN management software can manage the SAN devices from other vendors.[2]^: 180

In a SAN, data is transferred, stored and accessed on a block level. As such, a SAN does not provide data file abstraction, only block-level storage and operations. Server operating systems maintain their own file systems on their own dedicated, non-shared LUNs on the SAN, as though they were local to themselves. If multiple systems were simply to attempt to share a LUN, these would interfere with each other and quickly corrupt the data. Any planned sharing of data on different computers within a LUN requires software. File systems have been developed to work with SAN software to provide file-level access. These are known as shared-disk file system.

Video editing systems require very high data transfer rates and very low latency. SANs in media and entertainment are often referred to as serverless due to the nature of the configuration which places the video workflow (ingest, editing, playout) desktop clients directly on the SAN rather than attaching to servers. Control of data flow is managed by a distributed file system. Per-node bandwidth usage control, sometimes referred to as quality of service (QoS), is especially important in video editing as it ensures fair and prioritized bandwidth usage across the network.

SAN Storage QoS enables the desired storage performance to be calculated and maintained for network customers accessing the device. Some factors that affect SAN QoS are:

• Bandwidth – The rate of data throughput available on the system.
• Latency – The time delay for a read/write operation to execute.
• Queue depth – The number of outstanding operations waiting to execute to the underlying disks (traditional or solid-state drives).

Alternatively, over-provisioning can be used to provide additional capacity to compensate for peak network traffic loads. However, where network loads are not predictable, over-provisioning can eventually cause all bandwidth to be fully consumed and latency to increase significantly resulting in SAN performance degradation.

Storage virtualization is the process of abstracting logical storage from physical storage. The physical storage resources are aggregated into storage pools, from which the logical storage is created. It presents to the user a logical space for data storage and transparently handles the process of mapping it to the physical location, a concept called location transparency. This is implemented in modern disk arrays, often using vendor-proprietary technology. However, the goal of storage virtualization is to group multiple disk arrays from different vendors, scattered over a network, into a single storage device. The single storage device can then be managed uniformly.[8]

• List of networked storage hardware platforms
• List of storage area network management systems
• Massive array of idle disks (MAID)
• Storage hypervisor
• Storage resource management (SRM)
• Converged storage

• What Is a Storage Area Network (SAN)?
• Introduction to Storage Area Networks Exhaustive Introduction into SAN, IBM Redbook
• SAN vs. DAS: A Cost Analysis of Storage in the Enterprise at the Wayback Machine (archived 2011-10-30)
• SAS and SATA, solid-state storage lower data center power consumption
• SAN NAS Videos
• 20 most promising enterprise storage solution providers of 2018