Reliability, availability and serviceability

While RAS originated as a hardware-oriented term, systems thinking has extended the concept of reliability-availability-serviceability to systems in general, including software.[4]

• Reliability can be defined as the probability that a system will produce correct outputs up to some given time t.[5] Reliability is enhanced by features that help to avoid, detect and repair hardware faults. A reliable system does not silently continue and deliver results that include uncorrected corrupted data. Instead, it detects and, if possible, corrects the corruption, for example: by retrying an operation for transient (soft) or intermittent errors, or else, for uncorrectable errors, isolating the fault and reporting it to higher-level recovery mechanisms (which may failover to redundant replacement hardware, etc.), or else by halting the affected program or the entire system and reporting the corruption. Reliability can be characterized in terms of mean time between failures (MTBF), with reliability = exp(-t/MTBF).[5]
• Availability means the probability that a system is operational at a given time, i.e. the amount of time a device is actually operating as the percentage of total time it should be operating. High-availability systems may report availability in terms of minutes or hours of downtime per year. Availability features allow the system to stay operational even when faults do occur. A highly available system would disable the malfunctioning portion and continue operating at a reduced capacity. In contrast, a less capable system might crash and become totally nonoperational. Availability is typically given as a percentage of the time a system is expected to be available, e.g., 99.999 percent ("five nines").
• Serviceability or maintainability is the simplicity and speed with which a system can be repaired or maintained; if the time to repair a failed system increases, then availability will decrease. Serviceability includes various methods of easily diagnosing the system when problems arise. Early detection of faults can decrease or avoid system downtime. For example, some enterprise systems can automatically call a service center (without human intervention) when the system experiences a system fault. The traditional focus has been on making the correct repairs with as little disruption to normal operations as possible.

Note the distinction between reliability and availability: reliability measures the ability of a system to function correctly, including avoiding data corruption, whereas availability measures how often the system is available for use, even though it may not be functioning correctly. For example, a server may run forever and so have ideal availability, but may be unreliable, with frequent data corruption.[6]

Physical faults can be temporary or permanent.

• Permanent faults lead to a continuing error and are typically due to some physical failure such as metal electromigration or dielectric breakdown.
• Temporary faults include transient and intermittent faults.
  • Transient (a.k.a. soft) faults lead to independent one-time errors and are not due to permanent hardware faults: examples include alpha particles flipping a memory bit, electromagnetic noise, or power-supply fluctuations.
  • Intermittent faults occur due to a weak system component, e.g. circuit parameters degrading, leading to errors that are likely to recur.[5]

Transient and intermittent faults can typically be handled by detection and correction by e.g., ECC codes or instruction replay (see below). Permanent faults will lead to uncorrectable errors which can be handled by replacement by duplicate hardware, e.g., processor sparing, or by the passing of the uncorrectable error to high level recovery mechanisms. A successfully corrected intermittent fault can also be reported to the operating system (OS) to provide information for predictive failure analysis.

Example hardware features for improving RAS include the following, listed by subsystem:

• Processor:
  • Processor instruction error detection (e.g. residue checking of results[7]) with instruction retry e.g. alternative processor recovery in IBM mainframes,[8] or "Instruction replay technology" in Itanium systems.[9]
  • Processors running in lock-step to perform master-checker or voting schemes.
  • Machine Check Architecture and ACPI Platform Error Interface to report errors to the OS.
• Memory:
  • Parity or ECC (including single device correction) protection of memory components (cache and main memory); bad cache line disabling; memory scrubbing; memory sparing, memory mirroring;[10] bad page offlining; redundant bit steering; redundant array of independent memory (RAIM).
• I/O:
  • Cyclic redundancy check checksums for data transmission/retry and data storage, e.g. PCI Express (PCIe) Advanced Error Reporting (AER),[11] redundant I/O paths.
• Storage:
  • RAID configurations for hard disk drive and solid state drive storage.
  • Journaling file systems for file repair after crashes.
  • Checksums on both data and metadata, and background scrubbing.
  • Self-Monitoring, Analysis, and Reporting Technology for hard disk drive and solid state drive.
• Power/cooling:
  • Duplicating components to avoid single points of failure, e.g., power-supplies.
  • Over-designing the system for the specified operating ranges of clock frequency, temperature, voltage, vibration.
  • Temperature sensors to throttle operating frequency when temperature goes out of specification.
  • Surge protector, uninterruptible power supply, auxiliary power.
• System:
  • Hot swapping of components: CPUs, RAMs, hard disk drives and solid state drives.
  • Predictive failure analysis to predict which intermittent correctable errors will lead eventually to hard non-correctable errors.
  • Partitioning/domaining of computer components to allow one large system to act as several smaller systems.
  • Virtual machines to decrease the severity of operating system software faults.
  • Redundant I/O domains[12] or I/O partitions[13] for providing virtual I/O to guest virtual machines.
  • Computer clustering capability with failover capability, for complete redundancy of hardware and software.
  • Dynamic software updating to avoid the need to reboot the system for a kernel software update, for example Ksplice under Linux.
  • Independent management processor for serviceability: remote monitoring, alerting and control.

Fault-tolerant designs extended the idea by making RAS to be the defining feature of their computers for applications like stock market exchanges or air traffic control, where system crashes would be catastrophic. Fault-tolerant computers (e.g., see Tandem Computers and Stratus Technologies), which tend to have duplicate components running in lock-step for reliability, have become less popular, due to their high cost. High availability systems, using distributed computing techniques like computer clusters, are often used as cheaper alternatives.

• Machine Check Architecture (MCA)
• Machine-check exception (MCE)
• High availability (HA)
• Redundancy (engineering)
• Integrated logistics support
• RAMS (reliability, availability, maintainability and safety)

• Itanium Reliability, Availability and Serviceability (RAS) Features Overview of RAS features in general and specific features of the Itanium processor.
• POWER7 System RAS Key Aspects of Power Systems Reliability, Availability, and Serviceability. Daniel Henderson, Jim Mitchell, and George Ahrens. February 10, 2012 Overview of RAS features in Power processors.
• Intel Corp. Reliability, Availability, and Serviceability for the Always-on Enterprise (appendix B) and Intel Xeon Processor E7 Family: supporting next generation RAS servers. White paper. Overview of RAS features in Xeon processors.
• zEnterprise 196 System Overview. IBM Corp. (Chapter 10) Overview of RAS features of IBM z196 processor and zEnterprise 196 server.
• Maximizing Application Reliability and Availability with the SPARC M5-32 Server RAS features of Oracle’s SPARC M5-32 server