Null (SQL)

Null has been the focus of controversy and a source of debate because of its associated three-valued logic (3VL), special requirements for its use in SQL joins, and the special handling required by aggregate functions and SQL grouping operators. Computer science professor Ron van der Meyden summarized the various issues as: "The inconsistencies in the SQL standard mean that it is not possible to ascribe any intuitive logical semantics to the treatment of nulls in SQL."[1] Although various proposals have been made for resolving these issues, the complexity of the alternatives has prevented their widespread adoption.

Because Null is not a data value, but a marker for an absent value, using mathematical operators on Null gives an unknown result, which is represented by Null.[6] In the following example, multiplying 10 by Null results in Null:

10 * NULL          -- Result is NULL

This can lead to unanticipated results. For instance, when an attempt is made to divide Null by zero, platforms may return Null instead of throwing an expected "data exception – division by zero".[6] Though this behavior is not defined by the ISO SQL standard many DBMS vendors treat this operation similarly. For instance, the Oracle, PostgreSQL, MySQL Server, and Microsoft SQL Server platforms all return a Null result for the following:

NULL / 0

String concatenation operations, which are common in SQL, also result in Null when one of the operands is Null.[7] The following example demonstrates the Null result returned by using Null with the SQL || string concatenation operator.

'Fish ' || NULL || 'Chips'   -- Result is NULL

This is not true for all database implementations. In an Oracle RDBMS for example NULL and the empty string are considered the same thing and therefore 'Fish ' || NULL || 'Chips' results in 'Fish Chips'.

SQL three-valued logic is encountered in Data Manipulation Language (DML) in comparison predicates of DML statements and queries. The WHERE clause causes the DML statement to act on only those rows for which the predicate evaluates to True. Rows for which the predicate evaluates to either False or Unknown are not acted on by INSERT, UPDATE, or DELETE DML statements, and are discarded by SELECT queries. Interpreting Unknown and False as the same logical result is a common error encountered while dealing with Nulls.[9] The following simple example demonstrates this fallacy:

SELECT *
FROM t
WHERE i = NULL;

The example query above logically always returns zero rows because the comparison of the i column with Null always returns Unknown, even for those rows where i is Null. The Unknown result causes the SELECT statement to summarily discard each and every row. (However, in practice, some SQL tools will retrieve rows using a comparison with Null.)

Basic SQL comparison operators always return Unknown when comparing anything with Null, so the SQL standard provides for two special Null-specific comparison predicates. The IS NULL and IS NOT NULL predicates (which use a postfix syntax) test whether data is, or is not, Null.[11]

The SQL standard contains the optional feature F571 "Truth value tests" that introduces three additional logical unary operators (six in fact, if we count their negation, which is part of their syntax), also using postfix notation. They have the following truth tables:[12]

The F571 feature is orthogonal to the presence of the boolean datatype in SQL (discussed later in this article) and, despite syntactic similarities, F571 does not introduce boolean or three-valued literals in the language. The F571 feature was actually present in SQL92,[13] well before the boolean datatype was introduced to the standard in 1999. The F571 feature is implemented by few systems however; PostgreSQL is one of those implementing it.

The addition of IS UNKNOWN to the other operators of SQL's three-valued logic makes the SQL three-valued logic functionally complete,[14] meaning its logical operators can express (in combination) any conceivable three-valued logical function.

On systems which don't support the F571 feature, it is possible to emulate IS UNKNOWN p by going over every argument that could make the expression p Unknown and test those arguments with IS NULL or other NULL-specific functions, although this may be more cumbersome.

In SQL's three-valued logic the law of the excluded middle, p OR NOT p, no longer evaluates to true for all p. More precisely, in SQL's three-valued logic p OR NOT p is unknown precisely when p is unknown and true otherwise. Because direct comparisons with Null result in the unknown logical value, the following query

SELECT * FROM stuff WHERE ( x = 10 ) OR NOT ( x = 10 );

is not equivalent in SQL with

SELECT * FROM stuff;

if the column x contains any Nulls; in that case the second query would return some rows the first one does not return, namely all those in which x is Null. In classical two-valued logic, the law of the excluded middle would allow the simplification of the WHERE clause predicate, in fact its elimination. Attempting to apply the law of the excluded middle to SQL's 3VL is effectively a false dichotomy. The second query is actually equivalent with:

SELECT * FROM stuff;
-- is (because of 3VL) equivalent to:
SELECT * FROM stuff WHERE ( x = 10 ) OR NOT ( x = 10 ) OR x IS NULL;

Thus, to correctly simplify the first statement in SQL requires that we return all rows in which x is not null.

SELECT * FROM stuff WHERE x IS NOT NULL;

In view of the above, observe that for SQL's WHERE clause a tautology similar to the law of excluded middle can be written. Assuming the IS UNKNOWN operator is present, p OR (NOT p) OR (p IS UNKNOWN) is true for every predicate p. Among logicians, this is called law of excluded fourth.

There are some SQL expressions in which it is less obvious where the false dilemma occurs, for example:

SELECT 'ok' WHERE 1 NOT IN (SELECT CAST (NULL AS INTEGER))
UNION
SELECT 'ok' WHERE 1 IN (SELECT CAST (NULL AS INTEGER));

produces no rows because IN translates to an iterated version of equality over the argument set and 1<>NULL is Unknown, just as a 1=NULL is Unknown. (The CAST in this example is needed only in some SQL implementations like PostgreSQL, which would reject it with a type checking error otherwise. In many systems plain SELECT NULL works in the subquery.) The missing case above is of course:

SELECT 'ok' WHERE (1 IN (SELECT CAST (NULL AS INTEGER))) IS UNKNOWN;

Joins evaluate using the same comparison rules as for WHERE clauses. Therefore, care must be taken when using nullable columns in SQL join criteria. In particular a table containing any nulls is not equal with a natural self-join of itself, meaning that whereas ${\displaystyle R\bowtie R=R}$ is true for any relation R in relational algebra, a SQL self-join will exclude all rows having a Null anywhere.[15] An example of this behavior is given in the section analyzing the missing-value semantics of Nulls.

The SQL COALESCE function or CASE expressions can be used to "simulate" Null equality in join criteria, and the IS NULL and IS NOT NULL predicates can be used in the join criteria as well. The following predicate tests for equality of the values A and B and treats Nulls as being equal.

(A = B) OR (A IS NULL AND B IS NULL)

SQL provides two flavours of conditional expressions. One is called "simple CASE" and operates like a switch statement. The other is called a "searched CASE" in the standard, and operates like an if...elseif.

The simple CASE expressions use implicit equality comparisons which operate under the same rules as the DML WHERE clause rules for Null. Thus, a simple CASE expression cannot check for the existence of Null directly. A check for Null in a simple CASE expression always results in Unknown, as in the following:

SELECT CASE i WHEN NULL THEN 'Is Null'  -- This will never be returned
              WHEN    0 THEN 'Is Zero'  -- This will be returned when i = 0
              WHEN    1 THEN 'Is One'   -- This will be returned when i = 1
              END
FROM t;

Because the expression i = NULL evaluates to Unknown no matter what value column i contains (even if it contains Null), the string 'Is Null' will never be returned.

On the other hand, a "searched" CASE expression can use predicates like IS NULL and IS NOT NULL in its conditions. The following example shows how to use a searched CASE expression to properly check for Null:

SELECT CASE WHEN i IS NULL THEN 'Null Result'  -- This will be returned when i is NULL
            WHEN     i = 0 THEN 'Zero'         -- This will be returned when i = 0
            WHEN     i = 1 THEN 'One'          -- This will be returned when i = 1
            END
FROM t;

In the searched CASE expression, the string 'Null Result' is returned for all rows in which i is Null.

Oracle's dialect of SQL provides a built-in function DECODE which can be used instead of the simple CASE expressions and considers two nulls equal.

SELECT DECODE(i, NULL, 'Null Result', 0, 'Zero', 1, 'One') FROM t;

Finally, all these constructs return a NULL if no match is found; they have a default ELSE NULL clause.

SQL/PSM (SQL Persistent Stored Modules) defines procedural extensions for SQL, such as the IF statement. However, the major SQL vendors have historically included their own proprietary procedural extensions. Procedural extensions for looping and comparisons operate under Null comparison rules similar to those for DML statements and queries. The following code fragment, in ISO SQL standard format, demonstrates the use of Null 3VL in an IF statement.

IF i = NULL THEN
      SELECT 'Result is True'
ELSEIF NOT(i = NULL) THEN
      SELECT 'Result is False'
ELSE
      SELECT 'Result is Unknown';

The IF statement performs actions only for those comparisons that evaluate to True. For statements that evaluate to False or Unknown, the IF statement passes control to the ELSEIF clause, and finally to the ELSE clause. The result of the code above will always be the message 'Result is Unknown' since the comparisons with Null always evaluate to Unknown.

Constructs representing missing information, such as Codd tables, are actually intended to represent a set of relations, one for each possible instantiation of their parameters; in the case of Codd tables, this means replacement of Nulls with some concrete value. For example,

A construct (such as a Codd table) is said to be a strong representation system (of missing information) if any answer to a query made on the construct can be particularized to obtain an answer for any corresponding query on the relations it represents, which are seen as models of the construct. More precisely, if q is a query formula in the relational algebra (of "pure" relations) and if q is its lifting to a construct intended to represent missing information, a strong representation has the property that for any query q and (table) construct T, q lifts all the answers to the construct, i.e.:

${\displaystyle \mathop {\mathrm {Models} } ({\bar {q}}(T))=\{q(R)\,|R\in \mathop {\mathrm {Models} } (T)\}}$

(The above has to hold for queries taking any number of tables as arguments, but the restriction to one table suffices for this discussion.) Clearly Codd tables do not have this strong property if selections and projections are considered as part of the query language. For example, all the answers to

SELECT * FROM Emp WHERE Age = 22;

should include the possibility that a relation like EmpH22 may exist. However, Codd tables cannot represent the disjunction "result with possibly 0 or 1 rows". A device, mostly of theoretical interest, called conditional table (or c-table) can however represent such an answer:

where the condition column is interpreted as the row doesn't exist if the condition is false. It turns out that because the formulas in the condition column of a c-table can be arbitrary propositional logic formulas, an algorithm for the problem whether a c-table represents some concrete relation has a co-NP-complete complexity, thus is of little practical worth.

A weaker notion of representation is therefore desirable. Imielinski and Lipski introduced the notion of weak representation, which essentially allows (lifted) queries over a construct to return a representation only for sure information, i.e. if it's valid for all "possible world" instantiations (models) of the construct. Concretely, a construct is a weak representation system if

${\displaystyle \bigcap \mathop {\mathrm {Models} } ({\bar {q}}(T))=\bigcap \{q(R)\,|R\in \mathop {\mathrm {Models} } (T)\}}$

The right-hand side of the above equation is the sure information, i.e. information which can be certainly extracted from the database regardless of what values are used to replace Nulls in the database. In the example we considered above, it's easy to see that the intersection of all possible models (i.e. the sure information) of the query selecting WHERE Age = 22 is actually empty because, for instance, the (unlifted) query returns no rows for the relation EmpH37. More generally, it was shown by Imielinski and Lipski that Codd tables are a weak representation system if the query language is restricted to projections, selections (and renaming of columns). However, as soon as we add either joins or unions to the query language, even this weak property is lost, as evidenced in the next section.

Consider the following query over the same Codd table Emp from the previous section:

SELECT Name FROM Emp WHERE Age = 22
UNION
SELECT Name FROM Emp WHERE Age <> 22;

Whatever concrete value one would choose for the NULL age of Harriet, the above query will return the full column of names of any model of Emp, but when the (lifted) query is run on Emp itself, Harriet will always be missing, i.e. we have:

Thus when unions are added to the query language, Codd tables are not even a weak representation system of missing information, meaning that queries over them don't even report all sure information. It's important to note here that semantics of UNION on Nulls, which are discussed in a later section, did not even come into play in this query. The "forgetful" nature of the two sub-queries was all that it took to guarantee that some sure information went unreported when the above query was run on the Codd table Emp.

For natural joins, the example needed to show that sure information may be unreported by some query is slightly more complicated. Consider the table

and the query

SELECT F1, F3 FROM
     (SELECT F1, F2 FROM J) AS F12
     NATURAL JOIN
     (SELECT F2, F3 FROM J) AS F23;

The intuition for what happens above is that the Codd tables representing the projections in the subqueries lose track of the fact that the Nulls in the columns F12.F2 and F23.F2 are actually copies of the originals in the table J. This observation suggests that a relatively simple improvement of Codd tables (which works correctly for this example) would be to use Skolem constants (meaning Skolem functions which are also constant functions), say ω₁₂ and ω₂₂ instead of a single NULL symbol. Such an approach, called v-tables or Naive tables, is computationally less expensive that the c-tables discussed above. However, it is still not a complete solution for incomplete information in the sense that v-tables are only a weak representation for queries not using any negations in selection (and not using any set difference either). The first example considered in this section is using a negative selection clause, WHERE Age <> 22, so it is also an example where v-tables queries would not report sure information.

The NULLIF function accepts two parameters. If the first parameter is equal to the second parameter, NULLIF returns Null. Otherwise, the value of the first parameter is returned.

NULLIF(value1, value2)

Thus, NULLIF is an abbreviation for the following CASE expression:

CASE WHEN value1 = value2 THEN NULL ELSE value1 END

The COALESCE function accepts a list of parameters, returning the first non-Null value from the list:

COALESCE(value1, value2, value3, ...)

COALESCE is defined as shorthand for the following SQL CASE expression:

CASE WHEN value1 IS NOT NULL THEN value1
     WHEN value2 IS NOT NULL THEN value2
     WHEN value3 IS NOT NULL THEN value3
     ...
     END

Some SQL DBMSs implement vendor-specific functions similar to COALESCE. Some systems (e.g. Transact-SQL) implement an ISNULL function, or other similar functions that are functionally similar to COALESCE. (See Is functions for more on the IS functions in Transact-SQL.)

The Oracle NVL function accepts two parameters. It returns the first non-NULL parameter or NULL if all parameters are NULL.

A COALESCE expression can be converted into an equivalent NVL expression thus:

COALESCE ( val1, ... , val{n} )

turns into:

NVL( val1 , NVL( val2 , NVL( val3 , … , NVL ( val{n-1} , val{n} ) … )))

A use case of this function is to replace in an expression a NULL by a value like in NVL(SALARY, 0) which says, 'if SALARY is NULL, replace it with the value 0'.

There is, however, one notable exception. In most implementations, COALESCE evaluates its parameters until it reaches the first non-NULL one, while NVL evaluates all of its parameters. This is important for several reasons. A parameter after the first non-NULL parameter could be a function, which could either be computationally expensive, invalid, or could create unexpected side effects.

Misunderstanding of how Null works is the cause of a great number of errors in SQL code, both in ISO standard SQL statements and in the specific SQL dialects supported by real-world database management systems. These mistakes are usually the result of confusion between Null and either 0 (zero) or an empty string (a string value with a length of zero, represented in SQL as ''). Null is defined by the SQL standard as different from both an empty string and the numerical value 0, however. While Null indicates the absence of any value, the empty string and numerical zero both represent actual values.

A classic error is the attempt to use the equals operator = in combination with the keyword NULL to find rows with Nulls. According to the SQL standard this is an invalid syntax and shall lead to an error message or an exception. But most implementations accept the syntax and evaluate such expressions to UNKNOWN. The consequence is that no rows are found – regardless of whether rows with Nulls exist or not. The proposed way to retrieve rows with Nulls is the use of the predicate IS NULL instead of = NULL.

SELECT *
FROM sometable
WHERE num = NULL;  -- Should be "WHERE num IS NULL"

In a related, but more subtle example, a WHERE clause or conditional statement might compare a column's value with a constant. It is often incorrectly assumed that a missing value would be "less than" or "not equal to" a constant if that field contains Null, but, in fact, such expressions return Unknown. An example is below:

SELECT *
FROM sometable
WHERE num <> 1;  -- Rows where num is NULL will not be returned,
                 -- contrary to many users' expectations.

These confusions arise because the Law of Identity is restricted in SQL's logic. When dealing with equality comparisons using the NULL literal or the UNKNOWN truth-value, SQL will always return UNKNOWN as the result of the expression. This is a partial equivalence relation and makes SQL an example of a Non-Reflexive logic.[32]

Similarly, Nulls are often confused with empty strings. Consider the LENGTH function, which returns the number of characters in a string. When a Null is passed into this function, the function returns Null. This can lead to unexpected results, if users are not well versed in 3-value logic. An example is below:

SELECT *
FROM sometable
WHERE LENGTH(string) < 20; -- Rows where string is NULL will not be returned.

This is complicated by the fact that in some database interface programs (or even database implementations like Oracle's), NULL is reported as an empty string, and empty strings may be incorrectly stored as NULL.

The ISO SQL implementation of Null is the subject of criticism, debate and calls for change. In The Relational Model for Database Management: Version 2, Codd suggested that the SQL implementation of Null was flawed and should be replaced by two distinct Null-type markers. The markers he proposed were to stand for "Missing but Applicable" and "Missing but Inapplicable", known as A-values and I-values, respectively. Codd's recommendation, if accepted, would have required the implementation of a four-valued logic in SQL.[5] Others have suggested adding additional Null-type markers to Codd's recommendation to indicate even more reasons that a data value might be "Missing", increasing the complexity of SQL's logic system. At various times, proposals have also been put forth to implement multiple user-defined Null markers in SQL. Because of the complexity of the Null-handling and logic systems required to support multiple Null markers, none of these proposals have gained widespread acceptance.

Chris Date and Hugh Darwen, authors of The Third Manifesto, have suggested that the SQL Null implementation is inherently flawed and should be eliminated altogether,[33] pointing to inconsistencies and flaws in the implementation of SQL Null-handling (particularly in aggregate functions) as proof that the entire concept of Null is flawed and should be removed from the relational model.[34] Others, like author Fabian Pascal, have stated a belief that "how the function calculation should treat missing values is not governed by the relational model."

Another point of conflict concerning Nulls is that they violate the closed-world assumption model of relational databases by introducing an open-world assumption into it.[35] The closed world assumption, as it pertains to databases, states that "Everything stated by the database, either explicitly or implicitly, is true; everything else is false."[36] This view assumes that the knowledge of the world stored within a database is complete. Nulls, however, operate under the open world assumption, in which some items stored in the database are considered unknown, making the database's stored knowledge of the world incomplete.