Integer factorization

No algorithm has been published that can factor all integers in polynomial time, that is, that can factor a b-bit number n in time O(b^k) for some constant k. Neither the existence nor non-existence of such algorithms has been proved, but it is generally suspected that they do not exist and hence that the problem is not in class P.[4][5] The problem is clearly in class NP, but it is generally suspected that it is not NP-complete, though this has not been proven.[6]

There are published algorithms that are faster than O((1 + ε)^b) for all positive ε, that is, sub-exponential. As of 2022, the algorithm with best theoretical asymptotic running time is the general number field sieve (GNFS), first published in 1993,[7] running on a b-bit number n in time:

${\displaystyle \exp \left(\left({\sqrt[{3}]{\frac {64}{9}}}+o(1)\right)(\ln n)^{\frac {1}{3}}(\ln \ln n)^{\frac {2}{3}}\right).}$

For current computers, GNFS is the best published algorithm for large n (more than about 400 bits). For a quantum computer, however, Peter Shor discovered an algorithm in 1994 that solves it in polynomial time. This will have significant implications for cryptography if quantum computation becomes scalable. Shor's algorithm takes only O(b³) time and O(b) space on b-bit number inputs. In 2001, Shor's algorithm was implemented for the first time, by using NMR techniques on molecules that provide 7 qubits.[8]

It is not known exactly which complexity classes contain the decision version of the integer factorization problem (that is: does n have a factor smaller than k?). It is known to be in both NP and co-NP, meaning that both "yes" and "no" answers can be verified in polynomial time. An answer of "yes" can be certified by exhibiting a factorization n = d(n/d) with d ≤ k. An answer of "no" can be certified by exhibiting the factorization of n into distinct primes, all larger than k; one can verify their primality using the AKS primality test, and then multiply them to obtain n. The fundamental theorem of arithmetic guarantees that there is only one possible string of increasing primes that will be accepted, which shows that the problem is in both UP and co-UP.[9] It is known to be in BQP because of Shor's algorithm.

The problem is suspected to be outside all three of the complexity classes P, NP-complete, and co-NP-complete. It is therefore a candidate for the NP-intermediate complexity class. If it could be proved to be either NP-complete or co-NP-complete, this would imply NP = co-NP, a very surprising result, and therefore integer factorization is widely suspected to be outside both these classes.

In contrast, the decision problem "Is n a composite number?" (or equivalently: "Is n a prime number?") appears to be much easier than the problem of specifying factors of n. The composite/prime problem can be solved in polynomial time (in the number b of digits of n) with the AKS primality test. In addition, there are several probabilistic algorithms that can test primality very quickly in practice if one is willing to accept a vanishingly small possibility of error. The ease of primality testing is a crucial part of the RSA algorithm, as it is necessary to find large prime numbers to start with.

A special-purpose factoring algorithm's running time depends on the properties of the number to be factored or on one of its unknown factors: size, special form, etc. The parameters which determine the running time vary among algorithms.

An important subclass of special-purpose factoring algorithms is the Category 1 or First Category algorithms, whose running time depends on the size of smallest prime factor. Given an integer of unknown form, these methods are usually applied before general-purpose methods to remove small factors.[10] For example, naive trial division is a Category 1 algorithm.

• Trial division
• Wheel factorization
• Pollard's rho algorithm, which has two common flavors to identify group cycles: one by Floyd and one by Brent.
• Algebraic-group factorization algorithms, among which are Pollard's p − 1 algorithm, Williams' p + 1 algorithm, and Lenstra elliptic curve factorization
• Fermat's factorization method
• Euler's factorization method
• Special number field sieve

A general-purpose factoring algorithm, also known as a Category 2, Second Category, or Kraitchik family algorithm,[10] has a running time which depends solely on the size of the integer to be factored. This is the type of algorithm used to factor RSA numbers. Most general-purpose factoring algorithms are based on the congruence of squares method.

• Dixon's algorithm
• Continued fraction factorization (CFRAC)
• Quadratic sieve
• Rational sieve
• General number field sieve
• Shanks's square forms factorization (SQUFOF)

• Shor's algorithm, for quantum computers

Given an integer n that will be factored, where n is an odd positive integer greater than a certain constant. In this factoring algorithm the discriminant Δ is chosen as a multiple of n, Δ = −dn, where d is some positive multiplier. The algorithm expects that for one d there exist enough smooth forms in G_Δ. Lenstra and Pomerance show that the choice of d can be restricted to a small set to guarantee the smoothness result.

Denote by P_Δ the set of all primes q with Kronecker symbol ${\displaystyle \left({\tfrac {\Delta }{q}}\right)=1}$. By constructing a set of generators of G_Δ and prime forms f_q of G_Δ with q in P_Δ a sequence of relations between the set of generators and f_q are produced. The size of q can be bounded by ${\displaystyle c_{0}(\log |\Delta |)^{2}}$ for some constant ${\displaystyle c_{0}}$.

The relation that will be used is a relation between the product of powers that is equal to the neutral element of G_Δ. These relations will be used to construct a so-called ambiguous form of G_Δ, which is an element of G_Δ of order dividing 2. By calculating the corresponding factorization of Δ and by taking a gcd, this ambiguous form provides the complete prime factorization of n. This algorithm has these main steps:

Let n be the number to be factored.

1. Let Δ be a negative integer with Δ = −dn, where d is a multiplier and Δ is the negative discriminant of some quadratic form.
2. Take the t first primes ${\displaystyle p_{1}=2,p_{2}=3,p_{3}=5,\dots ,p_{t}}$, for some ${\displaystyle t\in {\mathbb {N} }}$.
3. Let ${\displaystyle f_{q}}$ be a random prime form of G_Δ with ${\displaystyle \left({\tfrac {\Delta }{q}}\right)=1}$.
4. Find a generating set X of G_Δ
5. Collect a sequence of relations between set X and {f_q : q ∈ P_Δ} satisfying: ${\displaystyle \left(\prod _{x\in X_{}}x^{r(x)}\right).\left(\prod _{q\in P_{\Delta }}f_{q}^{t(q)}\right)=1}$
6. Construct an ambiguous form ${\displaystyle (a,b,c)}$ that is an element f ∈ G_Δ of order dividing 2 to obtain a coprime factorization of the largest odd divisor of Δ in which ${\displaystyle \Delta =-4ac{\text{ or }}a(a-4c){\text{ or }}(b-2a)(b+2a)}$
7. If the ambiguous form provides a factorization of n then stop, otherwise find another ambiguous form until the factorization of n is found. In order to prevent useless ambiguous forms from generating, build up the 2-Sylow group Sll₂(Δ) of G(Δ).

To obtain an algorithm for factoring any positive integer, it is necessary to add a few steps to this algorithm such as trial division, and the Jacobi sum test.

The algorithm as stated is a probabilistic algorithm as it makes random choices. Its expected running time is at most ${\displaystyle L_{n}\left[{\tfrac {1}{2}},1+o(1)\right]}$.[14]