Convex optimization

A convex optimization problem is an optimization problem in which the objective function is a convex function and the feasible set is a convex set. A function ${\displaystyle f}$ mapping some subset of ${\displaystyle \mathbb {R} ^{n}}$into ${\displaystyle \mathbb {R} \cup \{\pm \infty \}}$ is convex if its domain is convex and for all ${\displaystyle \theta \in [0,1]}$ and all ${\displaystyle x,y}$ in its domain, the following condition holds: ${\displaystyle f(\theta x+(1-\theta )y)\leq \theta f(x)+(1-\theta )f(y)}$. A set S is convex if for all members ${\displaystyle x,y\in S}$ and all ${\displaystyle \theta \in [0,1]}$, we have that ${\displaystyle \theta x+(1-\theta )y\in S}$.

Concretely, a convex optimization problem is the problem of finding some ${\displaystyle \mathbf {x^{\ast }} \in C}$ attaining

${\displaystyle \inf\{f(\mathbf {x} ):\mathbf {x} \in C\}}$,

where the objective function ${\displaystyle f:{\mathcal {D}}\subseteq \mathbb {R} ^{n}\to \mathbb {R} }$ is convex, as is the feasible set ${\displaystyle C}$.[10] [11] If such a point exists, it is referred to as an optimal point or solution; the set of all optimal points is called the optimal set. If ${\displaystyle f}$ is unbounded below over ${\displaystyle C}$ or the infimum is not attained, then the optimization problem is said to be unbounded. Otherwise, if ${\displaystyle C}$ is the empty set, then the problem is said to be infeasible.[12]

The following are useful properties of convex optimization problems:[16][12]

• every local minimum is a global minimum;
• the optimal set is convex;
• if the objective function is strictly convex, then the problem has at most one optimal point.

These results are used by the theory of convex minimization along with geometric notions from functional analysis (in Hilbert spaces) such as the Hilbert projection theorem, the separating hyperplane theorem, and Farkas' lemma.

The following problem classes are all convex optimization problems, or can be reduced to convex optimization problems via simple transformations:[12][17]

A hierarchy of convex optimization problems. (LP: linear program, QP: quadratic program, SOCP second-order cone program, SDP: semidefinite program, CP: cone program.)

• Least squares
• Linear programming
• Convex quadratic minimization with linear constraints
• Quadratic minimization with convex quadratic constraints
• Conic optimization
• Geometric programming
• Second order cone programming
• Semidefinite programming
• Entropy maximization with appropriate constraints

Convex optimization has practical applications for the following.

• Portfolio optimization.[18]
• Worst-case risk analysis.[18]
• Optimal advertising.[18]
• Variations of statistical regression (including regularization and quantile regression).[18]
• Model fitting[18] (particularly multiclass classification[19]).
• Electricity generation optimization.[19]
• Combinatorial optimization.[19]
• Non-probabilistic modelling of uncertainty.[20]
• Localization using wireless signals [21]

Consider a convex minimization problem given in standard form by a cost function ${\displaystyle f(x)}$ and inequality constraints ${\displaystyle g_{i}(x)\leq 0}$ for ${\displaystyle 1\leq i\leq m}$. Then the domain ${\displaystyle {\mathcal {X}}}$ is:

${\displaystyle {\mathcal {X}}=\left\{x\in X\vert g_{1}(x),\ldots ,g_{m}(x)\leq 0\right\}.}$

The Lagrangian function for the problem is

${\displaystyle L(x,\lambda _{0},\lambda _{1},\ldots ,\lambda _{m})=\lambda _{0}f(x)+\lambda _{1}g_{1}(x)+\cdots +\lambda _{m}g_{m}(x).}$

For each point ${\displaystyle x}$ in ${\displaystyle X}$ that minimizes ${\displaystyle f}$ over ${\displaystyle X}$, there exist real numbers ${\displaystyle \lambda _{0},\lambda _{1},\ldots ,\lambda _{m},}$ called Lagrange multipliers, that satisfy these conditions simultaneously:

1. ${\displaystyle x}$ minimizes ${\displaystyle L(y,\lambda _{0},\lambda _{1},\ldots ,\lambda _{m})}$ over all ${\displaystyle y\in X,}$
2. ${\displaystyle \lambda _{0},\lambda _{1},\ldots ,\lambda _{m}\geq 0,}$ with at least one ${\displaystyle \lambda _{k}>0,}$
3. ${\displaystyle \lambda _{1}g_{1}(x)=\cdots =\lambda _{m}g_{m}(x)=0}$ (complementary slackness).

If there exists a "strictly feasible point", that is, a point ${\displaystyle z}$ satisfying

${\displaystyle g_{1}(z),\ldots ,g_{m}(z)<0,}$

then the statement above can be strengthened to require that ${\displaystyle \lambda _{0}=1}$.

Conversely, if some ${\displaystyle x}$ in ${\displaystyle X}$ satisfies (1)–(3) for scalars ${\displaystyle \lambda _{0},\ldots ,\lambda _{m}}$ with ${\displaystyle \lambda _{0}=1}$ then ${\displaystyle x}$ is certain to minimize ${\displaystyle f}$ over ${\displaystyle X}$.

Unconstrained convex optimization can be easily solved with gradient descent (a special case of steepest descent) or Newton's method, combined with line search for an appropriate step size; these can be mathematically proven to converge quickly, especially the latter method.[22] Convex optimization with linear equality constraints can also be solved using KKT matrix techniques if the objective function is a quadratic function (which generalizes to a variation of Newton's method, which works even if the point of initialization does not satisfy the constraints), but can also generally be solved by eliminating the equality constraints with linear algebra or solving the dual problem.[22] Finally, convex optimization with both linear equality constraints and convex inequality constraints can be solved by applying an unconstrained convex optimization technique to the objective function plus logarithmic barrier terms.[22] (When the starting point is not feasible - that is, satisfying the constraints - this is preceded by so-called phase I methods, which either find a feasible point or show that none exist. Phase I methods generally consist of reducing the search in question to yet another convex optimization problem.[22])

Convex optimization problems can also be solved by the following contemporary methods:[23]

• Bundle methods (Wolfe, Lemaréchal, Kiwiel), and
• Subgradient projection methods (Polyak),
• Interior-point methods,[1] which make use of self-concordant barrier functions [24] and self-regular barrier functions.[25]
• Cutting-plane methods
• Ellipsoid method
• Subgradient method
• Dual subgradients and the drift-plus-penalty method

Subgradient methods can be implemented simply and so are widely used.[26] Dual subgradient methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables.

Extensions of convex optimization include the optimization of biconvex, pseudo-convex, and quasiconvex functions. Extensions of the theory of convex analysis and iterative methods for approximately solving non-convex minimization problems occur in the field of generalized convexity, also known as abstract convex analysis.

• Duality
• Karush–Kuhn–Tucker conditions
• Optimization problem
• Proximal gradient method

• Bertsekas, Dimitri P.; Nedic, Angelia; Ozdaglar, Asuman (2003). Convex Analysis and Optimization. Belmont, MA.: Athena Scientific. ISBN 978-1-886529-45-8.
• Bertsekas, Dimitri P. (2009). Convex Optimization Theory. Belmont, MA.: Athena Scientific. ISBN 978-1-886529-31-1.
• Bertsekas, Dimitri P. (2015). Convex Optimization Algorithms. Belmont, MA.: Athena Scientific. ISBN 978-1-886529-28-1.
• Borwein, Jonathan; Lewis, Adrian (2000). Convex Analysis and Nonlinear Optimization: Theory and Examples, Second Edition (PDF). Springer. Retrieved 12 Apr 2021.
• Christensen, Peter W.; Anders Klarbring (2008). An introduction to structural optimization. Vol. 153. Springer Science & Business Media. ISBN 9781402086663.

• Hiriart-Urruty, Jean-Baptiste, and Lemaréchal, Claude. (2004). Fundamentals of Convex analysis. Berlin: Springer.
• Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). Convex analysis and minimization algorithms, Volume I: Fundamentals. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Vol. 305. Berlin: Springer-Verlag. pp. xviii+417. ISBN 978-3-540-56850-6. MR 1261420.
• Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). Convex analysis and minimization algorithms, Volume II: Advanced theory and bundle methods. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Vol. 306. Berlin: Springer-Verlag. pp. xviii+346. ISBN 978-3-540-56852-0. MR 1295240.
• Kiwiel, Krzysztof C. (1985). Methods of Descent for Nondifferentiable Optimization. Lecture Notes in Mathematics. New York: Springer-Verlag. ISBN 978-3-540-15642-0.
• Lemaréchal, Claude (2001). "Lagrangian relaxation". In Michael Jünger and Denis Naddef (ed.). Computational combinatorial optimization: Papers from the Spring School held in Schloß Dagstuhl, May 15–19, 2000. Lecture Notes in Computer Science. Vol. 2241. Berlin: Springer-Verlag. pp. 112–156. doi:10.1007/3-540-45586-8_4. ISBN 978-3-540-42877-0. MR 1900016. S2CID 9048698.
• Nesterov, Yurii; Nemirovskii, Arkadii (1994). Interior Point Polynomial Methods in Convex Programming. SIAM.
• Nesterov, Yurii. (2004). Introductory Lectures on Convex Optimization, Kluwer Academic Publishers
• Rockafellar, R. T. (1970). Convex analysis. Princeton: Princeton University Press.

• Ruszczyński, Andrzej (2006). Nonlinear Optimization. Princeton University Press.
• Schmit, L.A.; Fleury, C. 1980: Structural synthesis by combining approximation concepts and dual methods. J. Amer. Inst. Aeronaut. Astronaut 18, 1252-1260

Wikimedia Commons has media related to Convex optimization.

• EE364a: Convex Optimization I and EE364b: Convex Optimization II, Stanford course homepages
• 6.253: Convex Analysis and Optimization, an MIT OCW course homepage
• Brian Borchers, An overview of software for convex optimization
• Convex Optimization Book by Lieven Vandenberghe and Stephen P. Boyd