This series of complementary textbooks cover all aspects of continuous optimization, and its connections with discrete optimization via duality. The two convex optimization books deal primarily with convex, possibly nondifferentiable, problems and rely on convex analysis. By contrast the nonlinear programming book focuses primarily on analytical and computational methods for possibly nonconvex differentiable problems. It relies primarily on calculus and variational analysis, yet it still contains a detailed presentation of duality theory and its uses for both convex and nonconvex problems.
The books share notation, style, and prerequisites, and are aimed at students, researchers, and practitioners, roughly at the first year graduate level. They rely on rigorous analysis, but they also aim at an intuitive exposition that makes use of geometric visualization. They are supported by substantial internet-accessible material, including exercises with solutions, lecture slides, and course content from MIT Open Courseware. Three other complementary and more specialized books of the author can also be freely downloaded: the 1998 Network Optimization book, the 1989 Parallel and Distributed Computation book (coauthored with J. Tsitsiklis), and the 1982 Constrained Optimization and Lagrange Multiplier Methods book.
Instructors that adopt any one book as a main textbook for a course may contact the publishing company Athena Scientific for a free adoption copy.
This book provides an up-to-date, comprehensive, and rigorous account of nonlinear programming at the first year graduate student level. It covers descent algorithms for unconstrained and constrained optimization, Lagrange multiplier theory, interior point and augmented Lagrangian methods for linear and nonlinear programs, duality theory, and major aspects of large-scale optimization.
The third edition of the book is a thoroughly rewritten version of the 1999 second edition. New material was included, some of the old material was discarded, and a large portion of the remainder was reorganized or revised. The new material covers a variety of topics, such as first order methods, proximal algorithms, alternating direction methods of multipliers, and conic programming; also large-scale optimization topics of much current interest, such as incremental methods, and distributed asynchronous computation, and their applications in machine learning, signal processing, neural network training, and big data applications.
Another objective of the third edition to integrate its contents with those of the other books, together with internet-accessible material, so that they complement each other and form a unified whole. To this end, bridges to the other books have been provided, with extensive references to generalizations, discussions, and elaborations of the analysis.
Click here to download the Table of Contents and the Preface.
Click here to order the print version of the book from amazon.com.
Click here to order the Ebook version of the book from Google Play.
Click here to visit the book's web site at Athena Scientific for exercise solutions, slides, and other instructional material.Click here to download the Newton Method code referenced in the last appendix of the book.
This book aims at an insightful, concise, and rigorous treatment of the basic theory of convex sets and functions in finite dimensions, and the analytical/geometrical foundations of convex optimization and duality theory. It provides much of the mathematical background needed for the in-depth reading of the Nonlinear Programming and Convex Optimization Algorithms books.
Convexity theory is first developed in a simple accessible manner, using easily visualized proofs. Then the focus shifts to a transparent geometrical line of analysis to develop the fundamental duality between descriptions of convex functions in terms of points, and in terms of hyperplanes. Finally, convexity theory and abstract duality are applied to problems of constrained optimization, Fenchel and conic duality, and game theory to develop the sharpest possible duality results within a highly visual geometric framework.
The book is aimed at students, researchers, and practitioners, roughly at the first year graduate level. It is similar in style to the author's 2015 Convex Optimization Algorithms book, but can be read independently. The latter book focuses on algorithmic issues, while the 2009 Convex Optimization Theory book focuses on convexity theory and optimization duality. The two books share notation, and together cover the entire finite-dimensional convex optimization methodology.
The paper Stochastic Optimization Problems with Nondifferentiable Cost Functionals, JOTA, 1973, on differentiation of the expected value of nondifferentiable convex functions.
Mathematical statements of Danskin's Theorem, on differentiation of the maximum of an infinite collection of convex functions.
The duality framework introduced in the "Convex Optimization Theory" book is defined by a single set and two simple associated geometrical problems: the min common point problem and the max crossing point problem. The salient feature of the min common/max crossing (MC/MC) framework is its highly visual geometry, through which all the core issues of duality theory become apparent and can be analyzed in a unified way. The approach is to obtain a handful of broadly applicable theorems within the MC/MC framework, specialize them to particular types of problems (constrained optimization, Fenchel duality, minimax problems, etc), and then address all duality questions (existence of duality gap, existence of dual optimal solutions, structure of the dual optimal solution set), and other issues (subdifferential theory, theorems of the alternative, duality gap estimates, etc).
Click here for the survey paper Min Common-Max Crossing Duality: A Geometric View of Conjugacy in Convex Optimization. Click here for A 60-Year Journey in Convex Optimization, a videolecture on the history and the evolution of the subject at MIT, which includes a review of the MC/MC abstract duality framework. Click here for the Slides from the lecture.
Click here to download the entire book with an on-line supplement: The supplement is an extensive set of theoretical exercises (a total of 150) with detailed high-quality solutions, which significantly extend the range and value of the book.
Click here for the Ebook version of the book from Google Play. The Ebook also includes the set of theoretical exercises with detailed high-quality solutions.
Click here to order the print version of the book from amazon.com.
Click here to visit the book's web site at Athena Scientific slides, and other instructional material, or to order the book directly from the publisher.
Click here for an abbreviated summary that consists of the Statements of all the Mathematical Results.
This book provides a comprehensive and accessible presentation of algorithms for solving convex optimization problems. It relies on rigorous mathematical analysis, but also aims at an intuitive exposition that makes use of visualization where possible. This is facilitated by the extensive use of analytical and algorithmic concepts of duality, which by nature lend themselves to geometrical interpretation. The book places particular emphasis on modern developments, and their widespread applications in fields such as large-scale resource allocation problems, signal processing, and machine learning.
The book is aimed at students, researchers, and practitioners, roughly at the first year graduate level. It is similar in style to the author's 2009 Convex Optimization Theory book, but can be read independently. The latter book focuses on convexity theory and optimization duality, while the 2015 Convex Optimization Algorithms book focuses on algorithmic issues. The two books share notation, and together cover the entire finite-dimensional convex optimization methodology. To facilitate readability, the statements of definitions and results of the "theory book" are reproduced without proofs in Appendix B.
The print version of the book is available from the publishing company Athena Scientific, or from Amazon.com. It is also available as an Ebook from Google Play.
Click here to visit the book's web site at Athena Scientific for exercises with solutions, slides, and other instructional material, or to order the book directly from the publisher. Click here to download Preface and Table of Contents, and Chapter 1
This research monograph, first published in 1982 by Academic Press, and republished by Athena Scientific in 1996, is a comprehensive treatment of some of the most widely used constrained optimization methods, including the augmented Lagrangian/multiplier and sequential quadratic programming methods.
"This is an excellent reference book. The author has done a great job in at least three directions. First, he expertly, systematically and with ever-present authority guides the reader through complicated areas of numerical optimization. This is achieved by carefully explaining and illustrating (by figures, if necessary) the underlying principles and theory. Second, he provides extensive guidance on the merits of various types of methods. This is extremely useful to practitioners. Finally, this is truly a state of the art book on numerical optimization."
S. Zlobec, McGill University, in SIAM Review
It can be purchased from Athena Scientific or as EBOOK from Google Play. It can also be freely downloaded in scanned form
Constrained Optimization and Lagrange Multiplier Methods
The Method of Multipliers for Equality Constrained Problems
The Method of Multipliers for Inequality Constrained and Nondifferentiable Optimization Problems
Exact Penalty Methods and Lagrangian Methods
Nonquadratic Penalty Functions - Convex Programming
References
Index