• Formal Formulation of Optimization
• constraint
• types of conditions
  • FONC and SONC
• Derivatives
  • Directional Derivatives
  • numerical methods
    • Finite-Difference Method
      • Forward Difference
      • Central Difference
      • Backward Difference
    • Complex-Difference Method
  • exact methods: autodiff
    • Forward Accumulation
    • cooool: Dual Number Method
• Bracketing (one dimensional optimization schemes)
  • Fibonacci Search
  • Quadratic Search
  • Shubert-Piyavskill Method
• Descent Direction Iteration
  • Line Search
  • Approximate Line Search
    • Sufficient Decrease Condition
    • Curvature Condition
  • Trust Region Methods
• First-Order Methods
  • good ol gradient descent
  • Conjugate Gradient
  • Hyper-gradient Descent
• Second-Order Methods
  • Newton’s Method
  • or approximate it using Secant Method
• Direct Methods
  • Cyclic Coordinate Search
  • Accelerated Coordinate Search
  • Powell’s Method
  • Hooke-Jeeves Search
  • Generalized Pattern Search
    • opportunistic search
    • dynamic ordering
  • Nelder-Mead Simplex Method