• Multi-Objective Optimization
  • Pareto Optimality
    • Pareto Frontier
      • Solving for it
        • Constraint Method
        • Lexicographic Method
        • Weighted Method
        • Goal Programming
    • Utopia Point
  • Multi-Objective Population Method
    • Classic method
    • Non-Dominating Ranking
    • Pareto filter
    • Niche Technique
• Preference Elicitation
• Sampling Plans
  • Full Factorial
  • Random Sampling
  • Uniform Projection
  • Stratified Sampling
• Space-Filling Metrics
  • Pairwise Distances
  • Morris-Mitchell
• Space-Filling Subset
• Surrogate Models
  • Linear Model
  • Basis Functions
  • Regularization
• Probabilistic Surrogate Models
  • Gaussian Process
    • squared exponential kernel
• Surrogate Optimization
  • Prediction Based Exploration
  • Error Based Exploration
  • Lower Confidence Bound Exploration
  • Probability of Improvement Exploration and Expected Improvement Exploration
• optimization uncertainty
  • set-based uncertainty
  • probabilistic uncertainty
    • uncertainty expected value optimization
    • uncertainty variance optimization

Introduction

Reinforcement Learning with Human Feedback (RLHF) has demonstrated superb effect for aligning the performance of a language model (LM) against unsupervised human preference judgements of LM output trajectories ((Ziegler et al. 2020)). However, the original RLHF formulation has shown little direct improvements to the model’s toxicity without further prompting, yet conferred some advantage when prompted to specifically be respectful ((Ouyang et al. 2022)).

To specifically target the problem of the reduction of harmfulness in LM toxicity, varying approaches have been explored via contrastive learning ((Lu et al., n.d.)), or—though a combination of instruction-following RLHF and in-context learning (i.e. prompting)—sampling and LM self-correcting output trajectories ((Ganguli et al. 2023)).

• Stochastic Methods
  • Noisy Descent
  • Mesh Adaptive Direct Search
  • simulated annealing
    • Metropolis Criteria
    • Simulated Annealing Schedule
    • Cross Entropy Method
  • Natural Evolution
• Population Methods
  • Generic Algorithms
  • Particle Swarm Optimization
• constraints
  • active constraint
  • region-based constraints
  • Lagrange multiplier
    • primal problem
    • KKT Conditions
    • dual form of the primal problem
  • Penalty Methods
  • Interior Point Method
• Linear Program
  • FONC for linear program
  • linear program equality form
  • FONC for linear program
  • simplex algorithm
  • Dual Certificates

• Formal Formulation of Optimization
• constraint
• types of conditions
  • FONC and SONC
• Derivatives
  • Directional Derivatives
  • numerical methods
    • Finite-Difference Method
      • Forward Difference
      • Central Difference
      • Backward Difference
    • Complex-Difference Method
  • exact methods: autodiff
    • Forward Accumulation
    • cooool: Dual Number Method
• Bracketing (one dimensional optimization schemes)
  • Fibonacci Search
  • Quadratic Search
  • Shubert-Piyavskill Method
• Descent Direction Iteration
  • Line Search
  • Approximate Line Search
    • Sufficient Decrease Condition
    • Curvature Condition
  • Trust Region Methods
• First-Order Methods
  • good ol gradient descent
  • Conjugate Gradient
  • Hyper-gradient Descent
• Second-Order Methods
  • Newton’s Method
  • or approximate it using Secant Method
• Direct Methods
  • Cyclic Coordinate Search
  • Accelerated Coordinate Search
  • Powell’s Method
  • Hooke-Jeeves Search
  • Generalized Pattern Search
    • opportunistic search
    • dynamic ordering
  • Nelder-Mead Simplex Method