Gradient Descent and Convex Intuition TODO
Concept
Gradient descent is an optimization method that exploits the fact that the gradient of the objective function points in the direction of steepest ascent, and repeatedly moves a step of size equal to the learning rate in the opposite direction. A function is convex if the line segment joining any two points in its domain always lies on or above the function's graph; this property guarantees that a local minimum is also the global minimum, and that a point where the gradient is zero is optimal. For a smooth convex function whose gradient is L-Lipschitz continuous, choosing a learning rate at or below 1/L guarantees convergence, and adding strong convexity yields a faster, exponentially decaying error rate. A learning rate that's too large causes divergence, one that's too small is slow, and poor conditioning causes zig-zagging in narrow, elongated valleys — which is why techniques like momentum, adaptive learning rates, and preconditioning are used to mitigate it.
In numerical optimization beyond machine learning — market-making parameter tuning, calibration — you need to tell whether a failure to converge comes from the problem's non-convexity or from a learning-rate/conditioning issue.
Code & Formula
# Day 29 — Gradient Descent / Convex 직관
# 볼록함수 f(x) = (x - 3)^2 위에서 경사하강법을 돌려 최소점 x=3 으로 수렴하는 과정을 본다.
def f(x):
return (x - 3) ** 2
def f_prime(x):
return 2 * (x - 3)
x = 10.0 # 시작점
learning_rate = 0.1
history = [x]
for step in range(30):
grad = f_prime(x)
x = x - learning_rate * grad
history.append(x)
print("경사하강 진행 (x, f(x)):")
for i in [0, 1, 2, 5, 10, 20, len(history) - 1]:
xi = history[i]
print(f" step {i:2d}: x = {xi:.6f}, f(x) = {f(xi):.8f}")
print(f"\n최종 x = {history[-1]:.6f} (참값 x* = 3)")
print(f"최종 f(x) = {f(history[-1]):.10f} (참값 f(x*) = 0)")
Exercise
Construct a 2D quadratic function with a deliberately poor condition number, run both plain and momentum-based gradient descent on it across several learning rates, and compare the iteration counts and trajectories visually.
Practical Connection
The LMSR cost function is convex, so properties like well-defined prices and a bounded loss follow directly from convexity, and this same optimization toolkit is exactly what you use when calibrating parameters against real data.
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/math-50-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.