---
id: 2023-12-17
aliases: December 17, 2023
tags:
- link-note
- Data-Science
- Machine-Learning
- Optimization
---

# Optimization

## Math

### Partial Differentiation/Derivative

- Differentiate about a specific variable
- Consider others as constants
- $\dfrac{\partial y}{\partial x}$
- e.g., $f(x,y) = x^2 + xy + 3$

### Chain Rule

- $\dfrac{dy}{dx} = \dfrac{dy}{du}*\dfrac{du}{dx}$
- e.g., $y = ln(u), u = 2x + 4$

## Loss Function

### Mean Squared Error (MSE)

- $L = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y_i})^2$

## Parameter Calculation

### Least Square Method (LSM)

- Minimize error of data
- a: slope (coefficient)
- b: intercept
- $L = \sum_{i=1}^{N} (y_i - (ax_i + b))^2$

#### Method 1.

- $0 = \dfrac{\partial L}{\partial a} = \sum_{i=1}^{N} 2(y_i - (ax_i + b))(-x_i) = 2(a\sum_{i=1}^{N} x_i^2 + b\sum_{i=1}^{N} x_i - \sum_{i=1}^{N} x_iy_i)$
- $0 = \dfrac{\partial L}{\partial b} = \sum_{i=1}^{N} 2(y_i - (ax_i + b))(-1) = 2(a\sum_{i=1}^{N} x_i + b\sum_{i=1}^{N}1 - \sum_{i=1}^{N} y_i)$
- $a^* = \dfrac{\sum_{i=1}^{N}(x-\bar{x})(y-\bar{y})}{\sum_{i=1}^{N}(x-\bar{x})^2}$
- $b^* = \bar{y} - a^*\bar{x}$

#### Method 2.

- Partial differentiation with respect to matrix $||Y - WX||^2$
- $-2X^T(Y-WX) = 0$
- $W = (X^TX)^{-1}X^TY$