Regularization, Normalization

---

Normalization(정규화)

Scaling input features so that they have similar ranges or distributions. 

- 데이터 전처리 과정 중 하나.

- 데이터의 형태를 좀 더 의미있고, 학습에 적합하게 만드는 과정

- z-score, minmax scaler로 value를 0-1 사이로 분포를 조정

 

Normalization refers to scaling input features so that they have similar ranges or distributions. This can help prevent certain features from dominating others during training which could lead to biased model predictions. Normalizing inputs also makes it easier for optimization algorithms such as gradient descent to converge faster towards optimal solutions by reducing oscillations in loss function values.

 

---

Regularization(정칙화)

Involves adding a penalty term (e.g., L1/L2 norm) into objective functions used during training process with aim of discouraging overfitting

-  오버피팅(과적합) 해결을 위한 방법에 속함

-  train loss는 약간 증가하더라도 결과적으로 val loss와 test loss 감소가 목적임

-  L1, L2 Regularization, Dropout, Batch normalization

 

Regularization on the other hand involves adding a penalty term (e.g., L1/L2 norm) into objective functions used during training process with aim of discouraging overfitting i.e., when model becomes too complex/expressive relative amount data available leading poor generalizability beyond seen examples. Regularized models tend generalize better than non-regularized ones because they are less prone memorize noise present in dataset while still capturing important patterns underlying target variable(s).

 

---

BatchNormalization layer 의 위치

일반적으로는 Dense layer(activation function like ReLU)의 앞에 적용된다.

입력값이 활성화 함수로 전달되기 전에 정규화 되기를 목적으로 하기 때문이다.

입력데이터의 평균과 분산을 계싼하고 이를 정규화하게 되면, 

가중치 초기화와 함께 학습속도를 높이고 안정성도 향상시킬 수 있다.

 

 

https://subinium.github.io/introduction-to-normalization/

Introduction to Deep Learning Normalization