SA
Skip to main content

Maximum Likelihood Estimation

f(D∣θ)f(\mathbb{D} | \theta)

θ^ML=argmaxθg(x∣θ)=argmaxθln⁡g(x∣θ)\hat{\theta}_\text{ML} = \text{argmax}_{\theta} g(x|\theta) = \text{argmax}_{\theta} \ln g(x | \theta)

θ^ML=argmaxθg(x1, x2, ⋯ , xn∣θ)\hat{\theta}_\text{ML} = \text{argmax}_{\theta} g(x_1,~x_2,~\cdots,~x_n | \theta) =argmaxθ∏k=1ng(xk∣θ)= \text{argmax}_{\theta} \prod\limits_{k=1}^{n} g(x_k|\theta) — i.i.d. / r.s. =argmaxθ∑k=1nln⁡g(xk∣θ)= \text{argmax}_{\theta} \sum\limits_{k=1}^{n} \ln g(x_k|\theta)

∂L∂θ∣θ=θ^ML=0{\partial L \over \partial \theta} |_{\theta = \hat\theta_\text{ML}} = 0

∴\therefore Check ∂L∂θ∣θ=θ^ML<0{\partial L \over \partial \theta} |_{\theta = \hat\theta_\text{ML}} < 0

h(θ)^ML=h(θ^ML)\hat {h(\theta)}^\text{ML} = h(\hat\theta^\text{ML})

x1,⋯ ,xn∼Geometric(P)x_1, \cdots, x_n \sim \text{Geometric} (P) σ2^ML\hat{\sigma^2}^\text{ML}

1p^=Xn‾⇒p^=1x‾\hat{1 \over p} = \overline{X_n} \Rightarrow \hat{p} = {1 \over \overline{x}}

σ2^ML=qp2=1−pp2^ML=1−1xn‾1xn‾2{\hat{\sigma^2}}^\text{ML} = {q \over p^2} = {\hat{{1-p} \over p^2}}^\text{ML} = 1 - {{1 \over \overline{x_n}} \over {{1 \over \overline{x_n}}^2}}

Max-likelihood: Tries to give the best PDF.

Max-likelihood parameter as θ^\hat \theta

θ^ML=argmaxθf(x1,x2,⋯xn∣θ)=argmaxθln⁡f(x1,x2,⋯xn∣θ)=argmaxθL \hat \theta ^\text{ML} = \text{argmax}_{\theta} f(x_1, x_2, \cdots x^n | \theta) = \text{argmax}_{\theta} \ln f(x_1, x_2, \cdots x^n | \theta) = \text{argmax}_{\theta} L

Assuming IID

=ln⁡∏k=1nf(xk∣θ)=argmaxθ∑k=1nln⁡f(xk∣θ) = \ln \prod_{k=1}^{n} f(x_k | \theta) = \text{argmax}_{\theta} \sum_{k=1}^{n} \ln f(x_k | \theta)

Maximum Likelihood Estimation​

  1. consistent (convergent in probability)
  2. Asymptotically Normal
  3. Invariance Principle g(θ)ML^=g(θ^ML)\hat{g(\theta)_{ML}} = g(\hat\theta_{ML})

Links to This Note

Probability

- Maximum Likelihood Estimation

Maximum A Posteriori

... be thought of as "regularized" MLE (?)

2023-03-27

Maximum Likelihood Estimation