Skip to main content

Maximum Likelihood Estimation

f(Dโˆฃฮธ)f(\mathbb{D} | \theta)

ฮธ^ML=argmaxฮธg(xโˆฃฮธ)=argmaxฮธlnโกg(xโˆฃฮธ)\hat{\theta}_\text{ML} = \text{argmax}_{\theta} g(x|\theta) = \text{argmax}_{\theta} \ln g(x | \theta)

ฮธ^ML=argmaxฮธg(x1,ย x2,ย โ‹ฏโ€‰,ย xnโˆฃฮธ)\hat{\theta}_\text{ML} = \text{argmax}_{\theta} g(x_1,~x_2,~\cdots,~x_n | \theta) =argmaxฮธโˆk=1ng(xkโˆฃฮธ)= \text{argmax}_{\theta} \prod\limits_{k=1}^{n} g(x_k|\theta) โ€” i.i.d. / r.s. =argmaxฮธโˆ‘k=1nlnโกg(xkโˆฃฮธ)= \text{argmax}_{\theta} \sum\limits_{k=1}^{n} \ln g(x_k|\theta)

โˆ‚Lโˆ‚ฮธโˆฃฮธ=ฮธ^ML=0{\partial L \over \partial \theta} |_{\theta = \hat\theta_\text{ML}} = 0

โˆด\therefore Check โˆ‚Lโˆ‚ฮธโˆฃฮธ=ฮธ^ML<0{\partial L \over \partial \theta} |_{\theta = \hat\theta_\text{ML}} < 0

h(ฮธ)^ML=h(ฮธ^ML)\hat {h(\theta)}^\text{ML} = h(\hat\theta^\text{ML})

x1,โ‹ฏโ€‰,xnโˆผGeometric(P)x_1, \cdots, x_n \sim \text{Geometric} (P) ฯƒ2^ML\hat{\sigma^2}^\text{ML}

1p^=Xnโ€พโ‡’p^=1xโ€พ\hat{1 \over p} = \overline{X_n} \Rightarrow \hat{p} = {1 \over \overline{x}}

ฯƒ2^ML=qp2=1โˆ’pp2^ML=1โˆ’1xnโ€พ1xnโ€พ2{\hat{\sigma^2}}^\text{ML} = {q \over p^2} = {\hat{{1-p} \over p^2}}^\text{ML} = 1 - {{1 \over \overline{x_n}} \over {{1 \over \overline{x_n}}^2}}

Max-likelihood: Tries to give the best PDF.

Max-likelihood parameter as ฮธ^\hat \theta

ฮธ^ML=argmaxฮธf(x1,x2,โ‹ฏxnโˆฃฮธ)=argmaxฮธlnโกf(x1,x2,โ‹ฏxnโˆฃฮธ)=argmaxฮธL \hat \theta ^\text{ML} = \text{argmax}_{\theta} f(x_1, x_2, \cdots x^n | \theta) = \text{argmax}_{\theta} \ln f(x_1, x_2, \cdots x^n | \theta) = \text{argmax}_{\theta} L

Assuming IID

=lnโกโˆk=1nf(xkโˆฃฮธ)=argmaxฮธโˆ‘k=1nlnโกf(xkโˆฃฮธ) = \ln \prod_{k=1}^{n} f(x_k | \theta) = \text{argmax}_{\theta} \sum_{k=1}^{n} \ln f(x_k | \theta)

Maximum Likelihood Estimationโ€‹

  1. consistent (convergent in probability)
  2. Asymptotically Normal
  3. Invariance Principle g(ฮธ)ML^=g(ฮธ^ML)\hat{g(\theta)_{ML}} = g(\hat\theta_{ML})