lecture-8

Prévia do material em texto

A Studocu não é patrocinada ou endossada por nenhuma faculdade ou universidade
Lecture 8
Statistical Theory Ii (Missouri State University)
A Studocu não é patrocinada ou endossada por nenhuma faculdade ou universidade
Lecture 8
Statistical Theory Ii (Missouri State University)
Baixado por Ênio de Paula (jorgepont9999@gmail.com)
lOMoARcPSD|8960359
https://www.studocu.com/en-us?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
https://www.studocu.com/en-us/document/missouri-state-university/statistical-theory-ii/lecture-8/1029678?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
https://www.studocu.com/en-us/course/missouri-state-university/statistical-theory-ii/736983?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
https://www.studocu.com/en-us?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
https://www.studocu.com/en-us/document/missouri-state-university/statistical-theory-ii/lecture-8/1029678?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
https://www.studocu.com/en-us/course/missouri-state-university/statistical-theory-ii/736983?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
Math 541: Statistical Theory II
Connection between Method of Moment and Maximum Likelihood
Lecturer: Songfeng Zheng
In the parameter estimation problem, we have an i.i.d. random sample X1, · · · , Xn from the
probability distribution f(x|θ) with unknown parameter(s) θ. f(x|θ) could either be density
function or probability mass function depending on the problem at hand. The purpose is to
estimate the unknown parameter(s) θ from the sample X1, · · · , Xn.
We have so far investigated two methods for estimating parameters, namely, method of
moment and maximum likelihood. The two seemingly different methods have connections
between them.
Recall that in the method of moment, we estimate the parameter(s) by solving the equa-
tion(s):
E(Xk) = µk = mk =
1
n
n
∑
i=1
Xk
i
i.e.
1
n
n
∑
i=1
Xk
i
− µk = 0 (1)
While in the method of maximum likelihood, we want to maximize the log-likelihood function
with respect to the unknown parameter
l(θ) =
n
∑
i=1
log f(Xi|θ)
and this is equivalent to solving the following equation
n
∑
i=1
∂
∂θ
log f(Xi|θ) = 0 (2)
We can generalize the method of moment in the following way: let h(x) be a real-valued
function such that E[h(X)] exists. We define
µh = E[h(X)] =
∫
h(x)f(x|θ)dx and mh =
1
n
n
∑
i=1
h(Xi)
Usually, µh is a function of the unknown parameter θ. Solving the equation
µh = mh
1
Baixado por Ênio de Paula (jorgepont9999@gmail.com)
lOMoARcPSD|8960359
https://www.studocu.com/en-us?utm_campaign=shared-document&utm_source=studocu-document&utm_medium=social_sharing&utm_content=lecture-8
2
will give us an estimate of θ. Please note that if we let h(x) = xk − µk we will have the
original method of moment because in this case µh = E[h(X)] = E(X
k −µk) = µk −µk = 0;
mh =
1
n
∑
n
i=1
h(Xi) =
1
n
∑
n
i=1
(Xk
i
− µk) =
1
n
∑
n
i=1
Xk
i
− µk. Equating mh to µh, we will have
1
n
∑
n
i=1
Xk
i
− µk = 0, which is the method of moment equation (1).
Next we will prove that if we use h(x) = ∂
∂θ
log f(x|θ), we will have the method of maximum
likelihood. In this case, we have
µh = E
[
∂
∂θ
log f(X|θ)
]
=
∫
[
∂
∂θ
log f(x|θ)
]
f(x|θ)dx
=
∫ ∂
∂θ
f(x|θ)
f(x|θ)
f(x|θ)dx =
∫
∂
∂θ
f(x|θ)dx
=
∂
∂θ
∫
f(x|θ)dx =
∂1
∂θ
= 0
Here we assume the order of integral and differential can be changed, this is justified under
some reasonable conditions. We also have
mh =
n
∑
i=1
∂
∂θ
log f(Xi|θ)
Equating mh to µh, we will have
n
∑
i=1
∂
∂θ
log f(Xi|θ) = 0
which is exactly the equation for maximum likelihood estimation Eqn. 2.
In general, if h(x, θ) is selected such that E(h(X, θ)) = 0, then h(X, θ) is called score function.
Under this situation, the common framework for estimation equation is
1
n
n
∑
i=1
h(Xi, θ) = 0
In method of moments, the score function is h(X, θ) = Xk − µk(θ), and in maximum likeli-
hood method, the score function is h(X, θ) = l′(X|θ) = ∂
∂θ
log f(X|θ).
Baixado por Ênio de Paula (jorgepont9999@gmail.com)
lOMoARcPSD|8960359