从独立同分布的数据估计参数

假设我们要从独立同分布的随机变量 D = { x 1 , … , x n } D=\{x_1,\dots,x_n\} D={x1​,…,xn​}来估计出一个$\theta$
最大似然估计:
$$\begin{gathered} L(\theta )=P(D;\theta )=P(x_1,\dots ,x_n;\theta ) \\ =P(x_1;\theta )P(x_2;\theta )\dots P(x_n;\theta ) \\ ={\Pi }_i^{N}P(x_i;\theta ) \end{gathered}$$
$\hat{\theta }=\underset{\theta }{\mathrm{argmax}}L(\theta )=\underset{\theta }{\mathrm{argmax}}log(L(\theta ))$

例1：伯努利模型
$P(x)={\theta }^x(1-\theta {)}^{1-x}$
那么$L(\theta )=\Pi P(x_i;\theta )=\Pi ({\theta }^{x_i}(1-\theta {)}^{1-x_i})={\theta }^{\sum _{i=1}^N{x}_i}(1-\theta {)}^{\sum _{i=1}^N(1-x_i)}$
$\log L(\theta )=\log {\theta }^{\sum _{i=1}^N{x}_i}(1-\theta {)}^{\sum _{i=1}^N(1-x_i)}=n_h\log \theta +(N-n_h)\log (1-\theta )$
$\frac{\partial \log L(\theta )}{\partial \theta }=\frac{n_h}{\theta }-\frac{N-n_h}{1-\theta }=0$
$(1-\theta )n_h=\theta (N-n_h)\Rightarrow {\hat{\theta }}_{MLE}=\frac{n_h}{N}$

例2：单变量正态分布模型
P ( x ) = ( 2 π σ 2 ) − 1 2 exp ⁡ { − ( x − μ ) 2 / 2 σ 2 } P(x)=(2\pi\sigma^2)^{-\frac{1}{2}}\exp\{-(x-\mu)^2/2\sigma^2\} P(x)=(2πσ2)−21​exp{−(x−μ)2/2σ2}
L ( θ ) = Π P ( x i ) = Π ( 2 π σ 2 ) − 1 2 exp ⁡ { − ( x i − μ ) 2 / 2 σ 2 } L(\theta)=\Pi P(x_i)=\Pi (2\pi\sigma^2)^{-\frac{1}{2}}\exp\{-(x_i-\mu)^2/2\sigma^2\} L(θ)=ΠP(xi​)=Π(2πσ2)−21​exp{−(xi​−μ)2/2σ2}
$\log L(\theta )=-\frac{N}{2}\log 2\pi {\sigma }^2-\frac{1}{2{\sigma }^2}\sum _{i=1}^N(x_i-\mu {)}^2$
$\frac{\partial \log L(\theta )}{\partial \mu }=\frac{1}{{\sigma }^2}\sum _{i=1}^N(x_i-\mu )=0\Rightarrow {\mu }_{MLE}=\frac{\sum _{i=1}^N{x}_i}{N}$
$\frac{\partial \log L(\theta )}{\partial {\sigma }^2}=-\frac{N}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}\sum _{i=1}^N(x_i-\mu {)}^2=0\Rightarrow {\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^2$

为了防止过拟合（上面求解出来的太过于精确了）我们就会用${\hat{\theta }}_{ML}=\frac{n_h}{n_h+n_t}$

贝叶斯学习

$P(\theta |D)=\frac{P(D|\theta )P(\theta )}{P(D)}$，其中$P(\theta |D)$就是后验，$P(D|\theta )$就是上面求的似然估计，$P(\theta )$就是$\theta$的一个先验,$P(D)$就是相当于是一个归一项、常熟。
那么，$P(\theta |D)\propto P(D|\theta )P(\theta )$。
其中，最大后验概率就是${\hat{\theta }}_{MAP}=\underset{\theta }{\mathrm{argmax}}P(\theta |D)$。
当$\theta$的先验是一个均匀分布时，那么$P(\theta )$也是一个常熟，那么$P(\theta |D)\propto P(D|\theta )$即MLE=MAP

例1：Beta分布，似然是伯努利分布
给$\theta$一个先验是Beta分布，$P(\theta )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$
那么$\theta$的后验$P(\theta |D)\propto P(D|\theta )P(\theta )\propto {\theta }^{n_h}(1-\theta {)}^{N-n_h}\times {\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}={\theta }^{n_h+\alpha -1}(1-\theta {)}^{N-n_h+\beta -1}\propto Beta(\alpha +n_h,\beta +n_t)$
也就是先验和后验共轭了
例2：Beta分布，似然是二项分布
给$\theta$一个先验是Beta分布，$P(\theta )=\frac{\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}{\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}$
$P(D|\theta )=C_n^k{\theta }^k(1-\theta {)}^{n-k}$
那么后验概率$P(\theta |D)\propto P(D|\theta )P(\theta )\propto {\theta }^{\alpha -1}(1-\theta {)}^{\beta -1}{\theta }^k(1-\theta {)}^{n-k}\propto Beta(\alpha +k,\beta +n-k)$
例3：
多项分布：$P(x_1,\dots ,x_k;n,p_1,\dots ,p_k)=\frac{n!}{x_1!\dots x_n!}{p}_1^{x_1}\dots p_k^{x_k}$
狄利克雷分布：$P({\theta }_1,\dots ,{\theta }_K)=\frac{1}{B(\alpha )}{\Pi }_i^K{\theta }_i^{{\alpha }_i-1}$