Bayesian Estimation

Let x be an observation extracted from a distribution P(x, θ), dependent on an unknown parameter θ. Let us model θ as a random variable extracted from a known distribution p(θ). In the absence of any observation, the a priori distribution of x is the average of all distributions weighted by probability of the value of θ generating them.

P(x) = ∫ P(x, θ) dθ

We wish to estimate θ using both the knowledge of the distribution and an observation of x'.

P(θ | x') * P(x') = P(x' and θ) = P(x' | θ) * P(θ)

Hence it is possible to compute the conditional distribution of x given an observation x'.

P(x | x') = ∫ p(x, θ | x') dθ = ∫ P(x | θ, x') P(θ | x') dθ = ∫ P(x | θ) P(θ | x') dθ

Notice that P(x | θ, x') = P(x | θ) because θ, the parameter of the distribution is a sufficient statistic. Hence, the knowledge of x' does not change the a posteriori distribution of x.

Maximum A Posteriori estimate (MAP)

θ[MAP] = arg max p(θ | x)

Bayesian Estimator: A posteriori expected value of the parameter.

θ[Bayes] = E[θ | x] =  ∫ θ P(θ | x) dθ