Ex 1.12 - Pattern Recognition and ML by Christopher Bishop

Pre-read

Question

Solution from Book

Solution Using ChatGPT

Are you aware of the result E[xn, xm] = mu^2 + I . sigma ^ 2

I assume you meant $E[X_n \cdot X_m] = \mu^2 + I \cdot \sigma^2$, where $I$ is the Kronecker delta, which is 1 when $n = m$ and 0 otherwise.

In that case, $E[X_n \cdot X_m]$ simplifies to:

$E[X_n \cdot X_m] = \begin{cases} \mu^2 + \sigma^2 & \text{if } n = m \\ \mu^2 & \text{if } n \neq m \end{cases}$

This is because when $n = m$, you have the product of a variable with itself, and the additional $\sigma^2$ term comes from the covariance term. When $n \neq m$, the covariance term is zero, and you are left with the square of the mean.

If you have further questions or if there's anything specific you'd like to clarify, feel free to ask!

Can you derive this expression?

Certainly! Let's derive the expression for $E[X_n \cdot X_m]$ for two continuous random variables $X_n$ and $X_m$ from the same Gaussian distribution $N(\mu, \sigma^2)$ with mean $\mu$ and variance $\sigma^2$.

The expectation of the product is given by:

$E[X_n \cdot X_m] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x_n \cdot x_m \cdot f(x_n) \cdot f(x_m) \,dx_n \,dx_m$

Since $X_n$ and $X_m$ are from the same distribution, let's denote $f(x)$ as the probability density function (PDF) of $X$. The expression becomes:

$E[X_n \cdot X_m] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x_n \cdot x_m \cdot f(x) \cdot f(x) \,dx_n \,dx_m$

Now, split the integral into two cases: $n = m$ and $n \neq m$.

Case 1: $n = m$

$E[X_n \cdot X_m] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x_n^2 \cdot f(x) \cdot f(x) \,dx_n \,dx_m$

$= \left(\int_{-\infty}^{\infty} x_n^2 \cdot f(x) \,dx_n\right) \cdot \left(\int_{-\infty}^{\infty} f(x) \,dx_m\right)$

$= \mu^2 + \sigma^2$

This result comes from the fact that the integral of $x^2$ over the entire range for a Gaussian distribution with mean $\mu$ is $\mu^2 + \sigma^2$.

Case 2: $n \neq m$

$E[X_n \cdot X_m] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x_n \cdot x_m \cdot f(x) \cdot f(x) \,dx_n \,dx_m$

$= \left(\int_{-\infty}^{\infty} x_n \cdot f(x) \,dx_n\right) \cdot \left(\int_{-\infty}^{\infty} x_m \cdot f(x) \,dx_m\right)$

$= \mu \cdot \mu$

This result comes from the fact that the integral of $x$ over the entire range for a Gaussian distribution with mean $\mu$ is $\mu$.

Therefore, combining both cases:

$E[X_n \cdot X_m] = \mu^2 + I \cdot \sigma^2$

where $I$ is the Kronecker delta, equal to 1 when $n = m$ and 0 otherwise.

References

• give me math expression for: (1/n) * sum_for_n_equals_1_to_N(E[ (1/n^2) * sum_for_m_equals_1_to_N(sum_for_l_equals_1_to_N(xm*xl)) ])
• what is the mathematical expression for (1/N) * sum_n_equal_1_to_N(E[(2/N) * xn * sum_m_equals_1_to_N(xm)])

Tags: Mathematical Foundations for Data Science,Machine Learning,