Multivariante Gaussians and Mixtures of Gaussians (MoG)

In [11]:
import matplotlib.pyplot as plt
import numpy as np
from numpy import *
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline  

First, let's generate a "2D cloud" of points by independently generating x1's and x2's.

In [2]:
mu1 = 5
mu2 = 10
sigma1 = 17
sigma2 = 3

x1 = np.random.normal(loc=mu1, scale=sigma1, size = 10000)
x2 = np.random.normal(loc=mu2, scale=sigma2, size = 10000)
x = vstack((x1, x2))
In [3]:
plt.scatter(x1, x2)
Out[3]:
<matplotlib.collections.PathCollection at 0x7f6af444bf50>

What if we wanted for the x1 and x2 coordinates to be correlated? I.e., we want the x2 to be larger if x1 is larger, on average. That would make the cloud be tilted.

In [4]:
mu = [5, 10]

Sigma = array([[ 289,   30.    ],
               [  30.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
In [5]:
plt.scatter(x1, x2)
Out[5]:
<matplotlib.collections.PathCollection at 0x7f6af234bd10>

A higher covariance between x1 and x2 means more tilt and a "thinner" cloud.

In [6]:
mu = [5, 10]

Sigma = array([[ 289,   45.    ],
               [  45.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
plt.scatter(x1, x2)
Out[6]:
<matplotlib.collections.PathCollection at 0x7f6af22f8b90>

If the covariance is high enough, the cloud is very thin, since a higher x1 always implies a higher x2:

In [7]:
mu = [5, 10]

Sigma = array([[ 289,   51.    ],
               [  51.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
plt.scatter(x1, x2)
Out[7]:
<matplotlib.collections.PathCollection at 0x7f6af2227c10>

(Optional aside: we got 51 by using the fact that 1ρ(x1,x2)=Cov(x1,x2)Var(x1)Var(x2)1.

Diagonal Covariances

Of course, we can set Cov(x1,x2) to 0, in which case we'll get an "axis-aligned" cloud:

In [8]:
mu = [5, 10]

Sigma = array([[ 289,   0.    ],
               [  0.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
plt.scatter(x1, x2)
Out[8]:
<matplotlib.collections.PathCollection at 0x7f6af2153c90>

Mixtures of Gaussians

Let's now generate some Mixtures of Gaussians in 2d.

In [9]:
pi = array([.2, .3, .5])
mu = [[10, 5], [-10, -5], [-10, 5]]
sigma = [ [[9, 5], 
           [5, 9]],
         
         
          [[5, 0], 
           [0, 5]],
         
         
          [[9, -5], 
           [-5, 9]]
        ]

N = 10000
Ns = (N*pi).astype(int)
colors = ["r", "g", "b"]
for i in range(len(pi)):
    x1, x2 = np.random.multivariate_normal(mu[i], sigma[i], Ns[i]).T
    plt.scatter(x1, x2, color=colors[i])

Mixtures of Gaussians in 3d

Finally, let's plot a Mixture of Gaussians in 3d. For simplicity, let's make the clouds axis-aligned.

In [10]:
pi = array([.2, .3, .5])
mu = [[10, 5, 0], [-10, -5, 0], [-10, 5, 3]]
sigma = [ [[9, 0, 0], 
           [0, 9, 0],
           [0, 0, 9]],
         
          [[9, 0, 0], 
           [0, 9, 0],
           [0, 0, 9]],         
         
          [[9, 0, 0], 
           [0, 9, 0],
           [0, 0, 9]],        ]

N = 1000
Ns = (N*pi).astype(int)
colors = ["r", "g", "b"]

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

for i in range(len(pi)):
    x1, x2, x3 = np.random.multivariate_normal(mu[i], sigma[i], Ns[i]).T
    ax.scatter(x1, x2, x3, c=colors[i])