Gaussian Mixture Models

A Gaussian mixture model is a probabilistic model that assumes that data are generated from a finite mixture of Gaussians with unknown parameters. The model likelihood can be written as:

\[p(x|\theta) = \sum_{i=1}^{K}{\pi_i \mathcal{N}(x|\mu_i, \Sigma_i)}\]

where \(p(x|\theta)\) is probability distribution given \(\theta:=\{\pi_i, \mu_i, \Sigma_i\}_{i=1}^K\), \(K\) denotes number of mixture components, \(\pi_i\) denotes weight for \(i\)-th component, \(\mathcal{N}\) denotes a multivariate normal distribution with mean vector \(\mu_i\) and covariance matrix \(\Sigma_i\).

The expectation maximization (EM) algorithm is used to learn parameters of the model, via finding a local maximum of a lower bound on the likelihood.

See Chapter 20 in [Bar12] for a detailed introduction.

Example

We start by creating CDenseFeatures (here 64 bit floats aka RealFeatures) as

features_train = RealFeatures(f_feats_train)
features_train = RealFeatures(f_feats_train);
RealFeatures features_train = new RealFeatures(f_feats_train);
features_train = Modshogun::RealFeatures.new f_feats_train
features_train <- RealFeatures(f_feats_train)
features_train = modshogun.RealFeatures(f_feats_train)
RealFeatures features_train = new RealFeatures(f_feats_train);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);

We initialize CGMM, passing the desired number of mixture components.

num_components = 3
gmm = GMM(num_components)
num_components = 3;
gmm = GMM(num_components);
int num_components = 3;
GMM gmm = new GMM(num_components);
num_components = 3
gmm = Modshogun::GMM.new num_components
num_components <- 3
gmm <- GMM(num_components)
num_components = 3
gmm = modshogun.GMM(num_components)
int num_components = 3;
GMM gmm = new GMM(num_components);
auto num_components = 3;
auto gmm = some<CGMM>(num_components);

We provide training features to the CGMM object, train it by using EM algorithm and sample data-points from the trained model.

gmm.set_features(features_train)
gmm.train_em()
output = gmm.sample()
gmm.set_features(features_train);
gmm.train_em();
output = gmm.sample();
gmm.set_features(features_train);
gmm.train_em();
DoubleMatrix output = gmm.sample();
gmm.set_features features_train
gmm.train_em 
output = gmm.sample 
gmm$set_features(features_train)
gmm$train_em()
output <- gmm$sample()
gmm:set_features(features_train)
gmm:train_em()
output = gmm:sample()
gmm.set_features(features_train);
gmm.train_em();
double[] output = gmm.sample();
gmm->set_features(features_train);
gmm->train_em();
auto output = gmm->sample();

We extract parameters like \(\pi\), \(\mu_i\) and \(\Sigma_i\) for any componenet from the trained model.

component_num = 1
nth_mean = gmm.get_nth_mean(component_num)
nth_cov = gmm.get_nth_cov(component_num)
coef = gmm.get_coef()
component_num = 1;
nth_mean = gmm.get_nth_mean(component_num);
nth_cov = gmm.get_nth_cov(component_num);
coef = gmm.get_coef();
int component_num = 1;
DoubleMatrix nth_mean = gmm.get_nth_mean(component_num);
DoubleMatrix nth_cov = gmm.get_nth_cov(component_num);
DoubleMatrix coef = gmm.get_coef();
component_num = 1
nth_mean = gmm.get_nth_mean component_num
nth_cov = gmm.get_nth_cov component_num
coef = gmm.get_coef 
component_num <- 1
nth_mean <- gmm$get_nth_mean(component_num)
nth_cov <- gmm$get_nth_cov(component_num)
coef <- gmm$get_coef()
component_num = 1
nth_mean = gmm:get_nth_mean(component_num)
nth_cov = gmm:get_nth_cov(component_num)
coef = gmm:get_coef()
int component_num = 1;
double[] nth_mean = gmm.get_nth_mean(component_num);
double[,] nth_cov = gmm.get_nth_cov(component_num);
double[] coef = gmm.get_coef();
auto component_num = 1;
auto nth_mean = gmm->get_nth_mean(component_num);
auto nth_cov = gmm->get_nth_cov(component_num);
auto coef = gmm->get_coef();

We obtain log likelihood of belonging to clusters and being generated by this model.

log_likelihoods = gmm.cluster(nth_mean)
log_likelihoods = gmm.cluster(nth_mean);
DoubleMatrix log_likelihoods = gmm.cluster(nth_mean);
log_likelihoods = gmm.cluster nth_mean
log_likelihoods <- gmm$cluster(nth_mean)
log_likelihoods = gmm:cluster(nth_mean)
double[] log_likelihoods = gmm.cluster(nth_mean);
auto log_likelihoods = gmm->cluster(nth_mean);

We can also use Split-Merge Expectation-Maximization algorithm [UNGH00] for training.

gmm.train_smem()
gmm.train_smem();
gmm.train_smem();
gmm.train_smem 
gmm$train_smem()
gmm:train_smem()
gmm.train_smem();
gmm->train_smem();

References

Wikipedia: Mixture_model

Wikipedia: Expectation–maximization_algorithm

[Bar12]D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.
[UNGH00]N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton. Smem algorithm for mixture models. Neural Computation, 12(9):2109–2128, 2000.