Averaged Perceptron

The averaged Perceptron is an online binary classifier. It is an extension of the standard Perceptron algorithm; it uses the averaged weight and bias. Given a vector \(\mathbf{x}\), the predicted class is given by:

\[\theta\left(\mathbf{w} \cdot \mathbf{x}+b\right)\]

Here, \(\mathbf{w}\) is the average weight vector, \(b\) is the average bias and \(\theta\) is a step function:

\[\begin{split}\theta(x) = \begin{cases} 1 & x > 0 \\ 0 & x = 0 \\ -1 & x < 0 \end{cases}\end{split}\]

See chapter 17 in [Bar12] for a brief explanation of the Perceptron.

Example

Given a linearly separable dataset, we create some CDenseFeatures (RealFeatures, here 64 bit float values) and some CBinaryLabels to set up the training and validation sets.

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = BinaryLabels(f_labels_train)
labels_test = BinaryLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = BinaryLabels(f_labels_train);
labels_test = BinaryLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);
features_train = Modshogun::RealFeatures.new f_feats_train
features_test = Modshogun::RealFeatures.new f_feats_test
labels_train = Modshogun::BinaryLabels.new f_labels_train
labels_test = Modshogun::BinaryLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- BinaryLabels(f_labels_train)
labels_test <- BinaryLabels(f_labels_test)
features_train = modshogun.RealFeatures(f_feats_train)
features_test = modshogun.RealFeatures(f_feats_test)
labels_train = modshogun.BinaryLabels(f_labels_train)
labels_test = modshogun.BinaryLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CBinaryLabels>(f_labels_train);
auto labels_test = some<CBinaryLabels>(f_labels_test);

We create the CAveragedPerceptron instance by passing it the traning features and labels. We also set its learn rate and its maximum iterations.

learn_rate = 1.0
max_iter = 1000
perceptron = AveragedPerceptron(features_train, labels_train)
perceptron.set_learn_rate(learn_rate)
perceptron.set_max_iter(max_iter)
learn_rate = 1.0;
max_iter = 1000;
perceptron = AveragedPerceptron(features_train, labels_train);
perceptron.set_learn_rate(learn_rate);
perceptron.set_max_iter(max_iter);
double learn_rate = 1.0;
int max_iter = 1000;
AveragedPerceptron perceptron = new AveragedPerceptron(features_train, labels_train);
perceptron.set_learn_rate(learn_rate);
perceptron.set_max_iter(max_iter);
learn_rate = 1.0
max_iter = 1000
perceptron = Modshogun::AveragedPerceptron.new features_train, labels_train
perceptron.set_learn_rate learn_rate
perceptron.set_max_iter max_iter
learn_rate <- 1.0
max_iter <- 1000
perceptron <- AveragedPerceptron(features_train, labels_train)
perceptron$set_learn_rate(learn_rate)
perceptron$set_max_iter(max_iter)
learn_rate = 1.0
max_iter = 1000
perceptron = modshogun.AveragedPerceptron(features_train, labels_train)
perceptron:set_learn_rate(learn_rate)
perceptron:set_max_iter(max_iter)
double learn_rate = 1.0;
int max_iter = 1000;
AveragedPerceptron perceptron = new AveragedPerceptron(features_train, labels_train);
perceptron.set_learn_rate(learn_rate);
perceptron.set_max_iter(max_iter);
auto learn_rate = 1.0;
auto max_iter = 1000;
auto perceptron = some<CAveragedPerceptron>(features_train, labels_train);
perceptron->set_learn_rate(learn_rate);
perceptron->set_max_iter(max_iter);

Then we train the CAveragedPerceptron and we apply it to the test data, which gives the predicted CBinaryLabels.

perceptron.train()
perceptron.set_features(features_test)
labels_predict = perceptron.apply()
perceptron.train();
perceptron.set_features(features_test);
labels_predict = perceptron.apply();
perceptron.train();
perceptron.set_features(features_test);
Labels labels_predict = perceptron.apply();
perceptron.train 
perceptron.set_features features_test
labels_predict = perceptron.apply 
perceptron$train()
perceptron$set_features(features_test)
labels_predict <- perceptron$apply()
perceptron:train()
perceptron:set_features(features_test)
labels_predict = perceptron:apply()
perceptron.train();
perceptron.set_features(features_test);
Labels labels_predict = perceptron.apply();
perceptron->train();
perceptron->set_features(features_test);
auto labels_predict = perceptron->apply();

We can also extract the average weights \(\mathbf{w}\) and the bias \(b\).

weights = perceptron.get_w()
bias = perceptron.get_bias()
weights = perceptron.get_w();
bias = perceptron.get_bias();
DoubleMatrix weights = perceptron.get_w();
double bias = perceptron.get_bias();
weights = perceptron.get_w 
bias = perceptron.get_bias 
weights <- perceptron$get_w()
bias <- perceptron$get_bias()
weights = perceptron:get_w()
bias = perceptron:get_bias()
double[] weights = perceptron.get_w();
double bias = perceptron.get_bias();
auto weights = perceptron->get_w();
auto bias = perceptron->get_bias();

Finally, we can evaluate the performance, e.g. using CAccuracyMeasure.

eval = AccuracyMeasure()
accuracy = eval.evaluate(labels_predict, labels_test)
eval = AccuracyMeasure();
accuracy = eval.evaluate(labels_predict, labels_test);
AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);
eval = Modshogun::AccuracyMeasure.new 
accuracy = eval.evaluate labels_predict, labels_test
eval <- AccuracyMeasure()
accuracy <- eval$evaluate(labels_predict, labels_test)
eval = modshogun.AccuracyMeasure()
accuracy = eval:evaluate(labels_predict, labels_test)
AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);
auto eval = some<CAccuracyMeasure>();
auto accuracy = eval->evaluate(labels_predict, labels_test);

References

Wikipedia: Perceptron

[Bar12]D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.