Kernel Ridge Regression

Kernel ridge regression is a non-parametric form of ridge regression. The aim is to learn a function in the space induced by the respective kernel \(k\) by minimizing a squared loss with a squared norm regularization term.

The solution can be written in closed form as:

\[\alpha = \left({\bf K}+\tau{\bf I}\right)^{-1}{\bf y}\]

where \({\bf K}\) is the kernel matrix and \(\alpha\) is the vector of weights in the space induced by the kernel. The learned function can then be evaluated as \(f(x)=\sum_{i=1}^N\alpha_ik(x,x_i)\).

See Chapter 17 in [Bar12] for a detailed introduction.

Example

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CRegressionLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = RegressionLabels(f_labels_train)
labels_test = RegressionLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = RegressionLabels(f_labels_train);
labels_test = RegressionLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);
features_train = Modshogun::RealFeatures.new f_feats_train
features_test = Modshogun::RealFeatures.new f_feats_test
labels_train = Modshogun::RegressionLabels.new f_labels_train
labels_test = Modshogun::RegressionLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- RegressionLabels(f_labels_train)
labels_test <- RegressionLabels(f_labels_test)
features_train = modshogun.RealFeatures(f_feats_train)
features_test = modshogun.RealFeatures(f_feats_test)
labels_train = modshogun.RegressionLabels(f_labels_train)
labels_test = modshogun.RegressionLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CRegressionLabels>(f_labels_train);
auto labels_test = some<CRegressionLabels>(f_labels_test);

Choose an appropriate CKernel and instantiate it. Here we use a CGaussianKernel.

width = 1.0
kernel = GaussianKernel(features_train, features_train, width)
width = 1.0;
kernel = GaussianKernel(features_train, features_train, width);
double width = 1.0;
GaussianKernel kernel = new GaussianKernel(features_train, features_train, width);
width = 1.0
kernel = Modshogun::GaussianKernel.new features_train, features_train, width
width <- 1.0
kernel <- GaussianKernel(features_train, features_train, width)
width = 1.0
kernel = modshogun.GaussianKernel(features_train, features_train, width)
double width = 1.0;
GaussianKernel kernel = new GaussianKernel(features_train, features_train, width);
auto width = 1.0;
auto kernel = some<CGaussianKernel>(features_train, features_train, width);

We create an instance of CKernelRidgeRegression classifier by passing it \(\tau\), the kernel and labels.

tau = 0.001
krr = KernelRidgeRegression(tau, kernel, labels_train)
tau = 0.001;
krr = KernelRidgeRegression(tau, kernel, labels_train);
double tau = 0.001;
KernelRidgeRegression krr = new KernelRidgeRegression(tau, kernel, labels_train);
tau = 0.001
krr = Modshogun::KernelRidgeRegression.new tau, kernel, labels_train
tau <- 0.001
krr <- KernelRidgeRegression(tau, kernel, labels_train)
tau = 0.001
krr = modshogun.KernelRidgeRegression(tau, kernel, labels_train)
double tau = 0.001;
KernelRidgeRegression krr = new KernelRidgeRegression(tau, kernel, labels_train);
auto tau = 0.001;
auto krr = some<CKernelRidgeRegression>(tau, kernel, labels_train);

Then we train the regression model and apply it to test data to get the predicted CRegressionLabels.

krr.train()
labels_predict = krr.apply_regression(features_test)
krr.train();
labels_predict = krr.apply_regression(features_test);
krr.train();
RegressionLabels labels_predict = krr.apply_regression(features_test);
krr.train 
labels_predict = krr.apply_regression features_test
krr$train()
labels_predict <- krr$apply_regression(features_test)
krr:train()
labels_predict = krr:apply_regression(features_test)
krr.train();
RegressionLabels labels_predict = krr.apply_regression(features_test);
krr->train();
auto labels_predict = krr->apply_regression(features_test);

After training, we can extract \(\alpha\).

alpha = krr.get_alphas()
alpha = krr.get_alphas();
DoubleMatrix alpha = krr.get_alphas();
alpha = krr.get_alphas 
alpha <- krr$get_alphas()
alpha = krr:get_alphas()
double[] alpha = krr.get_alphas();
auto alpha = krr->get_alphas();

Finally, we can evaluate the CMeanSquaredError.

eval = MeanSquaredError()
mse = eval.evaluate(labels_predict, labels_test)
eval = MeanSquaredError();
mse = eval.evaluate(labels_predict, labels_test);
MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);
eval = Modshogun::MeanSquaredError.new 
mse = eval.evaluate labels_predict, labels_test
eval <- MeanSquaredError()
mse <- eval$evaluate(labels_predict, labels_test)
eval = modshogun.MeanSquaredError()
mse = eval:evaluate(labels_predict, labels_test)
MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);
auto eval = some<CMeanSquaredError>();
auto mse = eval->evaluate(labels_predict, labels_test);

References

Wikipedia: Kernel_method

[Bar12]D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.