Linear Models#
The topic of linear models encompasses a range of commonly used statistical models and fitting algorithms, including multiple linear regression, logistic regression, polynomial regression, and nonparametric regression.
The general form of a linear model fitting problem is as follows:
where \(X\) is an array of \(n_{\mathrm{samples}}\) observations with \(n_{\mathrm{features}}\) features, \(y\) is an array of \(n_{\mathrm{samples}}\) responses / labels, \(\phi\) is a set of (possibly nonlinear) basis functions, \(\beta\) is a set of weights / coefficients, \(g^{-1}\) is a (possibly nonlinear) activation / link function, and \(C_{\theta}\) is a cost / error function, which may depend on a set of (fixed) hyperparameters, \(\theta\).
Most linear models satisfy a form where the cost function can be split into a loss function (which measures quality of fit to the data) and a penalty term (which regularizes the model parameters). Regularization is also referred to as shrinkage because it tends to shrink the size of the parameter values and/or shrink the number of non-zero parameters.
The loss function is typically a function of the responses or labels (\(y\)), the features (\(X\)), and the model parameters (\(\beta\)), whereas the penalty term is typically only a function of the 1-norm and/or the 2-norm of the model parameters (\(\beta\)). Such linear models are often referred to as elastic nets.
Typically, in addition to the conditions above, \(\phi\) and \(g^{-1}\) are identity mappings and the general form for the cost function becomes,
where \(0\le\lambda, 0\le\alpha\le1\) are hyperparameters with \(\lambda\) setting the magnitude of the overall penalization and \(\alpha\) distributing its share across the \(\ell_1\) and \(\ell_2\) regularization terms, \(\lVert \beta \rVert_1\) and \(\lVert \beta \rVert_2\). \(L\) is known as the loss function. Linear models where \(\alpha=0\) are called ridge regression models. When \(\alpha=1\) the model is a lasso model, whereas if \(0 < \alpha < 1\) the model is an elastic net regression model.
Intercept
If a linear model contains an intercept term, the parameter vector \(\beta\) has dimension \(n_{\mathrm{features}}+1\) and the observations \(X_i\) are extended with a constant of \(1\). For example, if there is a single feature and the linear model has an intercept term the mean square error loss cost function becomes,
where \(x_i\) represents a single (scalar) observation and without any regularization terms.
The intercept can be added by setting the linear regression handle option intercept to 1. For further details see Linear model options.
Note
The intercept term is never regularized. In the example above only \(\beta_1\) is subject to regularization.
Available models#
Models can be classified by their loss function, the following subsections present the supported loss functions.
Mean square error#
This is the most basic model and its use is widespread across many applications. The cost function for a linear regression model where the fit (loss) is measured by the mean square error (MSE) is,
where \(X_i\) represents a single (multi-dimensional) observation, i.e., a row in a table of observations. Note that the intercept term \(\beta_0\) is not regularized.
Logistic regression#
Logistic regression is a type of supervised classification model aimed at assigning labels. In AOCL-DA, the labels are expected to be provided in a categorical response variable, \(y\), encoded by \(\{0, 1, 2, \ldots, K-1 \}\). The fit is based on maximizing the log-likelihood (loss function) of the probabilities that each observation \(i\) belongs to a given class, in turn defined by,
As an example, if \(K=2\), the loss function simplifies to,
As in the linear regression model, \(\ell_1\) or \(\ell_2\) regularization can be applied by adding the corresponding penalty term to the cost function.
Fitting methods#
Different methods are available to compute the models. The method is chosen automatically by default but can be set manually using the optional parameter optim method (see Linear model options).
Direct solvers
QR: the standard MSE linear regression model can be computed using the QR factorization of the data matrix if no regularization term is required.
\[X = QR,\]where \(Q\) is a \(n_{\mathrm{samples}} \times n_{\mathrm{features}}\) matrix with orthogonal columns and \(R\) is a \(n_{\mathrm{features}}\times n_{\mathrm{features}}\) triangular matrix.
SVD: the singular value decomposition can be used to compute standard MSE and ridge regression models.
\[X = UDV^T,\]where \(U\) is an orthogonal matrix of size \(n_{\mathrm{samples}}\times n_{\mathrm{samples}}\), \(D\) is a \(n_{\mathrm{samples}}\times n_{\mathrm{features}}\) diagonal matrix whose elements are the non-negative singular values of \(X\) and \(V^T\) is an orthogonal matrix of size \(n_{\mathrm{features}} \times n_{\mathrm{features}}\).
Cholesky: the Cholesky decomposition can be used for normal and ridge regression when the data matrix is full-rank. It factorizes the symmetric positive-definite normal equations matrix \(X^TX\) into two triangular matrices. In linear models it can be used to find coefficients expressed as:
\[\beta = (X^TX+\lambda)^{-1}X^Ty,\]where, after left multiplying by the expression inside of the inverse, we end up with a system of linear equations in the form \(Ax=B\). The left hand side can be factorized using the Cholesky decomposition as follows:
\[X^TX+\lambda = LL^T,\]where \(L\) is a lower triangular matrix with real and positive diagonal entries. This matrix is then used to find a solution to the system of linear equations.
Iterative solvers
L-BFGS-B: a solver aimed at minimizing smooth nonlinear functions (Liu and Nocedal [1989]). It can be used to compute both MSE and logistic models with or without \(\ell_2\) regularization. It is not suitable when an \(\ell_1\) regularization term is required.
Coordinate descent: a solver aimed at minimizing nonlinear functions. It is suitable for linear models with an \(\ell_1\) regularization term and elastic nets (Friedman et al. [2010], Friedman et al. [2007]).
Note
The coordinate descent method implemented is optimized to solve lasso or elastic net problems. For ridge or unregularized problems the use of any alternative methods is recommended.
Warning
The implemented coordinate descent method is designed to fit standardized data. As such, it should not be used with the scaling method set to anything other than
scaling only(default), orstandardize(see next section). Using any other type of scaling will result in lack of convergence or unexpected results.Conjugate gradient: a solver aimed at finding a solution to a system of linear equations. It can be used to compute linear regression with or without \(\ell_2\) regularization.
Scaling the data#
In many circumstances, the data used to perform a fit is badly scaled and rescaling can have numerical benefits. Furthermore, iterative solvers can show improved quality of the solution when fitting on rescaled data.
If scaling is requested, then the fitting routine takes care of rescaling the problem data. Once the model is trained, it reverts the scaling on the trained coefficients, so this process is transparent to the user.
The optional parameter scaling controls the scaling of the data. It defaults to auto, where it selects the type of scaling according to the optimization method used. The following table shows the default for each solver, while the table Linear model data scaling types shows the supported scaling types.
Method |
option optim method |
option scaling |
|---|---|---|
Conjugate Gradient Method |
|
|
Coordinate Descent Method |
|
|
Singular Value Decomposition |
|
|
QR Factorization |
|
|
Cholesky Factorization |
|
|
L-BFGS-B Solver |
|
|
When a solver provides two default options for scaling, then it is chosen based on problem characteristics, e.g., if the problem is underdetermined or if the model has an intercept.
Note
Scaling is applied prior to solving the problem and hence the regularization (if any) is done over the scaled problem and not on the unscaled version. This has implications on the trained coefficients and may differ from the regularized model trained with unscaled data.
The following table shows the supported scaling types. In the table, \(N\) is a shorthand for \(n_{\mathrm{samples}}\); \(\sigma_Z\) refers to the sample standard deviation of \(Z\); and \(\mu_Z\) represents the sample mean of \(Z\). Finally, \(\hat Z\) represents the scaled version of \(Z\). Specifically, in the table below columns labeled \(\hat Y\) and \(\hat X_j\) report the transforms performed on the response vector \(Y\) and the columns of the predictor matrix \(X\).
scaling value |
model intercept |
\(m_Y\) |
\(s_Y\) |
\(hat Y\) |
\(m_{X_j}\) |
\(s_{X_j}\) |
\(hat X_j\) |
|---|---|---|---|---|---|---|---|
|
yes/no |
\(0\) |
\(1\) |
\(Y\) |
\(0\) |
\(1\) |
\(X_j\) |
|
yes |
\(\mu_Y\) |
\(1\) |
\(Y - \mu_Y\) |
\(\mu_{X_j}\) |
\(1\) |
\(X_j - \mu_{X_j}\) |
|
no |
\(0\) |
\(1\) |
\(Y\) |
\(0\) |
\(1\) |
\(X_j\) |
|
yes |
\(\mu_Y\) |
\(\sigma_Y\) |
\(\frac{\frac{1}{\sqrt{N}}(Y-\mu_Y)}{\sigma_Y}\) |
\(\mu_{X_j}\) |
\(1\) |
\(\frac{1}{\sqrt{N}}(X_j-\mu_{X_j})\) |
|
no |
\(0\) |
\(\left\|\frac{1}{\sqrt{N}} Y\right\|\) |
\(\frac{\frac{1}{\sqrt{N}} Y}{s_Y}\) |
\(0\) |
\(1\) |
\(\frac{1}{\sqrt{N}} X_j\) |
|
yes |
\(\mu_Y\) |
\(\sigma_Y\) |
\(\frac{\frac{1}{\sqrt{N}}(Y-\mu_Y)}{\sigma_Y}\) |
\(\mu_{X_j}\) |
\(\sigma_{X_j}\) |
\(\frac{\frac{1}{\sqrt{N}} (X_j-\mu_{X_j})}{\sigma_{X_j}}\) |
|
no |
\(0\) |
\(\left\|\frac{1}{\sqrt{N}} Y\right\|\) |
\(\frac{\frac{1}{\sqrt{N}} Y}{s_Y}\) |
\(0\) |
\(\sigma_{X_j}\) |
\(\frac{\frac{1}{\sqrt{N}} X_j}{\sigma_{X_j}}\) |
Note
Scaling types scale only and standardize also affect the regularization penalty term \(\lambda\) in (1).
Iterative solvers can show the convergence progress by printing to standard output some progress metric for each iteration (as specified by the optional parameter print level). It is important to note that all reported metrics are based on the rescaled data.
When requesting information from the handle, after training the model, the reported metrics are also based on the scaling type used.
Initial coefficients#
For iterative solvers, it is possible to provide a warm-start for the process via an initial guess for the coefficients (for example, these may have been obtained via previous fit). When provided with too many coefficients, only the first \(k\) coefficients will be used, where \(k\) is the expected number of coefficients. The intercept will always be the last member of the provided array. If too few coefficients are provided, the initial guess will be ignored.
Warning
For the conjugate gradient solver in the underdetermined case (\(n_{\mathrm{features}} > n_{\mathrm{samples}}\)) we are solving the dual problem, and thus the initial coefficients should also be dual (so that the expected number of coefficients \(k\) is equal to \(n_{\mathrm{samples}}\) instead of \(n_{\mathrm{features}}\)).
Initial coefficients can be provided in the following way:
Provide x0 parameter when calling aoclda.linear_model.linmod.fit().
Provide coefs pointer to initial coefficients while calling da_linmod_fit_start_?.
Typical workflow for linear models#
The standard way of computing a linear model using AOCL-DA is as follows.
Initialize a
aoclda.linear_model.linmod()object with options set in the class constructor.Fit the linear model to your data using
aoclda.linear_model.linmod.fit().Extract results from the
aoclda.linear_model.linmod()object via its class attributes.
Initialize a
da_handlewithda_handle_typeda_handle_linmod.Pass data to the handle using da_linmod_define_features_?.
Customize the model using da_options_set_? (see below for a list of the available options).
Compute the linear model using da_linmod_fit_?.
Evaluate the model on new data using da_linmod_evaluate_model_?.
Extract results using da_handle_get_result_?. The following results are available:
Coefficients (
da_linmod_coef): the optimal coefficients of the fitted model.Some solvers provide extra information. (
da_linmod_rinfo), when available, contains the info[100] array with the following values:info[0]: loss value at current iterate, \(L(\beta_0, \beta)\),
info[1]: norm of the gradient of the loss function,
info[2]: number of iterations made (only for iterative solvers),
info[3]: compute time (wall clock time in seconds),
info[4]: number of model evaluations performed,
info[5]: infinity norm of the optimization metric (varies with the method used),
info[6]: infinity norm of of a given metric at the initial iterate (varies with the method used),
info[7]: number of cheap model evaluations (only relevant for Coordinate Descent Method) and indicates the number of low-rank updates used to evaluate model,
info[8-99]: reserved for future use.
Note
When information is not available, -1 is returned.
Linear model options#
The available Python options are detailed in the aoclda.linear_model.linmod() class constructor.
Various options can be set to customize the linear models by calling one of these functions. The following table details the available options, where \(\epsilon\) represents the machine precision.
Option name |
Type |
Default |
Description |
Constraints |
|---|---|---|---|---|
optim method |
string |
\(s=\) auto |
Select optimization method to use. |
\(s=\) auto, bfgs, cg, chol, cholesky, coord, lbfgs, lbfgsb, qr, sparse_cg, or svd. |
scaling |
string |
\(s=\) auto |
Scale or standardize feature matrix and response vector. Matrix is copied and then rescaled. Option key value auto indicates that rescaling type is chosen by the solver (this also includes no scaling). |
\(s=\) auto, centering, no, none, scale, scale only, standardise, or standardize. |
optim progress factor |
real |
\(r=\frac{10}{\sqrt{2\,\varepsilon}}\) |
Factor used to detect convergence of the iterative optimization step. See option in the corresponding optimization solver documentation. |
\(0 \le r\) |
optim convergence tol |
real |
\(r=10/2\sqrt{2\,\varepsilon}\) |
Tolerance to declare convergence for the iterative optimization step. See option in the corresponding optimization solver documentation. |
\(0 < r < 1\) |
print options |
string |
\(s=\) no |
Print options. |
\(s=\) no, or yes. |
lambda |
real |
\(r=0\) |
Penalty coefficient for the regularization terms: lambda( (1-alpha)/2 L2 + alpha L1 ). |
\(0 \le r\) |
alpha |
real |
\(r=0\) |
Coefficient of alpha in the regularization terms: lambda( (1-alpha)/2 L2 + alpha L1 ). |
\(0 \le r \le 1\) |
optim iteration limit |
integer |
\(i=10000\) |
Maximum number of iterations to perform in the optimization phase. Valid only for iterative solvers, e.g. L-BFGS-B, Coordinate Descent, etc. |
\(1 \le i\) |
intercept |
integer |
\(i=0\) |
Add intercept variable to the model. |
\(0 \le i \le 1\) |
print level |
integer |
\(i=0\) |
Set level of verbosity for the solver. |
\(0 \le i \le 5\) |
For the complete list of optional parameters see linear model options.
Examples#
The code below is supplied with your installation (see Python examples).
1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
2#
3# Redistribution and use in source and binary forms, with or without modification,
4# are permitted provided that the following conditions are met:
5# 1. Redistributions of source code must retain the above copyright notice,
6# this list of conditions and the following disclaimer.
7# 2. Redistributions in binary form must reproduce the above copyright notice,
8# this list of conditions and the following disclaimer in the documentation
9# and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11# may be used to endorse or promote products derived from this software without
12# specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25# pylint: disable = invalid-name
26
27"""
28Linear regression example Python script
29"""
30
31import sys
32from aoclda.linear_model import linmod
33import numpy as np
34
35
36def linmod_example():
37 """
38 Linear regression with an intercept variable
39 """
40 X = np.array([[1, 1], [2, 3], [3, 5], [4, 8], [5, 7], [6, 9]])
41 y = np.array([3., 6.5, 10., 12., 13., 19.])
42 lmod = linmod("mse", intercept=True)
43 lmod.fit(X, y)
44
45 # Extract coefficients
46 coef = lmod.coef
47 print(f"coefficients: [{coef[0]:.3f}, {coef[1]:.3f}, {coef[2]:.3f}]")
48 print('expected : [2.350, 0.350, 0.433]\n')
49
50 # Evaluate model on new data
51 X_test = np.array([[1, 1.1], [2.5, 3], [7, 9]])
52 pred = lmod.predict(X_test)
53 print(f"predictions: [{pred[0]:.3f}, {pred[1]:.3f}, {pred[2]:.3f}]")
54 print('expected : [3.168 7.358 20.0333]')
55
56
57if __name__ == "__main__":
58 try:
59 linmod_example()
60 except RuntimeError:
61 sys.exit(1)
The showcased examples sources and more, can be found in the examples folder of your installation.
1/*
2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without modification,
5 * are permitted provided that the following conditions are met:
6 * 1. Redistributions of source code must retain the above copyright notice,
7 * this list of conditions and the following disclaimer.
8 * 2. Redistributions in binary form must reproduce the above copyright notice,
9 * this list of conditions and the following disclaimer in the documentation
10 * and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 * may be used to endorse or promote products derived from this software without
13 * specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <assert.h>
30#include <iostream>
31
32int main() {
33
34 std::cout << "----------------------------------------" << std::endl;
35 std::cout << "Mean squared error model" << std::endl;
36 std::cout << "min ||Ax-b||^2; with A an 5x2 matrix" << std::endl << std::endl;
37 std::cout << std::fixed;
38 std::cout.precision(5);
39
40 // Problem data
41 da_int m = 5, n = 2;
42 double Al[10] = {1, 2, 3, 4, 5, 1, 3, 5, 1, 1};
43 double bl[5] = {1, 1, 1, 1, 1};
44 da_int nx = 2;
45 double x[2];
46
47 // Initialize the linear regression
48 da_handle handle = nullptr;
49 da_status status;
50 bool pass = true;
51 pass = pass && da_handle_init_d(&handle, da_handle_linmod) == da_status_success;
52 pass =
53 pass && da_linmod_select_model_d(handle, linmod_model_mse) == da_status_success;
54 pass = pass && da_linmod_define_features_d(handle, m, n, Al, bl) == da_status_success;
55 if (!pass) {
56 std::cout << "Something unexpected happened in the model definition\n";
57 da_handle_destroy(&handle);
58 return 1;
59 }
60
61 // Compute regression
62 status = da_linmod_fit_d(handle);
63 if (status == da_status_success) {
64 std::cout << "regression computed successfully!" << std::endl;
65 nx = 0; // Query the correct size
66 da_handle_get_result_d(handle, da_linmod_coef, &nx, x);
67 assert(nx == 2);
68 da_handle_get_result_d(handle, da_linmod_coef, &nx, x);
69 std::cout << "Coefficients: " << x[0] << " " << x[1] << std::endl;
70 std::cout << "(Expected : " << 0.199256 << " " << 0.130354 << ")" << std::endl;
71 } else {
72 std::cout << "Something wrong happened during MSE regression. Terminating"
73 << std::endl;
74 da_handle_destroy(&handle);
75 return 1;
76 }
77 std::cout << "----------------------------------------" << std::endl;
78
79 // Solve the same model with single precision
80 // Problem data
81 float As[10] = {1, 2, 3, 4, 5, 1, 3, 5, 1, 1};
82 float bs[5] = {1, 1, 1, 1, 1};
83 float xs[2];
84
85 std::cout.precision(2);
86 // Initialize the linear regression
87 da_handle handle_s = nullptr;
88 pass = true;
89 pass = pass && da_handle_init_s(&handle_s, da_handle_linmod) == da_status_success;
90 pass =
91 pass && da_linmod_select_model_s(handle_s, linmod_model_mse) == da_status_success;
92 pass =
93 pass && da_linmod_define_features_s(handle_s, m, n, As, bs) == da_status_success;
94 if (!pass) {
95 std::cout << "Something unexpected happened in the model definition\n";
96 da_handle_destroy(&handle);
97 da_handle_destroy(&handle_s);
98 return 1;
99 }
100
101 // Compute regression
102 status = da_linmod_fit_s(handle_s);
103 if (status == da_status_success) {
104 std::cout << "regression computed successfully!" << std::endl;
105 // status = da_linmod_s_get_coef(handle_s, &nx, xs);
106 nx = 0; // Query the correct size
107 da_handle_get_result_s(handle_s, da_linmod_coef, &nx, xs);
108 assert(nx == 2);
109 da_handle_get_result_s(handle_s, da_linmod_coef, &nx, xs);
110 std::cout << "Coefficients: " << xs[0] << " " << xs[1] << std::endl;
111 std::cout << "(Expected : " << 0.20 << " " << 0.13 << ")" << std::endl;
112 } else {
113 std::cout << "Something wrong happened during MSE regression. Terminating"
114 << std::endl;
115 da_handle_destroy(&handle);
116 da_handle_destroy(&handle_s);
117 return 1;
118 }
119 std::cout << "----------------------------------------" << std::endl;
120
121 da_handle_destroy(&handle);
122 da_handle_destroy(&handle_s);
123
124 return 0;
125}
1/*
2 * Copyright (C) 2023-2024 Advanced Micro Devices, Inc. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without modification,
5 * are permitted provided that the following conditions are met:
6 * 1. Redistributions of source code must retain the above copyright notice,
7 * this list of conditions and the following disclaimer.
8 * 2. Redistributions in binary form must reproduce the above copyright notice,
9 * this list of conditions and the following disclaimer in the documentation
10 * and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 * may be used to endorse or promote products derived from this software without
13 * specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <assert.h>
30#include <cmath>
31#include <iomanip>
32#include <iostream>
33#include <vector>
34
35/*
36 * Linear model elastic net regression example
37 * using the data set from
38 *
39 * EFRON, HASTIE, JOHNSTONE, and TIBSHIRANI (2004).
40 * Least angle regression (with discussion).
41 * Ann. Statist. 32 407–499. MR2060166
42 * https://hastie.su.domains/Papers/LARS/data64.txt
43 *
44 * The "diabetes data set" consists of 441 observations
45 * and 10 features, while the model chosen is linear and
46 * fitted with both L1 and L2 penalty terms.
47 *
48 * The example showcases how to use datastore framework to
49 * extract data, but it can be directly loaded using
50 * dense matrices using e.g., da_read_csv_d API.
51 */
52#ifndef DATA_DIR
53#define DATA_DIR "data"
54#endif
55
56int main() {
57
58 std::cout << "---------------------------------------------------" << std::endl;
59 std::cout << "Elastic net regression example using diabetes data" << std::endl;
60 std::cout << "---------------------------------------------------" << std::endl
61 << std::endl;
62 std::cout << std::fixed;
63 std::cout.precision(5);
64
65 // Problem data
66 // m: observations; n: features
67 da_int m = 442, n = 10;
68 da_int rhs_pos = 10;
69 std::vector<double> features;
70 std::vector<double> rhs;
71 std::vector<double> x;
72 // n features
73 features.resize(m * n);
74 rhs.resize(m);
75 x.resize(n + 1);
76 // Initial parameter estimates: n + intercept
77 // x.assign({0, 0, 700, 200, 100, 80, 160, 0, 300, 0});
78 x.assign(m + 1, 0);
79
80 // Reference solution
81 std::vector<double> x_ref(n + 1);
82 x_ref.assign({0, -19.4574064435, 4.3253307426, 0.6585289836, 0, 0, -1.6904339251, 0,
83 19.3039496667, 0});
84
85 da_status status;
86
87 // Load data from file
88 da_datastore csv;
89 const char filename[](DATA_DIR "/diabetes.csv");
90 da_datastore_init(&csv);
91 da_datastore_options_set_int(csv, "CSV whitespace delimiter", 1);
92 da_datastore_options_set_string(csv, "CSV comment", "#");
93 da_datastore_options_set_int(csv, "CSV use header row", 1);
94 status = da_data_load_from_csv(csv, filename);
95 if (status != da_status_success) {
96 da_datastore_print_error_message(csv);
97 return 1;
98 }
99 da_int nr, nc;
100 status = da_data_get_n_rows(csv, &nr);
101 status = da_data_get_n_cols(csv, &nc);
102 if (nr != 442 || nc != 11) {
103 std::cout << "Unexpected size for the loaded data: "
104 "(rows="
105 << nr << ", cols=" << nc << ")" << std::endl;
106 return 2;
107 }
108
109 // Extract the 10 features into a dense matrix
110 bool pass = true;
111 pass = pass && da_data_select_columns(csv, "features", 0, n - 1) == da_status_success;
112 pass = pass && da_data_extract_selection_real_d(csv, "features", features.data(),
113 m) == da_status_success;
114 // Extract response variable
115 pass = pass &&
116 da_data_select_columns(csv, "response", rhs_pos, rhs_pos) == da_status_success;
117 pass = pass && da_data_extract_selection_real_d(csv, "response", rhs.data(), m) ==
118 da_status_success;
119 da_datastore_destroy(&csv);
120 if (!pass) {
121 std::cout
122 << "Unexpected error in the feature and response matrices extraction.\n";
123 return 1;
124 }
125
126 // Initialize the linear regression
127 pass = true;
128 da_int nx = 0;
129 da_handle handle = nullptr;
130 pass = pass && da_handle_init_d(&handle, da_handle_linmod) == da_status_success;
131 pass =
132 pass && da_linmod_select_model_d(handle, linmod_model_mse) == da_status_success;
133 pass = pass &&
134 da_options_set_string(handle, "scaling", "standardise") == da_status_success;
135 pass = pass && da_options_set_real_d(handle, "alpha", 1) == da_status_success;
136 pass = pass && da_options_set_real_d(handle, "lambda", 4) == da_status_success;
137 pass = pass &&
138 da_options_set_string(handle, "print options", "yes") == da_status_success;
139 pass = pass && da_options_set_int(handle, "intercept", 0) == da_status_success;
140 pass = pass && da_options_set_int(handle, "print level", 1) == da_status_success;
141 pass = pass && da_linmod_define_features_d(handle, m, n, features.data(),
142 rhs.data()) == da_status_success;
143 if (!pass) {
144 std::cout << "Unexpected error in the model definition.\n";
145 da_handle_destroy(&handle);
146 return 1;
147 }
148
149 // Compute regression
150 status = da_linmod_fit_start_d(handle, n + 1, x.data());
151 bool ok = false;
152 if (status == da_status_success) {
153 std::cout << "Regression computed" << std::endl;
154 // Query the amount of coefficient in the model (n+intercept)
155 da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
156 x.resize(nx);
157 da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
158 std::cout << "Coefficients: " << std::endl;
159 std::cout.precision(3);
160
161 bool oki;
162 ok = !ok;
163 for (da_int i = 0; i < nx; i++) {
164 oki = std::abs(x[i] - x_ref[i]) <= 1.0e-3;
165 std::cout << " x[" << std::setw(2) << i << "] = " << std::setw(9) << x[i]
166 << " expecting " << std::setw(9) << x_ref[i]
167 << (oki ? " (OK)" : " [WRONG]") << std::endl;
168 ok &= oki;
169 }
170 } else {
171 std::cout << "Unexpected error:" << std::endl;
172 da_handle_print_error_message(handle);
173 }
174 std::cout << "----------------------------------------" << std::endl;
175
176 da_handle_destroy(&handle);
177
178 return ok ? 0 : 7;
179}