Linear Models#

The topic of linear models encompasses a range of commonly used statistical models and fitting algorithms, including multiple linear regression, logistic regression, polynomial regression, and nonparametric regression.

The general form of a linear model fitting problem is as follows:

\[\min_{\beta}\left[ C_{\theta}\left( y, g^{-1}(\ \beta \, \phi(X)\ ) \right) \right],\]

where \(X\) is an array of \(n_{\mathrm{samples}}\) observations with \(n_{\mathrm{features}}\) features, \(y\) is an array of \(n_{\mathrm{samples}}\) responses / labels, \(\phi\) is a set of (possibly nonlinear) basis functions, \(\beta\) is a set of weights / coefficients, \(g^{-1}\) is a (possibly nonlinear) activation / link function, and \(C_{\theta}\) is a cost / error function, which may depend on a set of (fixed) hyperparameters, \(\theta\).

Most linear models satisfy a form where the cost function can be split into a loss function (which measures quality of fit to the data) and a penalty term (which regularizes the model parameters). Regularization is also referred to as shrinkage because it tends to shrink the size of the parameter values and/or shrink the number of non-zero parameters.

The loss function is typically a function of the responses or labels (\(y\)), the features (\(X\)), and the model parameters (\(\beta\)), whereas the penalty term is typically only a function of the 1-norm and/or the 2-norm of the model parameters (\(\beta\)). Such linear models are often referred to as elastic nets.

Typically, in addition to the conditions above, \(\phi\) and \(g^{-1}\) are identity mappings and the general form for the cost function becomes,

(1)#\[C_{\{\lambda,\alpha\}} \left( \beta \right) = L(y, \beta X) + \lambda \bigg( \alpha \lVert \beta \rVert_1 + \frac{(1 - \alpha)}{2} \lVert \beta \rVert_2^2 \bigg),\]

where \(0\le\lambda, 0\le\alpha\le1\) are hyperparameters with \(\lambda\) setting the magnitude of the overall penalization and \(\alpha\) distributing its share across the \(\ell_1\) and \(\ell_2\) regularization terms, \(\lVert \beta \rVert_1\) and \(\lVert \beta \rVert_2\). \(L\) is known as the loss function. Linear models where \(\alpha=0\) are called ridge regression models. When \(\alpha=1\) the model is a lasso model, whereas if \(0 < \alpha < 1\) the model is an elastic net regression model.

Intercept

If a linear model contains an intercept term, the parameter vector \(\beta\) has dimension \(n_{\mathrm{features}}+1\) and the observations \(X_i\) are extended with a constant of \(1\). For example, if there is a single feature and the linear model has an intercept term the mean square error loss cost function becomes,

\[C\left( \beta_0, \beta_1 \right) = \sum_{i=0}^n (y_i - \beta_0 - \beta_1 x_i)^2,\]

where \(x_i\) represents a single (scalar) observation and without any regularization terms.

The intercept can be added by setting the linear regression handle option intercept to 1. For further details see Linear model options.

Note

The intercept term is never regularized. In the example above only \(\beta_1\) is subject to regularization.

Available models#

Models can be classified by their loss function, the following subsections present the supported loss functions.

Mean square error#

This is the most basic model and its use is widespread across many applications. The cost function for a linear regression model where the fit (loss) is measured by the mean square error (MSE) is,

\[\begin{split}C_{\{0<\lambda,0\le\alpha\le1\}}\left( \beta_0, \beta \right) & = L(y, \beta_0, \beta X) \\ & = \mathrm{MSE}(y, \beta_0, \beta X)= \frac{1}{2n} \sum_{i=1}^n (y_i - \beta_0 - \beta X_i)^2 + \lambda \bigg( \alpha \lVert \beta \rVert_1 + \frac{(1 - \alpha)}{2} \lVert \beta \rVert_2^2 \bigg),\end{split}\]

where \(X_i\) represents a single (multi-dimensional) observation, i.e., a row in a table of observations. Note that the intercept term \(\beta_0\) is not regularized.

Logistic regression#

Logistic regression is a type of supervised classification model aimed at assigning labels. In AOCL-DA, the labels are expected to be provided in a categorical response variable, \(y\), encoded by \(\{0, 1, 2, \ldots, K-1 \}\). The fit is based on maximizing the log-likelihood (loss function) of the probabilities that each observation \(i\) belongs to a given class, in turn defined by,

\[\begin{split}p(y_i=k\,|\,X_i, \beta) = \frac{ \exp(\beta_k X_i) }{ 1 + \sum_{l=0}^{K-2}\exp(\beta_l X_i) }, \mathrm{ for }\ 0 \leq k < K-1,\\ p(y_i=K-1\,|\,X_i, \beta) = \frac{ 1 }{ 1 + \sum_{l=0}^{K-2}\exp(\beta_l X_i) }.\end{split}\]

As an example, if \(K=2\), the loss function simplifies to,

\[C\left( \beta \right) = -L(y, \beta X) = \sum_{i=0}^n \bigg( y_i \log p(X_i, \beta) + (1 - y_i) \log \big( 1 - p(X_i, \beta) \big) \bigg).\]

As in the linear regression model, \(\ell_1\) or \(\ell_2\) regularization can be applied by adding the corresponding penalty term to the cost function.

Fitting methods#

Different methods are available to compute the models. The method is chosen automatically by default but can be set manually using the optional parameter optim method (see Linear model options).

Direct solvers

  • QR: the standard MSE linear regression model can be computed using the QR factorization of the data matrix if no regularization term is required.

    \[X = QR,\]

    where \(Q\) is a \(n_{\mathrm{samples}} \times n_{\mathrm{features}}\) matrix with orthogonal columns and \(R\) is a \(n_{\mathrm{features}}\times n_{\mathrm{features}}\) triangular matrix.

  • SVD: the singular value decomposition can be used to compute standard MSE and ridge regression models.

    \[X = UDV^T,\]

    where \(U\) is an orthogonal matrix of size \(n_{\mathrm{samples}}\times n_{\mathrm{samples}}\), \(D\) is a \(n_{\mathrm{samples}}\times n_{\mathrm{features}}\) diagonal matrix whose elements are the non-negative singular values of \(X\) and \(V^T\) is an orthogonal matrix of size \(n_{\mathrm{features}} \times n_{\mathrm{features}}\).

  • Cholesky: the Cholesky decomposition can be used for normal and ridge regression when the data matrix is full-rank. It factorizes the symmetric positive-definite normal equations matrix \(X^TX\) into two triangular matrices. In linear models it can be used to find coefficients expressed as:

    \[\beta = (X^TX+\lambda)^{-1}X^Ty,\]

    where, after left multiplying by the expression inside of the inverse, we end up with a system of linear equations in the form \(Ax=B\). The left hand side can be factorized using the Cholesky decomposition as follows:

    \[X^TX+\lambda = LL^T,\]

    where \(L\) is a lower triangular matrix with real and positive diagonal entries. This matrix is then used to find a solution to the system of linear equations.

Iterative solvers

  • L-BFGS-B: a solver aimed at minimizing smooth nonlinear functions (Liu and Nocedal [1989]). It can be used to compute both MSE and logistic models with or without \(\ell_2\) regularization. It is not suitable when an \(\ell_1\) regularization term is required.

  • Coordinate descent: a solver aimed at minimizing nonlinear functions. It is suitable for linear models with an \(\ell_1\) regularization term and elastic nets (Friedman et al. [2010], Friedman et al. [2007]).

    Note

    The coordinate descent method implemented is optimized to solve lasso or elastic net problems. For ridge or unregularized problems the use of any alternative methods is recommended.

    Warning

    The implemented coordinate descent method is designed to fit standardized data. As such, it should not be used with the scaling method set to anything other than scaling only (default), or standardize (see next section). Using any other type of scaling will result in lack of convergence or unexpected results.

  • Conjugate gradient: a solver aimed at finding a solution to a system of linear equations. It can be used to compute linear regression with or without \(\ell_2\) regularization.

Scaling the data#

In many circumstances, the data used to perform a fit is badly scaled and rescaling can have numerical benefits. Furthermore, iterative solvers can show improved quality of the solution when fitting on rescaled data.

If scaling is requested, then the fitting routine takes care of rescaling the problem data. Once the model is trained, it reverts the scaling on the trained coefficients, so this process is transparent to the user.

The optional parameter scaling controls the scaling of the data. It defaults to auto, where it selects the type of scaling according to the optimization method used. The following table shows the default for each solver, while the table Linear model data scaling types shows the supported scaling types.

Default scaling type when optional parameter scaling = auto#

Method

option optim method

option scaling

Conjugate Gradient Method

cg

centering or none

Coordinate Descent Method

coord

scale only

Singular Value Decomposition

svd

centering

QR Factorization

qr

centering

Cholesky Factorization

cholesky

centering or none

L-BFGS-B Solver

lbfgsb

centering or none

When a solver provides two default options for scaling, then it is chosen based on problem characteristics, e.g., if the problem is underdetermined or if the model has an intercept.

Note

Scaling is applied prior to solving the problem and hence the regularization (if any) is done over the scaled problem and not on the unscaled version. This has implications on the trained coefficients and may differ from the regularized model trained with unscaled data.

The following table shows the supported scaling types. In the table, \(N\) is a shorthand for \(n_{\mathrm{samples}}\); \(\sigma_Z\) refers to the sample standard deviation of \(Z\); and \(\mu_Z\) represents the sample mean of \(Z\). Finally, \(\hat Z\) represents the scaled version of \(Z\). Specifically, in the table below columns labeled \(\hat Y\) and \(\hat X_j\) report the transforms performed on the response vector \(Y\) and the columns of the predictor matrix \(X\).

Linear model data scaling types#

scaling value

model intercept

\(m_Y\)

\(s_Y\)

\(hat Y\)

\(m_{X_j}\)

\(s_{X_j}\)

\(hat X_j\)

none

yes/no

\(0\)

\(1\)

\(Y\)

\(0\)

\(1\)

\(X_j\)

centering

yes

\(\mu_Y\)

\(1\)

\(Y - \mu_Y\)

\(\mu_{X_j}\)

\(1\)

\(X_j - \mu_{X_j}\)

centering

no

\(0\)

\(1\)

\(Y\)

\(0\)

\(1\)

\(X_j\)

scale only

yes

\(\mu_Y\)

\(\sigma_Y\)

\(\frac{\frac{1}{\sqrt{N}}(Y-\mu_Y)}{\sigma_Y}\)

\(\mu_{X_j}\)

\(1\)

\(\frac{1}{\sqrt{N}}(X_j-\mu_{X_j})\)

scale only

no

\(0\)

\(\left\|\frac{1}{\sqrt{N}} Y\right\|\)

\(\frac{\frac{1}{\sqrt{N}} Y}{s_Y}\)

\(0\)

\(1\)

\(\frac{1}{\sqrt{N}} X_j\)

standardize

yes

\(\mu_Y\)

\(\sigma_Y\)

\(\frac{\frac{1}{\sqrt{N}}(Y-\mu_Y)}{\sigma_Y}\)

\(\mu_{X_j}\)

\(\sigma_{X_j}\)

\(\frac{\frac{1}{\sqrt{N}} (X_j-\mu_{X_j})}{\sigma_{X_j}}\)

standardize

no

\(0\)

\(\left\|\frac{1}{\sqrt{N}} Y\right\|\)

\(\frac{\frac{1}{\sqrt{N}} Y}{s_Y}\)

\(0\)

\(\sigma_{X_j}\)

\(\frac{\frac{1}{\sqrt{N}} X_j}{\sigma_{X_j}}\)

Note

Scaling types scale only and standardize also affect the regularization penalty term \(\lambda\) in (1).

Iterative solvers can show the convergence progress by printing to standard output some progress metric for each iteration (as specified by the optional parameter print level). It is important to note that all reported metrics are based on the rescaled data.

When requesting information from the handle, after training the model, the reported metrics are also based on the scaling type used.

Initial coefficients#

For iterative solvers, it is possible to provide a warm-start for the process via an initial guess for the coefficients (for example, these may have been obtained via previous fit). When provided with too many coefficients, only the first \(k\) coefficients will be used, where \(k\) is the expected number of coefficients. The intercept will always be the last member of the provided array. If too few coefficients are provided, the initial guess will be ignored.

Warning

For the conjugate gradient solver in the underdetermined case (\(n_{\mathrm{features}} > n_{\mathrm{samples}}\)) we are solving the dual problem, and thus the initial coefficients should also be dual (so that the expected number of coefficients \(k\) is equal to \(n_{\mathrm{samples}}\) instead of \(n_{\mathrm{features}}\)).

Initial coefficients can be provided in the following way:

Provide x0 parameter when calling aoclda.linear_model.linmod.fit().

Provide coefs pointer to initial coefficients while calling da_linmod_fit_start_?.

Typical workflow for linear models#

The standard way of computing a linear model using AOCL-DA is as follows.

  1. Initialize a aoclda.linear_model.linmod() object with options set in the class constructor.

  2. Fit the linear model to your data using aoclda.linear_model.linmod.fit().

  3. Extract results from the aoclda.linear_model.linmod() object via its class attributes.

  1. Initialize a da_handle with da_handle_type da_handle_linmod.

  2. Pass data to the handle using da_linmod_define_features_?.

  3. Customize the model using da_options_set_? (see below for a list of the available options).

  4. Compute the linear model using da_linmod_fit_?.

  5. Evaluate the model on new data using da_linmod_evaluate_model_?.

  6. Extract results using da_handle_get_result_?. The following results are available:

    • Coefficients (da_linmod_coef): the optimal coefficients of the fitted model.

    • Some solvers provide extra information. (da_linmod_rinfo), when available, contains the info[100] array with the following values:

      • info[0]: loss value at current iterate, \(L(\beta_0, \beta)\),

      • info[1]: norm of the gradient of the loss function,

      • info[2]: number of iterations made (only for iterative solvers),

      • info[3]: compute time (wall clock time in seconds),

      • info[4]: number of model evaluations performed,

      • info[5]: infinity norm of the optimization metric (varies with the method used),

      • info[6]: infinity norm of of a given metric at the initial iterate (varies with the method used),

      • info[7]: number of cheap model evaluations (only relevant for Coordinate Descent Method) and indicates the number of low-rank updates used to evaluate model,

      • info[8-99]: reserved for future use.

      Note

      When information is not available, -1 is returned.

Linear model options#

The available Python options are detailed in the aoclda.linear_model.linmod() class constructor.

Various options can be set to customize the linear models by calling one of these functions. The following table details the available options, where \(\epsilon\) represents the machine precision.

Linear models options#

Option name

Type

Default

Description

Constraints

optim method

string

\(s=\) auto

Select optimization method to use.

\(s=\) auto, bfgs, cg, chol, cholesky, coord, lbfgs, lbfgsb, qr, sparse_cg, or svd.

scaling

string

\(s=\) auto

Scale or standardize feature matrix and response vector. Matrix is copied and then rescaled. Option key value auto indicates that rescaling type is chosen by the solver (this also includes no scaling).

\(s=\) auto, centering, no, none, scale, scale only, standardise, or standardize.

optim progress factor

real

\(r=\frac{10}{\sqrt{2\,\varepsilon}}\)

Factor used to detect convergence of the iterative optimization step. See option in the corresponding optimization solver documentation.

\(0 \le r\)

optim convergence tol

real

\(r=10/2\sqrt{2\,\varepsilon}\)

Tolerance to declare convergence for the iterative optimization step. See option in the corresponding optimization solver documentation.

\(0 < r < 1\)

print options

string

\(s=\) no

Print options.

\(s=\) no, or yes.

lambda

real

\(r=0\)

Penalty coefficient for the regularization terms: lambda( (1-alpha)/2 L2 + alpha L1 ).

\(0 \le r\)

alpha

real

\(r=0\)

Coefficient of alpha in the regularization terms: lambda( (1-alpha)/2 L2 + alpha L1 ).

\(0 \le r \le 1\)

optim iteration limit

integer

\(i=10000\)

Maximum number of iterations to perform in the optimization phase. Valid only for iterative solvers, e.g. L-BFGS-B, Coordinate Descent, etc.

\(1 \le i\)

intercept

integer

\(i=0\)

Add intercept variable to the model.

\(0 \le i \le 1\)

print level

integer

\(i=0\)

Set level of verbosity for the solver.

\(0 \le i \le 5\)

For the complete list of optional parameters see linear model options.

Examples#

The code below is supplied with your installation (see Python examples).

 1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
 2#
 3# Redistribution and use in source and binary forms, with or without modification,
 4# are permitted provided that the following conditions are met:
 5# 1. Redistributions of source code must retain the above copyright notice,
 6#    this list of conditions and the following disclaimer.
 7# 2. Redistributions in binary form must reproduce the above copyright notice,
 8#    this list of conditions and the following disclaimer in the documentation
 9#    and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11#    may be used to endorse or promote products derived from this software without
12#    specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25# pylint: disable = invalid-name
26
27"""
28Linear regression example Python script
29"""
30
31import sys
32from aoclda.linear_model import linmod
33import numpy as np
34
35
36def linmod_example():
37    """
38    Linear regression with an intercept variable
39    """
40    X = np.array([[1, 1], [2, 3], [3, 5], [4, 8], [5, 7], [6, 9]])
41    y = np.array([3., 6.5, 10., 12., 13., 19.])
42    lmod = linmod("mse", intercept=True)
43    lmod.fit(X, y)
44
45    # Extract coefficients
46    coef = lmod.coef
47    print(f"coefficients: [{coef[0]:.3f}, {coef[1]:.3f}, {coef[2]:.3f}]")
48    print('expected    : [2.350, 0.350, 0.433]\n')
49
50    # Evaluate model on new data
51    X_test = np.array([[1, 1.1], [2.5, 3], [7, 9]])
52    pred = lmod.predict(X_test)
53    print(f"predictions: [{pred[0]:.3f}, {pred[1]:.3f}, {pred[2]:.3f}]")
54    print('expected   : [3.168  7.358 20.0333]')
55
56
57if __name__ == "__main__":
58    try:
59        linmod_example()
60    except RuntimeError:
61        sys.exit(1)

The showcased examples sources and more, can be found in the examples folder of your installation.

  1/*
  2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
  3 *
  4 * Redistribution and use in source and binary forms, with or without modification,
  5 * are permitted provided that the following conditions are met:
  6 * 1. Redistributions of source code must retain the above copyright notice,
  7 *    this list of conditions and the following disclaimer.
  8 * 2. Redistributions in binary form must reproduce the above copyright notice,
  9 *    this list of conditions and the following disclaimer in the documentation
 10 *    and/or other materials provided with the distribution.
 11 * 3. Neither the name of the copyright holder nor the names of its contributors
 12 *    may be used to endorse or promote products derived from this software without
 13 *    specific prior written permission.
 14 *
 15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 24 * POSSIBILITY OF SUCH DAMAGE.
 25 *
 26 */
 27
 28#include "aoclda.h"
 29#include <assert.h>
 30#include <iostream>
 31
 32int main() {
 33
 34    std::cout << "----------------------------------------" << std::endl;
 35    std::cout << "Mean squared error model" << std::endl;
 36    std::cout << "min ||Ax-b||^2; with A an 5x2 matrix" << std::endl << std::endl;
 37    std::cout << std::fixed;
 38    std::cout.precision(5);
 39
 40    // Problem data
 41    da_int m = 5, n = 2;
 42    double Al[10] = {1, 2, 3, 4, 5, 1, 3, 5, 1, 1};
 43    double bl[5] = {1, 1, 1, 1, 1};
 44    da_int nx = 2;
 45    double x[2];
 46
 47    // Initialize the linear regression
 48    da_handle handle = nullptr;
 49    da_status status;
 50    bool pass = true;
 51    pass = pass && da_handle_init_d(&handle, da_handle_linmod) == da_status_success;
 52    pass =
 53        pass && da_linmod_select_model_d(handle, linmod_model_mse) == da_status_success;
 54    pass = pass && da_linmod_define_features_d(handle, m, n, Al, bl) == da_status_success;
 55    if (!pass) {
 56        std::cout << "Something unexpected happened in the model definition\n";
 57        da_handle_destroy(&handle);
 58        return 1;
 59    }
 60
 61    // Compute regression
 62    status = da_linmod_fit_d(handle);
 63    if (status == da_status_success) {
 64        std::cout << "regression computed successfully!" << std::endl;
 65        nx = 0; // Query the correct size
 66        da_handle_get_result_d(handle, da_linmod_coef, &nx, x);
 67        assert(nx == 2);
 68        da_handle_get_result_d(handle, da_linmod_coef, &nx, x);
 69        std::cout << "Coefficients: " << x[0] << " " << x[1] << std::endl;
 70        std::cout << "(Expected   : " << 0.199256 << " " << 0.130354 << ")" << std::endl;
 71    } else {
 72        std::cout << "Something wrong happened during MSE regression. Terminating"
 73                  << std::endl;
 74        da_handle_destroy(&handle);
 75        return 1;
 76    }
 77    std::cout << "----------------------------------------" << std::endl;
 78
 79    // Solve the same model with single precision
 80    // Problem data
 81    float As[10] = {1, 2, 3, 4, 5, 1, 3, 5, 1, 1};
 82    float bs[5] = {1, 1, 1, 1, 1};
 83    float xs[2];
 84
 85    std::cout.precision(2);
 86    // Initialize the linear regression
 87    da_handle handle_s = nullptr;
 88    pass = true;
 89    pass = pass && da_handle_init_s(&handle_s, da_handle_linmod) == da_status_success;
 90    pass =
 91        pass && da_linmod_select_model_s(handle_s, linmod_model_mse) == da_status_success;
 92    pass =
 93        pass && da_linmod_define_features_s(handle_s, m, n, As, bs) == da_status_success;
 94    if (!pass) {
 95        std::cout << "Something unexpected happened in the model definition\n";
 96        da_handle_destroy(&handle);
 97        da_handle_destroy(&handle_s);
 98        return 1;
 99    }
100
101    // Compute regression
102    status = da_linmod_fit_s(handle_s);
103    if (status == da_status_success) {
104        std::cout << "regression computed successfully!" << std::endl;
105        // status = da_linmod_s_get_coef(handle_s, &nx, xs);
106        nx = 0; // Query the correct size
107        da_handle_get_result_s(handle_s, da_linmod_coef, &nx, xs);
108        assert(nx == 2);
109        da_handle_get_result_s(handle_s, da_linmod_coef, &nx, xs);
110        std::cout << "Coefficients: " << xs[0] << " " << xs[1] << std::endl;
111        std::cout << "(Expected   : " << 0.20 << " " << 0.13 << ")" << std::endl;
112    } else {
113        std::cout << "Something wrong happened during MSE regression. Terminating"
114                  << std::endl;
115        da_handle_destroy(&handle);
116        da_handle_destroy(&handle_s);
117        return 1;
118    }
119    std::cout << "----------------------------------------" << std::endl;
120
121    da_handle_destroy(&handle);
122    da_handle_destroy(&handle_s);
123
124    return 0;
125}
  1/*
  2 * Copyright (C) 2023-2024 Advanced Micro Devices, Inc. All rights reserved.
  3 *
  4 * Redistribution and use in source and binary forms, with or without modification,
  5 * are permitted provided that the following conditions are met:
  6 * 1. Redistributions of source code must retain the above copyright notice,
  7 *    this list of conditions and the following disclaimer.
  8 * 2. Redistributions in binary form must reproduce the above copyright notice,
  9 *    this list of conditions and the following disclaimer in the documentation
 10 *    and/or other materials provided with the distribution.
 11 * 3. Neither the name of the copyright holder nor the names of its contributors
 12 *    may be used to endorse or promote products derived from this software without
 13 *    specific prior written permission.
 14 *
 15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 24 * POSSIBILITY OF SUCH DAMAGE.
 25 *
 26 */
 27
 28#include "aoclda.h"
 29#include <assert.h>
 30#include <cmath>
 31#include <iomanip>
 32#include <iostream>
 33#include <vector>
 34
 35/*
 36 * Linear model elastic net regression example
 37 * using the data set from
 38 *
 39 * EFRON, HASTIE, JOHNSTONE, and TIBSHIRANI (2004).
 40 * Least angle regression (with discussion).
 41 * Ann. Statist. 32 407–499. MR2060166
 42 * https://hastie.su.domains/Papers/LARS/data64.txt
 43 *
 44 * The "diabetes data set" consists of 441 observations
 45 * and 10 features, while the model chosen is linear and
 46 * fitted with both L1 and L2 penalty terms.
 47 *
 48 * The example showcases how to use datastore framework to
 49 * extract data, but it can be directly loaded using
 50 * dense matrices using e.g., da_read_csv_d API.
 51 */
 52#ifndef DATA_DIR
 53#define DATA_DIR "data"
 54#endif
 55
 56int main() {
 57
 58    std::cout << "---------------------------------------------------" << std::endl;
 59    std::cout << "Elastic net regression example using diabetes data" << std::endl;
 60    std::cout << "---------------------------------------------------" << std::endl
 61              << std::endl;
 62    std::cout << std::fixed;
 63    std::cout.precision(5);
 64
 65    // Problem data
 66    // m: observations; n: features
 67    da_int m = 442, n = 10;
 68    da_int rhs_pos = 10;
 69    std::vector<double> features;
 70    std::vector<double> rhs;
 71    std::vector<double> x;
 72    // n features
 73    features.resize(m * n);
 74    rhs.resize(m);
 75    x.resize(n + 1);
 76    // Initial parameter estimates: n + intercept
 77    // x.assign({0, 0, 700, 200, 100, 80, 160, 0, 300, 0});
 78    x.assign(m + 1, 0);
 79
 80    // Reference solution
 81    std::vector<double> x_ref(n + 1);
 82    x_ref.assign({0, -19.4574064435, 4.3253307426, 0.6585289836, 0, 0, -1.6904339251, 0,
 83                  19.3039496667, 0});
 84
 85    da_status status;
 86
 87    // Load data from file
 88    da_datastore csv;
 89    const char filename[](DATA_DIR "/diabetes.csv");
 90    da_datastore_init(&csv);
 91    da_datastore_options_set_int(csv, "CSV whitespace delimiter", 1);
 92    da_datastore_options_set_string(csv, "CSV comment", "#");
 93    da_datastore_options_set_int(csv, "CSV use header row", 1);
 94    status = da_data_load_from_csv(csv, filename);
 95    if (status != da_status_success) {
 96        da_datastore_print_error_message(csv);
 97        return 1;
 98    }
 99    da_int nr, nc;
100    status = da_data_get_n_rows(csv, &nr);
101    status = da_data_get_n_cols(csv, &nc);
102    if (nr != 442 || nc != 11) {
103        std::cout << "Unexpected size for the loaded data: "
104                     "(rows="
105                  << nr << ", cols=" << nc << ")" << std::endl;
106        return 2;
107    }
108
109    // Extract the 10 features into a dense matrix
110    bool pass = true;
111    pass = pass && da_data_select_columns(csv, "features", 0, n - 1) == da_status_success;
112    pass = pass && da_data_extract_selection_real_d(csv, "features", features.data(),
113                                                    m) == da_status_success;
114    // Extract response variable
115    pass = pass &&
116           da_data_select_columns(csv, "response", rhs_pos, rhs_pos) == da_status_success;
117    pass = pass && da_data_extract_selection_real_d(csv, "response", rhs.data(), m) ==
118                       da_status_success;
119    da_datastore_destroy(&csv);
120    if (!pass) {
121        std::cout
122            << "Unexpected error in the feature and response matrices extraction.\n";
123        return 1;
124    }
125
126    // Initialize the linear regression
127    pass = true;
128    da_int nx = 0;
129    da_handle handle = nullptr;
130    pass = pass && da_handle_init_d(&handle, da_handle_linmod) == da_status_success;
131    pass =
132        pass && da_linmod_select_model_d(handle, linmod_model_mse) == da_status_success;
133    pass = pass &&
134           da_options_set_string(handle, "scaling", "standardise") == da_status_success;
135    pass = pass && da_options_set_real_d(handle, "alpha", 1) == da_status_success;
136    pass = pass && da_options_set_real_d(handle, "lambda", 4) == da_status_success;
137    pass = pass &&
138           da_options_set_string(handle, "print options", "yes") == da_status_success;
139    pass = pass && da_options_set_int(handle, "intercept", 0) == da_status_success;
140    pass = pass && da_options_set_int(handle, "print level", 1) == da_status_success;
141    pass = pass && da_linmod_define_features_d(handle, m, n, features.data(),
142                                               rhs.data()) == da_status_success;
143    if (!pass) {
144        std::cout << "Unexpected error in the model definition.\n";
145        da_handle_destroy(&handle);
146        return 1;
147    }
148
149    // Compute regression
150    status = da_linmod_fit_start_d(handle, n + 1, x.data());
151    bool ok = false;
152    if (status == da_status_success) {
153        std::cout << "Regression computed" << std::endl;
154        // Query the amount of coefficient in the model (n+intercept)
155        da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
156        x.resize(nx);
157        da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
158        std::cout << "Coefficients: " << std::endl;
159        std::cout.precision(3);
160
161        bool oki;
162        ok = !ok;
163        for (da_int i = 0; i < nx; i++) {
164            oki = std::abs(x[i] - x_ref[i]) <= 1.0e-3;
165            std::cout << " x[" << std::setw(2) << i << "] = " << std::setw(9) << x[i]
166                      << " expecting " << std::setw(9) << x_ref[i]
167                      << (oki ? " (OK)" : " [WRONG]") << std::endl;
168            ok &= oki;
169        }
170    } else {
171        std::cout << "Unexpected error:" << std::endl;
172        da_handle_print_error_message(handle);
173    }
174    std::cout << "----------------------------------------" << std::endl;
175
176    da_handle_destroy(&handle);
177
178    return ok ? 0 : 7;
179}

Further reading#

An introduction to linear models for Regression and Classification can be found in Chapters 3, 4 of Bishop [2006], or in Chapters 3-5 of Hastie et al. [2009].