Matrix Factorizations#

This chapter contains functions for decomposing a data matrix into the product of two or more matrices. Matrix factorizations are commonly used for dimensionality reduction and feature extraction.

Principal component analysis#

In a principal component analysis (PCA) a set of possibly correlated feature vectors (the columns of the data matrix) is transformed linearly into a new, uncorrelated coordinate system. The new coordinates (which are known as the principal components) are chosen such that the first coordinate accounts for the greatest variance in the data, the second coordinate accounts for the second greatest variance, etc. By using only the first few such coordinates, the data matrix can be reduced in dimension.

Prior to computing the PCA the data matrix is typically standardized by shifting each column so that it has a mean of zero. It can then be shown that the principal components are the eigenvalues of the covariance matrix corresponding to the mean-centered data matrix.

If the features of the data matrix vary greatly in magnitude, then in addition to mean-centering it can be useful to normalize each column by its standard deviation. In this case the principal components are the eigenvalues of the correlation matrix corresponding to the mean-centered data matrix.

The PCA is closely related to a matrix factorization known as the singular value decomposition (or SVD),

\[A = U\Sigma V^T,\]

where \(A\) is a (standardized) data matrix of size \(n_{\mathrm{samples}} \times n_{\mathrm{features}}\), \(\Sigma\) is a non-negative diagonal matrix of size \(n_{\mathrm{samples}} \times n_{\mathrm{features}}\) and \(U\) and \(V\) are orthogonal matrices of size \(n_{\mathrm{samples}} \times n_{\mathrm{samples}}\) and \(n_{\mathrm{features}} \times n_{\mathrm{features}}\) respectively. The nonzero entries of \(\Sigma\) are known as the singular values of \(A\).

AOCL-DA can compute the PCA via the SVD (with \(\Sigma\) and \(V\) truncated according to the number of principal components requested) or using an eigenvalue decomposition of the covariance/correlation matrix (which is faster for tall, thin matrices, but can be numerically unstable for ill-conditioned problems).

Outputs from the PCA#

After a PCA computation the following results are stored:

  • principal components - the “new coordinates” expressed in terms of the old coordinates. These are sorted in order of decreasing variance, and are given by the rows of \(V^T\).

  • scores - the data matrix expressed in terms of the new coordinates. This is given by \(U\Sigma\).

  • variance - the amount of variance explained by each of the principal components. Note that \(n\_samples -1\) degrees of freedom are used when computing variances.

  • total variance - the total variance across the whole dataset.

  • the SVD matrices - \(U\), \(V^T\) and \(\Sigma\) together with the column means and standard deviations.

After the PCA has been computed, two post-processing operations may be of interest:

  • transform - given a data matrix \(X\) in the same coordinates as the original data matrix \(A\), express \(X\) in terms of the new coordinates (the principal components of \(A\)). This is computed by applying any standardization used on \(A\) to \(X\) and post-multiplying by \(V\).

  • inverse transform - given a data matrix \(Y\) in the new coordinate system, express \(Y\) in terms of the original coordinates. This is computed by post-multiplying by \(V^T\) and inverting the standardization used on \(A\).

Typical workflow for PCA#

The standard way of computing the principal component analysis using AOCL-DA is as follows.

  1. Initialize a aoclda.factorization.PCA() object with options set in the class constructor.

  2. Compute the PCA for your data matrix using aoclda.factorization.PCA.fit().

  3. Perform further transformations in necessary using aoclda.factorization.PCA.transform() or aoclda.factorization.PCA.inverse_transform().

  4. Extract results from the aoclda.factorization.PCA() object via its class attributes.

  1. Initialize a da_handle with da_handle_type da_handle_pca.

  2. Pass data to the handle using da_pca_set_data_?.

  3. Set the number of principal components required and the type of PCA using da_options_set_? (see below).

  4. Compute the PCA using da_pca_compute_?.

  5. Perform further transformations as required, using da_pca_transform_? or da_pca_inverse_transform_?.

  6. Extract results using da_handle_get_result_?.

Options#

The available Python options are detailed in the aoclda.factorization.PCA() class constructor.

The following options can be set using da_options_set_?:

PCA options#

Option Name

Type

Default

Description

Constraints

pca method

string

\(s=\) covariance

Compute PCA based on the covariance or correlation matrix.

\(s=\) correlation, covariance, or svd.

degrees of freedom

string

\(s=\) unbiased

Whether to use biased or unbiased estimators for standard deviations and variances.

\(s=\) biased, or unbiased.

n_components

integer

\(i=1\)

Number of principal components to compute. If 0, then all components will be kept.

\(0 \le i\)

svd solver

string

\(s=\) auto

Which LAPACK routine to use for the underlying singular value decomposition.

\(s=\) auto, gesdd, gesvd, gesvdx, or syevd.

If the pca method option is set to svd then no standardization is performed. This option should be used if the input data is already standardized or if an explicit singular value decomposition is required. Note, however, that if the columns of the data matrix are not mean-centered, then the computed variance and total_variance will be meaningless.

If a full decomposition is required (so that all principal components are found) then svd solver should be set to gesdd. The LAPACK routines DGESDD or SGESDD (for double and single precision data respectively) will then be used. This choice offers the best performance, while maintaining high accuracy. Note that if internal heuristics determine that it is useful, a QR decomposition may be performed prior to the SVD.

If svd solver is set to syevd then the SVD will be found by explicitly forming the covariance or correlation matrix and using LAPACK routines DSYEVD or SSYEVD to perform an eigendecomposition. This is very fast for tall, thin data matrices but for wider matrices it requires a lot of memory. The method is also more susceptible to ill-conditioning so must be used with care. It is incompatible with the store U option.

svd solver should only be set to gesvd (so that the LAPACK routines DGESVD or SGESVD are used) if there is insufficient memory for the workspace requirements of gesdd, or if gesdd encounters convergence issues. If only one or two principal components are required then, depending on your data matrix, gesvdx may be faster (so that the LAPACK routines DGESVDX or SGESVDX are used).

If svd solver is set to auto, then DGESDD or SGESDD will be used unless internal heuristics determine that the eigendecomposition may be used.

If store U is set to 1, then the matrix \(U\) from the SVD will be stored and used to ensure deterministic results in the signs of the principal components. Note that there may be a small performance penalty in setting this option and it cannot be used if svd solver is set to syevd.

Examples#

The code below is supplied with your installation (see Python examples).

 1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
 2#
 3# Redistribution and use in source and binary forms, with or without modification,
 4# are permitted provided that the following conditions are met:
 5# 1. Redistributions of source code must retain the above copyright notice,
 6#    this list of conditions and the following disclaimer.
 7# 2. Redistributions in binary form must reproduce the above copyright notice,
 8#    this list of conditions and the following disclaimer in the documentation
 9#    and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11#    may be used to endorse or promote products derived from this software without
12#    specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25
26"""
27Principal component analysis example Python script
28"""
29
30import sys
31import numpy as np
32from aoclda.factorization import PCA
33
34
35def pca_example():
36    """
37    Principal component analysis example
38    """
39
40    # Define data arrays
41    a = np.array([[2., 2., 3., 2., 4.],
42                  [2., 5., 4., 8., 3.],
43                  [3., 2., 4., 4., 1.],
44                  [4., 8., 3., 6., 4.],
45                  [4., 3., 2., 9., 2.],
46                  [3., 2., 1., 5., 2.]])
47
48    x = np.array([[7., 4., 2., 9., 3.],
49                  [3., 2., 5., 6., 4.],
50                  [3., 3., 2., 4., 1.]])
51
52    print("\nPrincipal component analysis for a 6x5 data matrix\n")
53    try:
54        pca = PCA(n_components=3, store_U=True)
55        pca.fit(a)
56        x_transform = pca.transform(x)
57    except RuntimeError:
58        sys.exit(1)
59
60    # Print results
61    print("\nPrincipal components of a:\n")
62    print(pca.principal_components)
63    print("\nx_transform:\n")
64    print(x_transform)
65
66    # Check against expected results
67    expected_components = np.array([[-0.14907884486130418, -0.6612054163818867,
68                                     -0.031706610956396264, -0.7289116905829763,
69                                     -0.09091387966203135],
70                                    [-0.07220367025708045,  0.623738867070505,
71                                     0.20952521660694667, -0.6138062400926413,
72                                     0.4302063910917139],
73                                    [-0.38718653977350936, -0.06907631947413592,
74                                     0.8854125206703791,  0.1296593407398653,
75                                     -0.21106437194645863]])
76
77    expected_x_transform = np.array([[-3.250305270939447, -2.1581247424555086, -1.9477723543266676],
78                                     [0.6691223004872521, -0.21658703437771865,
79                                         1.7953216115607247],
80                                     [1.833601737126601, -0.2844305102179128, -0.5561178355649032]])
81
82    norm_components = np.linalg.norm(
83        pca.principal_components - expected_components)
84    norm_x_transform = np.linalg.norm(x_transform - expected_x_transform)
85
86    tol = 1.0e-12
87
88    if norm_components > tol or norm_x_transform > tol:
89        print("\nSolution is not within expected tolerance\n")
90        sys.exit(1)
91
92    print("\nPCA successfully computed\n")
93
94
95if __name__ == "__main__":
96    pca_example()

The code below can be found in pca.cpp in the examples folder of your installation.

  1/*
  2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
  3 *
  4 * Redistribution and use in source and binary forms, with or without modification,
  5 * are permitted provided that the following conditions are met:
  6 * 1. Redistributions of source code must retain the above copyright notice,
  7 *    this list of conditions and the following disclaimer.
  8 * 2. Redistributions in binary form must reproduce the above copyright notice,
  9 *    this list of conditions and the following disclaimer in the documentation
 10 *    and/or other materials provided with the distribution.
 11 * 3. Neither the name of the copyright holder nor the names of its contributors
 12 *    may be used to endorse or promote products derived from this software without
 13 *    specific prior written permission.
 14 *
 15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 24 * POSSIBILITY OF SUCH DAMAGE.
 25 *
 26 */
 27
 28#include "aoclda.h"
 29#include <iomanip>
 30#include <iostream>
 31
 32/* Basic PCA example
 33 *
 34 * This example computes a principal component
 35 * analysis for a small data matrix.
 36 */
 37
 38int main() {
 39
 40    // Initialize the handle
 41    da_handle handle = nullptr;
 42
 43    std::cout << "-----------------------------------------------------------------------"
 44              << std::endl;
 45    std::cout << "Basic PCA" << std::endl;
 46    std::cout << "Principal component analysis for a 6x5 data matrix" << std::endl
 47              << std::endl;
 48    std::cout << std::fixed;
 49    std::cout.precision(5);
 50
 51    int exit_code = 0;
 52    bool pass = true;
 53
 54    // Input data
 55    double A[30] = {2.0, 2.0, 3.0, 4.0, 4.0, 3.0, 2.0, 5.0, 2.0, 8.0,
 56                    3.0, 2.0, 3.0, 4.0, 4.0, 3.0, 2.0, 1.0, 2.0, 8.0,
 57                    4.0, 6.0, 9.0, 5.0, 4.0, 3.0, 1.0, 4.0, 2.0, 2.0};
 58
 59    da_int n_samples = 6, n_features = 5, n_components = 3, lda = 6;
 60
 61    // Create the handle and pass it the data matrix
 62    pass = pass && (da_handle_init_d(&handle, da_handle_pca) == da_status_success);
 63    pass = pass && (da_pca_set_data_d(handle, n_samples, n_features, A, lda) ==
 64                    da_status_success);
 65
 66    // Set options
 67    pass = pass && (da_options_set_string(handle, "PCA method", "covariance") ==
 68                    da_status_success);
 69    pass = pass && (da_options_set_int(handle, "n_components", n_components) ==
 70                    da_status_success);
 71    pass = pass && (da_options_set_int(handle, "store U", 1) == da_status_success);
 72
 73    // Compute the PCA
 74    pass = pass && (da_pca_compute_d(handle) == da_status_success);
 75
 76    // Transform another data matrix into the same feature space
 77    double X[15] = {7.0, 3.0, 3.0, 4.0, 2.0, 3.0, 2.0, 5.0,
 78                    2.0, 9.0, 6.0, 4.0, 3.0, 4.0, 1.0};
 79    da_int m_samples = 3, m_features = 5, ldx = 3, ldx_transform = 3;
 80    double *X_transform = new double[m_samples * n_components];
 81    pass = pass && (da_pca_transform_d(handle, m_samples, m_features, X, ldx, X_transform,
 82                                       ldx_transform) == da_status_success);
 83
 84    // Extract results from the handle
 85    da_int principal_components_dim = n_components * n_features;
 86    da_int scores_dim = n_samples * n_components;
 87    double *principal_components = new double[principal_components_dim];
 88    double *scores = new double[scores_dim];
 89
 90    pass = pass && (da_handle_get_result_d(handle, da_pca_principal_components,
 91                                           &principal_components_dim,
 92                                           principal_components) == da_status_success);
 93    pass = pass && (da_handle_get_result_d(handle, da_pca_scores, &scores_dim, scores) ==
 94                    da_status_success);
 95
 96    // Check status (we could do this after every function call)
 97    if (pass) {
 98        std::cout << "PCA computed successfully" << std::endl << std::endl;
 99
100        std::cout << "Principal components:" << std::endl;
101        for (da_int j = 0; j < n_components; j++) {
102            std::cout << std::left << " PC " << std::setw(6) << j + 1;
103        }
104        std::cout << std::endl;
105        for (da_int j = 0; j < n_features; j++) {
106            for (da_int i = 0; i < n_components; i++) {
107                std::cout << std::right << std::setw(8)
108                          << principal_components[n_components * i + j] << "  ";
109            }
110            std::cout << std::endl;
111        }
112        std::cout << std::endl;
113
114        // Check against expected results
115        double principal_components_exp[15] = {
116            -0.14907884486130418,  -0.07220367025708045, -0.38718653977350936,
117            -0.6612054163818867,   0.623738867070505,    -0.06907631947413592,
118            -0.031706610956396264, 0.20952521660694667,  0.8854125206703791,
119            -0.7289116905829763,   -0.6138062400926413,  0.1296593407398653,
120            -0.09091387966203135,  0.4302063910917139,   -0.21106437194645863};
121        double scores_exp[18] = {
122            3.797261129593253,    -2.5006179943446254,  2.431393931595693,
123            -3.383775820752579,   -2.0509494403116166,  1.7066881942198742,
124            1.8917911630360351,   -0.14051085079306697, -0.48911894407452433,
125            3.0345920645743383,   -2.9954589898464876,  -1.3012944428962916,
126            -0.10695425296598449, 1.5602497256676358,   1.2837835252499912,
127            -0.7771478863983585,  -0.5060720435855457,  -1.4538590679677388};
128        double X_transform_exp[9] = {
129            -3.250305270939447,  0.6691223004872521,   1.833601737126601,
130            -2.1581247424555086, -0.21658703437771865, -0.2844305102179128,
131            -1.9477723543266676, 1.7953216115607247,   -0.5561178355649032};
132
133        double tol = 1.0e-14;
134        double err = 0.0;
135        for (da_int i = 0; i < principal_components_dim; i++)
136            err = std::max(
137                err, std::abs(principal_components[i] - principal_components_exp[i]));
138        for (da_int i = 0; i < scores_dim; i++)
139            err = std::max(err, std::abs(scores[i] - scores_exp[i]));
140        for (da_int i = 0; i < m_samples * n_components; i++)
141            err = std::max(err, std::abs(X_transform[i] - X_transform_exp[i]));
142        if (err > tol) {
143            std::cout << "Solution is not within the expected tolerance: " << err
144                      << std::endl;
145            exit_code = 1;
146        }
147    } else {
148        exit_code = 1;
149    }
150
151    // Clean up
152    da_handle_destroy(&handle);
153    delete[] principal_components;
154    delete[] scores;
155    delete[] X_transform;
156
157    std::cout << "-----------------------------------------------------------------------"
158              << std::endl;
159
160    return exit_code;
161}