Matrix Factorizations#
This chapter contains functions for decomposing a data matrix into the product of two or more matrices. Matrix factorizations are commonly used for dimensionality reduction and feature extraction.
Principal component analysis#
In a principal component analysis (PCA) a set of possibly correlated feature vectors (the columns of the data matrix) is transformed linearly into a new, uncorrelated coordinate system. The new coordinates (which are known as the principal components) are chosen such that the first coordinate accounts for the greatest variance in the data, the second coordinate accounts for the second greatest variance, etc. By using only the first few such coordinates, the data matrix can be reduced in dimension.
Prior to computing the PCA the data matrix is typically standardized by shifting each column so that it has a mean of zero. It can then be shown that the principal components are the eigenvalues of the covariance matrix corresponding to the mean-centered data matrix.
If the features of the data matrix vary greatly in magnitude, then in addition to mean-centering it can be useful to normalize each column by its standard deviation. In this case the principal components are the eigenvalues of the correlation matrix corresponding to the mean-centered data matrix.
The PCA is closely related to a matrix factorization known as the singular value decomposition (or SVD),
where \(A\) is a (standardized) data matrix of size \(n_{\mathrm{samples}} \times n_{\mathrm{features}}\), \(\Sigma\) is a non-negative diagonal matrix of size \(n_{\mathrm{samples}} \times n_{\mathrm{features}}\) and \(U\) and \(V\) are orthogonal matrices of size \(n_{\mathrm{samples}} \times n_{\mathrm{samples}}\) and \(n_{\mathrm{features}} \times n_{\mathrm{features}}\) respectively. The nonzero entries of \(\Sigma\) are known as the singular values of \(A\).
AOCL-DA can compute the PCA via the SVD (with \(\Sigma\) and \(V\) truncated according to the number of principal components requested) or using an eigenvalue decomposition of the covariance/correlation matrix (which is faster for tall, thin matrices, but can be numerically unstable for ill-conditioned problems).
Outputs from the PCA#
After a PCA computation the following results are stored:
principal components - the “new coordinates” expressed in terms of the old coordinates. These are sorted in order of decreasing variance, and are given by the rows of \(V^T\).
scores - the data matrix expressed in terms of the new coordinates. This is given by \(U\Sigma\).
variance - the amount of variance explained by each of the principal components. Note that \(n\_samples -1\) degrees of freedom are used when computing variances.
total variance - the total variance across the whole dataset.
the SVD matrices - \(U\), \(V^T\) and \(\Sigma\) together with the column means and standard deviations.
After the PCA has been computed, two post-processing operations may be of interest:
transform - given a data matrix \(X\) in the same coordinates as the original data matrix \(A\), express \(X\) in terms of the new coordinates (the principal components of \(A\)). This is computed by applying any standardization used on \(A\) to \(X\) and post-multiplying by \(V\).
inverse transform - given a data matrix \(Y\) in the new coordinate system, express \(Y\) in terms of the original coordinates. This is computed by post-multiplying by \(V^T\) and inverting the standardization used on \(A\).
Typical workflow for PCA#
The standard way of computing the principal component analysis using AOCL-DA is as follows.
Initialize a
aoclda.factorization.PCA()object with options set in the class constructor.Compute the PCA for your data matrix using
aoclda.factorization.PCA.fit().Perform further transformations in necessary using
aoclda.factorization.PCA.transform()oraoclda.factorization.PCA.inverse_transform().Extract results from the
aoclda.factorization.PCA()object via its class attributes.
Initialize a
da_handlewithda_handle_typeda_handle_pca.Pass data to the handle using da_pca_set_data_?.
Set the number of principal components required and the type of PCA using da_options_set_? (see below).
Compute the PCA using da_pca_compute_?.
Perform further transformations as required, using da_pca_transform_? or da_pca_inverse_transform_?.
Extract results using da_handle_get_result_?.
Options#
The available Python options are detailed in the aoclda.factorization.PCA() class constructor.
The following options can be set using da_options_set_?:
Option Name |
Type |
Default |
Description |
Constraints |
|---|---|---|---|---|
pca method |
string |
\(s=\) covariance |
Compute PCA based on the covariance or correlation matrix. |
\(s=\) correlation, covariance, or svd. |
degrees of freedom |
string |
\(s=\) unbiased |
Whether to use biased or unbiased estimators for standard deviations and variances. |
\(s=\) biased, or unbiased. |
n_components |
integer |
\(i=1\) |
Number of principal components to compute. If 0, then all components will be kept. |
\(0 \le i\) |
svd solver |
string |
\(s=\) auto |
Which LAPACK routine to use for the underlying singular value decomposition. |
\(s=\) auto, gesdd, gesvd, gesvdx, or syevd. |
If the pca method option is set to svd then no standardization is performed. This option should be used if the input data is already standardized or if an explicit singular value decomposition is required. Note, however, that if the columns of the data matrix are not mean-centered, then the computed variance and total_variance will be meaningless.
If a full decomposition is required (so that all principal components are found) then svd solver should be set to gesdd. The LAPACK routines DGESDD or SGESDD (for double and single precision data respectively) will then be used. This choice offers the best performance, while maintaining high accuracy. Note that if internal heuristics determine that it is useful, a QR decomposition may be performed prior to the SVD.
If svd solver is set to syevd then the SVD will be found by explicitly forming the covariance or correlation matrix and using LAPACK routines DSYEVD or SSYEVD to perform an eigendecomposition. This is very fast for tall, thin data matrices but for wider matrices it requires a lot of memory. The method is also more susceptible to ill-conditioning so must be used with care. It is incompatible with the store U option.
svd solver should only be set to gesvd (so that the LAPACK routines DGESVD or SGESVD are used) if there is insufficient memory for the workspace requirements of gesdd, or if gesdd encounters convergence issues. If only one or two principal components are required then, depending on your data matrix, gesvdx may be faster (so that the LAPACK routines DGESVDX or SGESVDX are used).
If svd solver is set to auto, then DGESDD or SGESDD will be used unless internal heuristics determine that the eigendecomposition may be used.
If store U is set to 1, then the matrix \(U\) from the SVD will be stored and used to ensure deterministic results in the signs of the principal components. Note that there may be a small performance penalty in setting this option and it cannot be used if svd solver is set to syevd.
Examples#
The code below is supplied with your installation (see Python examples).
1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
2#
3# Redistribution and use in source and binary forms, with or without modification,
4# are permitted provided that the following conditions are met:
5# 1. Redistributions of source code must retain the above copyright notice,
6# this list of conditions and the following disclaimer.
7# 2. Redistributions in binary form must reproduce the above copyright notice,
8# this list of conditions and the following disclaimer in the documentation
9# and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11# may be used to endorse or promote products derived from this software without
12# specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25
26"""
27Principal component analysis example Python script
28"""
29
30import sys
31import numpy as np
32from aoclda.factorization import PCA
33
34
35def pca_example():
36 """
37 Principal component analysis example
38 """
39
40 # Define data arrays
41 a = np.array([[2., 2., 3., 2., 4.],
42 [2., 5., 4., 8., 3.],
43 [3., 2., 4., 4., 1.],
44 [4., 8., 3., 6., 4.],
45 [4., 3., 2., 9., 2.],
46 [3., 2., 1., 5., 2.]])
47
48 x = np.array([[7., 4., 2., 9., 3.],
49 [3., 2., 5., 6., 4.],
50 [3., 3., 2., 4., 1.]])
51
52 print("\nPrincipal component analysis for a 6x5 data matrix\n")
53 try:
54 pca = PCA(n_components=3, store_U=True)
55 pca.fit(a)
56 x_transform = pca.transform(x)
57 except RuntimeError:
58 sys.exit(1)
59
60 # Print results
61 print("\nPrincipal components of a:\n")
62 print(pca.principal_components)
63 print("\nx_transform:\n")
64 print(x_transform)
65
66 # Check against expected results
67 expected_components = np.array([[-0.14907884486130418, -0.6612054163818867,
68 -0.031706610956396264, -0.7289116905829763,
69 -0.09091387966203135],
70 [-0.07220367025708045, 0.623738867070505,
71 0.20952521660694667, -0.6138062400926413,
72 0.4302063910917139],
73 [-0.38718653977350936, -0.06907631947413592,
74 0.8854125206703791, 0.1296593407398653,
75 -0.21106437194645863]])
76
77 expected_x_transform = np.array([[-3.250305270939447, -2.1581247424555086, -1.9477723543266676],
78 [0.6691223004872521, -0.21658703437771865,
79 1.7953216115607247],
80 [1.833601737126601, -0.2844305102179128, -0.5561178355649032]])
81
82 norm_components = np.linalg.norm(
83 pca.principal_components - expected_components)
84 norm_x_transform = np.linalg.norm(x_transform - expected_x_transform)
85
86 tol = 1.0e-12
87
88 if norm_components > tol or norm_x_transform > tol:
89 print("\nSolution is not within expected tolerance\n")
90 sys.exit(1)
91
92 print("\nPCA successfully computed\n")
93
94
95if __name__ == "__main__":
96 pca_example()
The code below can be found in pca.cpp in the examples folder of your installation.
1/*
2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without modification,
5 * are permitted provided that the following conditions are met:
6 * 1. Redistributions of source code must retain the above copyright notice,
7 * this list of conditions and the following disclaimer.
8 * 2. Redistributions in binary form must reproduce the above copyright notice,
9 * this list of conditions and the following disclaimer in the documentation
10 * and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 * may be used to endorse or promote products derived from this software without
13 * specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <iomanip>
30#include <iostream>
31
32/* Basic PCA example
33 *
34 * This example computes a principal component
35 * analysis for a small data matrix.
36 */
37
38int main() {
39
40 // Initialize the handle
41 da_handle handle = nullptr;
42
43 std::cout << "-----------------------------------------------------------------------"
44 << std::endl;
45 std::cout << "Basic PCA" << std::endl;
46 std::cout << "Principal component analysis for a 6x5 data matrix" << std::endl
47 << std::endl;
48 std::cout << std::fixed;
49 std::cout.precision(5);
50
51 int exit_code = 0;
52 bool pass = true;
53
54 // Input data
55 double A[30] = {2.0, 2.0, 3.0, 4.0, 4.0, 3.0, 2.0, 5.0, 2.0, 8.0,
56 3.0, 2.0, 3.0, 4.0, 4.0, 3.0, 2.0, 1.0, 2.0, 8.0,
57 4.0, 6.0, 9.0, 5.0, 4.0, 3.0, 1.0, 4.0, 2.0, 2.0};
58
59 da_int n_samples = 6, n_features = 5, n_components = 3, lda = 6;
60
61 // Create the handle and pass it the data matrix
62 pass = pass && (da_handle_init_d(&handle, da_handle_pca) == da_status_success);
63 pass = pass && (da_pca_set_data_d(handle, n_samples, n_features, A, lda) ==
64 da_status_success);
65
66 // Set options
67 pass = pass && (da_options_set_string(handle, "PCA method", "covariance") ==
68 da_status_success);
69 pass = pass && (da_options_set_int(handle, "n_components", n_components) ==
70 da_status_success);
71 pass = pass && (da_options_set_int(handle, "store U", 1) == da_status_success);
72
73 // Compute the PCA
74 pass = pass && (da_pca_compute_d(handle) == da_status_success);
75
76 // Transform another data matrix into the same feature space
77 double X[15] = {7.0, 3.0, 3.0, 4.0, 2.0, 3.0, 2.0, 5.0,
78 2.0, 9.0, 6.0, 4.0, 3.0, 4.0, 1.0};
79 da_int m_samples = 3, m_features = 5, ldx = 3, ldx_transform = 3;
80 double *X_transform = new double[m_samples * n_components];
81 pass = pass && (da_pca_transform_d(handle, m_samples, m_features, X, ldx, X_transform,
82 ldx_transform) == da_status_success);
83
84 // Extract results from the handle
85 da_int principal_components_dim = n_components * n_features;
86 da_int scores_dim = n_samples * n_components;
87 double *principal_components = new double[principal_components_dim];
88 double *scores = new double[scores_dim];
89
90 pass = pass && (da_handle_get_result_d(handle, da_pca_principal_components,
91 &principal_components_dim,
92 principal_components) == da_status_success);
93 pass = pass && (da_handle_get_result_d(handle, da_pca_scores, &scores_dim, scores) ==
94 da_status_success);
95
96 // Check status (we could do this after every function call)
97 if (pass) {
98 std::cout << "PCA computed successfully" << std::endl << std::endl;
99
100 std::cout << "Principal components:" << std::endl;
101 for (da_int j = 0; j < n_components; j++) {
102 std::cout << std::left << " PC " << std::setw(6) << j + 1;
103 }
104 std::cout << std::endl;
105 for (da_int j = 0; j < n_features; j++) {
106 for (da_int i = 0; i < n_components; i++) {
107 std::cout << std::right << std::setw(8)
108 << principal_components[n_components * i + j] << " ";
109 }
110 std::cout << std::endl;
111 }
112 std::cout << std::endl;
113
114 // Check against expected results
115 double principal_components_exp[15] = {
116 -0.14907884486130418, -0.07220367025708045, -0.38718653977350936,
117 -0.6612054163818867, 0.623738867070505, -0.06907631947413592,
118 -0.031706610956396264, 0.20952521660694667, 0.8854125206703791,
119 -0.7289116905829763, -0.6138062400926413, 0.1296593407398653,
120 -0.09091387966203135, 0.4302063910917139, -0.21106437194645863};
121 double scores_exp[18] = {
122 3.797261129593253, -2.5006179943446254, 2.431393931595693,
123 -3.383775820752579, -2.0509494403116166, 1.7066881942198742,
124 1.8917911630360351, -0.14051085079306697, -0.48911894407452433,
125 3.0345920645743383, -2.9954589898464876, -1.3012944428962916,
126 -0.10695425296598449, 1.5602497256676358, 1.2837835252499912,
127 -0.7771478863983585, -0.5060720435855457, -1.4538590679677388};
128 double X_transform_exp[9] = {
129 -3.250305270939447, 0.6691223004872521, 1.833601737126601,
130 -2.1581247424555086, -0.21658703437771865, -0.2844305102179128,
131 -1.9477723543266676, 1.7953216115607247, -0.5561178355649032};
132
133 double tol = 1.0e-14;
134 double err = 0.0;
135 for (da_int i = 0; i < principal_components_dim; i++)
136 err = std::max(
137 err, std::abs(principal_components[i] - principal_components_exp[i]));
138 for (da_int i = 0; i < scores_dim; i++)
139 err = std::max(err, std::abs(scores[i] - scores_exp[i]));
140 for (da_int i = 0; i < m_samples * n_components; i++)
141 err = std::max(err, std::abs(X_transform[i] - X_transform_exp[i]));
142 if (err > tol) {
143 std::cout << "Solution is not within the expected tolerance: " << err
144 << std::endl;
145 exit_code = 1;
146 }
147 } else {
148 exit_code = 1;
149 }
150
151 // Clean up
152 da_handle_destroy(&handle);
153 delete[] principal_components;
154 delete[] scores;
155 delete[] X_transform;
156
157 std::cout << "-----------------------------------------------------------------------"
158 << std::endl;
159
160 return exit_code;
161}