Basic Statistics#

This chapter contains functions to compute basic statistical quantities such as the mean, variance or quantiles of a data matrix. Utility routines for standardizing the data are also available, and correlation and covariance matrices can also be computed. For more information on basic statistical quantities, see Rice [1995], Kokoska and Zwillinger [2000]. These functions operate on a n_rows \(\times\) n_cols matrix stored in column major format.

Choosing an axis#

Most statistical quantities can be computed by column, by row or for the data matrix overall.

This is specified using the axis function parameter.

The axis parameter can take the following values:

  • 'col' - statistical quantities will be computed for each column of the data matrix

  • 'row' - statistical quantities will be computed for each row of the data matrix

  • 'all' - statistical quantities will be computed for the whole data matrix

For example, if the function mean() is called with the axis argument set to col, then n_cols means will be computed, one for each column. If the axis argument is set to all, then a single mean will be computed.

This is specified using the da_axis enum.

The da_axis enum can take the following values:

  • da_axis_col - statistical quantities will be computed for each column of the data matrix

  • da_axis_row - statistical quantities will be computed for each row of the data matrix

  • da_axis_all - statistical quantities will be computed for the whole data matrix

For example, if the routine da_mean_s() is called with the da_axis argument set to da_axis_col, then n_cols means will be computed, one for each column. If the da_axis argument is set to da_axis_all, then a single mean will be computed.

Examples#

The code below is supplied with your installation (see Python examples).

 1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
 2#
 3# Redistribution and use in source and binary forms, with or without modification,
 4# are permitted provided that the following conditions are met:
 5# 1. Redistributions of source code must retain the above copyright notice,
 6#    this list of conditions and the following disclaimer.
 7# 2. Redistributions in binary form must reproduce the above copyright notice,
 8#    this list of conditions and the following disclaimer in the documentation
 9#    and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11#    may be used to endorse or promote products derived from this software without
12#    specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25"""
26Basic statistics example python script
27"""
28
29from aoclda.basic_stats import harmonic_mean, mean, variance, quantile, covariance_matrix, standardize
30import numpy as np
31import sys
32
33
34def basic_stats_example():
35    """
36    Basic statistics examples
37    """
38    a = np.array([[1.1, 2.213, 3.3], [4, 5.013, 6]],
39                 dtype=np.float32,
40                 order='F')
41    print("Dataset:\n")
42    print(a)
43
44    means = mean(a, axis="row")
45    print(f"\nRow means: {means}")
46
47    harmonic_means = harmonic_mean(a, axis="col")
48    print(f"\nColumn harmonic means: {harmonic_means}")
49
50    var = variance(a, axis="all")
51    print(f"\nOverall data variance: {var}")
52
53    medians = quantile(a, 0.5, axis="row")
54    print(f"\nRow medians: {medians}")
55
56    covar = covariance_matrix(a)
57    print(f"\nCovariance matrix:\n{covar}")
58
59    standardized = standardize(a)
60    print(f"\nStandardized matrix:\n{standardized}")
61
62    print("\nBasic statistics calculations successful")
63    print("---------------------------")
64
65
66if __name__ == "__main__":
67    try:
68        basic_stats_example()
69    except RuntimeError:
70        sys.exit(1)

The code below can be found in basic_statistics.cpp in the examples folder of your installation.

  1/*
  2 * Copyright (C) 2023-2024 Advanced Micro Devices, Inc. All rights reserved.
  3 *
  4 * Redistribution and use in source and binary forms, with or without modification,
  5 * are permitted provided that the following conditions are met:
  6 * 1. Redistributions of source code must retain the above copyright notice,
  7 *    this list of conditions and the following disclaimer.
  8 * 2. Redistributions in binary form must reproduce the above copyright notice,
  9 *    this list of conditions and the following disclaimer in the documentation
 10 *    and/or other materials provided with the distribution.
 11 * 3. Neither the name of the copyright holder nor the names of its contributors
 12 *    may be used to endorse or promote products derived from this software without
 13 *    specific prior written permission.
 14 *
 15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 24 * POSSIBILITY OF SUCH DAMAGE.
 25 *
 26 */
 27
 28#include "aoclda.h"
 29#include <cmath>
 30#include <iostream>
 31
 32/* Basic statistics example
 33 *
 34 * This example computes descriptive statistics
 35 * for a small dataset, together with the
 36 * correlation matrix and the standardized data
 37 */
 38
 39int main() {
 40
 41    std::cout << "-----------------------------------------------------------------------"
 42              << std::endl;
 43    std::cout << "Basic statistics" << std::endl;
 44    std::cout << "Descriptive statistics for a 4x5 data matrix" << std::endl << std::endl;
 45    std::cout << std::fixed;
 46    std::cout.precision(5);
 47
 48    // Problem data
 49    double X[20]{1.0, 2.0, 3.0, 4.0, 4.0, 3.0, 2.0, 1.0, 2.0, 8.0,
 50                 4.0, 6.0, 9.0, 5.0, 4.0, 3.0, 1.0, 1.0, 2.0, 2.0};
 51    da_int n_rows = 4, n_cols = 5, ldx = 4, ldcov = 5, dof = 0, mode = 0;
 52
 53    // Arrays for output data
 54    double harmonic_mean[5], mean[4], variance[4], kurtosis[4];
 55    double minimum[1], lower_hinge[1], median[1], upper_hinge[1], maximum[1];
 56    double cov[25];
 57    double *dummy = nullptr;
 58
 59    // Arrays for expected outputs
 60    double harmonic_mean_exp[5]{1.92, 1.92, 3.84, 4.472049689440994, 1.3333333333333333};
 61    double mean_exp[4]{3.4, 3.8, 3., 3.2};
 62    double variance_exp[4]{9.04, 6.16, 0.8, 2.96};
 63    double kurtosis_exp[4]{-0.4210588143159213, -0.9675324675324677, -1.7500000000000002,
 64                           -1.005478451424398};
 65    double minimum_exp[1]{1}, lower_hinge_exp[1]{2}, median_exp[1]{3},
 66        upper_hinge_exp[1]{4}, maximum_exp[1]{9};
 67    double cov_exp[25]{// clang-format off
 68                     1.6666666666666665, -1.6666666666666665,  1.3333333333333333,
 69                     -3.1666666666666665,  0.6666666666666666, -1.6666666666666665,
 70                     1.6666666666666665, -1.3333333333333333,  3.1666666666666665,
 71                     -0.6666666666666666,  1.3333333333333333, -1.3333333333333333,
 72                     6.666666666666666,  -4.333333333333333,   0.0,
 73                     -3.1666666666666665,  3.1666666666666665 ,-4.333333333333333,
 74                     6.916666666666666,  -1.1666666666666665,  0.6666666666666666,
 75                     -0.6666666666666666,  0.0,                -1.1666666666666665,
 76                     0.3333333333333333 };
 77    // clang-format on
 78    double X_exp[20]{-1.1618950038622251, -0.3872983346207417, 0.3872983346207417,
 79                     1.1618950038622251,  1.1618950038622251,  0.3872983346207417,
 80                     -0.3872983346207417, -1.1618950038622251, -1.1618950038622251,
 81                     1.1618950038622251,  -0.3872983346207417, 0.3872983346207417,
 82                     1.4258795636800752,  -0.0950586375786717, -0.4752931878933584,
 83                     -0.8555277382080452, -0.8660254037844387, -0.8660254037844387,
 84                     0.8660254037844387,  0.8660254037844387};
 85
 86    int exit_code = 0;
 87    bool pass = true;
 88
 89    // Compute column-wise harmonic means
 90    pass = pass && (da_harmonic_mean_d(da_axis_col, n_rows, n_cols, X, ldx,
 91                                       harmonic_mean) == da_status_success);
 92
 93    // Compute row-wise mean, variance and kurtosis
 94    pass = pass && (da_kurtosis_d(da_axis_row, n_rows, n_cols, X, ldx, mean, variance,
 95                                  kurtosis) == da_status_success);
 96
 97    // Compute overall max/min, median and hinges
 98    pass = pass && (da_five_point_summary_d(da_axis_all, n_rows, n_cols, X, ldx, minimum,
 99                                            lower_hinge, median, upper_hinge,
100                                            maximum) == da_status_success);
101
102    // Compute covariance matrix
103    pass = pass && (da_covariance_matrix_d(n_rows, n_cols, X, ldx, dof, cov, ldcov) ==
104                    da_status_success);
105
106    // Standardize the original data matrix
107    pass = pass && (da_standardize_d(da_axis_col, n_rows, n_cols, X, ldx, dof, mode,
108                                     dummy, dummy) == da_status_success);
109
110    // Check status (we could do this after every function call)
111    if (pass) {
112        std::cout << "Statistics computed successfully" << std::endl << std::endl;
113
114        // Print computed statistics
115        std::cout << "Column-wise harmonic means:  ";
116        for (da_int i = 0; i < n_cols; i++)
117            std::cout << "  " << harmonic_mean[i];
118        std::cout << std::endl << std::endl;
119
120        std::cout << "Row-wise means:  ";
121        for (da_int i = 0; i < n_rows; i++)
122            std::cout << "  " << mean[i];
123        std::cout << std::endl << std::endl;
124
125        std::cout << "Row-wise variances:  ";
126        for (da_int i = 0; i < n_rows; i++)
127            std::cout << "  " << variance[i];
128        std::cout << std::endl << std::endl;
129
130        std::cout << "Row-wise kurtoses:  ";
131        for (da_int i = 0; i < n_rows; i++)
132            std::cout << "  " << kurtosis[i];
133        std::cout << std::endl << std::endl;
134
135        std::cout << "Overall five-point summary statistics:  " << minimum[0] << "  "
136                  << lower_hinge[0] << "  " << median[0] << "  " << upper_hinge[0] << "  "
137                  << maximum[0] << std::endl
138                  << std::endl;
139
140        std::cout << "Covariance matrix:" << std::endl;
141        for (da_int j = 0; j < n_cols; j++) {
142            for (da_int i = 0; i < n_cols; i++) {
143                std::cout << cov[ldcov * i + j] << "  ";
144            }
145            std::cout << std::endl;
146        }
147        std::cout << std::endl;
148
149        std::cout << "Standardized data matrix:" << std::endl;
150        for (da_int j = 0; j < n_rows; j++) {
151            for (da_int i = 0; i < n_cols; i++) {
152                std::cout << X[ldx * i + j] << "  ";
153            }
154            std::cout << std::endl;
155        }
156        std::cout << std::endl;
157
158        // Check the outputs match the expected results
159        double tol = 1.0e-14;
160        double err = 0.0;
161        for (da_int i = 0; i < n_cols; i++)
162            err = std::max(err, std::abs(harmonic_mean[i] - harmonic_mean_exp[i]));
163        for (da_int i = 0; i < n_rows; i++)
164            err = std::max(err, std::abs(mean[i] - mean_exp[i]));
165        for (da_int i = 0; i < n_rows; i++)
166            err = std::max(err, std::abs(variance[i] - variance_exp[i]));
167        for (da_int i = 0; i < n_rows; i++)
168            err = std::max(err, std::abs(kurtosis[i] - kurtosis_exp[i]));
169        err = std::max(err, std::abs(minimum[0] - minimum_exp[0]));
170        err = std::max(err, std::abs(lower_hinge[0] - lower_hinge_exp[0]));
171        err = std::max(err, std::abs(median[0] - median_exp[0]));
172        err = std::max(err, std::abs(upper_hinge[0] - upper_hinge_exp[0]));
173        err = std::max(err, std::abs(maximum[0] - maximum_exp[0]));
174        for (da_int j = 0; j < n_cols; j++) {
175            for (da_int i = 0; i < n_cols; i++) {
176                err =
177                    std::max(err, std::abs(cov[ldcov * j + i] - cov_exp[ldcov * j + i]));
178            }
179        }
180        for (da_int j = 0; j < n_cols; j++) {
181            for (da_int i = 0; i < n_rows; i++) {
182                err = std::max(err, std::abs(X[ldx * j + i] - X_exp[ldx * j + i]));
183            }
184        }
185        if (err > tol) {
186            std::cout << "Solution is not within the expected tolerance: " << err
187                      << std::endl;
188            exit_code = 1;
189        }
190
191    } else {
192        exit_code = 1;
193    }
194
195    std::cout << "-----------------------------------------------------------------------"
196              << std::endl;
197
198    return exit_code;
199}