Data Management#

Introduction#

The AOCL-DA C API contains a suite of functions for loading and manipulating data in a straightforward manner before calling the library’s algorithmic routines. All functions in this suite use the da_datastore structure to store and manipulate data.

There are two ways to load data in the library:

  • directly into an array of pre-allocated memory of the correct type, or

  • into a data store, which contains functionality for cleaning and manipulating data.

Currently, the only supported format for loading data from hard memory is CSV (comma separated values).

Loading data from hard memory into dense arrays#

CSV files#

The AOCL-DA C API contains routines for reading data of a single type from a CSV (comma separated values) file into an array. In addition, a character array of column headings can optionally be read.

The routines take a da_datastore structure as their first argument, which must be initialized prior to the routine call using da_datastore_init(). This is used to store options but is not used to store the actual CSV data, which is instead returned in an array. By default the data is stored in column major order, so that the element in the \(i\)-th row and \(j\)-th column is stored in the [\(i + j \times\) n_rows] entry of the array. If the CSV data storage option is set to row major then data is stored in row major order, so that the element in the \(i\)-th row and \(j\)-th column is stored in the [\(j + i \times\) n_cols] entry of the array.

For more details on each of the available functions, see the API documentation.

Note

If you wish to load data directly from the CSV file to the da_datastore struct, then use da_data_load_from_csv().

Data manipulation#

The da_datastore structure can also be used to load and manipulate data. The life cycle of a data store typically follows these steps:

  • Initialize the data store structure.

  • Load data into the data store, from pre-allocated memory, CSV files or other data stores.

  • Edit the data.

  • Select a subsection of the store.

  • Extract data, either columns/rows or specific regions.

  • Destroy the structure.

For more details on each of the available functions, see the API documentation.

Loading data into a da_datastore#

Data can be loaded into a da_datastore by adding blocks of data from different sources. A typical example is to load data from a file and add columns that were allocated dynamically in your program. This can be achieved by calling da_data_load_from_csv() and da_data_load_col_int() consecutively, for example.

When calling any of the da_data_load_row_? or da_data_load_col_? functions on a da_datastore that is not empty, certain constraints must be respected.

  • While adding columns, the number of rows in the block to be added must match the current number of rows present in the da_datastore (da_data_get_num_rows() can be used to query the dimension).

  • New rows can be added in several sub-blocks. However:

    • the da_datastore will be locked until the current number of columns in the store matches the number of columns of the new block;

    • each sub-block has a minimum column size determined by the number of consecutive columns of the same type in the store. For example, if a given store already has two integer columns and a float column, new rows can be added in two sub-blocks (one with two integer columns and one with the remaining float column).

The final way to load data into a data store is from another da_datastore. Calling da_data_hconcat() will horizontally concatenate two da_datastore objects with matching numbers of rows.

Selecting and extracting data#

The da_datastore structure uses selections to select and extract subsets of the data. Note that column and row indices are always zero-based, meaning the first index is 0 and the indices of the last column and row are n_cols-1 and n_rows-1 respectively.

Selections

Selections are defined by a label and a set of column and row indices. If any of the functions in this subsection are called with a label that does not exist, a new selection is added to the store. Any number of selections can be defined at the same time in a given store.

da_data_select_columns(), da_data_select_rows() and da_data_select_slice() can be used to add respectively a set of column indices, a set of row indices, or the intersection of a set of rows and columns to a given selection label while da_data_select_non_missing() will remove all row indices containing missing data from the selection.

Extraction

Once the data is loaded into a data store, it can be extracted into dense blocks of contiguous memory suitable for the various algorithms of AOCL-DA. There are two ways to extract data from a da_datastore:

All extracted data will be given in column-major format that will be accepted by the rest of the algorithms in the library.

Options#

Various options can be set to customize the behavior of the data loading functions by calling one of these functions. The following table details the available options.

CSV file reading options#

Option Name

Type

Default

Description

Constraints

csv use header row

integer

\(i=0\)

Whether or not to interpret the first row as a header.

\(0 \le i \le 1\)

csv warn for missing data

integer

\(i=0\)

If set to 0, return error if missing data is encountered; if set to 1, issue a warning and store missing data as either a NaN (for floating point data) or the maximum value of the integer type being used.

\(0 \le i \le 1\)

csv skip footer

integer

\(i=0\)

Whether or not to ignore the last line when reading a CSV file.

\(0 \le i \le 1\)

csv delimiter

string

\(s=\) ,

The delimiter used when reading CSV files.

csv whitespace delimiter

integer

\(i=0\)

Whether or not to use whitespace as the delimiter when reading CSV files.

\(0 \le i \le 1\)

csv decimal

string

\(s=\) .

The character used to denote a decimal point in CSV files.

csv skip initial space

integer

\(i=0\)

Whether or not to ignore initial spaces in CSV file lines.

\(0 \le i \le 1\)

csv line terminator

string

empty

The character used to denote line termination in CSV files (leave this empty to use the default).

csv row start

integer

\(i=0\)

Ignore the specified number of lines from the top of the file (note that line numbers in CSV files start at 1).

\(0 \le i\)

csv comment

string

\(s=\) #

The character used to denote comments in CSV files (note, if a line in a CSV file is to be interpreted as only containing a comment, the comment character should be the first character on the line).

csv quote character

string

\(s=\) “

The character used to denote quotations in CSV files.

csv scientific notation character

string

\(s=\) e

The character used to denote powers of 10 in floating point values in CSV files.

csv escape character

string

\(s=\) \

The escape character in CSV files.

csv thousands

string

empty

The character used to separate thousands when reading numeric values in CSV files.

csv skip empty lines

integer

\(i=0\)

Whether or not to ignore empty lines in CSV files (note that caution should be used when using this in conjunction with options such as CSV skip rows since line numbers may no longer correspond to the original line numbers in the CSV file).

\(0 \le i \le 1\)

csv data storage

string

\(s=\) column major

Whether to store data from CSV files in row or column major format.

\(s=\) column major, or row major.

csv skip empty lines

integer

\(i=0\)

Whether or not to ignore empty lines in CSV files (note that caution should be used when using this in conjunction with options such as CSV skip rows since line numbers may no longer correspond to the original line numbers in the CSV file).

\(0 \le i \le 1\)

csv double quote

integer

\(i=0\)

Whether or not to interpret two consecutive quotechar characters within a field as a single quotechar character.

\(0 \le i \le 1\)

Note that, with the exception of the skip rows option, only single characters can be used in the string options above.

Examples#

Reading CSV#

Various files in the examples folder of your installation demonstrate the use of the CSV reading functions (e.g. linear_model.cpp).

Usage of da_datastore#

The source files datastore.cpp and linmod_diabetes.cpp showcase loading, selecting and using data with a da_datastore handle.

 1/*
 2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
 3 *
 4 * Redistribution and use in source and binary forms, with or without modification,
 5 * are permitted provided that the following conditions are met:
 6 * 1. Redistributions of source code must retain the above copyright notice,
 7 *    this list of conditions and the following disclaimer.
 8 * 2. Redistributions in binary form must reproduce the above copyright notice,
 9 *    this list of conditions and the following disclaimer in the documentation
10 *    and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 *    may be used to endorse or promote products derived from this software without
13 *    specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <iostream>
30#include <string>
31#include <vector>
32
33#ifndef DATA_DIR
34#define DATA_DIR "data"
35#endif
36
37/* Data store example
38 * This example demonstrates on a small data set how to:
39 * - load data from a CSV files
40 * - select and extract different subsets of the data
41 * - pass the extracted data to an AOCL-DA algorithms and compute a model
42 */
43
44int main() {
45    std::cout << "----------------------------------------" << std::endl;
46    std::cout << "Load data from a CSV file" << std::endl;
47
48    da_datastore store;
49    std::string filename = std::string(DATA_DIR) + "/" + "datastore_ex.csv";
50    da_status status;
51
52    // Load data
53    da_datastore_init(&store);
54    da_datastore_options_set_string(store, "CSV datatype", "double");
55    da_datastore_options_set_int(store, "CSV use header row", 1);
56    status = da_data_load_from_csv(store, filename.c_str());
57    if (status != da_status_success) {
58        std::cout << "Data loading unsuccessful" << std::endl;
59        return 1;
60    }
61
62    // Select the first 2 columns as the feature matrix and the last one as the response
63    std::vector<double> features(10), rhs(5);
64    da_data_select_columns(store, "features", 0, 1);
65    da_data_select_columns(store, "rhs", 2, 2);
66    da_data_extract_selection_real_d(store, "features", features.data(), 5);
67    da_data_extract_selection_real_d(store, "rhs", rhs.data(), 5);
68
69    // Define the regression problem to solve
70    da_handle handle;
71    da_handle_init_d(&handle, da_handle_linmod);
72    da_linmod_select_model_d(handle, linmod_model_mse);
73    da_linmod_define_features_d(handle, 5, 2, features.data(), rhs.data());
74
75    // Solve the problem
76    status = da_linmod_fit_d(handle);
77
78    int exit = 0;
79    if (status == da_status_success) {
80        std::cout << "Regression computed successfully!" << std::endl;
81        da_int nx = 2;
82        std::vector<double> x(2);
83        da_handle_get_result_d(handle, da_result::da_linmod_coef, &nx, x.data());
84        std::cout << "Coefficients: " << x[0] << " " << x[1] << std::endl;
85        std::cout << "(Expected   : " << 0.199256 << " " << 0.130354 << ")" << std::endl;
86    } else
87        exit = 1;
88
89    da_datastore_destroy(&store);
90    da_handle_destroy(&handle);
91
92    return exit;
93}
  1/*
  2 * Copyright (C) 2023-2024 Advanced Micro Devices, Inc. All rights reserved.
  3 *
  4 * Redistribution and use in source and binary forms, with or without modification,
  5 * are permitted provided that the following conditions are met:
  6 * 1. Redistributions of source code must retain the above copyright notice,
  7 *    this list of conditions and the following disclaimer.
  8 * 2. Redistributions in binary form must reproduce the above copyright notice,
  9 *    this list of conditions and the following disclaimer in the documentation
 10 *    and/or other materials provided with the distribution.
 11 * 3. Neither the name of the copyright holder nor the names of its contributors
 12 *    may be used to endorse or promote products derived from this software without
 13 *    specific prior written permission.
 14 *
 15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 24 * POSSIBILITY OF SUCH DAMAGE.
 25 *
 26 */
 27
 28#include "aoclda.h"
 29#include <assert.h>
 30#include <cmath>
 31#include <iomanip>
 32#include <iostream>
 33#include <vector>
 34
 35/*
 36 * Linear model elastic net regression example
 37 * using the data set from
 38 *
 39 * EFRON, HASTIE, JOHNSTONE, and TIBSHIRANI (2004).
 40 * Least angle regression (with discussion).
 41 * Ann. Statist. 32 407–499. MR2060166
 42 * https://hastie.su.domains/Papers/LARS/data64.txt
 43 *
 44 * The "diabetes data set" consists of 441 observations
 45 * and 10 features, while the model chosen is linear and
 46 * fitted with both L1 and L2 penalty terms.
 47 *
 48 * The example showcases how to use datastore framework to
 49 * extract data, but it can be directly loaded using
 50 * dense matrices using e.g., da_read_csv_d API.
 51 */
 52#ifndef DATA_DIR
 53#define DATA_DIR "data"
 54#endif
 55
 56int main() {
 57
 58    std::cout << "---------------------------------------------------" << std::endl;
 59    std::cout << "Elastic net regression example using diabetes data" << std::endl;
 60    std::cout << "---------------------------------------------------" << std::endl
 61              << std::endl;
 62    std::cout << std::fixed;
 63    std::cout.precision(5);
 64
 65    // Problem data
 66    // m: observations; n: features
 67    da_int m = 442, n = 10;
 68    da_int rhs_pos = 10;
 69    std::vector<double> features;
 70    std::vector<double> rhs;
 71    std::vector<double> x;
 72    // n features
 73    features.resize(m * n);
 74    rhs.resize(m);
 75    x.resize(n + 1);
 76    // Initial parameter estimates: n + intercept
 77    // x.assign({0, 0, 700, 200, 100, 80, 160, 0, 300, 0});
 78    x.assign(m + 1, 0);
 79
 80    // Reference solution
 81    std::vector<double> x_ref(n + 1);
 82    x_ref.assign({0, -19.4574064435, 4.3253307426, 0.6585289836, 0, 0, -1.6904339251, 0,
 83                  19.3039496667, 0});
 84
 85    da_status status;
 86
 87    // Load data from file
 88    da_datastore csv;
 89    const char filename[](DATA_DIR "/diabetes.csv");
 90    da_datastore_init(&csv);
 91    da_datastore_options_set_int(csv, "CSV whitespace delimiter", 1);
 92    da_datastore_options_set_string(csv, "CSV comment", "#");
 93    da_datastore_options_set_int(csv, "CSV use header row", 1);
 94    status = da_data_load_from_csv(csv, filename);
 95    if (status != da_status_success) {
 96        da_datastore_print_error_message(csv);
 97        return 1;
 98    }
 99    da_int nr, nc;
100    status = da_data_get_n_rows(csv, &nr);
101    status = da_data_get_n_cols(csv, &nc);
102    if (nr != 442 || nc != 11) {
103        std::cout << "Unexpected size for the loaded data: "
104                     "(rows="
105                  << nr << ", cols=" << nc << ")" << std::endl;
106        return 2;
107    }
108
109    // Extract the 10 features into a dense matrix
110    bool pass = true;
111    pass = pass && da_data_select_columns(csv, "features", 0, n - 1) == da_status_success;
112    pass = pass && da_data_extract_selection_real_d(csv, "features", features.data(),
113                                                    m) == da_status_success;
114    // Extract response variable
115    pass = pass &&
116           da_data_select_columns(csv, "response", rhs_pos, rhs_pos) == da_status_success;
117    pass = pass && da_data_extract_selection_real_d(csv, "response", rhs.data(), m) ==
118                       da_status_success;
119    da_datastore_destroy(&csv);
120    if (!pass) {
121        std::cout
122            << "Unexpected error in the feature and response matrices extraction.\n";
123        return 1;
124    }
125
126    // Initialize the linear regression
127    pass = true;
128    da_int nx = 0;
129    da_handle handle = nullptr;
130    pass = pass && da_handle_init_d(&handle, da_handle_linmod) == da_status_success;
131    pass =
132        pass && da_linmod_select_model_d(handle, linmod_model_mse) == da_status_success;
133    pass = pass &&
134           da_options_set_string(handle, "scaling", "standardise") == da_status_success;
135    pass = pass && da_options_set_real_d(handle, "alpha", 1) == da_status_success;
136    pass = pass && da_options_set_real_d(handle, "lambda", 4) == da_status_success;
137    pass = pass &&
138           da_options_set_string(handle, "print options", "yes") == da_status_success;
139    pass = pass && da_options_set_int(handle, "intercept", 0) == da_status_success;
140    pass = pass && da_options_set_int(handle, "print level", 1) == da_status_success;
141    pass = pass && da_linmod_define_features_d(handle, m, n, features.data(),
142                                               rhs.data()) == da_status_success;
143    if (!pass) {
144        std::cout << "Unexpected error in the model definition.\n";
145        da_handle_destroy(&handle);
146        return 1;
147    }
148
149    // Compute regression
150    status = da_linmod_fit_start_d(handle, n + 1, x.data());
151    bool ok = false;
152    if (status == da_status_success) {
153        std::cout << "Regression computed" << std::endl;
154        // Query the amount of coefficient in the model (n+intercept)
155        da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
156        x.resize(nx);
157        da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
158        std::cout << "Coefficients: " << std::endl;
159        std::cout.precision(3);
160
161        bool oki;
162        ok = !ok;
163        for (da_int i = 0; i < nx; i++) {
164            oki = std::abs(x[i] - x_ref[i]) <= 1.0e-3;
165            std::cout << " x[" << std::setw(2) << i << "] = " << std::setw(9) << x[i]
166                      << " expecting " << std::setw(9) << x_ref[i]
167                      << (oki ? " (OK)" : " [WRONG]") << std::endl;
168            ok &= oki;
169        }
170    } else {
171        std::cout << "Unexpected error:" << std::endl;
172        da_handle_print_error_message(handle);
173    }
174    std::cout << "----------------------------------------" << std::endl;
175
176    da_handle_destroy(&handle);
177
178    return ok ? 0 : 7;
179}