Data Management#
Introduction#
The AOCL-DA C API contains a suite of functions for loading and manipulating data in a straightforward
manner before calling the library’s algorithmic routines. All functions in this suite use the da_datastore structure to store and manipulate data.
There are two ways to load data in the library:
directly into an array of pre-allocated memory of the correct type, or
into a data store, which contains functionality for cleaning and manipulating data.
Currently, the only supported format for loading data from hard memory is CSV (comma separated values).
Loading data from hard memory into dense arrays#
CSV files#
The AOCL-DA C API contains routines for reading data of a single type from a CSV (comma separated values) file into an array. In addition, a character array of column headings can optionally be read.
The routines take a da_datastore structure as their first argument, which must be initialized prior
to the routine call
using da_datastore_init(). This is used to store options but is not used to store the actual
CSV data, which is instead returned in an array.
By default the data is stored in column major order, so that the element in the \(i\)-th row and \(j\)-th column is stored in the [\(i + j \times\) n_rows] entry of the array.
If the CSV data storage option is set to row major then data is stored in row major order, so that the element in the \(i\)-th row and \(j\)-th column is stored in the [\(j + i \times\) n_cols] entry of the array.
For more details on each of the available functions, see the API documentation.
Note
If you wish to load data directly from the CSV file to the da_datastore struct, then use da_data_load_from_csv().
Data manipulation#
The da_datastore structure can also be used to load and manipulate data. The life cycle of a data store typically follows these steps:
Initialize the data store structure.
Load data into the data store, from pre-allocated memory, CSV files or other data stores.
Edit the data.
Select a subsection of the store.
Extract data, either columns/rows or specific regions.
Destroy the structure.
For more details on each of the available functions, see the API documentation.
Loading data into a da_datastore#
Data can be loaded into a da_datastore by adding blocks of data from different sources.
A typical example is to load data from a file and add columns that were allocated dynamically in your program.
This can be achieved by calling da_data_load_from_csv() and da_data_load_col_int() consecutively, for example.
When calling any of the da_data_load_row_? or da_data_load_col_? functions on a da_datastore that is not empty, certain constraints must be
respected.
While adding columns, the number of rows in the block to be added must match the current number of rows present in the
da_datastore(da_data_get_num_rows()can be used to query the dimension).New rows can be added in several sub-blocks. However:
the
da_datastorewill be locked until the current number of columns in the store matches the number of columns of the new block;each sub-block has a minimum column size determined by the number of consecutive columns of the same type in the store. For example, if a given store already has two integer columns and a float column, new rows can be added in two sub-blocks (one with two integer columns and one with the remaining float column).
The final way to load data into a data store is from another da_datastore. Calling da_data_hconcat() will
horizontally concatenate two da_datastore objects with matching numbers of rows.
Selecting and extracting data#
The da_datastore structure uses selections to select and extract subsets of the data. Note that column and row indices are always zero-based, meaning the first index is 0 and the indices of the last column and row are n_cols-1 and n_rows-1 respectively.
Selections
Selections are defined by a label and a set of column and row indices. If any of the functions in this subsection are called with a label that does not exist, a new selection is added to the store. Any number of selections can be defined at the same time in a given store.
da_data_select_columns(), da_data_select_rows() and da_data_select_slice() can be used to add respectively a set of column indices, a set of row indices, or the intersection of a set of rows and columns to a given selection label while da_data_select_non_missing() will remove all row indices containing missing data from the selection.
Extraction
Once the data is loaded into a data store, it can be extracted into dense blocks of contiguous memory suitable for the various algorithms of AOCL-DA. There are two ways to extract data from a da_datastore:
Extract a specific column with one of the da_data_extract_column_? functions.
Extract a selection with a given label by calling one of the da_data_extract_selection_? functions.
All extracted data will be given in column-major format that will be accepted by the rest of the algorithms in the library.
Options#
Various options can be set to customize the behavior of the data loading functions by calling one of these functions. The following table details the available options.
Option Name |
Type |
Default |
Description |
Constraints |
|---|---|---|---|---|
csv use header row |
integer |
\(i=0\) |
Whether or not to interpret the first row as a header. |
\(0 \le i \le 1\) |
csv warn for missing data |
integer |
\(i=0\) |
If set to 0, return error if missing data is encountered; if set to 1, issue a warning and store missing data as either a NaN (for floating point data) or the maximum value of the integer type being used. |
\(0 \le i \le 1\) |
csv skip footer |
integer |
\(i=0\) |
Whether or not to ignore the last line when reading a CSV file. |
\(0 \le i \le 1\) |
csv delimiter |
string |
\(s=\) , |
The delimiter used when reading CSV files. |
|
csv whitespace delimiter |
integer |
\(i=0\) |
Whether or not to use whitespace as the delimiter when reading CSV files. |
\(0 \le i \le 1\) |
csv decimal |
string |
\(s=\) . |
The character used to denote a decimal point in CSV files. |
|
csv skip initial space |
integer |
\(i=0\) |
Whether or not to ignore initial spaces in CSV file lines. |
\(0 \le i \le 1\) |
csv line terminator |
string |
empty |
The character used to denote line termination in CSV files (leave this empty to use the default). |
|
csv row start |
integer |
\(i=0\) |
Ignore the specified number of lines from the top of the file (note that line numbers in CSV files start at 1). |
\(0 \le i\) |
csv comment |
string |
\(s=\) # |
The character used to denote comments in CSV files (note, if a line in a CSV file is to be interpreted as only containing a comment, the comment character should be the first character on the line). |
|
csv quote character |
string |
\(s=\) “ |
The character used to denote quotations in CSV files. |
|
csv scientific notation character |
string |
\(s=\) e |
The character used to denote powers of 10 in floating point values in CSV files. |
|
csv escape character |
string |
\(s=\) \ |
The escape character in CSV files. |
|
csv thousands |
string |
empty |
The character used to separate thousands when reading numeric values in CSV files. |
|
csv skip empty lines |
integer |
\(i=0\) |
Whether or not to ignore empty lines in CSV files (note that caution should be used when using this in conjunction with options such as CSV skip rows since line numbers may no longer correspond to the original line numbers in the CSV file). |
\(0 \le i \le 1\) |
csv data storage |
string |
\(s=\) column major |
Whether to store data from CSV files in row or column major format. |
\(s=\) column major, or row major. |
csv skip empty lines |
integer |
\(i=0\) |
Whether or not to ignore empty lines in CSV files (note that caution should be used when using this in conjunction with options such as CSV skip rows since line numbers may no longer correspond to the original line numbers in the CSV file). |
\(0 \le i \le 1\) |
csv double quote |
integer |
\(i=0\) |
Whether or not to interpret two consecutive quotechar characters within a field as a single quotechar character. |
\(0 \le i \le 1\) |
Note that, with the exception of the skip rows option, only single characters can be used in the string options above.
Examples#
Reading CSV#
Various files in the examples folder of your installation demonstrate the use of the CSV reading functions (e.g. linear_model.cpp).
Usage of da_datastore#
The source files datastore.cpp and linmod_diabetes.cpp showcase loading, selecting and using data with a da_datastore handle.
1/*
2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without modification,
5 * are permitted provided that the following conditions are met:
6 * 1. Redistributions of source code must retain the above copyright notice,
7 * this list of conditions and the following disclaimer.
8 * 2. Redistributions in binary form must reproduce the above copyright notice,
9 * this list of conditions and the following disclaimer in the documentation
10 * and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 * may be used to endorse or promote products derived from this software without
13 * specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <iostream>
30#include <string>
31#include <vector>
32
33#ifndef DATA_DIR
34#define DATA_DIR "data"
35#endif
36
37/* Data store example
38 * This example demonstrates on a small data set how to:
39 * - load data from a CSV files
40 * - select and extract different subsets of the data
41 * - pass the extracted data to an AOCL-DA algorithms and compute a model
42 */
43
44int main() {
45 std::cout << "----------------------------------------" << std::endl;
46 std::cout << "Load data from a CSV file" << std::endl;
47
48 da_datastore store;
49 std::string filename = std::string(DATA_DIR) + "/" + "datastore_ex.csv";
50 da_status status;
51
52 // Load data
53 da_datastore_init(&store);
54 da_datastore_options_set_string(store, "CSV datatype", "double");
55 da_datastore_options_set_int(store, "CSV use header row", 1);
56 status = da_data_load_from_csv(store, filename.c_str());
57 if (status != da_status_success) {
58 std::cout << "Data loading unsuccessful" << std::endl;
59 return 1;
60 }
61
62 // Select the first 2 columns as the feature matrix and the last one as the response
63 std::vector<double> features(10), rhs(5);
64 da_data_select_columns(store, "features", 0, 1);
65 da_data_select_columns(store, "rhs", 2, 2);
66 da_data_extract_selection_real_d(store, "features", features.data(), 5);
67 da_data_extract_selection_real_d(store, "rhs", rhs.data(), 5);
68
69 // Define the regression problem to solve
70 da_handle handle;
71 da_handle_init_d(&handle, da_handle_linmod);
72 da_linmod_select_model_d(handle, linmod_model_mse);
73 da_linmod_define_features_d(handle, 5, 2, features.data(), rhs.data());
74
75 // Solve the problem
76 status = da_linmod_fit_d(handle);
77
78 int exit = 0;
79 if (status == da_status_success) {
80 std::cout << "Regression computed successfully!" << std::endl;
81 da_int nx = 2;
82 std::vector<double> x(2);
83 da_handle_get_result_d(handle, da_result::da_linmod_coef, &nx, x.data());
84 std::cout << "Coefficients: " << x[0] << " " << x[1] << std::endl;
85 std::cout << "(Expected : " << 0.199256 << " " << 0.130354 << ")" << std::endl;
86 } else
87 exit = 1;
88
89 da_datastore_destroy(&store);
90 da_handle_destroy(&handle);
91
92 return exit;
93}
1/*
2 * Copyright (C) 2023-2024 Advanced Micro Devices, Inc. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without modification,
5 * are permitted provided that the following conditions are met:
6 * 1. Redistributions of source code must retain the above copyright notice,
7 * this list of conditions and the following disclaimer.
8 * 2. Redistributions in binary form must reproduce the above copyright notice,
9 * this list of conditions and the following disclaimer in the documentation
10 * and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 * may be used to endorse or promote products derived from this software without
13 * specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <assert.h>
30#include <cmath>
31#include <iomanip>
32#include <iostream>
33#include <vector>
34
35/*
36 * Linear model elastic net regression example
37 * using the data set from
38 *
39 * EFRON, HASTIE, JOHNSTONE, and TIBSHIRANI (2004).
40 * Least angle regression (with discussion).
41 * Ann. Statist. 32 407–499. MR2060166
42 * https://hastie.su.domains/Papers/LARS/data64.txt
43 *
44 * The "diabetes data set" consists of 441 observations
45 * and 10 features, while the model chosen is linear and
46 * fitted with both L1 and L2 penalty terms.
47 *
48 * The example showcases how to use datastore framework to
49 * extract data, but it can be directly loaded using
50 * dense matrices using e.g., da_read_csv_d API.
51 */
52#ifndef DATA_DIR
53#define DATA_DIR "data"
54#endif
55
56int main() {
57
58 std::cout << "---------------------------------------------------" << std::endl;
59 std::cout << "Elastic net regression example using diabetes data" << std::endl;
60 std::cout << "---------------------------------------------------" << std::endl
61 << std::endl;
62 std::cout << std::fixed;
63 std::cout.precision(5);
64
65 // Problem data
66 // m: observations; n: features
67 da_int m = 442, n = 10;
68 da_int rhs_pos = 10;
69 std::vector<double> features;
70 std::vector<double> rhs;
71 std::vector<double> x;
72 // n features
73 features.resize(m * n);
74 rhs.resize(m);
75 x.resize(n + 1);
76 // Initial parameter estimates: n + intercept
77 // x.assign({0, 0, 700, 200, 100, 80, 160, 0, 300, 0});
78 x.assign(m + 1, 0);
79
80 // Reference solution
81 std::vector<double> x_ref(n + 1);
82 x_ref.assign({0, -19.4574064435, 4.3253307426, 0.6585289836, 0, 0, -1.6904339251, 0,
83 19.3039496667, 0});
84
85 da_status status;
86
87 // Load data from file
88 da_datastore csv;
89 const char filename[](DATA_DIR "/diabetes.csv");
90 da_datastore_init(&csv);
91 da_datastore_options_set_int(csv, "CSV whitespace delimiter", 1);
92 da_datastore_options_set_string(csv, "CSV comment", "#");
93 da_datastore_options_set_int(csv, "CSV use header row", 1);
94 status = da_data_load_from_csv(csv, filename);
95 if (status != da_status_success) {
96 da_datastore_print_error_message(csv);
97 return 1;
98 }
99 da_int nr, nc;
100 status = da_data_get_n_rows(csv, &nr);
101 status = da_data_get_n_cols(csv, &nc);
102 if (nr != 442 || nc != 11) {
103 std::cout << "Unexpected size for the loaded data: "
104 "(rows="
105 << nr << ", cols=" << nc << ")" << std::endl;
106 return 2;
107 }
108
109 // Extract the 10 features into a dense matrix
110 bool pass = true;
111 pass = pass && da_data_select_columns(csv, "features", 0, n - 1) == da_status_success;
112 pass = pass && da_data_extract_selection_real_d(csv, "features", features.data(),
113 m) == da_status_success;
114 // Extract response variable
115 pass = pass &&
116 da_data_select_columns(csv, "response", rhs_pos, rhs_pos) == da_status_success;
117 pass = pass && da_data_extract_selection_real_d(csv, "response", rhs.data(), m) ==
118 da_status_success;
119 da_datastore_destroy(&csv);
120 if (!pass) {
121 std::cout
122 << "Unexpected error in the feature and response matrices extraction.\n";
123 return 1;
124 }
125
126 // Initialize the linear regression
127 pass = true;
128 da_int nx = 0;
129 da_handle handle = nullptr;
130 pass = pass && da_handle_init_d(&handle, da_handle_linmod) == da_status_success;
131 pass =
132 pass && da_linmod_select_model_d(handle, linmod_model_mse) == da_status_success;
133 pass = pass &&
134 da_options_set_string(handle, "scaling", "standardise") == da_status_success;
135 pass = pass && da_options_set_real_d(handle, "alpha", 1) == da_status_success;
136 pass = pass && da_options_set_real_d(handle, "lambda", 4) == da_status_success;
137 pass = pass &&
138 da_options_set_string(handle, "print options", "yes") == da_status_success;
139 pass = pass && da_options_set_int(handle, "intercept", 0) == da_status_success;
140 pass = pass && da_options_set_int(handle, "print level", 1) == da_status_success;
141 pass = pass && da_linmod_define_features_d(handle, m, n, features.data(),
142 rhs.data()) == da_status_success;
143 if (!pass) {
144 std::cout << "Unexpected error in the model definition.\n";
145 da_handle_destroy(&handle);
146 return 1;
147 }
148
149 // Compute regression
150 status = da_linmod_fit_start_d(handle, n + 1, x.data());
151 bool ok = false;
152 if (status == da_status_success) {
153 std::cout << "Regression computed" << std::endl;
154 // Query the amount of coefficient in the model (n+intercept)
155 da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
156 x.resize(nx);
157 da_handle_get_result_d(handle, da_linmod_coef, &nx, x.data());
158 std::cout << "Coefficients: " << std::endl;
159 std::cout.precision(3);
160
161 bool oki;
162 ok = !ok;
163 for (da_int i = 0; i < nx; i++) {
164 oki = std::abs(x[i] - x_ref[i]) <= 1.0e-3;
165 std::cout << " x[" << std::setw(2) << i << "] = " << std::setw(9) << x[i]
166 << " expecting " << std::setw(9) << x_ref[i]
167 << (oki ? " (OK)" : " [WRONG]") << std::endl;
168 ok &= oki;
169 }
170 } else {
171 std::cout << "Unexpected error:" << std::endl;
172 da_handle_print_error_message(handle);
173 }
174 std::cout << "----------------------------------------" << std::endl;
175
176 da_handle_destroy(&handle);
177
178 return ok ? 0 : 7;
179}