Nonlinear Data Fitting#
The topic of nonlinear data fitting encompasses a range of commonly used statistical models and fitting algorithms, including nonlinear least-squares and its variations.
The general form of a nonlinear least-squares fitting problem is as follows:
where \(x\) is the coefficient vector of size \(n_{coef}\) to be optimized, \(\Omega\) is a constraint set, \(r(x): R^{n_{coef}} \rightarrow R^{n_{res}}\) with \(\psi\) a loss function, \(W\) is a diagonal matrix of weights that defines the residual norm, and \(\sigma\) and \(p\) are the regularization parameters.
In most common use cases, the set \(\Omega\) is either absent or describes simple bound constraints (i.e., \(\ell_x \le x \le u_x\)), the loss function is absent and the weight matrix is the identity (making the residual norm \(\ell_2\)). The regularization is also absent, thus reducing the problem to
Given the nonlinear nature of the residual function, \(r(x)\), there is no closed formula as is the case for linear models. Here, the approach is to use an iterative optimization method that improves on an initial guess at every iteration.
Defining a nonlinear model#
A model is defined inside a handle, in which all the components of the model are configured. In particular, any model is defined via the residual function, \(r(x) = \theta(t, x) - y\), where the pair \((t, y)\) are the data points used to evaluate the model’s residual vector.
Residual functions
To train the model, the optimizer needs to make calls to the residual function which is
defined using da_nlls_define_residuals.
Some solvers require further information such as the
first order derivatives (residual Jacobian matrix) or even second order ones.
These are also defined with this function.
Refer to nonlinear least-squares callbacks for further details on the
residual function signatures.
Derivatives
A key requirement of this iterative optimizer is to have access to first order derivatives (residual Jacobian matrix) in order to calculate an improved solution. There is a strong relationship between the quality of the derivatives and the performance of the solver. If the user does not provide a call-back derivative function, either because it is not available or by choice, then the solver will approximate the derivatives matrix using the single-sided finite-differences method.
Finite-differences is a well established and numerically effective method to estimate missing derivatives. The method is expensive, requiring a number of residual function calls proportional to the number of variables (coefficients) in the model.
The implementation provides a single optional parameter ('finite differences step') that defines the perturbation step used
to estimate a derivative. The value of this step plays a crucial role in the quality of the approximation. The default
is a judicious value that works for most applications.
It is strongly recommended to relax the convergence tolerances (see options) when approximating derivatives. If it is observed that the solver “stagnates” or fails during the optimization process, tweaking the step value is encouraged.
Verifying derivatives
One of the most common problems while trying to train a model is having incorrect derivatives.
Writing the derivative call-back function is error-prone and to address this, a derivative
checker can be activated (set option 'Check derivatives' to 'yes') for checking the
derivatives provided by the call-back. The checker produces a table similar to
Begin Derivative Checker
Jacobian storage scheme (Fortran_Jacobian) = C (row-major)
Jac[ 0, 0] = -1.681939915944E-01 ~ -1.681939717973E-01 [ 1.177E-07], ( 0.165E-06)
Jac[ 2, 5] = 1.000000000000E+01 ~ -1.318047255339E+02 [ 1.076E+00], ( 0.100E-06) XT
...
Jac[ 3, 40] = -1.528131154992E+02 ~ -1.597095836470E+02 [ 4.318E-02], ( 0.100E-06) XT Skip
Derivative checker detected 106 likely error(s)
Note: derivative checker detected that 66 entries may correspond to the transpose.
Verify the Jacobian storage ordering is correct.
End Derivative Checker
The reported table has a few sections. The first column after the equal sign (=), is the derivative
returned by the user-supplied call-back. The column after the ~ sign is the approximated finite-difference
derivative. The value inside the brackets is the relative threshold
\(\frac{|\mathrm{approx} - \mathrm{exact}|}{\max(|\mathrm{approx}|,\; \mathrm{fd_ttol})}\),
(fd_ttol is defined by the option Derivative test tol). The value inside the parenthesis is the relative tolerance
to compare the relative threshold against.
The last column provides some flags: X to indicate that the threshold is larger than the tolerance and is deemed likely
to be wrong. T indicates that the value stored in \(J(i,j)\) corresponds the to the value belonging to the transposed Jacobian matrix,
providing a hint that possibly the storage sequence is incorrect. This implies that you should check in case the matrix is being stored in row-major format and that
the solver option 'Storage scheme' is set to column-major or vice-versa. Finally, Skip indicates that either the
associated variable is fixed (constrained to a fixed value) or the bounds on it are too tight to perform a finite-difference
approximation and thus the check for this entry cannot be performed and is skipped.
The derivative checker uses finite-differences to compare with the user-provided derivatives and as such the
quality of the approximation depends on the finite-difference step used (see option 'Finite difference step').
The option 'Derivative test tol' is involved in defining the relative tolerance to decide if the user-supplied
derivative is correct. A smaller value implies a more stringent test.
Under certain circumstances the checker may signal false-positives. Tweaking the options 'Finite difference step'
and 'Derivative test tol' can help prevent this.
It is highly recommended that during the writing or development of the derivative call-back, you set the option
'Check derivatives' to 'yes'.
After validating the residual Jacobian matrix, and to avoid performance impact, the option can then be reset to 'no'.
Residual weights
Under certain circumstances it is known that some residuals are more reliable than others. In such cases it is
desirable to give more importance to these. This is done by defining the weighting matrix, \(W\), using
da_nlls_define_weights. Note that \(W\) is a diagonal matrix with
positive elements. These elements
should correspond to the inverse of the variance of each residual.
Constraining the model
Some models aim to explain real-life phenomena where some coefficients may not make physical sense if
they take certain invalid
values, e.g. coefficient \(x_j\) representing a distance may not take negative values. For these cases, parameter
optimization needs to be constrained to valid values. In the previous distance example, the coefficient would be
bound constrained to the non-negative real half-space: \(0 \le x_j\).
These constraints are added to the model using da_nlls_define_bounds.
Adding regularization
Nonlinear models can have multiple local-minima that are undesirable, provide a biased solution or
even show signs of overfitting.
A practical way to tackle these scenarios is to introduce regularization.
Typically quadratic or cubic regularization (i.e., \(p=2, 3\)) yield best results. Note that \(\sigma\) and
\(p\) are hyperparameters and are not optimized by this model, so they have to be provided by the caller.
\(\sigma\) provides a transition between an unregularized local solution (\(\sigma=0\)) and the
zero-coefficient vector (\(\sigma \gg 0\)). Striking the correct balance may require trial and error
or a good understanding of the underlying model. Regularization is added by using the
optional parameters Regularization term (\(\sigma\)) and Regularization power (\(p\)),
see Nonlinear least-squares options.
Training the model
Once the model has been set up, the iterative training process is performed by calling the optimizer da_nlls_fit.
Typical workflow for nonlinear models#
The standard way of computing a nonlinear model using AOCL-DA is as follows.
Initialize a
aoclda.nonlinear_model.nlls()object with some options set in the class constructor.Fit a nonlinear model to the data using
aoclda.nonlinear_model.nlls.fit(). Here you will have to provide some functions that define the nonlinear model’s residual vector and a function to return the residual Jacobian matrix. The optimized parameters, \(x\), are modified in-place and returned on the interface ofaoclda.nonlinear_model.nlls.fit().Extract results from the
aoclda.nonlinear_model.nlls()object via its class attributes.
Initialize a
da_handlewithda_handle_typeda_handle_nlls.Pass the model to the handle using da_nlls_define_residuals_?.
Customize the model using da_options_set_? (see below for a list of the available options).
Train the nonlinear model using da_nlls_fit_? (you will have to provide an initial guess).
Optimized coefficients \(x\) are returned on the interface.
Extract results using da_handle_get_result_? using
da_result::da_rinfo.The following results are available in the
info[100]array:info[0]: objective value,
info[1]: gradient norm of objective,
info[2]: number of iterations,
info[3]: reserved for future use,
info[4]: number of function callback evaluations (includes
info[12]),info[5]: reserved for future use,
info[6]: reserved for future use,
info[7]: reserved for future use,
info[8]: number of gradient callback evaluations,
info[9]: number of Hessian callback evaluations,
info[10]: number of Hessian-vector callback evaluations,
info[11]: scaled gradient norm of objective,
info[12]: number of objective function callback evaluations used for approximating the derivatives or due to derivative checker,
info[13-99]: reserved for future use.
Nonlinear least-squares options#
The available Python options are detailed in the aoclda.nonlinear_model.nlls() class constructor and
the aoclda.nonlinear_model.nlls.fit() method.
Various options can be set to customize the nonlinear models by calling one of these functions. The following table details the available options, where \(\epsilon\) represents the machine precision.
Option name |
Type |
Default |
Description |
Constraints |
|---|---|---|---|---|
ralfit model |
string |
\(s=\) hybrid |
NLLS model to solve. |
\(s=\) gauss-newton, hybrid, quasi-newton, or tensor-newton. |
ralfit nlls method |
string |
\(s=\) galahad |
NLLS solver to use. |
\(s=\) aint, galahad, linear solver, more-sorensen, or powell-dogleg. |
ralfit globalization method |
string |
\(s=\) trust-region |
Globalization method to use. This parameter makes use of the regularization term and power option values. |
\(s=\) reg, regularization, tr, or trust-region. |
regularization power |
string |
\(s=\) quadratic |
Value of the regularization power term. |
\(s=\) cubic, or quadratic. |
regularization term |
real |
\(r=0\) |
Value of the regularization term. A value of 0 disables regularization. |
\(0 \le r\) |
ralfit iteration limit |
integer |
\(i=100\) |
Maximum number of iterations to perform. |
\(1 \le i\) |
ralfit convergence rel tol fun |
real |
\(r=10^{-8}\) |
Relative tolerance to declare convergence for the iterative optimization step. See details in optimization solver documentation. |
\(0 < r < 1\) |
ralfit convergence abs tol fun |
real |
\(r=10^{-8}\) |
Absolute tolerance to declare convergence for the iterative optimization step. See details in optimization solver documentation. |
\(0 < r < 1\) |
ralfit convergence rel tol grd |
real |
\(r=10^{-8}\) |
Relative tolerance on the gradient norm to declare convergence for the iterative optimization step. See details in optimization solver documentation. |
\(0 < r < 1\) |
ralfit convergence abs tol grd |
real |
\(r=10^{-5}\) |
Absolute tolerance on the gradient norm to declare convergence for the iterative optimization step. See details in optimization solver documentation. |
\(0 < r < 1\) |
ralfit convergence step size |
real |
\(r=\varepsilon/2\) |
Absolute tolerance over the step size to declare convergence for the iterative optimization step. See details in optimization solver documentation. |
\(0 < r < 1\) |
print level |
integer |
\(i=1\) |
Set level of verbosity for the solver: from 0, indicating no output, to 5, which is very verbose. |
\(0 \le i \le 5\) |
print options |
string |
\(s=\) no |
Print options list. |
\(s=\) no, or yes. |
storage scheme |
string |
\(s=\) c |
Define the storage scheme used to store multi-dimensional arrays (Jacobian matrix, etc). |
\(s=\) c, column-major, f, fortran, or row-major. |
check derivatives |
string |
\(s=\) no |
Check user-provided derivatives using finite-differences. |
\(s=\) no, or yes. |
finite differences step |
real |
\(r=10\;\sqrt{2\,\varepsilon}\) |
Size of step to use for estimating derivatives using finite-differences. |
\(0 < r < 10\) |
derivative test tol |
real |
\(r=10^{-4}\) |
Tolerance used to check user-provided derivatives by finite-differences. If <print level> is 1, then only the entries with larger discrepancy are reported, and if print level is greater than or equal to 2, then all entries are printed. |
\(0 < r \le 10\) |
Examples#
The code below is supplied with your installation (see Python examples).
1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
2#
3# Redistribution and use in source and binary forms, with or without modification,
4# are permitted provided that the following conditions are met:
5# 1. Redistributions of source code must retain the above copyright notice,
6# this list of conditions and the following disclaimer.
7# 2. Redistributions in binary form must reproduce the above copyright notice,
8# this list of conditions and the following disclaimer in the documentation
9# and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11# may be used to endorse or promote products derived from this software without
12# specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25# pylint: disable = invalid-name, import-error, unused-argument, missing-function-docstring, unused-variable
26
27"""
28Nonlinear data fitting example Python script
29"""
30
31import sys
32from aoclda.nonlinear_model import nlls
33import numpy as np
34
35def res(x, residuals, data=None) -> int:
36 (t, y) = data
37 x1 = x[0]
38 x2 = x[1]
39 residuals[:] = x1 * np.exp(x2 * t) - y
40 return 0
41
42
43def jac(x, jacobian, data) -> int:
44 x1 = x[0]
45 x2 = x[1]
46 (t, y) = data
47 jacobian[:] = np.column_stack((
48 np.exp(x2*t),
49 t * x1 * np.exp(x2*t)
50 ))
51 return 0
52
53
54def hes(x, r, Hr, data) -> int:
55 (t, y) = data
56 x1 = x[0]
57 x2 = x[1]
58 Hr[:] = np.zeros((2, 2))
59 Hr[0, 0] = 0.0 # H_11
60 v = t * np.exp(x2*t)
61 Hr[1, 0] = np.dot(r, v) # H_21
62 Hr[1, 1] = np.dot(r, (t*x1)*v) # H_22
63 return 0
64
65
66def nlls_example():
67 """
68 Nonlinear data fitting of a convolution model, the solution
69 provides the isolated parameters for the model:
70 yi = x1 e^(x2 ti)
71 for data vectors y : yi and t : ti (i=1:5)
72 """
73 # Data to be fitted
74 t = np.array([1.0, 2.0, 4.0, 5.0, 8.0])
75 y = np.array([3.0, 4.0, 6.0, 11.0, 20.0])
76
77 n_coef = 2
78 n_res = 5
79 xexp = np.array([2.54104549, 0.25950481], dtype=np.float64)
80 x = np.array([2.5, 0.25], dtype=np.float64)
81 w = 0.12 * np.array([1, 1, 1, 1, 1], dtype=np.float64)
82 blx = np.array([0.0, 0.0], dtype=np.float64)
83 bux = np.array([5.0, 3.0], dtype=np.float64)
84 ndf = nlls(n_coef, n_res, weights=w, lower_bounds=blx, upper_bounds=bux,
85 check_derivatives='yes', verbose=3)
86 ndf.fit(x, res, jac, hes, data=(t, y), abs_gtol=1e-7, gtol=1.e-9,
87 fd_ttol=1.1e-4, maxit=20)
88
89 print(f"Solution found in {ndf.n_iter} iterations")
90 print(f"Residual norm at solution: {ndf.metrics['obj']:.4f}")
91 print("Solution:")
92 for i in range(2):
93 ok = np.abs(x[i]-xexp[i]) <= 1.e-5
94 print(f"x[{i}]={x[i]:.5f} expected: ({xexp[i]:.5f}) OK? {ok}")
95
96if __name__ == "__main__":
97 try:
98 nlls_example()
99 except RuntimeError:
100 print("Something unexpected happened while running the example.")
101 sys.exit(1)
1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
2#
3# Redistribution and use in source and binary forms, with or without modification,
4# are permitted provided that the following conditions are met:
5# 1. Redistributions of source code must retain the above copyright notice,
6# this list of conditions and the following disclaimer.
7# 2. Redistributions in binary form must reproduce the above copyright notice,
8# this list of conditions and the following disclaimer in the documentation
9# and/or other materials provided with the distribution.
10# 3. Neither the name of the copyright holder nor the names of its contributors
11# may be used to endorse or promote products derived from this software without
12# specific prior written permission.
13#
14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23# POSSIBILITY OF SUCH DAMAGE.
24#
25# pylint: disable = invalid-name, import-error, unused-argument, missing-function-docstring, unused-variable
26
27"""
28Nonlinear data fitting example Python script
29"""
30
31import sys
32import math
33from aoclda.nonlinear_model import nlls
34import numpy as np
35
36n_res = 64
37
38# Define Normal and log-Normal distributions
39
40
41def lognormal(d, a, b, Al):
42 return Al/(d*b*np.sqrt(2*math.pi))*np.exp(-((np.log(d)-a)**2)/(2*b**2))
43
44
45def gaussian(d, mu, sigma, Ag):
46 return Ag*np.exp(-0.5*((d-mu)/sigma)**2)/(sigma*np.sqrt(2*math.pi))
47
48
49def res(x, residuals, data=None) -> int:
50 d = data['d']
51 y = data['y']
52 a = x[0]
53 b = x[1]
54 Al = x[2]
55 mu = x[3]
56 sigma = x[4]
57 Ag = x[5]
58 for i in range(n_res):
59 residuals[i] = lognormal(d[i], a, b, Al) + \
60 gaussian(d[i], mu, sigma, Ag) - y[i]
61 return 0
62
63
64def nlls_tracks():
65 """
66 Nonlinear data fitting for the convolution model
67 y ~ mixture of normal and log-normal
68 for data vectors y : yi and d : di (i=1:5)
69 using finite-differences
70 """
71 # Observations / data
72 diameter = range(1, n_res+1)
73 density = [
74 0.0722713864, 0.0575221239, 0.0604719764, 0.0405604720, 0.0317109145,
75 0.0309734513, 0.0258112094, 0.0228613569, 0.0213864307, 0.0213864307,
76 0.0147492625, 0.0213864307, 0.0243362832, 0.0169616519, 0.0095870206,
77 0.0147492625, 0.0140117994, 0.0132743363, 0.0147492625, 0.0140117994,
78 0.0140117994, 0.0132743363, 0.0117994100, 0.0132743363, 0.0110619469,
79 0.0103244838, 0.0117994100, 0.0117994100, 0.0147492625, 0.0110619469,
80 0.0132743363, 0.0206489676, 0.0169616519, 0.0169616519, 0.0280235988,
81 0.0221238938, 0.0235988201, 0.0221238938, 0.0206489676, 0.0228613569,
82 0.0184365782, 0.0176991150, 0.0132743363, 0.0132743363, 0.0088495575,
83 0.0095870206, 0.0073746313, 0.0110619469, 0.0036873156, 0.0051622419,
84 0.0058997050, 0.0014749263, 0.0022123894, 0.0029498525, 0.0014749263,
85 0.0007374631, 0.0014749263, 0.0014749263, 0.0007374631, 0.0000000000,
86 0.0000000000, 0.0000000000, 0.0000000000, 0.0000000000
87 ]
88 # Define the data structure to be passed to the callback functions
89 data = {'d': diameter, 'y': density}
90
91 n_coef = 6
92 xexp = np.array([2.02, 1.39, 0.693, 36.672,
93 6.96, 0.3368], dtype=np.float64)
94
95 x = np.array([1.63, 0.88, 1.0, 30, 1.52, 0.24], dtype=np.float64)
96 w = np.ones(n_res, dtype=np.float64)
97 w[55:63] = 5.0
98 w /= w.sum()
99 blx = np.zeros(n_coef, dtype=np.float64)
100 ndf = nlls(n_coef, n_res, weights=w, lower_bounds=blx)
101 ndf.fit(x, res, data=data, abs_gtol=1e-7, gtol=1.e-7, fd_step=5.e-7)
102
103 print(f"Solution found in {ndf.n_iter} iterations")
104 print(f"Residual norm at solution: {ndf.metrics['obj']:.4e}")
105 print("Solution:")
106 for i in range(n_coef):
107 ok = np.abs(x[i]-xexp[i]) <= 1.e-2
108 print(f"x[{i}]={x[i]:.3f} expected: ({xexp[i]:.3f}) OK? {ok}")
109
110
111if __name__ == "__main__":
112 try:
113 nlls_tracks()
114 except RuntimeError:
115 print("Something unexpected happened while running the example.")
116 sys.exit(1)
The code below can be found in nlls.cpp in the examples folder of your installation.
1/* Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
2 *
3 * Redistribution and use in source and binary forms, with or without modification,
4 * are permitted provided that the following conditions are met:
5 * 1. Redistributions of source code must retain the above copyright notice,
6 * this list of conditions and the following disclaimer.
7 * 2. Redistributions in binary form must reproduce the above copyright notice,
8 * this list of conditions and the following disclaimer in the documentation
9 * and/or other materials provided with the distribution.
10 * 3. Neither the name of the copyright holder nor the names of its contributors
11 * may be used to endorse or promote products derived from this software without
12 * specific prior written permission.
13 *
14 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
15 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
16 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
18 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
19 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
20 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
21 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
23 * POSSIBILITY OF SUCH DAMAGE.
24 *
25 */
26
27/* Fit the convolution model y_i = [Al * lognormal(a, b)]_i + [Ag * normal(mu, sigma)]_i
28 * given the density observations at the measured diameter sizes.
29 */
30
31#include "aoclda.h"
32#include <cmath>
33#include <iomanip>
34#include <iostream>
35#include <vector>
36
37struct udata_t {
38 const da_int *diameter;
39 const double *density;
40};
41
42const double pi = 3.14159265358979323846;
43
44// Empirical data
45const da_int diameter[64]{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
46 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
47 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
48 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64};
49const double density[64]{
50 0.0722713864, 0.0575221239, 0.0604719764, 0.0405604720, 0.0317109145, 0.0309734513,
51 0.0258112094, 0.0228613569, 0.0213864307, 0.0213864307, 0.0147492625, 0.0213864307,
52 0.0243362832, 0.0169616519, 0.0095870206, 0.0147492625, 0.0140117994, 0.0132743363,
53 0.0147492625, 0.0140117994, 0.0140117994, 0.0132743363, 0.0117994100, 0.0132743363,
54 0.0110619469, 0.0103244838, 0.0117994100, 0.0117994100, 0.0147492625, 0.0110619469,
55 0.0132743363, 0.0206489676, 0.0169616519, 0.0169616519, 0.0280235988, 0.0221238938,
56 0.0235988201, 0.0221238938, 0.0206489676, 0.0228613569, 0.0184365782, 0.0176991150,
57 0.0132743363, 0.0132743363, 0.0088495575, 0.0095870206, 0.0073746313, 0.0110619469,
58 0.0036873156, 0.0051622419, 0.0058997050, 0.0014749263, 0.0022123894, 0.0029498525,
59 0.0014749263, 0.0007374631, 0.0014749263, 0.0014749263, 0.0007374631, 0.0000000000,
60 0.0000000000, 0.0000000000, 0.0000000000, 0.0000000000};
61
62const struct udata_t udata = {diameter, density};
63
64// Scaled log-normal density distribution Al amplitude * log-normal(a, b)
65double lognormal(double d, double a, double b, double Al) {
66 return Al / (d * b * sqrt(2.0 * pi)) *
67 std::exp(-(pow(std::log(d) - a, 2.0)) / (2.0 * pow(b, 2.0)));
68}
69
70// Scaled normal density distribution Ag amplitude * normal(mu, sigma)
71double gaussian(double d, double mu, double sigma, double Ag) {
72 return Ag * exp(-0.5 * pow((d - mu) / sigma, 2)) / (sigma * sqrt(2.0 * pi));
73};
74
75// Residuals for the convolution model
76da_int eval_r([[maybe_unused]] da_int n_coef, da_int n_res, void *udata, double const *x,
77 double *r) {
78 double const a = x[0];
79 double const b = x[1];
80 double const Al = x[2];
81 double const mu = x[3];
82 double const sigma = x[4];
83 double const Ag = x[5];
84 da_int const *d = ((struct udata_t *)udata)->diameter;
85 double const *y = ((struct udata_t *)udata)->density;
86 for (da_int i = 0; i < n_res; ++i)
87 r[i] = lognormal(d[i], a, b, Al) + gaussian(d[i], mu, sigma, Ag) - y[i];
88 return 0;
89}
90
91// Jacobian matrix for the convolution model
92da_int eval_J(da_int n_coef, da_int n_res, void *udata, double const *x, double *J) {
93 double const a = x[0];
94 double const b = x[1];
95 double const Al = x[2];
96 double const mu = x[3];
97 double const sigma = x[4];
98 double const Ag = x[5];
99 da_int const *d = ((struct udata_t *)udata)->diameter;
100 for (da_int i = 0; i < n_res; ++i) {
101 double l = lognormal(d[i], a, b, Al);
102 J[i * n_coef + 0] = (log(d[i]) - a) / pow(b, 2.0) * l;
103 J[i * n_coef + 1] = (pow(log(d[i]) - a, 2.0) - pow(b, 2)) / pow(b, 3) * l;
104 J[i * n_coef + 2] = lognormal(d[i], a, b, 1.0);
105 double g = gaussian(d[i], mu, sigma, Ag);
106 J[i * n_coef + 3] = (d[i] - mu) / pow(sigma, 2.0) * g;
107 J[i * n_coef + 4] = (pow(d[i] - mu, 2.0) - pow(sigma, 2.0)) / pow(sigma, 3.0) * g;
108 J[i * n_coef + 5] = gaussian(d[i], mu, sigma, 1.0);
109 }
110 return 0;
111}
112int main(void) {
113 std::cout << " ----------------------------------------" << std::endl;
114 std::cout << " Nonlinear Least-Squares example" << std::endl;
115 std::cout << " ----------------------------------------" << std::endl;
116
117 const da_int n_coef = 6; /* vector (a, b, Al, mu, sigma, Ag) */
118 const da_int n_res = 64;
119 std::vector<double> coef({1.65, 0.9, 1.0, 30.0, 1.5, 0.25});
120 const double coef_exp[n_coef]{1.99, 1.37, 0.68, 36.64, 7.08, 0.34};
121
122 const double tol{1.0e-2};
123 std::vector<double> lower_bounds(n_res, 0.0);
124 std::vector<double> weights(n_res, 1.0);
125 for (da_int j = 55; j <= 63; ++j)
126 weights[j] = 5.0;
127 double wsum = 1.0 * (n_res - (63 - 55)) + 5.0 * (63 - 55);
128 for (da_int j = 55; j <= 63; ++j)
129 weights[j] = 5.0;
130 // Normalize weights
131 for (da_int j = 0; j < 64; ++j)
132 weights[j] /= wsum;
133
134 // Initialize handle for nonlinear regression
135 da_handle handle = nullptr;
136
137 bool pass = true;
138 pass &= da_handle_init_d(&handle, da_handle_nlls) == da_status_success;
139 pass &= da_nlls_define_residuals_d(handle, n_coef, n_res, eval_r, eval_J, nullptr,
140 nullptr) == da_status_success;
141 pass &= da_nlls_define_bounds_d(handle, n_coef, lower_bounds.data(), nullptr) ==
142 da_status_success;
143 pass &= da_nlls_define_weights_d(handle, n_res, weights.data()) == da_status_success;
144 if (!pass) {
145 std::cout << "Something unexpected happened in the model definition\n";
146 da_handle_destroy(&handle);
147 return 1;
148 }
149 pass &= da_options_set_int(handle, "print level", (da_int)2) == da_status_success;
150 pass &= da_options_set_real_d(handle, "finite differences step", 1e-7) ==
151 da_status_success;
152 pass &= pass &=
153 da_options_set_real_d(handle, "derivative test tol", 1e-3) == da_status_success;
154 pass &=
155 da_options_set_string(handle, "check derivatives", "yes") == da_status_success;
156 if (!pass) {
157 std::cout << "Something unexpected happened while setting options\n";
158 da_handle_destroy(&handle);
159 return 2;
160 }
161
162 std::cout << "\n ** Computing regression with exact first derivatives **\n"
163 << std::endl;
164 // Compute regression
165 da_status status;
166 status = da_nlls_fit_d(handle, n_coef, coef.data(), (void *)&udata);
167 bool ok{false};
168 const auto default_precision{std::cout.precision()};
169 if (status == da_status_success) {
170 std::cout << "Regression computed successfully!" << std::endl;
171 std::cout << "Coefficients: Idx x x*\n";
172 ok = true;
173 for (auto i = 0; i < n_coef; ++i) {
174 double gap = std::abs(coef[i] - coef_exp[i]);
175 bool oki = gap <= tol;
176 ok &= oki;
177 std::cout << std::setprecision(6);
178 std::cout << " " << i << std::setw(12) << coef[i] << " "
179 << std::setw(1) << " " << std::setw(12) << coef_exp[i]
180 << std::setw(8) << (oki ? "PASS (" : "FAIL (") << std::setw(8)
181 << std::setprecision(3) << gap << std::setw(1) << ")\n";
182 }
183 std::cout << std::setprecision(3);
184 std::cout << std::setprecision(default_precision);
185 } else {
186 std::cout << "Something wrong happened during the fit. Terminating. Message:"
187 << std::endl;
188 char *mesg{nullptr};
189 da_handle_get_error_message(handle, &mesg);
190 std::cout << mesg << std::endl;
191 free(mesg);
192 da_handle_destroy(&handle);
193 return 3;
194 }
195
196 // Get info out of handle
197 std::vector<double> info(1);
198 da_int size = info.size();
199 status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
200 if (status == da_status_operation_failed) {
201 info.resize(size);
202 }
203 status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
204 if (status == da_status_success) {
205 std::cout << "Fit error : " << info[0] << std::endl;
206 std::cout << "Norm of residual gradient: " << info[1] << std::endl;
207 }
208
209 // Now we solve again using finite differences
210 std::cout << "\n\n ** Computing regression with finite differences **\n" << std::endl;
211 coef.assign({1.65, 0.9, 1.0, 30.0, 1.5, 0.25});
212 pass &= da_nlls_define_residuals_d(handle, n_coef, n_res, eval_r, nullptr, nullptr,
213 nullptr) == da_status_success;
214 if (!pass) {
215 std::cout << "Something unexpected happened while defining residuals\n";
216 da_handle_destroy(&handle);
217 return 5;
218 }
219 status = da_nlls_fit_d(handle, n_coef, coef.data(), (void *)&udata);
220 if (status == da_status_success) {
221 const auto default_precision{std::cout.precision()};
222 std::cout << "Regression computed successfully!" << std::endl;
223 std::cout << "Coefficients: Idx x x*\n";
224 for (auto i = 0; i < n_coef; ++i) {
225 double gap = std::abs(coef[i] - coef_exp[i]);
226 bool oki = gap <= tol;
227 ok &= oki;
228 std::cout << std::setprecision(6);
229 std::cout << " " << i << std::setw(12) << coef[i] << " "
230 << std::setw(1) << " " << std::setw(12) << coef_exp[i]
231 << std::setw(8) << (oki ? "PASS (" : "FAIL (") << std::setw(8)
232 << std::setprecision(3) << gap << std::setw(1) << ")\n";
233 }
234 std::cout << std::setprecision(3);
235 std::cout << std::setprecision(default_precision);
236 } else {
237 std::cout << "Something wrong happened during the fit. Terminating. Message:"
238 << std::endl;
239 char *mesg{nullptr};
240 da_handle_get_error_message(handle, &mesg);
241 std::cout << mesg << std::endl;
242 free(mesg);
243 da_handle_destroy(&handle);
244 return 6;
245 }
246
247 // Get info out of handle
248 status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
249 if (status == da_status_operation_failed) {
250 info.resize(size);
251 }
252 status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
253 if (status == da_status_success) {
254 std::cout << "Fit error : " << info[0] << std::endl;
255 std::cout << "Norm of residual gradient: " << info[1] << std::endl;
256 }
257
258 da_handle_destroy(&handle);
259
260 std::cout << "Regressions were computed " << (ok ? "SUCCESSFULLY" : "with errors")
261 << std::endl;
262
263 return ok ? 0 : 4;
264}
1/*
2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
3 *
4 * Redistribution and use in source and binary forms, with or without modification,
5 * are permitted provided that the following conditions are met:
6 * 1. Redistributions of source code must retain the above copyright notice,
7 * this list of conditions and the following disclaimer.
8 * 2. Redistributions in binary form must reproduce the above copyright notice,
9 * this list of conditions and the following disclaimer in the documentation
10 * and/or other materials provided with the distribution.
11 * 3. Neither the name of the copyright holder nor the names of its contributors
12 * may be used to endorse or promote products derived from this software without
13 * specific prior written permission.
14 *
15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
24 * POSSIBILITY OF SUCH DAMAGE.
25 *
26 */
27
28#include "aoclda.h"
29#include <cmath>
30#include <iomanip>
31#include <iostream>
32#include <vector>
33
34const da_int n_res = 24;
35const da_int n_coef = 6;
36
37struct udata_t {
38 const double *t;
39 const double *y;
40};
41const double t[n_res]{0.00000E+00, 5.00000E-02, 1.00000E-01, 1.50000E-01, 2.00000E-01,
42 2.50000E-01, 3.00000E-01, 3.50000E-01, 4.00000E-01, 4.50000E-01,
43 5.00000E-01, 5.50000E-01, 6.00000E-01, 6.50000E-01, 7.00000E-01,
44 7.50000E-01, 8.00000E-01, 8.50000E-01, 9.00000E-01, 9.50000E-01,
45 1.00000E+00, 1.05000E+00, 1.10000E+00, 1.15000E+00};
46const double y[n_res]{2.5134E+00, 2.0443E+00, 1.6684E+00, 1.3664E+00, 1.1232E+00,
47 0.9269E+00, 0.7679E+00, 0.6389E+00, 0.5338E+00, 0.4479E+00,
48 0.3776E+00, 0.3197E+00, 0.2720E+00, 0.2325E+00, 0.1997E+00,
49 0.1723E+00, 0.1493E+00, 0.1301E+00, 0.1138E+00, 0.1000E+00,
50 0.0883E+00, 0.0783E+00, 0.0698E+00, 0.0624E+00};
51const struct udata_t udata = {t, y};
52
53da_int eval_r(da_int n_coef, da_int n_res, void *udata, double const *x, double *r) {
54 double const *t = ((struct udata_t *)udata)->t;
55 double const *y = ((struct udata_t *)udata)->y;
56
57 // r_i = y_i - x_1 e^(-x_2 t_i) - x_3 e^(-x_4 t_i) - x_5 e^(-x_6 t_i)
58 for (da_int i = 0; i < n_res; i++)
59 r[i] = y[i] - x[0] * exp(-x[1] * t[i]) - x[2] * exp(-x[3] * t[i]) -
60 x[4] * exp(-x[n_coef - 1] * t[i]);
61
62 return 0;
63}
64
65int main(void) {
66 std::cout << "--------------------------------------------------------------------"
67 << std::endl;
68 std::cout << " Nonlinear Least-Squares basic example (using finite differences)"
69 << std::endl;
70 std::cout << "--------------------------------------------------------------------"
71 << std::endl;
72
73 double coef[n_coef]{1.2, 0.3, 5.6, 5.5, 6.5, 7.6};
74 const double coef_exp[n_coef]{8.6811E-002, 0.954955, 0.843990,
75 2.951558, 1.582591, 4.986342};
76
77 const double tol{5.0e-5};
78
79 // Initialize handle for nonlinear regression
80 da_handle handle = nullptr;
81
82 bool pass = true;
83 pass &= da_handle_init_d(&handle, da_handle_nlls) == da_status_success;
84 pass &= da_nlls_define_residuals_d(handle, n_coef, n_res, eval_r, nullptr, nullptr,
85 nullptr) == da_status_success;
86 if (!pass) {
87 std::cout << "Something unexpected happened in the model definition\n";
88 da_handle_destroy(&handle);
89 return 1;
90 }
91
92 pass &= da_options_set_string(handle, "ralfit globalization method",
93 "regularization") == da_status_success;
94 pass &= da_options_set_int(handle, "ralfit iteration limit", (da_int)200) ==
95 da_status_success;
96 pass &= da_options_set_real_d(handle, "finite differences step", 1e-5) ==
97 da_status_success;
98 if (!pass) {
99 std::cout << "Something unexpected happened while setting options\n";
100 da_handle_destroy(&handle);
101 return 2;
102 }
103
104 // Compute regression
105 da_status status;
106 status = da_nlls_fit_d(handle, n_coef, coef, (void *)&udata);
107
108 bool ok{false};
109 if (status == da_status_success) {
110 const auto default_precision{std::cout.precision()};
111 std::cout << "Regression computed successfully!" << std::endl;
112 std::cout << "Coefficients: Idx x x*\n";
113 ok = true;
114 for (auto i = 0; i < n_coef; ++i) {
115 double gap = std::abs(coef[i] - coef_exp[i]);
116 bool oki = gap < tol;
117 ok &= oki;
118 std::cout << std::setprecision(6);
119 std::cout << " " << i << std::setw(12) << coef[i] << " "
120 << std::setw(1) << " " << std::setw(12) << coef_exp[i]
121 << std::setw(8) << (oki ? "PASS (" : "FAIL (") << std::setw(8)
122 << std::setprecision(3) << gap << std::setw(1) << ")\n";
123 }
124 std::cout << std::setprecision(3);
125 std::cout << std::setprecision(default_precision);
126 } else {
127 std::cout << "Something wrong happened during the fit. Terminating. Message:"
128 << std::endl;
129 char *mesg{nullptr};
130 da_handle_get_error_message(handle, &mesg);
131 std::cout << mesg << std::endl;
132 free(mesg);
133 da_handle_destroy(&handle);
134 return 3;
135 }
136
137 // Get info out of handle
138 std::vector<double> info(1);
139 da_int size = info.size();
140 status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
141 if (status == da_status_operation_failed) {
142 info.resize(size);
143 }
144 status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
145 if (status == da_status_success) {
146 std::cout << "Fit error : " << info[0] << std::endl;
147 std::cout << "Norm of residual gradient : " << info[1] << std::endl;
148 std::cout << "Objective fun calls : " << info[4] << std::endl;
149 std::cout << "Objective fun calls (fin diff) : " << info[12] << std::endl;
150 }
151
152 da_handle_destroy(&handle);
153
154 return ok ? 0 : 7;
155}