Nonlinear Data Fitting#

The topic of nonlinear data fitting encompasses a range of commonly used statistical models and fitting algorithms, including nonlinear least-squares and its variations.

The general form of a nonlinear least-squares fitting problem is as follows:

\[\underset{x \in \Omega}{\text{minimize }} F(x) := \frac{1}{2} \|\psi(r(x))\|_W^2 + \frac{\sigma}{p} \|x\|_2^p,\]

where \(x\) is the coefficient vector of size \(n_{coef}\) to be optimized, \(\Omega\) is a constraint set, \(r(x): R^{n_{coef}} \rightarrow R^{n_{res}}\) with \(\psi\) a loss function, \(W\) is a diagonal matrix of weights that defines the residual norm, and \(\sigma\) and \(p\) are the regularization parameters.

In most common use cases, the set \(\Omega\) is either absent or describes simple bound constraints (i.e., \(\ell_x \le x \le u_x\)), the loss function is absent and the weight matrix is the identity (making the residual norm \(\ell_2\)). The regularization is also absent, thus reducing the problem to

\[\underset{\ell_x \le x \le u_x}{\text{minimize }} F(x) := \frac{1}{2} \|r(x)\|_2^2.\]

Given the nonlinear nature of the residual function, \(r(x)\), there is no closed formula as is the case for linear models. Here, the approach is to use an iterative optimization method that improves on an initial guess at every iteration.

Defining a nonlinear model#

A model is defined inside a handle, in which all the components of the model are configured. In particular, any model is defined via the residual function, \(r(x) = \theta(t, x) - y\), where the pair \((t, y)\) are the data points used to evaluate the model’s residual vector.

Residual functions

To train the model, the optimizer needs to make calls to the residual function which is defined using da_nlls_define_residuals. Some solvers require further information such as the first order derivatives (residual Jacobian matrix) or even second order ones. These are also defined with this function. Refer to nonlinear least-squares callbacks for further details on the residual function signatures.

Derivatives

A key requirement of this iterative optimizer is to have access to first order derivatives (residual Jacobian matrix) in order to calculate an improved solution. There is a strong relationship between the quality of the derivatives and the performance of the solver. If the user does not provide a call-back derivative function, either because it is not available or by choice, then the solver will approximate the derivatives matrix using the single-sided finite-differences method.

Finite-differences is a well established and numerically effective method to estimate missing derivatives. The method is expensive, requiring a number of residual function calls proportional to the number of variables (coefficients) in the model.

The implementation provides a single optional parameter ('finite differences step') that defines the perturbation step used to estimate a derivative. The value of this step plays a crucial role in the quality of the approximation. The default is a judicious value that works for most applications.

It is strongly recommended to relax the convergence tolerances (see options) when approximating derivatives. If it is observed that the solver “stagnates” or fails during the optimization process, tweaking the step value is encouraged.

Verifying derivatives

One of the most common problems while trying to train a model is having incorrect derivatives. Writing the derivative call-back function is error-prone and to address this, a derivative checker can be activated (set option 'Check derivatives' to 'yes') for checking the derivatives provided by the call-back. The checker produces a table similar to

Begin Derivative Checker

   Jacobian storage scheme (Fortran_Jacobian) = C (row-major)

   Jac[     0,     0] =  -1.681939915944E-01 ~  -1.681939717973E-01  [ 1.177E-07], ( 0.165E-06)
   Jac[     2,     5] =   1.000000000000E+01 ~  -1.318047255339E+02  [ 1.076E+00], ( 0.100E-06)  XT
   ...
   Jac[     3,    40] =  -1.528131154992E+02 ~  -1.597095836470E+02  [ 4.318E-02], ( 0.100E-06)  XT   Skip

   Derivative checker detected    106 likely error(s)

   Note: derivative checker detected that     66 entries may correspond to the transpose.
   Verify the Jacobian storage ordering is correct.

End Derivative Checker

The reported table has a few sections. The first column after the equal sign (=), is the derivative returned by the user-supplied call-back. The column after the ~ sign is the approximated finite-difference derivative. The value inside the brackets is the relative threshold \(\frac{|\mathrm{approx} - \mathrm{exact}|}{\max(|\mathrm{approx}|,\; \mathrm{fd_ttol})}\), (fd_ttol is defined by the option Derivative test tol). The value inside the parenthesis is the relative tolerance to compare the relative threshold against. The last column provides some flags: X to indicate that the threshold is larger than the tolerance and is deemed likely to be wrong. T indicates that the value stored in \(J(i,j)\) corresponds the to the value belonging to the transposed Jacobian matrix, providing a hint that possibly the storage sequence is incorrect. This implies that you should check in case the matrix is being stored in row-major format and that the solver option 'Storage scheme' is set to column-major or vice-versa. Finally, Skip indicates that either the associated variable is fixed (constrained to a fixed value) or the bounds on it are too tight to perform a finite-difference approximation and thus the check for this entry cannot be performed and is skipped.

The derivative checker uses finite-differences to compare with the user-provided derivatives and as such the quality of the approximation depends on the finite-difference step used (see option 'Finite difference step').

The option 'Derivative test tol' is involved in defining the relative tolerance to decide if the user-supplied derivative is correct. A smaller value implies a more stringent test.

Under certain circumstances the checker may signal false-positives. Tweaking the options 'Finite difference step' and 'Derivative test tol' can help prevent this.

It is highly recommended that during the writing or development of the derivative call-back, you set the option 'Check derivatives' to 'yes'. After validating the residual Jacobian matrix, and to avoid performance impact, the option can then be reset to 'no'.

Residual weights

Under certain circumstances it is known that some residuals are more reliable than others. In such cases it is desirable to give more importance to these. This is done by defining the weighting matrix, \(W\), using da_nlls_define_weights. Note that \(W\) is a diagonal matrix with positive elements. These elements should correspond to the inverse of the variance of each residual.

Constraining the model

Some models aim to explain real-life phenomena where some coefficients may not make physical sense if they take certain invalid values, e.g. coefficient \(x_j\) representing a distance may not take negative values. For these cases, parameter optimization needs to be constrained to valid values. In the previous distance example, the coefficient would be bound constrained to the non-negative real half-space: \(0 \le x_j\). These constraints are added to the model using da_nlls_define_bounds.

Adding regularization

Nonlinear models can have multiple local-minima that are undesirable, provide a biased solution or even show signs of overfitting. A practical way to tackle these scenarios is to introduce regularization. Typically quadratic or cubic regularization (i.e., \(p=2, 3\)) yield best results. Note that \(\sigma\) and \(p\) are hyperparameters and are not optimized by this model, so they have to be provided by the caller. \(\sigma\) provides a transition between an unregularized local solution (\(\sigma=0\)) and the zero-coefficient vector (\(\sigma \gg 0\)). Striking the correct balance may require trial and error or a good understanding of the underlying model. Regularization is added by using the optional parameters Regularization term (\(\sigma\)) and Regularization power (\(p\)), see Nonlinear least-squares options.

Training the model

Once the model has been set up, the iterative training process is performed by calling the optimizer da_nlls_fit.

Typical workflow for nonlinear models#

The standard way of computing a nonlinear model using AOCL-DA is as follows.

  1. Initialize a aoclda.nonlinear_model.nlls() object with some options set in the class constructor.

  2. Fit a nonlinear model to the data using aoclda.nonlinear_model.nlls.fit(). Here you will have to provide some functions that define the nonlinear model’s residual vector and a function to return the residual Jacobian matrix. The optimized parameters, \(x\), are modified in-place and returned on the interface of aoclda.nonlinear_model.nlls.fit().

  3. Extract results from the aoclda.nonlinear_model.nlls() object via its class attributes.

  1. Initialize a da_handle with da_handle_type da_handle_nlls.

  2. Pass the model to the handle using da_nlls_define_residuals_?.

  3. Customize the model using da_options_set_? (see below for a list of the available options).

  4. Train the nonlinear model using da_nlls_fit_? (you will have to provide an initial guess).

  5. Optimized coefficients \(x\) are returned on the interface.

  6. Extract results using da_handle_get_result_? using da_result::da_rinfo.

    • The following results are available in the info[100] array:

      • info[0]: objective value,

      • info[1]: gradient norm of objective,

      • info[2]: number of iterations,

      • info[3]: reserved for future use,

      • info[4]: number of function callback evaluations (includes info[12]),

      • info[5]: reserved for future use,

      • info[6]: reserved for future use,

      • info[7]: reserved for future use,

      • info[8]: number of gradient callback evaluations,

      • info[9]: number of Hessian callback evaluations,

      • info[10]: number of Hessian-vector callback evaluations,

      • info[11]: scaled gradient norm of objective,

      • info[12]: number of objective function callback evaluations used for approximating the derivatives or due to derivative checker,

      • info[13-99]: reserved for future use.

Nonlinear least-squares options#

The available Python options are detailed in the aoclda.nonlinear_model.nlls() class constructor and the aoclda.nonlinear_model.nlls.fit() method.

Various options can be set to customize the nonlinear models by calling one of these functions. The following table details the available options, where \(\epsilon\) represents the machine precision.

Nonlinear data fitting options#

Option name

Type

Default

Description

Constraints

ralfit model

string

\(s=\) hybrid

NLLS model to solve.

\(s=\) gauss-newton, hybrid, quasi-newton, or tensor-newton.

ralfit nlls method

string

\(s=\) galahad

NLLS solver to use.

\(s=\) aint, galahad, linear solver, more-sorensen, or powell-dogleg.

ralfit globalization method

string

\(s=\) trust-region

Globalization method to use. This parameter makes use of the regularization term and power option values.

\(s=\) reg, regularization, tr, or trust-region.

regularization power

string

\(s=\) quadratic

Value of the regularization power term.

\(s=\) cubic, or quadratic.

regularization term

real

\(r=0\)

Value of the regularization term. A value of 0 disables regularization.

\(0 \le r\)

ralfit iteration limit

integer

\(i=100\)

Maximum number of iterations to perform.

\(1 \le i\)

ralfit convergence rel tol fun

real

\(r=10^{-8}\)

Relative tolerance to declare convergence for the iterative optimization step. See details in optimization solver documentation.

\(0 < r < 1\)

ralfit convergence abs tol fun

real

\(r=10^{-8}\)

Absolute tolerance to declare convergence for the iterative optimization step. See details in optimization solver documentation.

\(0 < r < 1\)

ralfit convergence rel tol grd

real

\(r=10^{-8}\)

Relative tolerance on the gradient norm to declare convergence for the iterative optimization step. See details in optimization solver documentation.

\(0 < r < 1\)

ralfit convergence abs tol grd

real

\(r=10^{-5}\)

Absolute tolerance on the gradient norm to declare convergence for the iterative optimization step. See details in optimization solver documentation.

\(0 < r < 1\)

ralfit convergence step size

real

\(r=\varepsilon/2\)

Absolute tolerance over the step size to declare convergence for the iterative optimization step. See details in optimization solver documentation.

\(0 < r < 1\)

print level

integer

\(i=1\)

Set level of verbosity for the solver: from 0, indicating no output, to 5, which is very verbose.

\(0 \le i \le 5\)

print options

string

\(s=\) no

Print options list.

\(s=\) no, or yes.

storage scheme

string

\(s=\) c

Define the storage scheme used to store multi-dimensional arrays (Jacobian matrix, etc).

\(s=\) c, column-major, f, fortran, or row-major.

check derivatives

string

\(s=\) no

Check user-provided derivatives using finite-differences.

\(s=\) no, or yes.

finite differences step

real

\(r=10\;\sqrt{2\,\varepsilon}\)

Size of step to use for estimating derivatives using finite-differences.

\(0 < r < 10\)

derivative test tol

real

\(r=10^{-4}\)

Tolerance used to check user-provided derivatives by finite-differences. If <print level> is 1, then only the entries with larger discrepancy are reported, and if print level is greater than or equal to 2, then all entries are printed.

\(0 < r \le 10\)

Examples#

The code below is supplied with your installation (see Python examples).

  1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
  2#
  3# Redistribution and use in source and binary forms, with or without modification,
  4# are permitted provided that the following conditions are met:
  5# 1. Redistributions of source code must retain the above copyright notice,
  6#    this list of conditions and the following disclaimer.
  7# 2. Redistributions in binary form must reproduce the above copyright notice,
  8#    this list of conditions and the following disclaimer in the documentation
  9#    and/or other materials provided with the distribution.
 10# 3. Neither the name of the copyright holder nor the names of its contributors
 11#    may be used to endorse or promote products derived from this software without
 12#    specific prior written permission.
 13#
 14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 23# POSSIBILITY OF SUCH DAMAGE.
 24#
 25# pylint: disable = invalid-name, import-error, unused-argument, missing-function-docstring, unused-variable
 26
 27"""
 28Nonlinear data fitting example Python script
 29"""
 30
 31import sys
 32from aoclda.nonlinear_model import nlls
 33import numpy as np
 34
 35def res(x, residuals, data=None) -> int:
 36    (t, y) = data
 37    x1 = x[0]
 38    x2 = x[1]
 39    residuals[:] = x1 * np.exp(x2 * t) - y
 40    return 0
 41
 42
 43def jac(x, jacobian, data) -> int:
 44    x1 = x[0]
 45    x2 = x[1]
 46    (t, y) = data
 47    jacobian[:] = np.column_stack((
 48        np.exp(x2*t),
 49        t * x1 * np.exp(x2*t)
 50    ))
 51    return 0
 52
 53
 54def hes(x, r, Hr, data) -> int:
 55    (t, y) = data
 56    x1 = x[0]
 57    x2 = x[1]
 58    Hr[:] = np.zeros((2, 2))
 59    Hr[0, 0] = 0.0                    # H_11
 60    v = t * np.exp(x2*t)
 61    Hr[1, 0] = np.dot(r, v)           # H_21
 62    Hr[1, 1] = np.dot(r, (t*x1)*v)    # H_22
 63    return 0
 64
 65
 66def nlls_example():
 67    """
 68    Nonlinear data fitting of a convolution model, the solution
 69    provides the isolated parameters for the model:
 70        yi = x1 e^(x2 ti)
 71    for data vectors y : yi and t : ti (i=1:5)
 72    """
 73    # Data to be fitted
 74    t = np.array([1.0, 2.0, 4.0,  5.0,  8.0])
 75    y = np.array([3.0, 4.0, 6.0, 11.0, 20.0])
 76
 77    n_coef = 2
 78    n_res = 5
 79    xexp = np.array([2.54104549, 0.25950481], dtype=np.float64)
 80    x = np.array([2.5, 0.25], dtype=np.float64)
 81    w = 0.12 * np.array([1, 1, 1, 1, 1], dtype=np.float64)
 82    blx = np.array([0.0,  0.0], dtype=np.float64)
 83    bux = np.array([5.0,  3.0], dtype=np.float64)
 84    ndf = nlls(n_coef, n_res, weights=w, lower_bounds=blx, upper_bounds=bux,
 85               check_derivatives='yes', verbose=3)
 86    ndf.fit(x, res, jac, hes, data=(t, y), abs_gtol=1e-7, gtol=1.e-9,
 87            fd_ttol=1.1e-4, maxit=20)
 88
 89    print(f"Solution found in {ndf.n_iter} iterations")
 90    print(f"Residual norm at solution: {ndf.metrics['obj']:.4f}")
 91    print("Solution:")
 92    for i in range(2):
 93        ok = np.abs(x[i]-xexp[i]) <= 1.e-5
 94        print(f"x[{i}]={x[i]:.5f} expected: ({xexp[i]:.5f}) OK? {ok}")
 95
 96if __name__ == "__main__":
 97    try:
 98        nlls_example()
 99    except RuntimeError:
100        print("Something unexpected happened while running the example.")
101        sys.exit(1)
  1# Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
  2#
  3# Redistribution and use in source and binary forms, with or without modification,
  4# are permitted provided that the following conditions are met:
  5# 1. Redistributions of source code must retain the above copyright notice,
  6#    this list of conditions and the following disclaimer.
  7# 2. Redistributions in binary form must reproduce the above copyright notice,
  8#    this list of conditions and the following disclaimer in the documentation
  9#    and/or other materials provided with the distribution.
 10# 3. Neither the name of the copyright holder nor the names of its contributors
 11#    may be used to endorse or promote products derived from this software without
 12#    specific prior written permission.
 13#
 14# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 15# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 16# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 17# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 18# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 19# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 20# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 21# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 23# POSSIBILITY OF SUCH DAMAGE.
 24#
 25# pylint: disable = invalid-name, import-error, unused-argument, missing-function-docstring, unused-variable
 26
 27"""
 28Nonlinear data fitting example Python script
 29"""
 30
 31import sys
 32import math
 33from aoclda.nonlinear_model import nlls
 34import numpy as np
 35
 36n_res = 64
 37
 38# Define Normal and log-Normal distributions
 39
 40
 41def lognormal(d, a, b, Al):
 42    return Al/(d*b*np.sqrt(2*math.pi))*np.exp(-((np.log(d)-a)**2)/(2*b**2))
 43
 44
 45def gaussian(d, mu, sigma, Ag):
 46    return Ag*np.exp(-0.5*((d-mu)/sigma)**2)/(sigma*np.sqrt(2*math.pi))
 47
 48
 49def res(x, residuals, data=None) -> int:
 50    d = data['d']
 51    y = data['y']
 52    a = x[0]
 53    b = x[1]
 54    Al = x[2]
 55    mu = x[3]
 56    sigma = x[4]
 57    Ag = x[5]
 58    for i in range(n_res):
 59        residuals[i] = lognormal(d[i], a, b, Al) + \
 60            gaussian(d[i], mu, sigma, Ag) - y[i]
 61    return 0
 62
 63
 64def nlls_tracks():
 65    """
 66    Nonlinear data fitting for the convolution model
 67    y ~ mixture of normal and log-normal
 68    for data vectors y : yi and d : di (i=1:5)
 69    using finite-differences
 70    """
 71    # Observations / data
 72    diameter = range(1, n_res+1)
 73    density = [
 74        0.0722713864, 0.0575221239, 0.0604719764, 0.0405604720, 0.0317109145,
 75        0.0309734513, 0.0258112094, 0.0228613569, 0.0213864307, 0.0213864307,
 76        0.0147492625, 0.0213864307, 0.0243362832, 0.0169616519, 0.0095870206,
 77        0.0147492625, 0.0140117994, 0.0132743363, 0.0147492625, 0.0140117994,
 78        0.0140117994, 0.0132743363, 0.0117994100, 0.0132743363, 0.0110619469,
 79        0.0103244838, 0.0117994100, 0.0117994100, 0.0147492625, 0.0110619469,
 80        0.0132743363, 0.0206489676, 0.0169616519, 0.0169616519, 0.0280235988,
 81        0.0221238938, 0.0235988201, 0.0221238938, 0.0206489676, 0.0228613569,
 82        0.0184365782, 0.0176991150, 0.0132743363, 0.0132743363, 0.0088495575,
 83        0.0095870206, 0.0073746313, 0.0110619469, 0.0036873156, 0.0051622419,
 84        0.0058997050, 0.0014749263, 0.0022123894, 0.0029498525, 0.0014749263,
 85        0.0007374631, 0.0014749263, 0.0014749263, 0.0007374631, 0.0000000000,
 86        0.0000000000, 0.0000000000, 0.0000000000, 0.0000000000
 87    ]
 88    # Define the data structure to be passed to the callback functions
 89    data = {'d': diameter, 'y': density}
 90
 91    n_coef = 6
 92    xexp = np.array([2.02, 1.39, 0.693, 36.672,
 93                     6.96, 0.3368], dtype=np.float64)
 94
 95    x = np.array([1.63, 0.88, 1.0, 30, 1.52, 0.24], dtype=np.float64)
 96    w = np.ones(n_res, dtype=np.float64)
 97    w[55:63] = 5.0
 98    w /= w.sum()
 99    blx = np.zeros(n_coef, dtype=np.float64)
100    ndf = nlls(n_coef, n_res, weights=w, lower_bounds=blx)
101    ndf.fit(x, res, data=data, abs_gtol=1e-7, gtol=1.e-7, fd_step=5.e-7)
102
103    print(f"Solution found in {ndf.n_iter} iterations")
104    print(f"Residual norm at solution: {ndf.metrics['obj']:.4e}")
105    print("Solution:")
106    for i in range(n_coef):
107        ok = np.abs(x[i]-xexp[i]) <= 1.e-2
108        print(f"x[{i}]={x[i]:.3f} expected: ({xexp[i]:.3f}) OK? {ok}")
109
110
111if __name__ == "__main__":
112    try:
113        nlls_tracks()
114    except RuntimeError:
115        print("Something unexpected happened while running the example.")
116        sys.exit(1)

The code below can be found in nlls.cpp in the examples folder of your installation.

  1/* Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
  2 *
  3 * Redistribution and use in source and binary forms, with or without modification,
  4 * are permitted provided that the following conditions are met:
  5 * 1. Redistributions of source code must retain the above copyright notice,
  6 *    this list of conditions and the following disclaimer.
  7 * 2. Redistributions in binary form must reproduce the above copyright notice,
  8 *    this list of conditions and the following disclaimer in the documentation
  9 *    and/or other materials provided with the distribution.
 10 * 3. Neither the name of the copyright holder nor the names of its contributors
 11 *    may be used to endorse or promote products derived from this software without
 12 *    specific prior written permission.
 13 *
 14 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 15 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 16 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 17 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 18 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 19 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 20 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 21 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 22 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 23 * POSSIBILITY OF SUCH DAMAGE.
 24 *
 25 */
 26
 27/* Fit the convolution model y_i = [Al * lognormal(a, b)]_i + [Ag * normal(mu, sigma)]_i
 28 * given the density observations at the measured diameter sizes.
 29 */
 30
 31#include "aoclda.h"
 32#include <cmath>
 33#include <iomanip>
 34#include <iostream>
 35#include <vector>
 36
 37struct udata_t {
 38    const da_int *diameter;
 39    const double *density;
 40};
 41
 42const double pi = 3.14159265358979323846;
 43
 44// Empirical data
 45const da_int diameter[64]{1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13, 14, 15, 16,
 46                          17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
 47                          33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
 48                          49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64};
 49const double density[64]{
 50    0.0722713864, 0.0575221239, 0.0604719764, 0.0405604720, 0.0317109145, 0.0309734513,
 51    0.0258112094, 0.0228613569, 0.0213864307, 0.0213864307, 0.0147492625, 0.0213864307,
 52    0.0243362832, 0.0169616519, 0.0095870206, 0.0147492625, 0.0140117994, 0.0132743363,
 53    0.0147492625, 0.0140117994, 0.0140117994, 0.0132743363, 0.0117994100, 0.0132743363,
 54    0.0110619469, 0.0103244838, 0.0117994100, 0.0117994100, 0.0147492625, 0.0110619469,
 55    0.0132743363, 0.0206489676, 0.0169616519, 0.0169616519, 0.0280235988, 0.0221238938,
 56    0.0235988201, 0.0221238938, 0.0206489676, 0.0228613569, 0.0184365782, 0.0176991150,
 57    0.0132743363, 0.0132743363, 0.0088495575, 0.0095870206, 0.0073746313, 0.0110619469,
 58    0.0036873156, 0.0051622419, 0.0058997050, 0.0014749263, 0.0022123894, 0.0029498525,
 59    0.0014749263, 0.0007374631, 0.0014749263, 0.0014749263, 0.0007374631, 0.0000000000,
 60    0.0000000000, 0.0000000000, 0.0000000000, 0.0000000000};
 61
 62const struct udata_t udata = {diameter, density};
 63
 64// Scaled log-normal density distribution Al amplitude * log-normal(a, b)
 65double lognormal(double d, double a, double b, double Al) {
 66    return Al / (d * b * sqrt(2.0 * pi)) *
 67           std::exp(-(pow(std::log(d) - a, 2.0)) / (2.0 * pow(b, 2.0)));
 68}
 69
 70// Scaled normal density distribution Ag amplitude * normal(mu, sigma)
 71double gaussian(double d, double mu, double sigma, double Ag) {
 72    return Ag * exp(-0.5 * pow((d - mu) / sigma, 2)) / (sigma * sqrt(2.0 * pi));
 73};
 74
 75// Residuals for the convolution model
 76da_int eval_r([[maybe_unused]] da_int n_coef, da_int n_res, void *udata, double const *x,
 77              double *r) {
 78    double const a = x[0];
 79    double const b = x[1];
 80    double const Al = x[2];
 81    double const mu = x[3];
 82    double const sigma = x[4];
 83    double const Ag = x[5];
 84    da_int const *d = ((struct udata_t *)udata)->diameter;
 85    double const *y = ((struct udata_t *)udata)->density;
 86    for (da_int i = 0; i < n_res; ++i)
 87        r[i] = lognormal(d[i], a, b, Al) + gaussian(d[i], mu, sigma, Ag) - y[i];
 88    return 0;
 89}
 90
 91// Jacobian matrix for the convolution model
 92da_int eval_J(da_int n_coef, da_int n_res, void *udata, double const *x, double *J) {
 93    double const a = x[0];
 94    double const b = x[1];
 95    double const Al = x[2];
 96    double const mu = x[3];
 97    double const sigma = x[4];
 98    double const Ag = x[5];
 99    da_int const *d = ((struct udata_t *)udata)->diameter;
100    for (da_int i = 0; i < n_res; ++i) {
101        double l = lognormal(d[i], a, b, Al);
102        J[i * n_coef + 0] = (log(d[i]) - a) / pow(b, 2.0) * l;
103        J[i * n_coef + 1] = (pow(log(d[i]) - a, 2.0) - pow(b, 2)) / pow(b, 3) * l;
104        J[i * n_coef + 2] = lognormal(d[i], a, b, 1.0);
105        double g = gaussian(d[i], mu, sigma, Ag);
106        J[i * n_coef + 3] = (d[i] - mu) / pow(sigma, 2.0) * g;
107        J[i * n_coef + 4] = (pow(d[i] - mu, 2.0) - pow(sigma, 2.0)) / pow(sigma, 3.0) * g;
108        J[i * n_coef + 5] = gaussian(d[i], mu, sigma, 1.0);
109    }
110    return 0;
111}
112int main(void) {
113    std::cout << " ----------------------------------------" << std::endl;
114    std::cout << "     Nonlinear Least-Squares example" << std::endl;
115    std::cout << " ----------------------------------------" << std::endl;
116
117    const da_int n_coef = 6; /* vector (a, b, Al, mu, sigma, Ag) */
118    const da_int n_res = 64;
119    std::vector<double> coef({1.65, 0.9, 1.0, 30.0, 1.5, 0.25});
120    const double coef_exp[n_coef]{1.99, 1.37, 0.68, 36.64, 7.08, 0.34};
121
122    const double tol{1.0e-2};
123    std::vector<double> lower_bounds(n_res, 0.0);
124    std::vector<double> weights(n_res, 1.0);
125    for (da_int j = 55; j <= 63; ++j)
126        weights[j] = 5.0;
127    double wsum = 1.0 * (n_res - (63 - 55)) + 5.0 * (63 - 55);
128    for (da_int j = 55; j <= 63; ++j)
129        weights[j] = 5.0;
130    // Normalize weights
131    for (da_int j = 0; j < 64; ++j)
132        weights[j] /= wsum;
133
134    // Initialize handle for nonlinear regression
135    da_handle handle = nullptr;
136
137    bool pass = true;
138    pass &= da_handle_init_d(&handle, da_handle_nlls) == da_status_success;
139    pass &= da_nlls_define_residuals_d(handle, n_coef, n_res, eval_r, eval_J, nullptr,
140                                       nullptr) == da_status_success;
141    pass &= da_nlls_define_bounds_d(handle, n_coef, lower_bounds.data(), nullptr) ==
142            da_status_success;
143    pass &= da_nlls_define_weights_d(handle, n_res, weights.data()) == da_status_success;
144    if (!pass) {
145        std::cout << "Something unexpected happened in the model definition\n";
146        da_handle_destroy(&handle);
147        return 1;
148    }
149    pass &= da_options_set_int(handle, "print level", (da_int)2) == da_status_success;
150    pass &= da_options_set_real_d(handle, "finite differences step", 1e-7) ==
151            da_status_success;
152    pass &= pass &=
153        da_options_set_real_d(handle, "derivative test tol", 1e-3) == da_status_success;
154    pass &=
155        da_options_set_string(handle, "check derivatives", "yes") == da_status_success;
156    if (!pass) {
157        std::cout << "Something unexpected happened while setting options\n";
158        da_handle_destroy(&handle);
159        return 2;
160    }
161
162    std::cout << "\n ** Computing regression with exact first derivatives **\n"
163              << std::endl;
164    // Compute regression
165    da_status status;
166    status = da_nlls_fit_d(handle, n_coef, coef.data(), (void *)&udata);
167    bool ok{false};
168    const auto default_precision{std::cout.precision()};
169    if (status == da_status_success) {
170        std::cout << "Regression computed successfully!" << std::endl;
171        std::cout << "Coefficients: Idx           x            x*\n";
172        ok = true;
173        for (auto i = 0; i < n_coef; ++i) {
174            double gap = std::abs(coef[i] - coef_exp[i]);
175            bool oki = gap <= tol;
176            ok &= oki;
177            std::cout << std::setprecision(6);
178            std::cout << "                " << i << std::setw(12) << coef[i] << " "
179                      << std::setw(1) << " " << std::setw(12) << coef_exp[i]
180                      << std::setw(8) << (oki ? "PASS (" : "FAIL (") << std::setw(8)
181                      << std::setprecision(3) << gap << std::setw(1) << ")\n";
182        }
183        std::cout << std::setprecision(3);
184        std::cout << std::setprecision(default_precision);
185    } else {
186        std::cout << "Something wrong happened during the fit. Terminating. Message:"
187                  << std::endl;
188        char *mesg{nullptr};
189        da_handle_get_error_message(handle, &mesg);
190        std::cout << mesg << std::endl;
191        free(mesg);
192        da_handle_destroy(&handle);
193        return 3;
194    }
195
196    // Get info out of handle
197    std::vector<double> info(1);
198    da_int size = info.size();
199    status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
200    if (status == da_status_operation_failed) {
201        info.resize(size);
202    }
203    status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
204    if (status == da_status_success) {
205        std::cout << "Fit error                : " << info[0] << std::endl;
206        std::cout << "Norm of residual gradient: " << info[1] << std::endl;
207    }
208
209    // Now we solve again using finite differences
210    std::cout << "\n\n ** Computing regression with finite differences **\n" << std::endl;
211    coef.assign({1.65, 0.9, 1.0, 30.0, 1.5, 0.25});
212    pass &= da_nlls_define_residuals_d(handle, n_coef, n_res, eval_r, nullptr, nullptr,
213                                       nullptr) == da_status_success;
214    if (!pass) {
215        std::cout << "Something unexpected happened while defining residuals\n";
216        da_handle_destroy(&handle);
217        return 5;
218    }
219    status = da_nlls_fit_d(handle, n_coef, coef.data(), (void *)&udata);
220    if (status == da_status_success) {
221        const auto default_precision{std::cout.precision()};
222        std::cout << "Regression computed successfully!" << std::endl;
223        std::cout << "Coefficients: Idx           x            x*\n";
224        for (auto i = 0; i < n_coef; ++i) {
225            double gap = std::abs(coef[i] - coef_exp[i]);
226            bool oki = gap <= tol;
227            ok &= oki;
228            std::cout << std::setprecision(6);
229            std::cout << "                " << i << std::setw(12) << coef[i] << " "
230                      << std::setw(1) << " " << std::setw(12) << coef_exp[i]
231                      << std::setw(8) << (oki ? "PASS (" : "FAIL (") << std::setw(8)
232                      << std::setprecision(3) << gap << std::setw(1) << ")\n";
233        }
234        std::cout << std::setprecision(3);
235        std::cout << std::setprecision(default_precision);
236    } else {
237        std::cout << "Something wrong happened during the fit. Terminating. Message:"
238                  << std::endl;
239        char *mesg{nullptr};
240        da_handle_get_error_message(handle, &mesg);
241        std::cout << mesg << std::endl;
242        free(mesg);
243        da_handle_destroy(&handle);
244        return 6;
245    }
246
247    // Get info out of handle
248    status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
249    if (status == da_status_operation_failed) {
250        info.resize(size);
251    }
252    status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
253    if (status == da_status_success) {
254        std::cout << "Fit error                : " << info[0] << std::endl;
255        std::cout << "Norm of residual gradient: " << info[1] << std::endl;
256    }
257
258    da_handle_destroy(&handle);
259
260    std::cout << "Regressions were computed " << (ok ? "SUCCESSFULLY" : "with errors")
261              << std::endl;
262
263    return ok ? 0 : 4;
264}
  1/*
  2 * Copyright (C) 2024 Advanced Micro Devices, Inc. All rights reserved.
  3 *
  4 * Redistribution and use in source and binary forms, with or without modification,
  5 * are permitted provided that the following conditions are met:
  6 * 1. Redistributions of source code must retain the above copyright notice,
  7 *    this list of conditions and the following disclaimer.
  8 * 2. Redistributions in binary form must reproduce the above copyright notice,
  9 *    this list of conditions and the following disclaimer in the documentation
 10 *    and/or other materials provided with the distribution.
 11 * 3. Neither the name of the copyright holder nor the names of its contributors
 12 *    may be used to endorse or promote products derived from this software without
 13 *    specific prior written permission.
 14 *
 15 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 16 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 17 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
 18 * IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 19 * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
 20 * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 21 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 22 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 23 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 24 * POSSIBILITY OF SUCH DAMAGE.
 25 *
 26 */
 27
 28#include "aoclda.h"
 29#include <cmath>
 30#include <iomanip>
 31#include <iostream>
 32#include <vector>
 33
 34const da_int n_res = 24;
 35const da_int n_coef = 6;
 36
 37struct udata_t {
 38    const double *t;
 39    const double *y;
 40};
 41const double t[n_res]{0.00000E+00, 5.00000E-02, 1.00000E-01, 1.50000E-01, 2.00000E-01,
 42                      2.50000E-01, 3.00000E-01, 3.50000E-01, 4.00000E-01, 4.50000E-01,
 43                      5.00000E-01, 5.50000E-01, 6.00000E-01, 6.50000E-01, 7.00000E-01,
 44                      7.50000E-01, 8.00000E-01, 8.50000E-01, 9.00000E-01, 9.50000E-01,
 45                      1.00000E+00, 1.05000E+00, 1.10000E+00, 1.15000E+00};
 46const double y[n_res]{2.5134E+00, 2.0443E+00, 1.6684E+00, 1.3664E+00, 1.1232E+00,
 47                      0.9269E+00, 0.7679E+00, 0.6389E+00, 0.5338E+00, 0.4479E+00,
 48                      0.3776E+00, 0.3197E+00, 0.2720E+00, 0.2325E+00, 0.1997E+00,
 49                      0.1723E+00, 0.1493E+00, 0.1301E+00, 0.1138E+00, 0.1000E+00,
 50                      0.0883E+00, 0.0783E+00, 0.0698E+00, 0.0624E+00};
 51const struct udata_t udata = {t, y};
 52
 53da_int eval_r(da_int n_coef, da_int n_res, void *udata, double const *x, double *r) {
 54    double const *t = ((struct udata_t *)udata)->t;
 55    double const *y = ((struct udata_t *)udata)->y;
 56
 57    // r_i = y_i - x_1 e^(-x_2 t_i) - x_3 e^(-x_4 t_i) - x_5 e^(-x_6 t_i)
 58    for (da_int i = 0; i < n_res; i++)
 59        r[i] = y[i] - x[0] * exp(-x[1] * t[i]) - x[2] * exp(-x[3] * t[i]) -
 60               x[4] * exp(-x[n_coef - 1] * t[i]);
 61
 62    return 0;
 63}
 64
 65int main(void) {
 66    std::cout << "--------------------------------------------------------------------"
 67              << std::endl;
 68    std::cout << " Nonlinear Least-Squares basic example (using finite differences)"
 69              << std::endl;
 70    std::cout << "--------------------------------------------------------------------"
 71              << std::endl;
 72
 73    double coef[n_coef]{1.2, 0.3, 5.6, 5.5, 6.5, 7.6};
 74    const double coef_exp[n_coef]{8.6811E-002, 0.954955, 0.843990,
 75                                  2.951558,    1.582591, 4.986342};
 76
 77    const double tol{5.0e-5};
 78
 79    // Initialize handle for nonlinear regression
 80    da_handle handle = nullptr;
 81
 82    bool pass = true;
 83    pass &= da_handle_init_d(&handle, da_handle_nlls) == da_status_success;
 84    pass &= da_nlls_define_residuals_d(handle, n_coef, n_res, eval_r, nullptr, nullptr,
 85                                       nullptr) == da_status_success;
 86    if (!pass) {
 87        std::cout << "Something unexpected happened in the model definition\n";
 88        da_handle_destroy(&handle);
 89        return 1;
 90    }
 91
 92    pass &= da_options_set_string(handle, "ralfit globalization method",
 93                                  "regularization") == da_status_success;
 94    pass &= da_options_set_int(handle, "ralfit iteration limit", (da_int)200) ==
 95            da_status_success;
 96    pass &= da_options_set_real_d(handle, "finite differences step", 1e-5) ==
 97            da_status_success;
 98    if (!pass) {
 99        std::cout << "Something unexpected happened while setting options\n";
100        da_handle_destroy(&handle);
101        return 2;
102    }
103
104    // Compute regression
105    da_status status;
106    status = da_nlls_fit_d(handle, n_coef, coef, (void *)&udata);
107
108    bool ok{false};
109    if (status == da_status_success) {
110        const auto default_precision{std::cout.precision()};
111        std::cout << "Regression computed successfully!" << std::endl;
112        std::cout << "Coefficients: Idx           x            x*\n";
113        ok = true;
114        for (auto i = 0; i < n_coef; ++i) {
115            double gap = std::abs(coef[i] - coef_exp[i]);
116            bool oki = gap < tol;
117            ok &= oki;
118            std::cout << std::setprecision(6);
119            std::cout << "                " << i << std::setw(12) << coef[i] << " "
120                      << std::setw(1) << " " << std::setw(12) << coef_exp[i]
121                      << std::setw(8) << (oki ? "PASS (" : "FAIL (") << std::setw(8)
122                      << std::setprecision(3) << gap << std::setw(1) << ")\n";
123        }
124        std::cout << std::setprecision(3);
125        std::cout << std::setprecision(default_precision);
126    } else {
127        std::cout << "Something wrong happened during the fit. Terminating. Message:"
128                  << std::endl;
129        char *mesg{nullptr};
130        da_handle_get_error_message(handle, &mesg);
131        std::cout << mesg << std::endl;
132        free(mesg);
133        da_handle_destroy(&handle);
134        return 3;
135    }
136
137    // Get info out of handle
138    std::vector<double> info(1);
139    da_int size = info.size();
140    status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
141    if (status == da_status_operation_failed) {
142        info.resize(size);
143    }
144    status = da_handle_get_result_d(handle, da_result::da_rinfo, &size, info.data());
145    if (status == da_status_success) {
146        std::cout << "Fit error                      : " << info[0] << std::endl;
147        std::cout << "Norm of residual gradient      : " << info[1] << std::endl;
148        std::cout << "Objective fun calls            : " << info[4] << std::endl;
149        std::cout << "Objective fun calls (fin diff) : " << info[12] << std::endl;
150    }
151
152    da_handle_destroy(&handle);
153
154    return ok ? 0 : 7;
155}

Further reading#

An introduction to nonlinear least-squares methods can be found in Nocedal and Wright [2006]. In-depth literature on modern trust-region solvers can be reviewed in: Gould et al. [2017], Kanzow et al. [2004], Adachi et al. [2015], Conn et al. [2000], and Gould et al. [2003].