Enhanced Levenberg-Marquardt Algorithm with Residual Function Support #1116

diluculo · 2025-03-21T12:20:23Z

This PR implements:

Support for direct residual functions in Levenberg-Marquardt algorithm
Improved MINPACK-style control flow for damping parameter updates
Comprehensive statistical metrics in NonlinearMinimizationResult
Numerical differentiation through dedicated NumericalJacobian class

…r reflect its purpose in modeling optimization problems

…elated interfaces

- Add configurable accuracy with orders 1-6 for both model and residual functions - Use NumericalJacobian class for more reliable derivative approximation

…objective function

…izer

…veModel

- TStatistics and PValues for parameter significance testing - ConfidenceIntervalHalfWidths for parameter precision - Dependencies to measure parameter correlations - Goodness-of-fit statistics

…stics

diluculo · 2025-03-22T05:13:00Z

Lastly, I've added a new ParameterStatistics class that computes all the key statistical measures for regression parameters, including standard errors, t-statistics, p-values, confidence intervals, and dependency analysis.

Ready for review and merge.

diluculo · 2025-03-28T12:03:50Z

I've just added the ILeastSquaresMinimizer interface, which unifies our API for nonlinear least squares minimizers by providing two overloads for the FindMinimum method (one for Vector and one for double[]).

Please review and merge when ready. Thank you!

diluculo · 2025-03-31T04:40:29Z

In the meantime, I found a critical bug in the Trust Region optimization implementation. The subproblem solvers were using unscaled gradient and Hessian values directly from the IObjectiveModel, while they should have been using the scaled values that NonlinearMinimizerBase.EvaluateJacobian produces.

The last commit fixes this issue by modifying the ITrustRegionSubproblem interface to accept the correctly scaled gradient and Hessian as parameters.

diluculo · 2025-03-31T11:07:54Z

Summary of Key Changes

Enhanced Levenberg-Marquardt Algorithm
- Support for direct residual functions
- Improved MINPACK-style control flow for damping parameter updates
Comprehensive Statistical Analysis
- New ParameterStatistics class for regression parameters analysis
- Calculates standard errors, t-statistics, p-values, confidence intervals, and dependencies
API Improvements
- Added ILeastSquaresMinimizer interface to unify nonlinear least squares minimizers
Bug Fixes
- Fixed a bug in Trust Region optimization where subproblem solvers were using unscaled gradient and Hessian values
- Modified ITrustRegionSubproblem interface to accept correctly scaled gradient and Hessian as parameters

febkor · 2025-04-01T15:53:45Z

Hi @diluculo,
Thank you for the trying to improve the optimizers here.
There's a few independent changes in this PR which makes review difficult however.

Please could this bit be pulled out to a separate PR for now?

Fixed a bug in Trust Region optimization where subproblem solvers were using unscaled gradient and Hessian values

diluculo · 2025-04-02T00:24:08Z

Hi @febkor

Thanks for your review. You're right, this PR contains several independent changes that should be separated. I've split it into 3 separate PRs:

Enhanced Levenberg-Marquardt Algorithm with residual support (including ParameterStatistics)
ILeastSquaresMinimizer interface : Add Basin Hopping Global Optimization Algorithm with New Least Squares Interface #1122
Trust Region optimization bug fix : Fix and Extend Trust Region Subproblem Implementations #1120
~~ParameterStatistics for parameter inference~~

I initially planned to separate ParameterStatistics as well, but since it depends on changes to IObjectiveModelEvaluation in the LM enhancements, I've kept it in this PR to maintain functionality.

Edit:
In addition to splitting the PR, I've also added new features that I had been working on:

The Basin Hopping global optimization algorithm in PR Add Basin Hopping Global Optimization Algorithm with New Least Squares Interface #1122
The Exact Trust Region subproblem solver in PR Fix and Extend Trust Region Subproblem Implementations #1120

Hi @cdrnet,
This ended up being more work than initially anticipated, but I've completed everything I had prepared. I hope these PRs can be included in v6.0.

All PRs are now ready for review.

diluculo · 2025-04-02T08:23:45Z

Regarding the failing test in AppVeyor: This is unrelated to the Levenberg-Marquardt enhancements - it's a floating-point precision issue in the statistics tests for .NET 8.0 (off by ~1.03E-09).

diluculo · 2025-04-02T12:23:23Z

Hi @febkor

Here is the mathematical background document I've prepared to explain the residual function support. It illustrates the relationship between the objective function (χ²) and the residual vector (R). I hope this helps in understanding the purpose and implementation of this PR.

febkor

Just a brief review, more from a performance angle.
Hope you find them useful.

febkor · 2025-07-07T09:38:41Z

src/Numerics/Optimization/NonlinearMinimizationResult.cs

        public NonlinearMinimizationResult(IObjectiveModel modelInfo, int iterations, ExitCondition reasonForExit)
        {
            ModelInfoAtMinimum = modelInfo;
            Iterations = iterations;
            ReasonForExit = reasonForExit;

            EvaluateCovariance(modelInfo);
+            EvaluateGoodnessOfFit(modelInfo);


Both EvaluateCovariance, EvaluateGoodnessOfFit seem like they should be optional.
The user might not need these, but they are relatively expensive to calculate.
On the master branch, EvaluateCovariance for example shows up in profiles as being quite significant (esp due to its allocations).

febkor · 2025-07-07T09:41:10Z

src/Numerics/Optimization/LevenbergMarquardtMinimizer.cs

            while (iterations < maximumIterations && exitCondition == ExitCondition.None)
            {
                iterations++;

                while (true)
                {
+                    // Store current Hessian diagonal for restoration if step is rejected
+                    var savedDiagonal = Hessian.Diagonal().Clone();


Is the separate Clone() required?
Matrix.Diagonal() produces a new vector with values copied in from the matrix.

febkor · 2025-07-07T09:42:49Z

src/Numerics/Optimization/LevenbergMarquardtMinimizer.cs

+            var mu = initialMu * Hessian.Diagonal().Max(); // Damping parameter μ
+            var nu = 2.0; // Multiplication factor ν for mu updates
+
+            // Counters for successful and failed iterations


The ncsuc counter seems to get incremented, but is not used for any logic?

febkor · 2025-07-07T09:59:00Z

src/Numerics/Optimization/ObjectiveFunctions/NonlinearObjectiveModel.cs

+
+                // Create adapter function that returns the model value for current observation
+                // when given parameters array
+                double funcAdapter(double[] p)


It seems it'd be more natural to extend the NumericalJacobian to support modelFunctions of the shape needed here?
The usage here seems more complex than needed and some inefficiencies:

_modelFunction returns a whole vector, but only a single value is used

this funcAdapter lambda creates an allocation per observation (minor in the context of the rest of the code, but still avoidable)

febkor · 2025-07-07T16:11:31Z

src/Numerics/Optimization/ObjectiveFunctions/NonlinearObjectiveModel.cs

+                weights = weights.PointwiseAbs();
+            }
+
+            Weights = (weights == null)


Does the Weights matrix get used anywhere?
Apart from some trivial null checks, the usages I noticed all only use the diagonal, i.e. the weights vector. Are there any I might have missed?
It can be quite expensive in memory terms to create it.

diluculo added 9 commits March 21, 2025 10:27

Rename NonlinearObjectiveFunction to NonlinearObjectiveModel to bette…

44e41dc

…r reflect its purpose in modeling optimization problems

Add direct residual function support to NonlinearObjectiveModel and r…

597917e

…elated interfaces

Improve numerical differentiation in NonlinearObjectiveModel

8bc80ad

- Add configurable accuracy with orders 1-6 for both model and residual functions - Use NumericalJacobian class for more reliable derivative approximation

Add factor of 2.0 to covariance calculation to compensate for 1/2 in …

ea74979

…objective function

Apply MINPACK-style cubic damping and trust-region update to LM optim…

88c09c3

…izer

Fix missing variable copies in CreateNew() method of NonlinearObjecti…

66f1b56

…veModel

Add direct residual circle fitting test to NonLinearCurveFittingTests

a0d434b

Add comprehensive statistical metrics to NonlinearMinimizationResult

9e0f585

- TStatistics and PValues for parameter significance testing - ConfidenceIntervalHalfWidths for parameter precision - Dependencies to measure parameter correlations - Goodness-of-fit statistics

Add ParameterStatistics class for computing parameter inference stati…

1306738

…stics

diluculo force-pushed the LMfromResidual branch from ce4ba1c to 1306738 Compare April 2, 2025 08:06

febkor reviewed Jul 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enhanced Levenberg-Marquardt Algorithm with Residual Function Support #1116

Enhanced Levenberg-Marquardt Algorithm with Residual Function Support #1116

Uh oh!

diluculo commented Mar 21, 2025

Uh oh!

diluculo commented Mar 22, 2025 •

edited

Loading

Uh oh!

diluculo commented Mar 28, 2025

Uh oh!

diluculo commented Mar 31, 2025

Uh oh!

diluculo commented Mar 31, 2025

Uh oh!

febkor commented Apr 1, 2025

Uh oh!

diluculo commented Apr 2, 2025 •

edited

Loading

Uh oh!

diluculo commented Apr 2, 2025

Uh oh!

diluculo commented Apr 2, 2025

Uh oh!

febkor left a comment

Uh oh!

febkor Jul 7, 2025

Uh oh!

febkor Jul 7, 2025

Uh oh!

febkor Jul 7, 2025

Uh oh!

febkor Jul 7, 2025

Uh oh!

febkor Jul 7, 2025

Uh oh!

Uh oh!

Enhanced Levenberg-Marquardt Algorithm with Residual Function Support #1116

Are you sure you want to change the base?

Enhanced Levenberg-Marquardt Algorithm with Residual Function Support #1116

Uh oh!

Conversation

diluculo commented Mar 21, 2025

Uh oh!

diluculo commented Mar 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

diluculo commented Mar 28, 2025

Uh oh!

diluculo commented Mar 31, 2025

Uh oh!

diluculo commented Mar 31, 2025

Uh oh!

febkor commented Apr 1, 2025

Uh oh!

diluculo commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

diluculo commented Apr 2, 2025

Uh oh!

diluculo commented Apr 2, 2025

Uh oh!

febkor left a comment

Choose a reason for hiding this comment

Uh oh!

febkor Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

febkor Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

febkor Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

febkor Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

febkor Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

diluculo commented Mar 22, 2025 •

edited

Loading

diluculo commented Apr 2, 2025 •

edited

Loading