Does Accord has gradient free, nonlinear regression classes?

[dotChris90]

• question
• bug report
• feature request

Are there any "Non-Linear"! and Gradient free Regression classes in Accord?

Hello Accord Team :D
I am very impressed of your work and find amazing that it already works for .NET Core! Amazing work.
Since I come from MATLAB and Python area I was thinking "hey why not ACCORD in Powershell." And actual I tried with Powershell and Powershell Core. It worked. Linear Regression - really, really nice.

But how about non linear models? So tried non-linear - if I compute the Gradient function - ok - i can compute. But how about : Gradient is too complex to compute?

So in other words : does ACCORD give user the possibility to use "http://accord-framework.net/docs/html/T_Accord_Statistics_Models_Regression_NonlinearRegression.htm" (this example) but without! computing the Gradient? And do Accord has some classes for finding the global Minimum of the corresponding Regression error function (not just local)?

Do not worry - just is for curiosity ;)

Wish you all a nice week. :D

[cesarsouza]

Hi @dotChris90!

Sorry for the delay in answering! However, the answer to your question is yes - there indeed is a method that can be used to implement gradient free regression in Accord.NET. The method is called Cobyla, and its documentation can be found here.

However, Cobyla is a quite general optimization algorithm. In order to use it to implement regression, you might need to pass a loss function to it (i.e. sum of squares error) in order to perform parameter optimization.

Hope it helps!

Cesar

[cesarsouza]

By the way - I will leave this issue open until an example on how to perform non-linear regression fitting using Cobyla is added to the documentation.

[dotChris90]

hi @cesarsouza

thanks for Reply. :) oh my god how I could miss this. Yeah it is Logical.

I think for a Cobyla solution you have to do a partial apply - is this possible with C#?
The General cost would depend on Input (my measurements), Output (also my measurements) and Parameter (I want to know). In e.g. F# you would partial apply your measurements to the cost function
and at end can compute the Parameter for min. costs.

Do you guys think it would be a good idea to have a "learnGradientFree()" method for NonlinearLeastSquares class? If it is a good idea maybe I could try to implement it. :)

[dotChris90]

hey @cesarsouza

ok I tried - I followed http://mikehadlow.blogspot.de/2015/09/partial-application-in-c.html and yes C# can partial application like F#. A lot more complex but works. I could construct the total error function and apply Input and Output.

Just ... I find that a Gradient free learning / Regression makes it more easy to use ACCORD for other People. Most People I know just want simple Regression Things for measurement fits or sth like that. So this Gradient free Version would be extrem easy to use.

So if you think "yes - Gradient free makes sense" : I would offer my help and Code for implement. :) Just Need to select where and if as Extension method or true method. :)

[cesarsouza]

Yes please! What I had mentioned with "I will leave this issue open until an example on how to perform non-linear regression fitting" was exactly referring to that :-)

I would suggest implementing it as a class that implements the NonlinearLeastSquares class. Or better, it could be a modification of this class that, when the gradient function had not been selected, would use Cobyla automatically.

However, please feel free to contribute whatever you would like and that you think would make more sense for your application - I can take care of transforming it into a proper class, or finding someone who can take care of transforming it.

Thanks!

Regards,
Cesar

[dotChris90]

@cesarsouza - ok will deal with it (hopefully) this week.

By the way : this is one of the most amazing libraries I saw in .NET world. It has nearly really everything a data Analyst wish. It can *.mat files, Regression, transformations, optim, ... really good (I started my skills with MATLAB - so every language that can read mat is impressive hehe)!

Just 1 more question (even not belong to this issue) :

• did you guys considered to use it in Powershell? Honestly I really think with ACCORD.NET Powershell could be a quite good alternative to Python. :D

[dotChris90]

Good evening @cesarsouza

First of all - Are you using VS or VS Code for implementing ACCORD?
I tried it on VS Code and got quite many "red lines" i.e. Errors in the csproj files - just want to know if maybe my configuration is wrong or maybe because ACCORD is used with VS (so maybe VS do some Background work).

So because of this I started to implement a child class (just for try out everything).

I started implementing on https://gitlab.com/scienceNumericFun/Accord.Feature_GradientFreeNonlinSQR (you can check - if you feel this is the right way to use).

The unit Test at https://gitlab.com/scienceNumericFun/Accord.Feature_GradientFreeNonlinSQR/blob/master/test/Accord.Statistics/Models/Regression/Fitting.Test/NonLinsqrtTester.cs Shows how I think a Gradient free Version could look like. I choose NelderMead because I think I remember that MATLABs Curve Fitting Toolbox is using it for all this non linear Regression. But it was implemented as generic so you can take any other optimization algorithm.

:)

[dotChris90]

@cesarsouza

Hello - just made an update for this.

I made a git mirror to gitlab of ACCORD.NET and added a sln and some csproj files. They were added for a better Visual Studio Code use. Accord.Core, Accord.Math + Accord.Math.Core and Accord.Statistics.
With this I was able to simple use a "dotnet build" command and the DLLs are compiled.
For some reasons (I still dont know - why) I could not build the Net Standard Solution without Visual Studio - so thats the reason why I made. It could be maybe possible that some $buildVariables in the MSBuild XML files are not generated from the CLI Tools.

The repo / fork is at https://gitlab.com/dotChris/accord-framework/tree/development
The NonlinearLeastSquares class got now 2 new Interfaces for Learn method. One generic where you can use generic to select the optimization algorithm or one normal where 3rd Argument is the optimization. The non generic I added because I tried it also on Powershell and like most Interpreter ... it has no generics ....

My question now is : do you accept pull requests so we somehow can merge it? I think we could merge to the development branch because nothing was deleted and just method and files were added.

[nerndt]

Hello @cesarsouza and dotChris90. I have tried to continue with your work of making a 4 paramter and 5 paramter nonlinear curve fitting gradient free sample for 4PL and 5PL dose response curves:
// GraphPad https://www.graphpad.com/guides/prism/8/curve-fitting/reg_dr_stim_variable.htm Sigmoidal, 4PL, X is log(concentration)
// Y=Bottom + (Top-Bottom)/(1+10^((LogEC50-X)*HillSlope)) // log agonist
nls4PL.Function = (w, x) => w[3] + (w[0] - w[3]) / (1.0 + Math.Pow(10.0, (w[2] - x[0]) * w[1]));

        // GraphPad https://www.graphpad.com/guides/prism/8/curve-fitting/reg_equations_commonly_used_for_in.htm Sigmoidal, 4PL, X is concentration 
        // Y=Bottom + (X^Hillslope)*(Top-Bottom)/(X^HillSlope + EC50^HillSlope) // agonist
        //Function = (w, x) => w[3] + (Math.Pow(x[0], w[1]) * (w[0] - w[3])) / (Math.Pow(x[0], w[1]) +  Math.Pow(w[2], w[1])),

and
// NGE0731202 GraphPad Asymmetric Sigmoidal, 5PL, X is log(concentration) https://www.graphpad.com/guides/prism/8/curve-fitting/reg_asymmetric_dose_response_ec.htm
//LogXb = LogEC50 + (1/HillSlope)*Log((2^(1/S))-1)
// Numerator = Top - Bottom
// Denominator = (1 + 10 ^ ((LogXb - X) * HillSlope)) ^ S
// Y = Bottom + (Numerator / Denominator)

            // NGE07312020 GraphPad Equation: Asymmetric Sigmoidal, 5PL, X is concentration https://www.graphpad.com/guides/prism/8/curve-fitting/reg_asymmetric_dose_response_ec_2.htm
            // Denominator = (1+(2^(1/S)-1)*((EC50/X)^HillSlope))^S
            // Numerator = Top - Bottom
            // Y = Bottom + (Numerator / Denominator)

I have good test data, but for the 4PL the results constantly change but once in a while it is correct.

I am happy to share all code and test data if I can get help to figure out what is still needed to get convergence.
I think it has something to do with the number of iterations and not constraining the various parameters to help it converge.

Any help/suggestions would be greatly appreciated.

PS - I work for a small biotech start up that simulates the human body outside the body (in vitro) accurately enough to grow human cells quite well and we want to run 4PL and 5PL dose response curves on impedance readings.

Take care, Nick

[nickexcellbio]

I have been able to use Cobyla to solve the equation. I do not know how to apply constraints which would definitely help!

        // Non-linear regression can also be solved using arbitrary models
        // that can be defined by the user. For example, let's say we know
        // the overall model for the outputs but we do not know the value
        // of its parameters: log(w0  * x) / sqrt(w1 * y + w2)

        //Func<double[], double[], double> model = (double[] x, double[] w)
        //    => Math.Log(w[0] * x[0]) / Math.Sqrt(w[1] * x[1] + w[2]);

        Func<double[], double[], double> model = (double[] x, double[] w)
            => w[3] + (w[0] - w[3]) / (Math.Pow((1.0 + (Math.Pow(2.0, (1.0 / w[4])) - 1.0) * (Math.Pow((w[2] / x[0]), w[1]))), w[4]));

        // Now that we have the model, we want to find which values we
        // can plug in its parameters such that the error when evaluating
        // in our data is as close to zero as possible. Mathematically, we
        // would like to find the best parameters w that minimizes:

        Func<double[], double> objective = (double[] w) =>
        {
            double sumOfSquares = 0.0;
            for (int i = 0; i < inputs.Length; i++)
            {
                double expected = outputs[i];
                double actual = model(inputs[i], w);
                sumOfSquares += Math.Pow(expected - actual, 2);
            }
            return sumOfSquares;
        };

        ///// This problem is taken from page 415 of Luenberger's book Applied
        ///// Nonlinear Programming. It is to maximize the area of a hexagon of
        ///// unit diameter.
        ///// 
        //var function = new NonlinearObjectiveFunction(9, x =>
        //    -0.5 * (x[0] * x[3] - x[1] * x[2] + x[2] * x[8]
        //    - x[4] * x[8] + x[4] * x[7] - x[5] * x[6]));

        //NonlinearConstraint[] constraints =
        //{
        //    new NonlinearConstraint(9, x => 1.0 - x[2] * x[2] - x[3] * x[3]),
        //    new NonlinearConstraint(9, x =>  1.0 - x[8] * x[8]),
        //    new NonlinearConstraint(9, x =>  1.0 - x[4] * x[4] - x[5] * x[5]),
        //    new NonlinearConstraint(9, x =>  1.0 - x[0] * x[0] - Math.Pow(x[1] - x[8], 2.0)),
        //    new NonlinearConstraint(9, x =>  1.0 - Math.Pow(x[0] - x[4], 2.0) - Math.Pow(x[1] - x[5], 2.0)),
        //    new NonlinearConstraint(9, x =>  1.0 - Math.Pow(x[0] - x[6], 2.0) - Math.Pow(x[1] - x[7], 2.0)),
        //    new NonlinearConstraint(9, x =>  1.0 - Math.Pow(x[2] - x[4], 2.0) - Math.Pow(x[3] - x[5], 2.0)),
        //    new NonlinearConstraint(9, x =>  1.0 - Math.Pow(x[2] - x[6], 2.0) - Math.Pow(x[3] - x[7], 2.0)),
        //    new NonlinearConstraint(9, x =>  1.0 - x[6] * x[6] - Math.Pow(x[7] - x[8], 2.0)),
        //    new NonlinearConstraint(9, x =>  x[0] * x[3] - x[1] * x[2]),
        //    new NonlinearConstraint(9, x =>  x[2] * x[8]),
        //    new NonlinearConstraint(9, x =>  -x[4] * x[8]),
        //    new NonlinearConstraint(9, x =>  x[4] * x[7] - x[5] * x[6]),
        //    new NonlinearConstraint(9, x =>  x[8]),
        //};

        //Cobyla cobyla = new Cobyla(function, constraints);

        // Now, let's use a gradient-free optimization algorithm to 
        // find the best parameters for our model's equations:
        double[] solutionStart = new double[5] { initialParameters[0], initialParameters[1], initialParameters[2], initialParameters[3], initialParameters[4] };
        var cobyla = new Cobyla(numberOfVariables: 5) // we have 5 parameters: w0, w1, w2, w3 and w4
        {
            Function = objective,
            MaxIterations = 100000, // 1000
            Solution = solutionStart // new double[] { 1.0, 6.4, 100 } // start with some random values
        };

        bool success = cobyla.Minimize(); // should be true
        // double[] solution = cobyla.Solution;
        double[] solution = new double[5] { cobyla.Solution[0], cobyla.Solution[1], cobyla.Solution[2], cobyla.Solution[3], cobyla.Solution[4] };

        // Get machine's predictions for inputs
        double[] prediction = inputs.Apply(x => model(x, solution));

        // Compute the error in the prediction (should be 0.0)
        double error = new SquareLoss(outputs).Loss(prediction);

        Debug.WriteLine("Cobyla error:" + error.ToString("N5")); // should be 0.000