Error rate for SVM

[HadeelHassan]

What would you like to submit? (put an 'x' inside the bracket that applies)

• question
• bug report
• feature request

Issue description

If you would like to submit an issue, please include a sample of the code you are trying to execute, as well as any data you might be using in your experiment, such as data tables, images, anything that might be necessary to reproduce the problem.

Note: If you would like to support the development for this feature or resolution of this bug, consider adding a bounty to it later in https://www.bountysource.com/teams/accord-net/issues

[HadeelHassan]

Hi
I made face recognition system using BoW and Chi-square multiclass SVM with big help from cesar and his powerful library Accord.Net
there is a little problem with this program , it gives class for input face anyway , even if that face is not available in database , how can i give an error rate for predicted result or making threshold for that
can anyone help me please
thanks

[cesarsouza]

Hi @HadeelHassan,

The easiest way to achieve what you want is by adding an extra class in your classification problem to model everything that is not a face (the background). In the same way that you have trained the system for multiple people in your database, add a new category containing only images that do not contain a face.

Then, if in the end the SVM says that the class for a given image should be this extra class, you can just conclude it is not a face.

Hope it helps,
Cesar

[HadeelHassan]

@cesarsouza
grate idea ..... I will try it and let you know
Thanks a lot.

[HadeelHassan]

@cesarsouza
but do not you think that human face not similar in features with backgrounds, the predicted result will be any other human class in my database rather than a background class. i was thinking about something like this code to calculate predicted error
error = new ZeroOneLoss(outputs).Loss(tree.Decide(inputs));
so if error rate more than a specific threshold then this face will ignored because it is not found in my database. is there something like that ? or any other idea

[blaisexen]

interesting...

[HadeelHassan]

@cesarsousa
i tried this way it is not working , any other opinion please ??

[cesarsouza]

Hi @HadeelHassan,

Please, may I ask more details about how you are testing this approach?

It should really work, at least to some extent. It has been used before in countless other problems and domains.

Just to reiterate a little bit: You should be grabing examples of the situation that you are currently testing (not exactly the same data, but the same situation, e.g. similar places, images or videos) that do not contain an human face and add those samples as a separate, but fake, "person" in your dataset of people. This should work as long as you have added enough samples to this "background", or "fake person", category.

What has happened after you added those samples? Has the performance decreased? Are "no faces" still being recognized as "faces" in your problem domain?

In the worst case, if this do not work for you at all even though you have really tried, you can just use a face detector to filter out faces from no-faces in your images, for example using the HaarObjectDetector class.

Hope it helps,
Cesar

[HadeelHassan]

@cesarsouza
i have no problem with background because i made Skin examination for detected face, so the false positive of Viola Jones reduced, also i captured some background images and gave them one class to be more efficient , i was speaking for human faces that not included in dataset
for example , let say that this program is for catching criminals in streets , and i have database of criminal faces, any face in street will be detected, but it will be classified by SVM , the classifier will not tell me that this person is not found in database , i hope you understand me now , i am sorry for my bad english language

[cesarsouza]

Oh, now I understand. Well, it might be possible, in practice, to disregard samples that the SVM is not completely confident about. However, here, there are a number of possibilities to achieve what you need. I can suggest two ways:

1. Grab a dataset of faces on the wild such as the Labeled Faces in the Wild dataset. Use all faces from this dataset as the "fake person" class in your dataset, as I had explained above for background samples. If the image is recognized as belonging to this "fake person" class, you can interpret it as belonging to someone outside your dataset. Or:

2. If I remember correctly, you are using a MulticlassSupportVectorMachine right? If that's the case, please set the decision method of this machine to Voting using

svm.Method = MulticlassComputeMethod.Voting;

Now, you should be able to get probability estimates from the SVMs using

svm.Probability(input_image);

Using those probabilities, you may be able to filter out samples that the SVMs are not really sure about. For example, if the probability of a given input image is less than 0.5, you may discard it as not belonging from any of the faces in your dataset. However, this may or may not work for your problem, and you might need to find this optimum threshold for your dataset by hand.

I would advise trying to build a dataset of faces that do not belong to any person in your dataset like in method 1. You can try to use a face detector to obtain all faces that do not correspond to any of the people currently present in your dataset and use those to try to learn the "fake person" class I had mentioned above.

Hope it helps,
Cesar

[HadeelHassan]

@cesarsouza
thanks a lot
i think that the second one is good solution , but i will try both of them to see which one will give me better accuracy .
this is my code
Bitmap[] testimage = new Bitmap[1];

                testimage[0] = newimage;//new image is detected face by Viola Jones
                var ff = bow.Transform(testimage);//Extract Bag of Word
                var predicted = machine.Decide(ff);// machine is MulticlassSupportVectorMachine <ChiSquare>
                                
                for (int r = 1; r <= sizeofdataset; r++)// search in dataset of faces images
                {
                    if (predicted[0] == Convert.ToInt16(workSheet.Cells[r, 3].Value.ToString()))//clsss number is in this row of this excel sheet 
                    {
                        listBox1.SelectedItem = r;
                        pictureBox2.Image = newimage;
                        string pppp = (workSheet.Cells[r, 1]).Value.ToString();
                        pictureBox3.Image = Bitmap.FromFile(pppp);
                        MessageBox.Show("Test" + r);
                        break;
                    }
                }

Where should i put
machine.Method = MulticlassComputeMethod.Voting;
var prob = machine.Probability(ff);
many many thanks

[blaisexen]

@HadeelHassan

hi,
I'm interested into your work using accord.net for face recognition with svm,

here's what I got with Accord_3.4.0
http://www.mediafire.com/file/udq4konr4csdsq0/Blaise_Face_Recognition_Setup.exe
if it message AV detected....it's a false alarm.

I hope to get answers from you if my test are higher than yours?

thank you

[HadeelHassan]

@blaisexen
sorry ,I paused my antivirus but windows still refuse to install it, i am afraid it will put my PC at risk
{Windows Defender SmartScreen prevented an unrecognized app from starting. Running this app might put your PC at risk.}
if you want , can you tell me what your program does?

[HadeelHassan]

@cesarsouza
i tried the first method as you recommended, and gave it class 0, as this code :
if (predicted[0] == Convert.ToInt16(workSheet.Cells[r, 3].Value.ToString())&&(!( predicted[0]==0)))
the efficiency got better but not best , i'ld like to try the second method can you please tell me how to implement it
many many thanks

[cesarsouza]

Hi Hadeel,

Let's say you used the .Decide function to determine to which class a face belongs. That's how you are doing right now, right?

So, after you have called the .Decide function, call also the .Probability function. This should give you a value between 0 and 1 determining how confident the model is that the image you gave to the decide method belongs to any of the classes in your dataset.

If this probability is very low, it means that the model is uncertain whether the class is really from the class the Decide method said it should be. This should mean that the image may not be a face from your dataset at all.

What you can do is to filter out those images where the model is uncertain and label them as not being faces from your dataset. For example, you can only consider a face to be from your dataset if its probability is higher than 0.9 or (90%).

However, there might be lots of trial and error before you can determine an appropriate threshold for your application, and there are also other ways this probability value can be computed. As a start, please make sure that your model has been set to use the Voting decision method using

svm.Method = MulticlassComputeMethod.Voting;

as this is the easiest way to get the Probability method working.

Regards,
Cesar

[blaisexen]

@HadeelHassan

I understand you, anyway

We have the same concept to get the exact recognition we want,

can you send a compiled application so I can test here if it's good than mine, or not?

[blaisexen]

@ALL,
Did someone test it?
as I said its a false alarm from AV.

anyway,
@HadeelHassan
let me have what's your thoughts about the application I send or just the image I attached.

[HadeelHassan]

@cesarsouza
Hi cesar
thank you very mach for your help, i tried both approaches that you gave, but the program find correct results and non correct results with the same ratio , for example it gave 0.636 probability ratio for my face and gave the same ratio for wrong classified,i tried to give 0.7 as threshold but no face is recognized, i think that is not because the program but because the image quality, i am using web camera of my laptop for create dataset and use the same for my program is that effective, i mean is the resolution of image effect the results of my program ??
many thanks
Hadeel

[HadeelHassan]

Hi @blaisexen
my program work very well for static image, but not very good for real time face recognition, it seems that your program is better if it work well for real time.
by the way how did you evaluate accuracy of your program?
thanks
Hadeel

[blaisexen]

@HadeelHassan
**actually it's not really accurate at all times.,sometimes it recognized other faces, that''s the problem I'm facing,,
and ALSO.... I think we have the same problem here, like the last message you have @cesarsouza

[cesarsouza]

@HadeelHassan, @blaisexen,

In this case, instead of using the voting method, switch to MultilabelSupportVectorMachine (instead of the MulticlassSupportVectorMachine you were using before), calibrate your machines using ProbabilisticOutputCalibration, set the Method property of the multi-label machine to "SumsToOneWithEmphasisOnWinner" and try again.

I will try to give an example below:

// Let's say we have the following data to be classified
// into three possible classes. Those are the samples:
// 
double[][] inputs =
{
    //               input         output
    new double[] { 0, 1, 1, 0 }, //  0 
    new double[] { 0, 1, 0, 0 }, //  0
    new double[] { 0, 0, 1, 0 }, //  0
    new double[] { 0, 1, 1, 0 }, //  0
    new double[] { 0, 1, 0, 0 }, //  0
    new double[] { 1, 0, 0, 1 }, //  1
    new double[] { 0, 0, 0, 1 }, //  1
    new double[] { 0, 0, 0, 1 }, //  1
    new double[] { 1, 0, 1, 1 }, //  2
    new double[] { 1, 1, 0, 1 }, //  2
    new double[] { 0, 1, 1, 1 }, //  2
    new double[] { 1, 1, 1, 1 }, //  2
};

int[] outputs = // those are the class labels
{
    0, 0, 0, 0, 0,
    1, 1, 1,
    2, 2, 2, 2,
};

// Create the multi-class learning algorithm for the machine
var teacher = new MultilabelSupportVectorLearning<Gaussian>() // Note: this is multi-label, not multi-class (keep the same kernel you were using before, do not change to a Gaussian, this is just an example)
{
    Learner = (param) => new SequentialMinimalOptimization<Gaussian>() // Configure in the same way you were doing before, you do not really need to use kernel estimation if you were not using before
    {
        UseKernelEstimation = true
    }
};

// Learn the machine
var machine = teacher.Learn(inputs, outputs);

// Create multi-label calibration algorithm for the machine
var calibration = new MultilabelSupportVectorLearning<Gaussian>() // again, use the same kernel you were using before
{
    Model = machine, // We will start with an existing machine

    // Configure the learning algorithm to use Platt's output calibration
    Learner = (param) => new ProbabilisticOutputCalibration<Gaussian>()
    {
        Model = param.Model // Start with an existing machine
    }
};

// Calibrate the machine
calibration.Learn(inputs, outputs);

// Now, set the machine probability method:
machine.Method = MultilabelProbabilityMethod.SumsToOneWithEmphasisOnWinner;

// Get class probabilities for each class & sample
double[][] prob = machine.Probabilities(inputs);

// Check if you can discern faces from non-faces using the probabilities

I have not actually tested the code above, so I really hope it works! Also, please read the comments I tried to put carefully, if you have questions please ask!

Regards,
Cesar

[HadeelHassan]

@cesarsouza
hi cesar
I dont know how to say thank you, I will try it hoping results will be better
many many and many thanks
Hadeel

[HadeelHassan]

@blaisexen
hi
I am sorry , i will share program with you when i gain good accuracy
Regards
Hadeel

[HadeelHassan]

@cschiano
Hi Cesar, I hop you fine
i am sorry for being late, i have had some problem with the code, finally i installed Accord.Net 3.7
but why the predicted bool or double ?
how to make matching with class label? previously i was making matching with them because predicted was int
if (predicted[0] == Convert.ToInt16(workSheet.Cells[r, 3].Value.ToString()))
{
listBox1.SelectedItem = r;
string inDatabase = workSheet.Cells[r, 1].Value.ToString();
workSheet1.Cells[i + 1, 2] = inDatabase;// pathofdataset + "\" +
listBox1.SelectedItem.ToString();
workSheet1.Cells[i + 1, 3] = predicted[0].ToString();
break;
}
and this is the error for the first line in the code
Operator '==' cannot be applied to operands of type 'bool[]' and 'short'
if (predicted[0] == Convert.ToInt16(workSheet.Cells[r, 3].Value.ToString()))
what can i do please? ( I think I am dummy 👎 )
Regards
Hadeel

[blaisexen]

@HadeelHassan
do like this:

                    bool[] xxx = new Boolean[0];
                    Int16 yyy=0;
                    if (Convert.ToInt16(xxx[0]) == 0) 
OR
                    if (xxx[0] == Convert.ToBoolean(yyy)) 

[HadeelHassan]

@blaisexen
thank you very much but i think that not solve the problem, classes are from 0....k where k is the number of individuals in dataset not 0 or 1
thanks a lot .

Hadeel

[blaisexen]

@HadeelHassan,
you can index your class number,
then you can put
for(int i=0; i<classnums;i++)
{
if (predicted[i] == here.... then begin get and break;;
}

[HadeelHassan]

@blaisexen
i have recorded dataset in Excel sheet , first column is the location of image in PC, second column is the name of individual, third is class number as in attached image
i am now trying to test input images( tested images) that is inside list box
the result should recorded in another excel sheet (record the location of tested images and the location of matched image and class number)
by this way i test many images at the same time and can calculate the accuracy in second excel sheet
so the purpose of my code is to find results and record them in excel sheet
and this is the code

private void button18_Click(object sender, EventArgs e)
{
// Create the multi-class learning algorithm for the machine
var teacher = new MulticlassSupportVectorLearning()
{
// Configure the learning algorithm to use SMO to train the
// underlying SVMs in each of the binary class subproblems.
Learner = (param) => new SequentialMinimalOptimization()
{
// Estimate a suitable guess for the Gaussian kernel's parameters.
// This estimate can serve as a starting point for a grid search.
UseKernelEstimation = true
}
};

        // Configure parallel execution options
        teacher.ParallelOptions.MaxDegreeOfParallelism = 1;

        // Learn a machine
        var machine = teacher.Learn(features, outputs);
        try
        {
            var watchTest = System.Diagnostics.Stopwatch.StartNew();
            for (int i = 0; i < listBox2.Items.Count - 1; i++)
            {
                Bitmap[] testimage = new Bitmap[1];
                listBox2.SelectedIndex = i;
                testimage[0] = new Bitmap(pathoftestset + "\\" + listBox2.SelectedItem.ToString());
                var ff = bow.Transform(testimage);
                var predicted = machine.Decide(ff);

                int r = 1;
                for (r = 1; r <= sizeofdataset; r++)
                {
                    if (predicted[0] == Convert.ToInt16(workSheet.Cells[r, 3].Value.ToString()))
                    {
                        listBox1.SelectedItem = r;
                        string inDatabase = workSheet.Cells[r, 1].Value.ToString();
                        workSheet1.Cells[i + 1, 2] = inDatabase;// pathofdataset + "\\" + listBox1.SelectedItem.ToString();
                        workSheet1.Cells[i + 1, 3] = predicted[0].ToString();
                       
                        
                        //MessageBox.Show("Test" + r);
                        break;
                    }
                }

            }

            watchTest.Stop();
            var elapsedMs = watchTest.ElapsedMilliseconds;
            MessageBox.Show("Time for Testing is: " + elapsedMs.ToString() + "  ms");
        }
        catch (Exception ex) { MessageBox.Show("Error" + ex); }

    }

this cod was working because it was multiclass no multilablel , the predict variable was int so i was make matching easily

[blaisexen]

try to change code from

Learner = (param) => new SequentialMinimalOptimization()
{
// Estimate a suitable guess for the Gaussian kernel's parameters.
// This estimate can serve as a starting point for a grid search.
UseKernelEstimation = true
}
};

to

Learner = (param) =>
                {
                    return new SequentialMinimalOptimization()
                    {
                        Complexity = 1,
                        Tolerance = 0.01000,
                        CacheSize = 1000,
                        Strategy = SelectionStrategy.Sequential,
                    };
                }

[HadeelHassan]

@blaisexen
i tried it , there is no change
thanks

[blaisexen]

@HadeelHassan

is that the issue?

Operator '==' cannot be applied to operands of type 'bool[]' and 'short'

is that actually the error, or some things you want?

because I think we have the same way we want!

[HadeelHassan]

@blaisexen
yes, now this is my error
I want to find if multi label SVM can give me better results than multiclass SVM
i cannot find results with this error

[blaisexen]

@HadeelHassan
I'll try....
....
//should be like this...
...
...
int r = 1;
for (r = 1; r <= sizeofdataset; r++)
{
bool predicted = machine.Decide(ff, r); //MOVE HERE, and should be like that bool
if (Convert.ToInt16(predicted).ToString() == Convert.ToInt16(workSheet.Cells[r, 3].Value.ToString()))
{
listBox1.SelectedItem = r;
...
...

[HadeelHassan]

@blaisexen
not working , there is error
Error 1 The best overloaded method match for 'Accord.MachineLearning.MultilabelScoreClassifierBase<double[]>.Decide(double[][], double[])' has some invalid arguments C:\Users\User\Documents\Visual Studio 2012\Projects\LearningWithDecisionTree\LearningWithDecisionTree\Form1.cs 789 37 LearningWithDecisionTree
Error 2 Argument 2: cannot convert from 'int' to 'double[]' C:\Users\User\Documents\Visual Studio 2012\Projects\LearningWithDecisionTree\LearningWithDecisionTree\Form1.cs 789 56 LearningWithDecisionTree

[blaisexen]

@HadeelHassan
I think we should wait from @cesarsouza

I have also a question that is connected from your project too,

thank you for the time

[cesarsouza]

Hi @blaisexen, @HadeelHassan,

Sorry, I have not yet read all the replies, but I just wanted to let you know that, since this is a multi-label support vector machine, it should be able to output a set of boolean labels for each class in your classification problem. So for example, if you have 26 classes, the output of the decide function will be a bool[] array of size 26, where if the value of the boolean at position i is true it means class i is present in the image you have given to the Decide function.

You can convert a multi-label classifier into a multi-class one using the .ToMulticlass() method of the multi-label SVM. For example, if you have a multi-label svm named ksvm after calling:

ksvm = calibration.Learn(facevalues.ToArray(), faceclassnums.ToArray());

you can convert it back to a multi-class classifier using

msvm = ksvm.ToMulticlass();

after this, you should be able to use Decide and Probability as normal as you had been doing before with multi-class SVMs.

Regards,
Cesar

[HadeelHassan]

@ceasarsouza
ok, get it, I will try it tomorrow, and let you know.
thanks a lot
also thanks @blaisexen for your trying to help
Hadeel

[HadeelHassan]

@cesarsouza
I tried it, not a big difference, in multi class, the correct decision was 120 from 135, while in multi label it became 123 from 135. all that in static images. any suggestion?
thanks a lot
Hadeel

[blaisexen]

@HadeelHassan
yes indeed they are almost no difference at all.

[cesarsouza]

Hi @HadeelHassan, @blaisexen,

Yes, the performance should be similar, but the probabilty estimates should be better than before. You might be able to use this improved probability to detect whether something was really a face or not by checking whether it is very low (lower than some threshold you might need to find manually).

Regards,
Cesar

[blaisexen]

@HadeelHassan,
I suggest to closed this question issue.

[blaisexen]

@HadeelHassan

please tell me if this application of image recognition is better than what you have there!

this is a cards or card image recognition,
http://www.mediafire.com/file/crwb2bpmvopqquf/Blaise_Imager.exe

for testing...
you can select images inside the program files folder \Blaise_imageR\sampletest\

I hope you can test it!

thank you