Confusion Matrix → nOw No MoRe CoNfUsIoN + How it helps Cyber World.

Confusion matrix

The confusion matrix is one of the most popular and widely used performance measurement techniques for classification models. While it is super easy to understand, its terminology can be a bit confusing.
Therefore, keeping the above premise under consideration, this article aims to clear the “fog” around this amazing model evaluation system.
To get things started, I have included a working example of this matrix to make things simple and easy to understand. However, the application of this technique can be used in more than two classes.

Understanding Accuracy Of Confusion Matrix

Accuracy is defined as the number of true positives and negatives divided by the sum of the true positives and negatives and the sum of false positives and negatives.

               number of true positives + number of true negatives
Acurracy=    ----------------------------------------------------
               (number of true positives + number of true negatives               + number of false positives + number of false negatives)

Basic Terms to Keep in Mind

1. True Positive (TP): These are the events that were correctly predicted by the model as “occurred = Yes.”
2. True Negative (TN): These are the events that were correctly predicted by the model as “not occurred = No.”
3. False Positive (FP): These are the events that were predicted as “occurred = Yes,” but in reality, it was “not occurred = No.”
4. False Negative (FN): This is the opposite of FP, i.e. predicted as “not occurred = No,” but in reality, it was “occurred = Yes.”

True positive rate (TPR)

The True positive rate (TPR) gives the proportion of correct predictions in predictions of the positive class.
The True positive rate is calculated by the following formula:

        number of true positives 
TPR =  --------------------------------------------------------
       (number of true positives + number of false   negatives)

False-positive rate (FPR)

The false-positive rate gives the proportion of incorrect predictions in the positive class.
The false-positive rate is calculated as the total number of false positives divided by the number of false positives and the number of true negatives.

                        number of false positives
False positive rate =  ---------------------------------------------
                         (number of false positives + number of true   .                         negatives)

Recall

Recall gives the proportion of correct predictions in positive class.
Recall (R) is defined as the number of true positives (Tp) over the number of true positives plus the number of false negatives (Fn).

            number of true positives 
Recall =  ----------------------------------------------------              `          (number of true positives + number of false negatives)

Precision

Precision gives the proportion of correct predictions in predictions of positive class.
Precision (P) is defined as the number of true positives (Tp) over the number of true positives plus the number of false positives (Fp).

                 number of true positives
Precision =_________________________________________________________
          (number of true positives + the number of false positives)

F1-Measure

F1-Measure gives the harmonic mean of precision and recall.
The F1-Measure is the weighted harmonic average, or mean, of precision and recall.

            (precision * recall)
F1 = 2 *  ____________________
            (precision + recall)

Need for Confusion Matrix in Machine learning

Here are the advantages of utilizing a confusion matrix:

1. It gives knowledge not just into the errors which are made by a classifier yet into additional of mistakes that are being made.
2. Each row of the confusion matrix addresses the occasions of the real class.
3. Each column of the confusion matrix addresses the cases of that anticipated class.
4. This breakdown encourages you to defeats the impediment of utilizing classification precision alone.
5. The confusion matrix does not just give you knowledge of the mistakes being made by your classifier yet also the kinds of errors that are being made.
6. It shows how any classification model is confused when it makes forecasts.

Key Terms Worth Knowing

Positive Predictive Value: This is very similar to precision with a difference of the fact that it takes prevalence into consideration. In the case where the classes are perfectly balanced (prevalence in 50%), the positive predictive value (PPV) is equivalent to precision.

Null Error Rate: This term defines how often your prediction would be wrong if you always predicted the majority class. In the above example, the null error rate would be 60/100 = 60%, because if you always predicted yes, you would on be wrong for the 60 “no” events. This can be considered and used as a baseline metric to compare your classifier against. However, the best classifier for a particular application will sometimes have a higher error rate than the null error rate.

F Score: This is a weighted average of the true positive (recall) and precision.

Roc Curve: This shows the true positive rates (an event) against false positives (a non-event marked as an event) rate at various cut points. A ROC Curve demonstrates a trade-off between sensitivity (recall and specificity or the true negative rate) and an increase in sensitivity, which will introduce a reduction in specificity. The closer the curve follows to the left-hand border and then top border of the ROC plot, the more accurate the test.

Confusion Matrix or a method used in Machine Learning: How benefits Cyber-security.

Threat Detection and Classification

Machine learning algorithms are used in applications to detect and respond to attacks. This can be achieved by analysing big data sets of security events and identifying patterns of malicious activities. ML works so that when similar events are detected, they are automatically dealt with by the trained ML model.

For instance, the dataset to feed a machine learning model can be created by using Indicators of Compromise (IOCs). These can help monitor, identify, and respond to threats in real time. ML classification algorithms can be used using IOC data sets to classify the behavior of malwares.

An example of such a use is evident in a report from Darktrace, an ML based Enterprise Immune Solution, that claims to have prevented attacks during the WannaCry ransomware crisis. According to David Palmer, Director of Technology at Darktrace, “Our algorithms spotted the attack within seconds in one NHS agency’s network, and the threat was mitigated without causing any damage to that organization,” he said of the ransomware, which infected more than 200,000 victims across 150 countries.

Phishing

Traditional phishing detection techniques alone lacks the speed and accuracy to detect and differentiate between harmless and malicious URLs. Latest ML algorithm predictive URL classification models can identify patterns that reveal malicious emails. To do this, the models are trained on features such as email headers, body-data, punctuation patterns, and more to classify and differentiate the malicious from the harmless.

WebShell

WebShell is a piece of code that is maliciously loaded into a website to provide access to make modifications on the web root directory of the server. This allows attackers to gain access of the database. Which, in turn, enables the bad actor to collect personal information. By using ML, a normal shopping cart behaviour can be detected, and the model can be trained to differentiate between normal and malicious behaviour.

The same goes for User Behaviour Analytics (UBA), which forms a supplementary layer to standard security measures, to provide complete visibility, detect account compromises, and mitigate and detect malicious or anomalous insider activity. By using ML algorithms, patterns of user behaviour are categorised, to understand what constitutes normal behaviour, and to detect abnormal activity. If an unusual action is made on a device on a given network, such as an employee login late at night, inconsistent remote access, or an unusually high number of downloads, the action and user is given a risk score based on their activity, patterns and time.

Network Risk Scoring

Use of quantitative measures to assign risk scores to sections of networks, help organisations to prioritise resources. ML can be used to analyse previous cyber-attack datasets and determine which areas of networks were mostly involved in particular attacks. This score can help quantify the likelihood, and impact of an attack, with respect to a given network area. Thus, helping organisations to reduce the risk of being victimized by further attacks.

When you are doing business profiling, you have to decipher what area, if compromised, is going to destroy your business. It could be a Customer Relationship Management (CRM) system, your accounting system, or your sales system. It’s about knowing, within your specific business environment, what area is most vulnerable. Say, for instance, HR goes down, this may have a low risk score within your company. But if your oil trading platform goes down, that could bring down your entire business. Every company has a different way of doing security. And once you understand the specifics of an organisation, you know what to really protect. And if there is a hack, you know what to prioritise.