Photo: Ox Street

What are 3 metrics of evaluation?

Metrics like accuracy, precision, recall are good ways to evaluate classification models for balanced datasets, but if the data is imbalanced then other methods like ROC/AUC perform better in evaluating the model performance.

analyticsvidhya.com - Evaluation Metrics For Classification Model - Analytics Vidhya

Can ibuprofen cure prostatitis?

It's commonly recommended to patients to take over the counter nonsteroidal anti-inflammatory (NSAID) products such as ibuprofen or acetaminophen,...

Last updated Jan 28, 2022

At what age does DHT drop?

Results. We found that serum testosterone, DHT, and E2 decline gradually from ages 35 onwards with a more marked decline after 80 years of age....

Last updated Sep 10, 2020

Promotion

This article was published as a part of the Data Science Blogathon

Image source https://www.kaggle.com/nkitgupta/evaluation-metrics-for-multi-class-classification

Abstract

The most important task in building any machine learning model is to evaluate its performance. So, the question arises that how would one measure the success of a machine learning model? How would we know that when to stop the training and evaluation and when to call it good?

With the help of this article, I’ll try to answer these questions. This article is organized as follows: In the first section, I’ll start with introducing what we mean by “ML Model Evaluation” and why it’s actually necessary to evaluate any machine learning model. In other subsequent section, we’ll discuss various evaluation metrics available to specific use cases in subsequent sections to better understand these metrics.

Introduction

Evaluation metrics are tied to machine learning tasks. There are different metrics for the tasks of classification and regression. Some metrics, like precision-recall, are useful for multiple tasks. Classification and regression are examples of supervised learning, which constitutes a majority of machine learning applications. Using different metrics for performance evaluation, we should be able to improve our model’s overall predictive power before we roll it out for production on unseen data. Without doing a proper evaluation of the Machine Learning model by using different evaluation metrics, and only depending on accuracy, can lead to a problem when the respective model is deployed on unseen data and may end in poor predictions. In the next section, I’ll discuss the Classification evaluation metrics that could help in the generalization of the ML classification model.

Classification Metrics in Machine Learning

Classification is about predicting the class labels given input data. In binary classification, there are only two possible output classes(i.e., Dichotomy). In multiclass classification, more than two possible classes can be present. I’ll focus only on binary classification. A very common example of binary classification is spam detection, where the input data could include the email text and metadata (sender, sending time), and the output label is either “spam” or “not spam.” (See Figure) Sometimes, people use some other names also for the two classes: “positive” and “negative,” or “class 1” and “class 0.” Figure — Email spam detection is a binary classification problem (source: From Book — Evaluating Machine Learning Model — O’Reilly) There are many ways for measuring classification performance. Accuracy, confusion matrix, log-loss, and AUC-ROC are some of the most popular metrics. Precision-recall is a widely used metrics for classification problems.

Accuracy

Accuracy simply measures how often the classifier correctly predicts. We can define accuracy as the ratio of the number of correct predictions and the total number of predictions. When any model gives an accuracy rate of 99%, you might think that model is performing very good but this is not always true and can be misleading in some situations. I am going to explain this with the help of an example. Consider a binary classification problem, where a model can achieve only two results, either model gives a correct or incorrect prediction. Now imagine we have a classification task to predict if an image is a dog or cat as shown in the image. In a supervised learning algorithm, we first fit/train a model on training data, then test the model on testing data. Once we have the model’s predictions from the X_test data, we compare them to the true y_values (the correct labels).

Image Source – https://www.kdnuggets.com/2020/04/performance-evaluation-metrics-classification.html

Is olive oil good for the brain?

The last brain health benefit of olive oil is that it can ward off cognitive decline. Higher intake of monounsaturated fats improves memory and...

Last updated Oct 8, 2018

How do sedated patients poop?

A tube may be inserted into the patient's rectum in order to empty the bowels before the procedure. In some cases, a suppository may be used. After...

Last updated Apr 1, 2018

Promotion

We feed the image of the dog into the training model. Suppose the model predicts that this is a dog, and then we compare the prediction to the correct label. If the model predicts that this image is a cat and then we again compare it to the correct label and it would be incorrect. We repeat this process for all images in X_test data. Eventually, we’ll have a count of correct and incorrect matches. But in reality, it is very rare that all incorrect or correct matches hold equal value. Therefore one metric won’t tell the entire story. Accuracy is useful when the target class is well balanced but is not a good choice for the unbalanced classes. Imagine the scenario where we had 99 images of the dog and only 1 image of a cat present in our training data. Then our model would always predict the dog, and therefore we got 99% accuracy. In reality, Data is always imbalanced for example Spam email, credit card fraud, and medical diagnosis. Hence, if we want to do a better model evaluation and have a full picture of the model evaluation, other metrics such as recall and precision should also be considered.

Confusion Matrix

Confusion Matrix is a performance measurement for the machine learning classification problems where the output can be two or more classes. It is a table with combinations of predicted and actual values. A confusion matrix is defined as thetable that is often used to describe the performance of a classification model on a set of the test data for which the true values are known.

Image Source — https://www.roelpeters.be/glossary/what-is-a-confusion-matrix/

It is extremely useful for measuring the Recall, Precision, Accuracy, and AUC-ROC curves. Let’s try to understand TP, FP, FN, TN with an example of pregnancy analogy. True Positive: We predicted positive and it’s true. In the image, we predicted that a woman is pregnant and she actually is. True Negative: We predicted negative and it’s true. In the image, we predicted that a man is not pregnant and he actually is not. False Positive (Type 1 Error)- We predicted positive and it’s false. In the image, we predicted that a man is pregnant but he actually is not. False Negative (Type 2 Error)- We predicted negative and it’s false. In the image, we predicted that a woman is not pregnant but she actually is. We discussed Accuracy, now let’s discuss some other metrics of the confusion matrix 1. Precision —Precision explains how many of the correctly predicted cases actually turned out to be positive. Precision is useful in the cases where False Positive is a higher concern than False Negatives. The importance of Precision is in music or video recommendation systems, e-commerce websites, etc. where wrong results could lead to customer churn and this could be harmful to the business. Precision for a label is defined as the number of true positives divided by the number of predicted positives. 2. Recall (Sensitivity) — Recall explains how many of the actual positive cases we were able to predict correctly with our model. It is a useful metric in cases where False Negative is of higher concern than False Positive. It is important in medical cases where it doesn’t matter whether we raise a false alarm but the actual positive cases should not go undetected! Recall for a label is defined as the number of true positives divided by the total number of actual positives.

Is whisky good for enlarged prostate?

Researchers also concluded that any benefits of drinking alcohol for an enlarged prostate may be outweighed by the negative effect it has on...

Last updated Jun 10, 2018

How do I know if my prostatitis is chronic?

Prostatitis is an inflammation of the prostate that can often be painful. It can be chronic or acute: Chronic prostatitis develops gradually and...

Last updated Jul 22, 2022

Imagine Your Moment Of Freedom!

Fluxactive Complete is conveniently packed with over 14 essential prostate powerhouse herbs, vitamins and grade A nutrients which work synergistically to help you support a healthy prostate faster

Learn More »

Promotion

3. F1 Score — It gives a combined idea about Precision and Recall metrics. It is maximum when Precision is equal to Recall.

F1 Score is the harmonic mean of precision and recall.

The F1 score punishes extreme values more. F1 Score could be an effective evaluation metric in the following cases:

When FP and FN are equally costly.

True Negative is high

4. AUC-ROC — The Receiver Operator Characteristic (ROC) is a probability curve that plots the TPR(True Positive Rate) against the FPR(False Positive Rate) at various threshold values and separates the ‘signal’ from the ‘noise’. The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish between classes. From the graph, we simply say the area of the curve ABDE and the X and Y-axis. From the graph shown below, the greater the AUC, the better is the performance of the model at different threshold points between positive and negative classes. This simply means that When AUC is equal to 1, the classifier is able to perfectly distinguish between all Positive and Negative class points. When AUC is equal to 0, the classifier would be predicting all Negatives as Positives and vice versa. When AUC is 0.5, the classifier is not able to distinguish between the Positive and Negative classes.

Image Source— https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/

Working of AUC —In a ROC curve, the X-axis value shows False Positive Rate (FPR), and Y-axis shows True Positive Rate (TPR). Higher the value of X means higher the number of False Positives(FP) than True Negatives(TN), while a higher Y-axis value indicates a higher number of TP than FN. So, the choice of the threshold depends on the ability to balance between FP and FN. 5. Log Loss — Log loss (Logistic loss) or Cross-Entropy Loss is one of the major metrics to assess the performance of a classification problem. For a single sample with true label y∈{0,1} and a probability estimate p=Pr(y=1), the log loss is:

Conclusion

Understanding how well a machine learning model will perform on unseen data is the main purpose behind working with these evaluation metrics. Metrics like accuracy, precision, recall are good ways to evaluate classification models for balanced datasets, but if the data is imbalanced then other methods like ROC/AUC perform better in evaluating the model performance. ROC curve isn’t just a single number but it’s a whole curve that provides nuanced details about the behavior of the classifier. It is also hard to quickly compare many ROC curves to each other. The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

analyticsvidhya.com - Evaluation Metrics For Classification Model - Analytics Vidhya

Why does my pee feel cold?

Takeaway. Pee shivers might be due to a sudden drop in body temperature, or mixed signals in your nervous system. They are harmless. This doesn't...

Last updated Jun 12, 2021

What does turmeric do for arteries?

Turmeric Turmeric is one of nature's most potent anti-inflammatories, due to a compound called curcumin. This not only reduces arterial...

Last updated Jun 10, 2021

Promotion

Can black olives lower blood pressure?

Olives can help manage high blood pressure. Due to their high oleic acid and antioxidant polyphenol content, olives have also been found to be the...

Last updated Nov 18, 2018

Imagine Your Moment Of Freedom!

Fluxactive Complete is conveniently packed with over 14 essential prostate powerhouse herbs, vitamins and grade A nutrients which work synergistically to help you support a healthy prostate faster

Learn More »

Promotion

Why do they keep surgery rooms so cold?

Historically, it was believed that cold temperatures in the OR helped minimize the potential for infections. While that has been disproven, ORs are...

Last updated Feb 10, 2019