Those terms are formally known as Type I and Type II Errors to statisticians and data scientists. As you explore data science and statistics, you will come across these terms so it is extremely important that you know the difference between the two. After learning a little bit more about the terms, you will start to notice Type I and Type II errors as you go about your daily life.
Where do Type I and Type II Errors Come From?
Like other statistical terms, Type I and Type II Errors come from statistical hypothesis testing. Usually in one of these tests we are able to arrive to some sort of conclusion. This conclusion is not always right and when the conclusion is wrong this can result in an error, a Type of error. Depending on the result we are concluding and the error we are committing, this error can be either a Type I or Type II error.
In more specific terms, we commit a Type I Error when we reject the null hypothesis when, in fact, it was true. A Type II Error occurs when we accept the null hypothesis when, in fact, it was false.
False Positive or False Negative?
You may be wondering which error is a False Positive and a False Negative. Well here it is:
False Positive = Type I Error
False Negative = Type II Error
What Do These Errors Mean?
Type I Error (False Positive)
A common, yet effective example of a Type I Error or False Positive is usually the pregnancy test. A False Positive would be telling someone they are pregnant when in fact they are not, like telling a man they are pregnant.
Type II Error (False Negative)
Continuing with the pregnancy example, a Type II Error or False Negative would be telling a pregnant woman they are not pregnant.
All of these errors have various consequences when committed. The result of these errors can be dire but determining which one is worse is entirely up to the problem.
Example of Committing Errors and Consequences
Let’s say you are doing an experiment in trying to classify if an object is an egg or not. You are given a dozens of real eggs and dozens of egg shaped rocks. Now the kicker here is you can only classify them visually without touching them.
So let’s fast forward through this experiment and arrive to the section where you created a machine that can classify an egg with about 90% accuracy. This is where the errors may potentially come in. The consequences associated with each error is dependent on the problem.
Egging a House
For example, a Type I Error or False Positive would be if our machine were to classify an egg shaped rock as an egg. This could be worse than a Type II error in several ways. What if we were a bunch of kids looking to egg a house, and we ended up throwing a rock and breaking a window. Something that would have just cost time and water to clean now could have potentially injured someone and turned into a serious act of vandalism.
A Type II Error of False Negative in this instance would be if our machine classified an egg as not an egg. In our example, this would not be much of an issue because we would just not pick this non-egg. Now, as a bunch of kids, we would not use this non-egg because we want to egg a house without destroying anything.
Which is Worse?
In this example, we can see that a Type I error is more costly and consequential than a Type II error. Here, a Type I error is worse than a Type II. We would hope to not misclassify an egg shaped rock as an egg because of the purpose we are using these eggs for.
Now if we changed the purpose of these eggs, then the worse type of error we can commit can change also. How would a Type II error be worse than a Type I?
How about if we wanted to find as many eggs as possible because each egg is extremely valuable. Maybe the eggs contain gold or something of equal value. In this case, if our machine misclassifies an egg as not an egg, then we potentially missed out on some gold. However, even if our machine misclassified a rock as an egg, then the mistake would not be as costly as in the egging example.
Determining the Worst Error
As you can see from the examples above, each error has its own set of issues depending on the problem. There is no worst error to commit because each problem brings its own set of complications to the table. Errors will happen throughout projects and experiments. It is up to the project designer, test maker, or data scientist to determine which error needs to be reduced the most.
Because each problem has its own context and obstacles, you will need to take them all into consideration when designing your experiments and projects to know which error is the worst one to commit. Hopefully, after reading through this, you have a better understanding of the types of errors and the difference between False Positives and False Negatives.