In terms of compression and information theory, the entropy of a source is the average amount of information (in bits) that symbols from the source can convey. The diagram on the right predicts image of a horse with a relatively high confidence (lower entropy) while the classifier on the left can not really distinguish (higher entropy) whether it's a Horse, a Cow, or a Giraffe. The diagrams show a comparison of entropy values of predictions from two classifier models. Here the entropy is used as a measure of how confident the classifier model is in its prediction. I made this visualization to describe relationship between entropy and confidence of the predicted class in an animal image classifier model (machine learning). This average amount of surprise could also be used as a measure for how uncertain we are. Lets assume that we have a function called Surprise(x) that would give us the amount of surprise for each outcome then we can average the amount of surprise on a probability distribution. On the other hand, it won't be too surprising if we got a head as we already have a 99 percent chance of getting a head. Since there is only a one percent chance of getting a tail, we would be very surprised if we actually get a tail. Lets say we have a bent coin that gives us a head 99% of the time and a tail 1% of the time.

We can also describe entropy as how surprised we would be if we get an outcome after we made our initial prediction. Here is a great alternate explanation for entropy in information theory.Įntropy is a measure of uncertainty involved in making a My favorite definition, with a more practical focus, is found in Chapter 1 of the excellent book The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas: That means the more random the text is, the lesser you can compress it. The more the entropy, the lesser the compression ratio. Entropy in data compression may denote the randomness of the data that you are inputing to the compression algorithm.

Entropy formula