S0F1 Figure 1: Joint probability P(X,H) where X ranges over 5 discrete values and H over two values. Each entry contains the number of hits for the cell (xi,hj). The joint probability P(xi,hj) is the number of hits divided by the total number of hits (22). See text for more details. In Fig. S0F1 we have P(h2 | x3)=3/7. Note that ∑jP(hj | xi)=∑jP(xi,hj)P(xi)=1P(xi)∑jP(xi,hj)=P(xi)/P(xi)=1. Likewise, the conditional probability P(xi | hj)=P(xi,hj)/P(hj) is the number of hits in cell (i,j) normalized by the number of hits in the j’th row and represents the probability of receiving BMI=xi given the class label H=hj (over-weight or not) of the person. In Fig. S0F1 we have P(x3 | h2)=3/8 which is the probability of receiving BMI=x3 given that the person is known to be of normal weight. Note that ∑iP(xi | hj)=1. If we draw the posteriors P(h1 |x) and P(h2 | x) using the probability distribution array in Fig. S0F1 we will see that P(h1 |x)>P(h2 | x) for all values of X smaller than a value which is in between x3 and x4. In Fig. S0F1 we have that P(h1)=14/22,P(h2)=8/22 that is there is a higher prior probability of a person being over-weight than being of normal weight. Also P(x3)=7/22 is the highest meaning that we encounter BMI=x3 with the highest probability. The RE measure is not a distance metric as it is not symmetric, D(\bf x||\bf y)≠D(\bf y||\bf x), and does not satisfy the triangle inequality. Nevertheless, it has several interesting properties which make it a fundamental measure in statistical inference.

Introduction to Machine Learning: Class Notes 67577

Reader Guidelines

Editor Guidelines