6.034 Artificial Intelligence - Recitations, fall 2004 online slides on learning

Next: Training Examples Previous: Information Theory and Decision

Information Gain

$Gain(S,A)$ = expected reduction in entropy due to sorting on $A$

Look at decrease in information of correct answer after split.



\begin{displaymath}Gain(S,A) \equiv Entropy(S) - \sum_{v \in Values(A)} \frac{\vert S_{v}\vert}{\vert S\vert} Entropy(S_{v}) \end{displaymath}