A decision tree is a tree-like structure where each internal node represents a test on a specific attribute, each branch represents the outcome of the test or decision, and each leaf node represents a class label or a numerical value.
Here is a basic decision tree showing how it might classify whether a patient has low or high heart attack risk according to different attributes.
There are many types of decision tree algorithms but we will be focusing on two of the most prominent ones:
1. ID3 (Iterative Dichotomiser 3):
ID3 was one of the earliest decision tree algorithms developed by Ross Quinlan. It uses entropy and information gain to determine the best split at each node.
2. CART (Classification and Regression Trees):
CART is a versatile decision tree algorithm introduced by Leo Breiman. It can be used for both classification and regression tasks. For classification, it uses Gini impurity, and for regression, it uses mean squared error to determine the best splits.