Why Prefer Short Hypotheses?

Occam's razor

• Occam's razor: is the problem-solving principle that the simplest solution tends to be the right one. When presented with competing hypotheses to solve a problem, one should select the solution with the fewest assumptions.
• Occam's razor: “Prefer the simplest hypothesis that fits the data”.

 Argument in favour of Occam’s razor:

• Fewer short hypotheses than long ones:
• Short hypotheses fits the training data which are less likely to be coincident.
• Longer hypotheses fits the training data might be coincident.
• Many complex hypotheses that fit the current training data but fail to generalize correctly to subsequent data.

Argument opposed:

• There are few small trees, and our priori chance of finding one consistent with an arbitrary set of data is therefore small. The difficulty here is that there are very many small sets of hypotheses that one can define but understood by fewer learner.
• The size of a hypothesis is determined by the representation used internally by the learner. Occam's razor will produce two different hypotheses from the same training examples when it is applied by two learners, both justifying their contradictory conclusions by Occam's razor. On this basis we might be tempted to reject Occam's razor altogether.