Usually, these design decisions are not documented in any specification and can only be gleaned from working group discussion archives that happen to be publicly accessible, or explained by the spec authors themselves. However, in this particular case, HTML 3.2 does state the following:
Except within literal text (e.g. the PRE
element), HTML treats contiguous sequences of white space characters as being equivalent to a single space character (ASCII decimal 32). These rules allow authors considerable flexibility when editing the marked-up text directly. Note that future revisions to HTML may allow for the interpretation of the horizontal tab character (ASCII decimal 9) with respect to a tab rule defined by an associated style sheet.
The behavior you see today is of course much more complicated than what was specified in HTML 3.2, but I believe the reasoning still applies. One example of where this flexibility can be useful is when you have a long paragraph that you intend to hard-wrap and indent:
<H1>Lorem ipsum</H1>
<P>Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Fastidii oportere
consulatu no quo. Vix saepe labores an, pri illud mentitum et, ex suas quas
duo. Sit utinam volutpat ea, id vis cibo meis dolorum, eam docendi
accommodare voluptatibus no. Id quaeque electram vim, ut sed singulis
neglegentur, ne graece alterum has. Simul partiendo quaerendum et his.
If whitespace wasn't collapsed, you would end up with a paragraph with unusually large gaps where the text is hard-wrapped due to the indentation.
No other HTML specification suggests any sort of reasoning behind this design decision. In particular HTML 4 only describes the collapsing behavior, and HTML5 and the living spec both defer to CSS, which doesn't explain anything either. Earlier versions of HTML also do not contain any explanation, although the following excerpt does appear in an example snippet in HTML 2.0:
<OL>
...
<UL COMPACT>
...
<LI> Whitespace may be used to assist in reading the
HTML source.
</UL>
...
</OL>