Here's the key point -- there can be more types of packets than INTERNET traffic. You could be using IPX, which is non-routable. How do clients identify each other? By the MAC address.
Routing != Addressing, which is really where the MAC comes into play.
In order to be routed, the OSI model adds a layer to allow for path discovery to the next gateway. This layer is responsible for routing, but knows nothing about the MAC address.
As a side note, at the hardware level, MAC addresses ARE used by switches, but not for routing. From How Stuff Works:
The switch gets the first packet of data from Node A. It reads the MAC
address and saves it to the lookup table for Segment A. The switch now
knows where to find Node A anytime a packet is addressed to it. This
process is called learning.
In this way, a switch can make sure that traffic is only outputted to the correct port. This isn't accomplishing routing so much as reducing network congestion. Only broadcasts and traffic destined specifically for that MAC address should be sent out the port.