If you're running a server, then ServerSocket is the proper solution. It will manage everything better than you doing it by hand through recycling and a host of other optimizations intended for running a server with Java.
Closing the socket disconnects the Java object from the operating system, which means that it isn't taking up any resources outside of the JVM, so it really shouldn't be a problem. But if the minimal overhead from Java's garbage collection/finalization scheme is too big of a burden, then Java isn't a valid solution (since your problem isn't specific to socket programming any more). Although I have to say that an efficient garbage collector is not much worse than explicitly managing memory (and can actually perform better).