Although hit the pool is correct; I really haven't heard anyone using that expression. People will say: I'm going to hit the gym (when they're talking about working out). When speaking in slang about swimming, most people would use a phrase like: I'm going to take a dip (you would only use "take a dip" of you're near a pool, however)
I disagree as 'take a dip' (which is definitely used) implies a relaxing, casual swim. 'Hit the pool' as with 'hit the gym' implies a more vigorous workout. You're right in that 'hit the gym' is the more common of the two expressions though.