I hate reply to my own posts, but:
I opened a TAC case on this - it seems that for updates that are already in your CSM install directory ( \CSCOpx\MDC\IPS\updates ) the "download latest updates" button is only checking MD5 sums and not re-downloading the files.
Once I ran the Download Latest Updates via the CSM Client while Remote Desktop'd to the CSM Server, it finished in about 20 minutes. I think this is because (at least in my update directory) there are 2Gb worth of files, and the CSM client needs to bring them down to your machine to compute the hash. If you're on the server the client has 'local' access to them, thus computing the hashs takes far less time.
So if you're having the same problem, just run the job