I don't see why this is occuring.
But once the call goes to speaker, don't press the speaker button, otherwise the call will disconnect.
After it goes to speaker and you want the call to go back to the handset, press the hook key where the handset is normally parked (don't press the speaker button), the call will then route to the handset.