If you're using the password command on the line, you won't be able to do anything other than service password encryption. If you need md5, your best bet is to use local accounts and aaa authentication:

new aaa-model

aaa authentication local default local

username routertest priv 15 secret testpassword

line vty 0 4

login authentication local

This will allow you to have md5 encrypted passwords, and you won't have to do anything special on the line (other than what I put up there).

Anytime you're doing aaa configuration, be sure to be logged in one window and test in another. It's very possible to lock yourself out of a router.

As far as configuring the password command as md5 directly on the line, I don't believe it's possible.

You can also try:

Router(config)# key config-key password-encrypt testpassword

Router(config)# password aes

The above encrypts isakmp passwords, but I can't remember if it affects all of the passwords that match whatever your "password-encrypt" line is.

Here's a doc:

http://www.cisco.com/en/US/tech/tk583/tk372/technologies_configuration_example09186a00801f2336.shtml

HTH,

John