Probably the easiest way to avoid passwords is ssh keys. SSH keys are a public/private key system that can be used instead of passwords to authenticate yourself to remote ssh servers. SSH provides a number of nice systems to manage your keys. For example you can store them in memory using 'ssh-agent'. This way, you only need to enter the key passphrase once. Even better: Keep your private ssh keys on a USB stick, connect them to the PC only once to add them to your ssh-agent. Once this is done, the keys will stay protected in memory and you can disconnect the USB stick again.

Limit logins to ssh keys, whcih eliminates the problem of password brute forcing. SSH keys can be used to limit access by IP address, or you can limit a user to execute a specific command based on what ssh key they use (great for automatic backups).