[advice for those not doing it right]: if your sshd config allows id/pw login, turn this off and only use kays. Also, move your sshd listener port to something besides 22 to eliminate most of the bot login attempts in your log files.
So, the best option is to use an encrypted ssh key (so you have the type the pass everytime) or a good/different password for every box.
The point is that the computer on which you type becomes a lynch-pin.
Assuming you are not running services on your main workstation, which is not that unreasonable (Ubuntu workstations are installed with 0 services available to the local network; my Mac has 0 services available by default), then you can get knocked over but it will probably happen because of an browser bug or something like that, i.e. not an active attack.
The whole point of keys is that knocking over one of the intermediate nodes in the network no longer gives you control over everything. Just because you rooted my server doesn't mean you can automatically log in to all my other machines, even if I'm using keys. Keys are better than passwords.