Well, I don't think there's a confirmed source, because of obvious reasons. Probably the same used on earth by us common folk. Maybe AES because of the deep government roots
Probably none. Why should telemetry data sent from Mars be encrypted? Encryption makes the system more complex and requires more computations and power on Mars.
I'm pretty sure telemetry is encrypted. Telemetry is not only position or velocity, there's a bunch of other information from sensors that should be handled with secrecy. Also gives more flexibility with announcements, given that the mission can fail anytime it's bad if some random person leaks that the rover stopped working suddenly or had a RAD