The posters interpretation of why Envoy is using downstream on listener side and upstream on backend side is wrong. There's a formal definition in RFC 2616:

  upstream/downstream
    Upstream and downstream describe the flow of a message: all messages flow from upstream to downstream.

Now HTTP is obviously bi-directional but since for most reverse proxy usecases responses far outweigh the requests upstream means "the backend"