I retract my previous statement about bad ICMP code (and other comments where I posted it). I was under the impression that mtr was actually doing ICMP echo requests to individual hops with decreasing TTLs, but it's just relying on the TTL being generated for the end to end echo request. However, this is just still a terrible indicator for packet loss, for example by wifi router heavily deprioritises generating TTL exceeded packets but will respond to a flood of echo requests no issue. My main contention is the per hop loss indicator is a useless and misleading metric and you should be measuring these things end to end with traceroute and ping separately.

EDIT: Re-Reading. I think I am some degree of a networker underestimating network complexity. I'll stand by that. Please make fun of me for only speaking in IPs and Ports.

Yeh. There is a very achievable level of knowledge about networking that's enough to make a lot of practical problems solvable.

Like, my practically acquired patchwork of knowledge about subnets, routing, some DNS, some VPN tech, maybe some ideas of masquerading and NAT'ing is easily enough to run a multi-site production environment across a number of networking stacks. And I wouldn't really call these things hard. I don't like people who are like "I don't know networking" once you say "routing table". The hardest part there is to understand how things are often a very large amount of very local decisions and a bunch of crossed fingers to get a packet from A to B. Oh an no one thinks about return paths until they run a site to site VPN.

But just a few steps beyond that is a cliff dropping into a terrifying abyss of complexity. LIke I know acronyms like BGP, CGNAT, ideas like Anycast DNS and kinda what they do, but it turns into very dark and different magik rather quickly. I say if we need that, we need a networker.

> I'm 100% sure the only reason so many programmers know how NAT works is because NAT breaks video games.

... and filesharing, from the days when bittorrent was huuuuge.

The way you write it makes it seems like you're blaming the tool for misleading results when that's the nature of traceroute itself.

MPLS don't have to hide routers though, up to the operator, even if they do it will give you idea of where things went wrong and you can contact the correct people. Load balancing links is either lacp or ecmp, first case doesn't really matter and in the second you'll just see multiple responses on a hop. Neither really had any impact on how useful traceroute is and doesn't really mislead.

It is possible to to reveal the impacts of asymmetric routing through other tools, for instance ThousandEyes can do this by performing a time synchronized bidirectional trace (among other things it can do that MTR cannot). This can be very valuable.

That said, in practice for the majority of end users, they will not be directly impacted by asymmetric routing, if only because so many services are now cloud-based and the major cloud devices are direct peered with all of the major ISPs at regional meeting points in most countries. As an example, on my connection in Denver on Comcast, going to most applications in AWS will enter the AWS network /in Denver/ and without traversing any transit provider, meaning effectively my traffic never goes across "the Internet", it goes from Comcast (my provider) directly to AWS (the provider for the application).

While it's always good to be mindful of the complexities of real-world routing, for the vast majority of common use cases now, entry-points to the target application are so widely distributed that the most impactful routing is inside the private network of the cloud provider, not across the larger Internet.

Disclaimer: Opinions are my own.

Which is why any network engineer worth their salt with ask for a trace in both directions (if available). Asymmetric routing can be an issue especially when going through stateful devices like firewalls.

They are only misleading if you allow yourself to be misled by them. It's an extremely informative measurement if you are aware of how it works and don't misinterpret the results.

I tried mtr (I usually use ping and traceroute). 87% packet loss at my router and the next hop, then 0% loss. WTF I say to myself and uninstall.

It's one of the best tools to troubleshoot packetloss on the internet and generally routed networks. It gives you way more information than ping or traceroute could potentially give.

If you run it in TCP or UDP mode you can even nail down the physical interface that's erroring in a LAG/LACP bundle due to being able to manipulate the 5 tuples very well.

I'm also curious about the flags you used for ping and mtr that showed you this discrapancy.

Yes, you can. It's basically a terminal case of something being unintuitive. Whether something is misleading is in the eye of the beholder.

Recently my mother felt misled by a car commercial. Her position was that saying things like "under this many years or that many miles" is misleading, because it suggests that it's a set of options she can pick from (which of course ended up not being the case).

Unfortunately for her, this is a natural language construct - whether she understands it correctly or not depends on how aligned her common sense regarding it is with people at large. She understood it differently and thus felt misled. But you may notice that ultimately it was her own mistaken understanding of the common parlance that misled her. So when she said this was misleading the only thing I could reasonably say was exactly this. That I did not find the phrasing misleading, and I'm sorry she'd been misled by it (irrespective of whether that was on her or on the world, as that doesn't really matter).

It's completely on people how they want to handle this. You can find people being misled by stuff like this to be unreasonable and just tell them so, or you can put out a disclaimer regardless. Depends completely per case. This goes all the way to having multiple mechanical interlocks at places with heavy duty xray sources, or preferring machine checked memory management.

> Right, so control-plane packet rates are rate limited (to some definition of sane), but they are applied to all applications, traceroutes, pings alike.

I'm not sure I understand what you're saying, but in this case control-plane packet rates are different for generating TTL exceeded vs Echo Response, where one is giving 80% loss and the other is giving 0% loss at similar rates. Gripe #1 why are we even testing control plane in the first place, it's a useless metric that doesn't have utility at measuring end to end latency/loss.

> An argument could be made for a device configured as such to show loss on ping but not on mtr if you configure the rate limits so that the icmp reply rate is lower than ttl expired rates. Which tool would be wrong than? Would you blame ping for producing misleading results?

Sure that would be a problem, but any combination could be misleading if the data path is yielding 0% loss for high rates of ICMP end to end. This is why it's not a very particularly helpful metric and can be downright misleading (usually not to me, but I've seen plenty people make incorrect inferences from bunk MTR results because the tool isn't intuitive).

> The running counters and the ability to pick out the obvious rate limiting when the loss doesn't cascade into the hops to me is akin to traceroutes * * * output. It doesn't always mean that the packets are blackholed, connectivity is broken, it just means the tool is producing an artifact due to network configuration or network characteristics. Further investigation is needed to figure out what's going on.

Sure that's great, not particularly helpful to the masses who misunderstand the tool. I worked as a network engineer for a decade receiving bunk MTR reports where people freak out because they're seeing "packet loss" which was inexistent on the data forwarding plane (you know the one that actually matters).

> MTR imho is giving you much more insight into the network than traceroute or ping separately. It doesn't resolve the usual firewall/rate limiting artifacts, but gives you way more information about paths if you know how to interpret them.

Time shouldn't be wasted measuring the control path and then investigating to confirm it is the control path and not data path. You cannot make these mistakes using traceroute and ping separately because traceroute doesn't have a notion of a "per-hop" loss indicator and ping doesn't involve intermediate hops (unless an intermediate hop generates an ICMP diagnostic for an echo request).