Use rsync. It now preserves the buffer cache status that files had before so it does not stomp on your allocations.

http://insights.oetiker.ch/linux/fadvise/

I suspect the reason to be that the system has noticed that your apps haven't touched their memory for a long time and thus "don't need it". This is a reasonably valid assumption on servers: if you've got some daemons that are backgrounded for minutes or hours at a time, keeping their memory resident is a waste. However, on the desktop responsiveness (latency) is more important than throughput. Just because you only switch between apps on a timescale of minutes or hours, doesn't mean the kernel should swap them out. So the algorithm needs different weighting.

I used to experience this problem, but I haven't lately. I suspect what's going on is that the "-desktop" kernel variant of OpenSUSE uses a differently weighted swappiness algorithm. If your distro offers a choice of different kernel variants, you could try them; otherwise (or if that doesn't help), you could track down the knobs you need to tweak to make the problem go away.

I guess that the kernel cannot know that the application (in your case scp) will not try to touch the data ever again.

The current default behavior, which you call crazy, does however favor programs which do actually need the data that was just read (e.g., databases).

Would mmapping those files instead of reading them provide saner behavior, or does the kernel still do that?

I'm guessing you're either running a rather old distribution/kernel, or your swappiness is set far too aggressively. Try setting it to 0.

Thanks! I am glad you learned something, and happy to get great feedback!

They have not changed in any way, however there is a patchset proposed currently to change how NUMA works a bit. Unclear if it will change this situation.

Pretty hard to make you happy, eh?

I agree with you on a purely technical basis, however, this article was written for the MySQL community, was tested (only) on MySQL, and has primarily affected my only on MySQL systems, which I (and the others referenced in the article) primarily run on Linux.

Thats a very good point, these types of problems exist to some extent on other OSes but this case seems pretty specific to Linux. I suspect testing on Solaris/FreeBSD would show better results in this area.

This applies to any case where you want a single process to use more than half the server's RAM.

Yes, absolutely. In fact one of the most common longer term referrers for that post is about MongoDB, not MySQL:

http://www.mongodb.org/display/DOCS/NUMA

Except that's not the TL;DR at all.

From the article:

"An aside on zone_reclaim_mode

The zone_reclaim_mode tunable in /proc/sys/vm can be used to fine-tune memory reclamation policies in a NUMA system. Subject to some clarifications from the linux-mm mailing list, it doesn’t seem to help in this case."

The real TL;DR is "run your mysql command under the auspices of '/usr/bin/numactl --interleave all' so that your big pool allocation is split evenly across nodes"

And an even better solution would be if _only_ the big pool allocation use interleaved allocation, and all the rest used normal node-bound allocation. This would require some sort of change to the malloc calls though, yes? All of the solutions listed in the article operate at the granularity of a process (or higher), not down to the individual allocation.

Even beyond that: it's desirable for a machine to be able to compute arbitrarily large data sets. If the data set can't efficiently fit in memory, the machine should still make progress, just more slowly, using disk.

It is not desirable for a machine to have a "wall" which, upon being hit, becomes a harsh restriction on its capabilities. This is because we often encounter the "wall" unexpectedly, at a time that might be critical.

I like the erlang technique of just dying when something bad happens. That is, if you write your app right then you can afford to just die at any point (without dropping any data on the floor at all). I agree with the grandparent--I've started to disable swap on my production servers because I reason that if I run out of ram then I haven't I configured something correctly. A real server shouldn't ever swap--heavy swapping grinds the whole world to a halt which means requests are being serviced way too slowly, if at all...

An application that is aware of the half dozen of so caching layers from register to platter can perform dramatically better than a naive program. Two wrinkles:

1) it needs to either be told the various sizes, speeds, and quirks on each server to make best use. (just some work)

2) it needs to coordinate with the other processes running on the system to divide up the resources. This is hard. Generally people bail and just assign some share of RAM and hope for the best with the other layers.

I agree of course, but I think the current situation is worse. Applications can allocate more than the amount of physical memory, at which point the system can become unusable. If on the other hand an allocation would have been refused at an earlier stage, there would have been no critical data to drop.

I guess I want to argue that with the currently typical amounts of RAM, all critical data should fit in RAM and stay there. The idea of virtual memory was to abstract over the difference between RAM and disk, but perhaps this has become a harmful abstraction now that RAM is big enough while the disadvantage of slow disks remains. RAM and disks are fundamentally different parts of the memory hierarchy, and should be treated completely differently by applications.

This is what I suspect, but in practice it seems not a good idea to just turn off swap:

> DO NOT TURN OFF SWAP to prevent this. Your box will crawl, kswapd will chew up a lot of the processor, Linux needs swap enabled, lets just hope its not used.

(from one of the blogs linked in the article).

However, I can't find a clear explanation of why this is so.

Yes, they are two distinct issues which are somewhat conflated: the actual "swap insanity" and swap in general, and NUMA performance effects. I think for my purposes, the negative effects of NUMA are relatively hidden, so we only see the side effects. That is, it's impossible to know at this point whether optimization for NUMA would help MySQL, since no one has made a serious effort to do it.

I suspect it would help quite a bit, if done right, and for the right query workload.

The two may be less distinct in the face of some approaches to minimizing remote reads/writes, e.g., replication of pages across domains. Apart from ensuring the cost of such replication is warranted, the dominant pressure against replication is the effect it has in increasing the memory footprint of the application, and thus possibly triggering more page faults.