I had not considered using a VM instead of strace when a program can detect ptrace(2) being used - good idea.

> Normally when reverse engineering a program, it is common to use tracing programs like strace. These tracing programs are quite useful, but they suffer from a design flaw: they use ptrace(2) to accomplish the tracing, which can be detected by the program being traced.

One way to do this would be to call ptrace() from your program and check if it returns the error EPERM. From the man page:

       EPERM  The specified process cannot be traced.  This could be because the tracer has  insufficient  privileges
              (the  required  capability  is CAP_SYS_PTRACE); unprivileged processes cannot trace processes that they
              cannot send signals to or those running set-user-ID/set-group-ID programs, for obvious reasons.  Alter‐
              natively, the process may already be being traced, or (on kernels before 2.6.26) be init(1) (PID 1).

However, this is not the best solution, as if your system has a security policy already in place for ptrace() detection, your process might get detected and killed. Other methods from the calling process might involve timing mechanisms, breakpoint detection, or checking other factors in the process' environment. One problem with the workaround suggested in this post (running a process from qemu-user) is that if it is truly security hardened, it might rely on timing differences smaller than the speed of VM instruction execution.

As a user or sysadmin, one way to detect ptrace is to use Yama [1], a Linux kernel module that creates an entry in /proc/sys/kernel/yama/ptrace_scope to configure a user's desired level of ptrace protection, from 0 (normal - any process can call ptrace() on another process owned by the same user) to 3 (completely disabling ptrace).

[1] https://www.kernel.org/doc/html/latest/admin-guide/LSM/Yama....

Prior thread (with comments): https://news.ycombinator.com/item?id=27046272