We used to place an INT3 vectored exception handler on function entry points and do everything interesting inside the exception handler. This made the execution stack basically invisible to every debugger since it doesn't debug exception handlers. You can enable/disable interrupts and tracing and whatever you need to do inside the exception handler to guarantee that nobody can see what you are doing and/or verify that no other program has registered another exception handler before doing anything interesting.

If you need to hook functions in third party software, this trick can be used to hook the function without modifying any of the functions code. All you need to do is modify some pointer used by the function to zero, and it will raise an exception as soon as something like p-> is executed on that pointer, then your exception handler can execute whatever code you need (i.e. write over stack, write to memory, exfiltrate handles) and on exit all you need to do is restore the correct register containing the pointer and wind back the execution counter by the size of the de-reference instruction.

Please don't use this knowledge to hurt people ...

This "self-debugging" technique is common in the Windows world too:

http://profile.maff1t.com/AntiDebugging/

Interestingly enough, VirtualBox does this too, and they call it "hardening", but IMHO it's quite an unexpected and hostile behaviour which is more characteristic of malware.

Thankfully this is easy to circumvent: have your debugger catch the ptrace syscall and lie about the result. Also, if antivirus programs haven't already added a signature for any programs that do that, they should.

Windows also has many similar evasion techniques, like checking if there is a top level exception handler. I use scyllahide, but even on gdb you can break at ptrace and patch it or for automated analysis, just flag anything that used ptrace but isn't a debugger and run it in a sandbox without ptracing it. Audit subsystem might be enough.

https://github.com/x64dbg/ScyllaHide

If I remember it right, only 1 ptrace can be attached. So this seems easy to fix:. Just ptrace yourself, and nobody else will.

As a bonus, write important state to memory supposed to be read-only. If someone hooked your ptrace, the hook has to reimplement ptrace in a lot more detail. Or use breakpoints as a mechanism to call subroutines.

In an old $DAY_JOB I used a variant of this as a way to make certain internal errors execute a breakpoint when debugged, and generate an error log if not debugged. iirc this did not happen until after an error had already occurred, so it was not very useful as anti-RE.

I've seen this used preemptively - have the process ptrace itself on startup (and then do nothing with it) to make it impossible (or at least far-from-trivial) for other interested parties to ptrace it.

   int ptrace(int request, int pid, void *addr, void *data) {
       return 0;
   }

  gcc -shared myptrace.c -o myptrace.so

  LD_PRELOAD=./mytrace.so ./thebinary     # shell
  ltrace -S -l ./mytrace.so ./thebinary   # strace in shell

  set environment LD_PRELOAD=./mytrace.so

Some android malware also checks TracerPid in /proc/*/status https://github.com/Cloudef/android2gnulinux/blob/master/src/...

There are also differences in the handling of `SIGTRAP`. For a serious use, this can implement `breakpoint()` function.

Tricks for windows

https://anti-debug.checkpoint.com