Because we know our code is compiled with ASan, we chose to leak the offset between printf, a function that is always allowed, and mmap, a function that gives us arbitrary assembly code execution. We can thus bypass the school's checks and call any syscall, in our case execve to get shell access.

We did not investigate further and simply reported this possible issue to the school.

I see no trivial way of patching this.

The flaw seems to be in the submission system.

The submission system could build/relink your code to a trampoline library with the body of each function to be banned replaced with error reporting and abort. In your example, it would trap:

  char *code =
    notmmap.fn(NULL, 4096, PROT_READ | PROT_WRITE | PROT_EXEC,
               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

and prevent you from using the PROT_EXEC flag during the execution phase of submission validation.

According to AI, on linux: "... a seccomp-BPF filter applied to each process before it starts running the program. It can inspect the prot argument to mmap and return EPERM when PROT_EXEC is set."

Additionally, on linux: "systemd offers MemoryDenyWriteExecute=yes for services. That is less restrictive than banning every executable mapping: it targets writable+executable mappings and related ways of making memory executable. "

( IF ... i read the article correctly )

ps. Submit the proposed solutions to your department head or other authority figure who may throw you some sort of bone such that your status in life improves.

And then afterwards remind yourself that you are in school. We are all, always, in school. V