Show HN: Compile for Arm at native speeds in an emulated system

pm215 4 years ago |

This is the same trick used by the old 'scratchbox' development environment that Nokia used for the maemo SDK. If the stuff you compile in it includes classic "configure and make" C/C++ projects then you'll probably also find that replacing the /bin/sh binary with an x86 one improves speeds -- configure and make spend quite a lot of time spawning shells to run bits of shell script.

valkmit 4 years ago | |

Hey, thanks for the insight - I didn’t realize this was a thing for some Nokia envs already - pretty cool!

re: x86 make and shell, it probably won’t work well here because once a process tree goes to x86 land, qemu cannot see it and it can’t “natively” run an arm binary.

Essentially we can only hook execve when emulating - when we run a native binary we no longer have any insight into any execve call it may make. (Though with some additional work I suppose one could also use LD_PRELOAD to hook execve)

This is why binfmt_misc and registering the interpreter with the kernel would be superior, if not for kernel level access requirement

pm215 4 years ago | | |

QEMU doesn't hook execve anyway, at least not upstream -- when a QEMU emulated process invokes another we rely on the binfmt-misc magic to cause the kernel to run QEMU on the child. We don't special case "the binary being exec'd seems to also be for the guest".

It looks like maybe you've made some local patches to QEMU that relate to execve?

bsdimp 4 years ago | | |

I know the qemu-bsd-user fork has some hacks for the bsd-user stuff that looks at the binary to be exec'd and does different things based on the target since there's some optimizations you can get out of this if you don't rely on the kernel 100%... Bits of them may even be in my latest patch-set :)

valkmit 4 years ago | | |

Cool stuff! Do you have a public branch you're working off of? I'd love to follow the progress as I wasn't aware that qemu user emulation for BSD was even a thing.

I didn't get a chance to peruse beyond briefly skimming the source on GH, but I couldn't find any special handling around execve - could you point me in the right direction?

I was looking at bsd-user/syscall.c, whose equivalent in linux-user is where my repo's execve hook lives, if that makes any difference.

valkmit 4 years ago | | |

Yep! I have a patch to qemu based on one that never made it to master upstream. It’s a sub module in the repo

pabs3 4 years ago |

I like how cross-compiling works on Debian, you install the native cross-compiler on your amd64 system, auto-install the cross-dependencies built natively on arm64 systems onto your amd64 system and then build. Of course, not all dependencies are cross-installable and not all packages are cross-buildable, which is why there is a QA effort for automatically doing test builds.

https://wiki.debian.org/CrossCompiling http://crossqa.debian.net/

Ecco 4 years ago |

This should be at the top: https://github.com/valkmit/aws-graviton2-on-intel#justificat...

valkmit 4 years ago | |

Thanks, I’ll add it now

parsecs 4 years ago | | |

I think they are suggesting to move the justification section of the github page to the top of the github readme. (Instead of posting another comment on HN)

valkmit 4 years ago |

Justification:

https://github.com/valkmit/aws-graviton2-on-intel#justificat...

bsdimp 4 years ago |

FreeBSD has used this same trick for half a dozen years or so to build arm, mips and riscv binaries using fast Intel servers. The FreeBSD build system can build itself to produce binaries for the target (so the compilers run natively) and qemu handles the rest with user-mode emulation via bsd-user (which I happen to be in the process of upstreaming).

gizdan 4 years ago |

Any chance you can add a high level explanation on how this works?