Right, so it would be necessary to force the NS records to point to registrar-owned nameservers which would be the point of enforcement. This is of course bad for a variety of reasons, not least how convention-violating it is.
Thanks for your comments - my goal with the writeup was to attempt to sound out why the idea of ".nonpublic" would or would not work, and I think it's clear that it's not really viable. That's a pity, as the current state of affairs isn't great. It was interesting to explore though.