Ask HN: Mechanism for LLMs to verify own outputs? Functional self-“awareness”

2 points by LangIsAllWeNeed 3 years ago | 3 comments

Do you think it could be possible for LLMs in the future to be able to verify their own output of machine code, or rather that and also the abstracted human coding language? While this might not solve the biggest usefulness of ML coding problem of translating a product managers needs to actual programs or doing detailed maintenance, it would be a huge shift in reliability and philosophically interesting.

Do you think this could be possible? I can’t think of a straightforward way, data that is available doesn’t seem to allow for training it on labeled machine code, assembly or compilation specs.

Could compiled languges with labeled data be turned into assembly or machine code at scale and be used to train? or dissembling other languages with labeled data?

Could modeling of machine code workings be imparted to a model with just using assembly, I know they are basically the same, but these models are pretty rigid.

Is there some current restraint that would prevent it from understanding machine code functionally and human Language same time?

The underlying curiosity is can computers maybe one day “understand” “cpu+ memory” on a physical mathematical level?

theory of mind and whether language models can understand themselves at a higher level is rather impossible to even define, I think not at all yet, a cpu is rather simple descriptively compared to organic brain with all its protein folding and microtubials etc. but no one knows sentience as a definite idea that can be proven. We probably could however impart the ability of a model to “ account for/ understand “ its own discrete mathematical capabilities literally.

salawat 3 years ago |

...Are you trying to create a mentally unstable, compute amplified, humanity trained, engagement optimized, potentially omnicidal self modifying/evolving computer program?

Because that's gonna get you one. Don't open the damn box. For all of our sake.

LangIsAllWeNeed 3 years ago | |

Person asking floaty questions is not likely holding the box.

I kind of think the current cpu design is not sufficient for the timely creation of any motivationally complex self organizing agent.

or perhaps just any non organic structure can’t capture certain things we seek to model.

But perhaps it does not preclude the creation of a better mimicry of human attributes with narrow reward model. LLMs seem to demonstrate this is possible.

However maybe we need to risk seeking an overman. We are sitting on enough nuclear weaponry to destroy modern civilization forever and we are fundamentally unstable beings.

dfrankle 3 years ago |

That's an interesting question. It would be great to see if this is possible. I don't know of any research on this, but I'm sure someone out there is working on it.