The issue here isn’t that the chatbot doesn’t understand plain English but (1) it doesn’t really have facilities for executive control and (2) has a finite attention window and eventually your instructions will scroll outside the window and it gets stuck.

To accomplish what you’re trying to do you need to introduce an executive control layer that might be written in some ordinary programming language that manipulates the input to the chatbot so that it always supplies the same instructions at the beginning and keeps a limited context of the chat each time it runs that model, and the chatbot supplies the right outputs to the outer system, e.g. the chatbot instructs the framework to increment or decrement the score. It is like having the LLM write a Python program and run that to solve a math problem but here the Python program writes prompts for the LLM.