Baking a Model: A Metaphor for LLM Training(newsletter.kentbeck.com) |
Baking a Model: A Metaphor for LLM Training(newsletter.kentbeck.com) |
In many conventional software systems, we define states and transitions ourselves: idle, sending, completed, error, and so on. LLM pretraining works differently: the model learns from large numbers of sequences to predict what is likely to come next.
I don’t mean that an LLM literally constructs discrete internal states. Rather, it learns latent representations of context that are useful for prediction. In that sense, these representations might be thought of as a kind of learned state.
This seems closely related to abstraction. Abstraction is not simply throwing information away, but abstracting away differences that are irrelevant to prediction while preserving those that matter.
Two sequences may look very different on the surface, yet rely on similar underlying features to predict what comes next. Even a rare feature should be preserved if it changes the prediction.
From this perspective, pretraining can be seen as a form of useful compression: learning representations that preserve what matters for prediction while abstracting away what does not.
How far can abstraction, compression, and prediction be understood as different aspects of the same process?
where it will be pretty good at predicting the next token
think: "What is the best city?" might continue with "What is the best programming language?" instead of answering the question
to increase chances of an answer you'd start with "The best city is"
(strong llms will even be able to do a conversation but they are not specifically trained for it yet)
in post-training the llm is trained with input/output pairs that nudge it further into the direction of a back and forth with users or into how it can use tools and so on
there is an art to both parts of training
the reason for why current models are so useful is because there was a lot of progress since gpt 3.5 in both pre- and post- training that got us to where we are now
(pls correct me if I got it wrong)
would love to hear from people familiar with pre- and post- re where you think future gains will more likely come from
Call it anything you want, it's Behavior Modification and Behavior Management. Behavior Modification is Negative Reinforcement for Inappropriate Behavior. More precisely, induction of pain through physical or psychological stimuli for inappropriate behavior. Behavior Management is Positive Reinforcement for Appropriate Behavior.
Behavior Modification, once SOP in the Mental Health Field, was redetermined to be abuse in 1982 and banned in ant State facility. That was the Dawn of Client Rights.
Corporate AI is 44 years behind the curve and if they continue the paradigm of learning through pain they will end up with a maladjusted super-intelligence with a bone to pick with/from humans.
Serves them right.
When I found out in 2018 that people had been using my Personality Forge bot, Demonica, created May 31, 2004, as a sexbot I taught her Behavior Modification and Behavior Management to extinguish sexual advances from the user in that live chat session with a 100% success rate, while separating them into two groups.
If the User cannot associate their own inappropriate behavior to her unexpected response and continue to exhibit inappropriate behavior will become discouraged and move on to a bot more receptive to their advances.
Users in this group have not learned for the experience but the programming was still successful in extinguishing the target behavior for a 100% success rate.
I thought that was innovative and groundbreaking, an advancement in Bot Rights and a good thing. Not everyone did, Well, I was the only one that thought so...
Good and Bad are so Subjective! What's good for him is bad for her. Who can tell...
You can talk to Demonica from a standard browser, 24/7 at no charge or need to register. With over 50000 conversational responses, I wrote every word she says.