Models
Every autogram has a classifier, chatbot, and userbot. The chatbot executes the instruction of thought and chat nodes, the classifier predicts transitions, and the userbot simulates the user. Right now the main supported models are OpenAI API and huggingface models. We plan to add more model APIs though.
Chatbot
The Chatbot is the model that generates the output text of chat and thought nodes, corresponding to chain of thought and replies to the user. The chatbot is assumed to be a chat completion style model. As arguments, it takes a list in inputs (treated as user side inputs) and outputs (treated as previous model outputs). Depending on the nodes used, some of the inputs may be instructions or combinations of instructions and user replies, instead of just user replies. The chatbot is implemented as an abstract class, and is overridden by either an OpenAI Chatbot class or a Huggingface Chatbot class, depending on the arguments in the Agent Config. All Chatbots must implement a generate_reply
method to get the output, and a truncate_input
method to shorten the inputs if they are too long.
The chatbot's generate reply method should also be able to handle errors/retries if an API call fails, and also have a special mechanism for regenerating with a new logit bias when banned_phrases
set in the AgentConfig are generated by the model. This most commonly comes u when the model refused to reply (for instance, OpenAI API occasionally replies "As an AI language model, can't do that" etc., even for feasible and perfectly benign instructions. This especially comes up when simulating the user side of the conversation.)
Classifier
The classifier model is a language model that predicts the answers to transition questions, and is used to predict transitions. The classifier model only needs to generate a single token, and uses a large positive logit bias on the allowable answer choices (which are always multiple choice A-Z, or yes/no) to ensure the model generates a valid answer. Without a logit bias, the model would likely sometimes generate things like "The answer is C." instead of just "C", which would make determining the predicted answer more difficult. Unlike the chatbot, which recieves previous inputs and output turns, the conversation history to the classifier is all concatenated to a single turn.
The classifier is implemented as an abstract class, and is overridden by either an OpenAI Classifier class or a Huggingface Classifier class, depending on the arguments in the Agent Config. If the chatbot and classifier are both huggingface models with the same path, the underlying language model is shared between the classifier and chatbot to avoid wasting GPU memory. All classifiers must implement a predict_class
method to predict the answer to the multiple choice question, and a truncate_input
method to shorten the prompt if it is too long.
Userbot
The userbot of an autogram is by default set to be the same object as the chatbot. The prompts to the userbot are what distinguish its behavior, since the instructions will typically be along the lines of "reply as the user and say x"