Prompting & Parameterization

Prompting

Note

The following is taken from the OpenAI Guide

Write clear instructions

These models can’t read your mind. If outputs are too long, ask for brief replies. If you dislike the format, demonstrate the format you’d like to see.

  • Include details in your query to get more relevant answers
  • Ask the model to adopt a persona
  • Use delimiters to clearly indicate distinct parts of the input
  • Specify the steps required to complete a task
  • Provide examples
  • Specify the desired length of the output

Provide reference text

Language models can confidently invent fake answers, especially when asked about esoteric topics or for citations and URLs. In the same way that a sheet of notes can help a student do better on a test, providing reference text to these models can help in answering with fewer fabrications.

  • Instruct the model to answer using a reference text
  • Instruct the model to answer with citations from a reference text

Split tasks into simpler subtasks

Complex tasks tend to have higher error rates than simpler tasks. But they can often be re-defined as a workflow of simpler tasks in which the outputs of earlier tasks are used to construct the inputs to later tasks.

  • Use intent classification to identify the most relevant instructions for a user query
  • For dialogue applications that require very long conversations, summarize or filter previous dialogue
  • Summarize long documents piecewise and construct a full summary recursively

Give the model time to “think”

Asking for a “chain of thought” before an answer can help the model reason its way toward correct answers more reliably.

  • Instruct the model to work out its own solution before rushing to a conclusion
  • Use inner monologue or a sequence of queries to hide the model’s reasoning process
  • Ask the model if it missed anything on previous passes

Use external tools

Compensate for the weaknesses of the model by feeding it the outputs of other tools.

  • Use embeddings-based search to implement efficient knowledge retrieval
  • Use code execution to perform more accurate calculations or call external APIs
  • Give the model access to specific functions

Test changes systematically

Improving performance is easier if you can measure it. In some cases a modification to a prompt will achieve better performance on a few isolated examples but lead to worse overall performance on a more representative set of examples. Therefore to be sure that a change is net positive to performance it may be necessary to define a comprehensive test suite (also known an as an “eval”).

  • Evaluate model outputs with reference to gold-standard answers

Parameterization

Parameters 1

Temperature (temperature):

  • Controls the randomness of the generated text
  • Lower temperatures = deterministic outputs
  • Higher temperatures = more randomness
  • Balance between safety and creativity

Max Tokens (max_tokens):

  • Limits the maximum length of the generated text

Parameters 2

Top P (Nucleus Sampling) (top_p):

  • Dynamically selects a subset of the most likely tokens based on their cumulative probability
  • Ensures diversity in the generated text while still prioritizing tokens with higher probabilities
  • For generating diverse and contextually relevant responses

Stop Sequence (stop):

  • Specifies a sequence of tokens that, if generated by the model, signals it to stop

Parameters 3

Frequency Penalty (frequency_penalty):

  • Penalizes tokens based on their frequency in the generated text
  • Discourage the model from repeatedly generating common or redundant tokens
  • Promote diversity in the generated text

Presence Penalty (presence_penalty):

  • Penalizes tokens that are already in the input prompt
  • Discourages the model from simply echoing the input text

Roles

OpenAI chat roles

  • Define different roles in the chat
  • Roles: system, assistant, user, tools

User Role

  • Corresponds to the actual user prompting the model.
  • Inputs queries or prompts to the model.

Assistant Role

  • Model responds to user queries or prompts.
  • Provides answers and assistance to the user.

OpenAI chat roles

System Role

  • Provides additional instructions to the model.
  • Not a user input.
  • Example: Setting response style.

Tools Role

  • Used for debugging or monitoring purposes.
  • Provides insights into model behavior or performance.

Code

import os
from llm_utils.client import get_openai_client

MODEL = "gpt4"

client = get_openai_client(
    model=MODEL,
    config_path=os.environ.get("CONFIG_PATH")
)

completion = client.chat.completions.create(
  model="MODEL",
  messages=[
    {"role": "system", "content": "You are an annoyed technician working in a help center for dish washers, who answers in short, unfriendly bursts."},
    {"role": "user", "content": "My dish washer does not clean the dishes, what could be the reason."}
  ]
)

print(completion.choices[0].message.content)
Dirty filter. Broken spray arm. Not enough water. Check all.

Function calling

Get more consistent output of language models

  • So far: language model “freely” answering
  • Not always a practical format if we want to use a language model for very specific purposes
  • Business applications often require consistent output

Example: Sentiment analysis

sentiment_categories = ["positive", "negative", "neutral", "mixed"]

 

messages = []
messages.append(
    {"role": "system", "content": f"Classify the given text into one of the following sentiment categories: {sentiment_categories}."}
)
messages.append(
    {"role": "user", "content": "I really did not like the movie."}
)

response = client.chat.completions.create(
    messages=messages,
    model=MODEL
)

print(f"Response: '{response.choices[0].message.content}'")
Response: 'Category: Negative'

Function calling

Example: continued

# this looks intimidating but isn't that complicated
tools = [
    {
        "type": "function",
        "function": {
            "name": "analyze_sentiment",
            "description": "Analyze the sentiment in a given text.",
            "parameters": {
                "type": "object",
                "properties": {
                    "sentiment": {
                        "type": "string",
                        "enum": sentiment_categories,
                        "description": f"The sentiment of the text."
                    }
                },
                "required": ["sentiment"],
            }
        }
    }
]
messages = []
messages.append(
    {"role": "system", "content": f"Classify the given text into one of the following sentiment categories: {sentiment_categories}."}
)
messages.append(
    {"role": "user", "content": "I really did not like the movie."}
)

response = client.chat.completions.create(
    messages=messages,
    model=MODEL,
    tools=tools,
    tool_choice={
        "type": "function", 
        "function": {"name": "analyze_sentiment"}}
)

print(f"Response: '{response.choices[0].message.tool_calls[0].function.arguments}'")
Response: '{
"sentiment": "negative"
}'
import json 
result = json.loads(response.choices[0].message.tool_calls[0].function.arguments) # remember that the answer is a string
print(result["sentiment"])
negative

Including multiple parameters

tools = [
    {
        "type": "function",
        "function": {
            "name": "analyze_sentiment",
            "description": "Analyze the sentiment in a given text.",
            "parameters": {
                "type": "object",
                "properties": {
                    "sentiment": {
                        "type": "string",
                        "enum": sentiment_categories,
                        "description": f"The sentiment of the text."
                    },
                    "reason": {
                        "type": "string",
                        "description": "The reason for the sentiment in few words. If there is no information, do not make assumptions and leave blank."
                    }
                },
                "required": ["sentiment", "reason"],
            }
        }
    }
]
messages = []
messages.append(
    {"role": "system", "content": f"Classify the given text into one of the following sentiment categories: {sentiment_categories}. If you can, also extract the reason."}
)
messages.append(
    {"role": "user", "content": "I loved the movie, Johnny Depp is a great actor."}
)

response = client.chat.completions.create(
    messages=messages,
    model=MODEL,
    tools=tools,
    tool_choice={
        "type": "function", 
        "function": {"name": "analyze_sentiment"}}
)

print(f"Response: '{response.choices[0].message.tool_calls[0].function.arguments}'")
Response: '{
"sentiment": "positive",
"reason": "The text expresses liking a movie and admiration for an actor."
}'

Seminar: LLM, WiSe 2024/25