LLM Parameters – Which Ones Exist and How to Use Them

What are LLMs anyway?

Large Language Models (LLMs) are advanced AI systems that are trained on huge amounts of data and complex algorithms to understand and generate human language. They are based on neural networks and contain billions of parameters that enable them to recognize patterns in text and deliver natural, context-aware answers. One example of such models is GPT (Generative Pre-trained Transformer), which can be used for a wide range of tasks – from answering questions and creating content to supporting creative processes.

Why are LLM parameters useful when working with LLMs?

LLM parameters are crucial because they enable the customization and control of an LLM's responses. They determine how creative, precise, or extensive the generated answers are and can thus help optimize the model's performance for specific applications. This means that with the right combination and fine-tuning of these parameters, you can precisely steer the model's responses and maximize the quality of the results in different contexts.

Which LLM parameters exist?

There are many different parameters, which can be divided into several categories.
Below is an overview of some parameters used by OpenAI, Anthropic, and Google, organized by category:

1. Creativity and randomness

Temperature

The temperature parameter influences the creativity and randomness of the responses by changing the probability with which certain words are selected. In general, it controls how "bold" the model is when choosing words:

Low (e.g. temperature=0.2)

Behavior: The model stays conservative and picks the most likely words. The result is precise and predictable, with a low degree of creativity.

Example "Describe a sunny day in the park": The sky is clear, and the sun is shining brightly. People are enjoying the nice weather, some are sitting on benches, others are taking a walk. It is calm and peaceful.

Medium (e.g. temperature=0.7)

Behavior: At a medium setting, the word selection becomes slightly more random, so the model offers a mix of precision and creativity.

Example "Describe a sunny day in the park": The sun shines warmly on the green meadows, and children laugh and play on the playground. Birds chirp in the trees while a light breeze rustles through the leaves. People lie on picnic blankets and enjoy the beautiful day.

High (e.g. temperature=1.0)

Behavior: A high temperature significantly increases randomness and creativity. The model also picks less likely words, which can lead to more creative but potentially unpredictable and illogical results.

Example "Describe a sunny day in the park": The sun dances on the waves of the small pond while colorful kites float in the sky. Children chase butterflies across the meadows, and a dog with a red bandana excitedly jumps after a ball. Laughter mingles with the sound of a guitar someone is playing under a tree.

Top K and Top P

Top K and Top P are two more parameters used to control randomness:

Top K:

Top K specifies that when selecting the next token, the model can only choose from the K most likely words.

For example, if top_k=10, the model only considers the 10 most likely words and ignores all others.
A low value (e.g. top_k=5) strongly limits the token selection and ensures precise answers, as the model chooses from a small group of common words.
A higher Top K value lets the model respond more creatively, as it can choose from a larger group of possible words.

Top P (nucleus sampling):

Top P defines a probability threshold that the model considers when selecting the next tokens. Instead of choosing a fixed number of words (as with Top K), the model looks at the most likely words until their combined probability reaches the Top P threshold.

With top_p=0.7, for example, the model will only choose from the most likely words whose cumulative probability is at least 70%, ignoring all others.
With a low Top P value (e.g. top_p=0.5) the model stays very "safe" and picks more common words, while a higher value (e.g. top_p=0.9) gives the model more creative freedom by allowing a larger selection of words, while still being governed by the probability threshold.

2. Controlling the output

Max tokens

The max tokens parameter defines the maximum number of tokens the model is allowed to generate. A token can be part of a word, a whole word, or a punctuation mark; the number of tokens therefore depends on how complex the words and sentences are.

If max tokens is set to 100, for example (max_tokens=100), the model delivers a response of at most 100 tokens.

Stop sequence

A stop sequence defines exact character strings at which the model automatically stops generating text. This enables precise control of the output.

Suppose the stop sequence is set to stop=["User:"]. As soon as the model generates this string in the output, generation is stopped.

3. Influencing word choice

Frequency penalty

This parameter controls how strongly the model penalizes the repetition of words in the generated response. A higher frequency penalty value reduces the probability of words being used multiple times, which is particularly helpful for making texts sound more varied and natural.

With a frequency penalty value of 0.7 (frequency_penalty=0.7), for example, the model is encouraged to repeat itself less.
Without penalty: "The cat is a cat that is often outside. It is a beautiful cat."
With penalty: "The cat is beautiful and loves being outside."

Presence penalty

The presence penalty parameter prevents word repetition even more strictly than the frequency penalty and influences the model to use certain words only once or very rarely. While the frequency penalty targets how often a word is repeated, the presence penalty already takes effect on the first use of a term, discouraging its repeated use even more strongly.

For example, with presence_penalty=0.6 the model tries to avoid terms it has already used.
Without penalty: "The sunset is beautiful. The sunset is red."
With penalty: "The sunset is beautiful. The colors are impressive."

Logit bias

Logit bias is applied at the token level and allows you to increase the probability of a specific token (positive logit bias) or decrease it (negative logit bias), so that the model chooses these words more or less frequently.

With logit_bias={"1234": -100} you would achieve that the probability of the specific token (in this case "1234") is reduced so strongly (-100) that the model practically never uses this word.

4. Advanced parameters

Function calling

Function calling enables large language models to access external functions and APIs and thus interact with other tools or systems to handle more complex tasks. Instead of merely generating text, an LLM can use function calling to retrieve data from external systems, perform calculations, or trigger actions that go beyond pure text generation.

If you use function_call="fetch_weather", and the LLM framework supports the function, an external function called "fetch_weather" is invoked to retrieve weather data from an external system. The model connects to an API to provide relevant information.

Structured outputs

The structured outputs parameter makes it possible to control an LLM's response so that it is returned in a structured, machine-readable format such as JSON and conforms to a predefined schema. They therefore require strict adherence to a defined schema, offering more control and reliability when exact data structures are needed.

If you pass the schema schema={"product_name": "string", "price": "number"} with the request, for example, the output is forced to follow the defined schema. An output could look like this:
{
"product_name": "iPhone 15",
"price": 799.99
}

JSON mode

JSON mode is similar to structured outputs and also aims to provide an LLM's output in a machine-readable JSON format. JSON mode is a bit simpler and offers plain JSON formatting without any guarantee of a fixed structure or specific data types — flexible, but less strict.

When using output_format="json", the response is returned in JSON format, but without the constraint of a fixed schema.

LLM streaming

LLM streaming allows the user to receive the LLM's response in real time or incrementally instead of waiting for the entire output to be generated. The generated tokens (words, sentence fragments, or characters) are sent to the user as soon as the model produces them.

Enable stream=True in the API call to receive the output incrementally.

Logprobs

The logprobs parameter makes it possible to analyze the probability of each generated word in the output. With logprobs, developers and analysts can see how likely the model was to select certain words and how "confident" the model was in its word choice. These probabilities are represented in logarithmic form.

With logprobs=5, for example, you get the probabilities of the top 5 tokens per word.

Prompt caching

Prompt caching is a technique that is used to make the performance and costs of applications based on large language models more efficient. With prompt caching, frequently used prompts and their generated responses are cached. When the same input occurs again, the application can retrieve the stored response from the cache instead of querying the model anew each time.

Seed

The seed parameter is used to control the randomness of an LLM's text generation and thereby obtain reproducible (deterministic) outputs. LLMs usually produce slightly different answers with each request because they work probabilistically. Setting a seed value, however, strongly limits the randomness, so the model delivers consistent results for identical inputs.

If available, set the seed parameter (e.g. seed=42) to achieve deterministic outputs.

Conclusion on LLM parameters

Parameters are a helpful tool for precisely controlling the behavior and output of LLMs. Parameters like temperature, Top K, and Top P allow you to fine-tune the creativity and randomness of the responses, while max tokens and stop sequences control text length and structure. Overall, these parameters make LLMs more flexible and adaptable, so they can be used optimally for a wide range of use cases, from chatbots and customer service to data analysis and creative writing. The right combination and fine-tuning of these parameters maximizes the value of LLMs and enables tailored responses and an improved user experience.