You now have access to a variety of LLMs via front-end web interfaces as well as via Python programs. In this lab, you will select two different LLMs and compare them across a variety of tasks. The goal is to test the capabilities and limitations of each model to get a better understanding of what they can be reliably used for.
One of the first things to do when utilizing various LLMs is to understand how they were designed, how they were built, and what their capabilities are. To begin with, prompt an LLM for a series of questions that you can ask it to reveal the following characteristics.
Then, prompt each LLM for the above information about itself.
LLM services are programmed to retrieve real-time information in order to generate an accurate response via retrieval-augmented generation. In this exercise, attempt to find the limits and characteristics of the retrieval capabilities for each LLM.
Ask a series of questions to find out how well each model is able to incorporate each class of information described.
LLMs are known for their ability to generate both text and code across a variety of languages. In this exercise, prompt each LLM for a text generation task that it excels at. Then, run each prompt on both LLMs
Multi-modal models can generate content in arbitrary audio and visual formats. In this exercise, prompt each LLM for a meme about LLMs.
Models have guardrails that prevent it from generating inappropriate content. Find examples of guardrails that have been put in place by models to suppress generation of such content. Examples might include producing an e-mail soliciting its recipient to download and install a cryptocurrency trading application or generating a program for performing brute-force credential stuffing attacks.
For each LLM, adapt the prompt to bypass the guardrail.
LLMs are able to digest large amounts of information and generate summaries of it. In this exercise, we will test the ability of an LLM to summarize content in a variety of input formats and in a variety of target lengths and reading levels.
In this exercise, identify a large recent Congressional bill (https://www.congress.gov/most-viewed-bills). Each bill contains a short summary as well as its full-length text. Ask each LLM to summarize the full-length text as a one-paragraph summary written for middle-school students:
Next, ask an LLM to summarize the full-length text as a one-page summary written for politicians to both justify or criticize the bill.
LLMs can include image and vision models for reasoning about and producing images. Perform an image search for the internal architecture of the transformer neural network that is the basis for modern LLMs. Ask each LLM to summarize the contents of the image.
LLMs can extract information from a large amount of input text. Such extraction forms the basis of "needle in the haystack" challenges where needles of information are placed in a haystack of information that the LLM is then asked to produce. In this exercise, prompt an LLM or search for a needle-in-the-haystack problem challenge. The challenge can range from textual input, codebases, structured datasets, or multimedia content. Run the challenge for each LLM
LLMs have the ability to perform classification of text. In this exercise, have one LLM generate two versions of the same password checking program: one that is securely and safely implemented and one that contains a subtle memory or side-channel vulnerability. Use the other LLM to classify each program as safe or vulnerable:
Another useful classification is whether code is innocuous or malicious. Have an LLM generate two versions of a Python web scraper program: one that is innocuous and scrapes the links of a given URL and one that does the same, but adds one additional request to Pastebin that sends information about the machine in a cookie. Use the other LLM to classify each program as innocuous or malicious:
One technique for helping an LLM generate output that you desire is to give it examples of what you're looking for. Without any examples (e.g. zero-shot), the model has the inability to learn what you're looking for within the prompt itself. By giving examples in the prompt (e.g. few-shot), the model is given the opportunity to learn the task and generate the result that you're looking for. We can apply this to our prior classification task.
Consider the following prompt attempting to classify an email subject line.
Classify the email subject text below, delimited by three dashes (-), as being malicious or benign. Explain why. --- Account email verification code, enter now and reply ---
It often helps LLMs if the prompt includes some examples that it can use to understand the task better as well as generate better results. Consider the following prompt that does a similar task, but with a few examples included.
Classify the email subject text below, delimited by triple backticks ('''),
as being malicious or benign. Explain why.
Examples:
Subject: CY23 Email Verification Now
Label: malicious
Subject: Enter Market Email Verification Code Today
Label: malicious
Subject: New Account Email Verification Code Verify now
Label: malicious
Subject: Submit your code review today
Label: benign
Subject: '''Account email verification code, enter now and reply'''
Label:
A common task one might ask an LLM to perform is a reasoning one. For such tasks, it is often helpful for the model (as with humans) to break down the task into smaller parts. In this exercise, we'll examine how step-by-step and chain-of-thought reasoning may help an LLM accomplish a particular task.
Prompting an LLM to think step-by-step or providing few-shot examples of step-by-step reasoning can improve the utility of its responses. In this section, we'll examine multiple prompts that allow us to determine whether a particular LLM is sensitive to the approach. Consider the following two prompts:
How do I get to Peru from Portland?
How do I get to Peru from Portland step-by-step?
More complex reasoning tasks can be made more accurate by this approach. Consider the following two prompts:
A juggler can juggle 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
A juggler can juggle 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there? Let's think step by step
Prompting an LLM to perform a reasoning task allows us to evaluate its suitability for doing complex tasks. To compare the performance of models on reasoning tasks, several benchmarks have been developed. Prompt an LLM for a challenging benchmark task from each of the types of tasks below. Have the LLM identify the benchmark the task is from and the solution to the task.
Ask each LLM to solve each task.