Skip to content 
DSPy stands for Declarative Self-improving Python. Moving away from brittle prompts, you can now write compositional Python code and use DSPy to teach your Large Language Model (LLM) to deliver high-quality output.
Approaches like Chain of Thought (CoT) and Reasoning and Acting (ReAct) are powerful, but they still rely on carefully constructed examples and prompts. If you switch from Model A to Model B or from version 1 to version 2, you might find that your old prompts are no longer working well.
This article will offer everything you need to know to get started with DSPy in an enterprise context.
Overall architecture
Common Patterns to Improve Prompts
There are some simple, commonly used patterns that can help improve a prompt. To name a couple:
- Few-Shots examples: providing examples helps the LLM grasp the essence of the question;
- Chain-of-Thoughts (CoT): pushing the LLM to think step by step, so that the it can generate its own reasoning trace and pay attention to this trace in order to generate a better final output
In this article, we will focus on the simple patterns to illustrate the essence of evaluation and optimisation.
There are other, more advanced patterns, which involve external data sources and environments. Notable examples include Retrieval Augmented Generation (RAG) and ReAct, both of which could also be automated with DSPy but are not the subject of this blog post specifically.
Example prompts
In an earlier article where we talked about reliable, scalable AI for enterprise, we looked at an example of building a financial adviser. Here, we are going to build on the same context and develop the same concept further.
The prompt we have is as follows:
Question: I’m a 40-year-old investor with a moderate risk tolerance. I have $50,000 to invest, and I would like to grow it over the next 10 years for my retirement. Can you run financial simulations (like Monte Carlo) to estimate potential returns and recommend suitable fund products?
The goal here is to automate a prompt, which could work across different models and strategies.
Experiment set-up
For this experiment, we will be using:
- LLM: Claude Sonnet 3.7 via AWS BedRock;
- Retrieval model (RM): In-memory Faiss or dockerised Weaviate;
- DSPy Version: 2.6.17.
A visual representation of the set-up looks like so:
The set-up leverages:
- Provide training data: An LM would require golden example,( i.e. example questions with correct answers), whereas an RM would require relevant contexts (e.g. a corpus).
- Program: Define your task as a programme, which consists of at least:
- Signature: A declarative specification of input / output behaviour of a DSPy module;
- Module: A prompting technique which can generalise to any signature;
- Validate: The task is then validated against pre-defined metrics;
- Compile: After validation, a programme can be compiled with an optimiser to produce the optimal prompt for a particular language model, retriever model, and strategy.
- Iterate: The prompt is only as good as the models and strategies selected, so users should iterate across different models and strategies for the best results.
Let’s use the financial adviser model to explain each step one by one.
Prompt automation (programming)
Signature instead of prompt
When working with LangChain or LlamaIndex, the first step is often to craft the prompt. In DSPy however, we do not directly write the prompt; instead, we provide a signature.
The signature above could be declared inline: 'question: str -> response: str'. However, we opt for a class-based declaration to demonstrate the reuse across strategies.
To do this, we create a class named QASignature with a constructor which takes the question as input field, and the response as output field.
Let’s compare that with a LangChain implementation:
Now, let’s compare it with a LlamaIndex Implementation:
We can see that LangChain focuses more on composable chains, whereas LlamaIndex focuses on retrievers when it comes to prompt. In any case, the approach remains similar.
Users would be crafting their own prompt, which fits towards a specific LLM model (e.g. AWS BedRock), and a specific Strategy (e.g. Chain-of-Thoughts). There is no material difference between LangChain and LlamaIndex when it comes to prompt set-up.
With DSPy, on the other hand, the user writes a signature, which declares the input and output through the LLM. The signature is completely agnostic to models and strategies, so you can reuse the exact signature for different strategies and models. This provides a loose coupling between the interface and the implementation, which is a common technique in software engineering.
Signature + Module -> Prompt
A DSPy module is a building block for programmes that use LLMs. Here, the module takes a signature, together with other parameters, to produce the LLM’s prompt and response. The relationship between a signature, a module, and an LLM looks like this:
The dspy.Predict in the previous code example is a module that provides the simplest strategy. The module takes in the signature, compiles to the prompt, and generates a response from LLM:
We can see this is a zero-shot prompt.
If we opt to use CoT, DSPy allows us to keep the same signature, QASignature.
The compilation works with a different module as well, the CoT.
Here is the compiled prompt:
Note how the reason field is being inserted into the prompt, and comes with the necessary steps to ask the LLM to apply reasoning. This in turn yields a better response from LLM, where the latter is actually trying to do the simulation instead of ignoring the user’s request.
Prompt evaluation (validating)
We have got the above CoT prompt without writing our own prompt and with a reusable signature, which is a step forward. However, we are yet to evaluate the quality of the prompt.
Manual evaluation would require a human to judge the prompt and determine whether it is effective in obtaining the accurate response from the LLM. Automated evaluation would involve asking the LLM to determine whether the response is good or not, given the provided criteria.
To allow the LLM to ground its judgment, we would also provide the golden response for some example questions.
Golden examples h3
Golden examples, or ground truths, are examples of questions and answers pairs which are prepared in advance and known to be accurate.
First, we look at how examples are prepared for training:
Here is a sample of the golden example, which contains both a question and its accurate response:
For this experiment, we prepare the response leveraging data augmentation, which involves a few human responses coupled with Gen AI. If you are aiming for higher quality and have the resources, you can prepare samples using purely human responses.
Metrics
Golden examples allow for an easy comparison with the response from the LLM, so we can provide a metric score.
A commonly used metric is F1 Score, which is the harmonic mean of:
- Precision: How cleanly the LLM avoided extra words;
- Recall: How many correct words the LLM predicted.
Hence, an F1 Score of 1 would imply that the LLM produces an identical response compared with the golden example. The lower the score, the poorer the performance.
Here is how to calculate an F1 score using LLM self-scoring:
And with the metric available, we can easily evaluate the module, e.g. Simple CoT, as follows:
In this example, we have obtained an average F1 score of 0.269, or 26.9%, out of Chain-of-Thought.
Prompt optimisation (Compiling)
Optimiser
Once we have a reliable metric, and run an evaluation over enough examples, we can come up with a baseline performance. In our case, it is 30.6% accurate.
We would like to tune our prompt to improve the metric, and the performance of the LLM respectively. Moreover, we would like the tuning to be completely automatic and data-driven, so that it can easily be re-run over different models and strategies. What we need is not more quirky prompt hacking, but an optimiser.
A DSPy optimiser is an algorithm that can tune the parameters of a DSPy programme (i.e. the prompts and / or the LM weights) to maximise the metrics you specify, such as accuracy.
With a few lines of code and several examples, we have achieved an F1 score of 0.766, or 76.6%, which is more than double the original CoT.
Resulting prompt
Here, you can see the optimised prompt response, together with the response:
One can see that the resulting prompt is rather detailed with dense examples, and is not easy for humans to read, let alone handcraft. Moreover, this prompt could change substantially across different strategies and models. An optimiser flow helps us ensure good results every time.
We can also see that the new prompt produces a better result, not only in recall and precision, but also in quality: we have got the LLM to actually perform a Monte Carlo simulation and select a legit fund.
Iterating across models and strategies
Iterate over strategies
In order to illustrate the optimiser with a minimum viable flow, we did not focus on various other strategies, such as RAG, which creates a context for the LLM to reason and can further improve the score. RAG often proves a good strategy if you can find enough context and get that indexed in a vector database.
Another strategy that warrants you attention is using a ReAct agent (also supported by DSPy). We will be covering that approach in a separate blog post, so stay tuned.
Since modules in DSPy are composable, one can easily chain up new strategies and re-run the optimiser for better results.
Iterate over the optimiser
DSPy works with a range of optimisers. We have opted for Multiprompt Instruction Proposal Optimizer Version 2 or MIPROv2, which runs evaluation trials, scoring each demo-prompt pair against the criteria we specify (i.e. F1 Score). By using Bayesian Optimisation, MIPROv2 quickly hones in on the best-performing prompt variation.
If you are not perfectly happy with MIPROv2, you can opt for another optimiser, such as BootstrapFinetune, and iterate for better performance.
Iterate over LLMs
Perhaps most importantly, the framework allows developers to quickly iterate over different LLMs. These include not only different versions of the same LLM (e.g. from GPT3.5 to GPT4) but also across vendors (from GPT to Claude), while ensuring the same performance.
DSPy takeaway
DSPy turns prompt engineering into a declarative, test-driven, model-agnostic software loop or writing I/O contracts, measuring, auto-improving, repeating.
Using a financial adviser example, we explored the ease of the approach and the capabilities it could enable.
This article is part of a series on reliable Gen AI for enterprises.
Explore related content:
