Extracting information with LLMs: GRPO to improve performance

Large language models are increasingly used for information extraction, where unstructured text needs to be converted into structured data for downstream systems.

Looking to make an LLM more efficient and reliable at this task, we ran an experiment with Group Relative Policy Optimisation (GRPO). In this blog post, we will evaluate this reinforcement learning technique and see if it does make LLMs better at extracting information.

The problem

LLMs already write fluent text thanks to pre-training and supervised fine-tuning. But extraction does not mainly test language; it tests discipline: following rules, making sensible trade-offs, and behaving consistently when inputs change slightly.

Supervised fine-tuning teaches imitation, which works well when there is a single, clear answer. Extraction, on the other hand, often is not like that. You can have several plausible outputs, but some are clearly better for production use.

Finance and insurance context

In domains like finance and insurance, information extraction is everywhere, from contracts and policies to claims notes and regulatory documents. The extracted entities (counterparties, organisations, dates, locations, identifiers) usually feed downstream systems like databases, compliance checks, audit trails, and reporting pipelines.

In this context, the key question is whether the output is accurate and consistent enough to be safely embedded inside a data pipeline.

Our solution

Reinforcement learning works well for our purposes because it lets the model generate multiple candidates and learn which ones to prefer based on scoring signals, which we have defined during training.

Instead of copying a single ‘gold’ output, the model learns to prefer behaviours that matter in practice: consistent formatting, precision, and rule compliance. This is especially valuable when labelled data is expensive and many outputs are not fully wrong, just partially correct.

To test this idea, we experiment with Group Relative Policy Optimisation (GRPO), a relatively new reinforcement learning technique that makes this approach more practical than earlier methods.

GRPO learns from relative comparisons. For the same input, the model generates a group of outputs. We then score them using simple rules, and reinforce the better ones, without the need for detailed human feedback.

Moreover, DeepSeek-R1 uses GRPO, and this has helped bring back attention to reinforcement-style training as a way to improve model behaviour.
To clarify, we still assume downstream validation and controls, particularly in regulated environments. The goal is not guaranteed correctness. It is to reduce avoidable errors and improve baseline reliability so the rest of the pipeline has less work to do.

To keep the experiment controlled, we use the CoNLL-2003 dataset for Named Entity Recognition (people, organisations, locations, misc). It is a good fit because the labels can later be adapted to domain-specific meanings while keeping the extraction mechanics like prompts, scoring rules, and output structure largely the same.

A closer look at the CoNLL-2003 Named Entity Recognition dataset

Below is a simplified example of how a single sentence is represented in the CoNLL-2003 NER dataset. This example is deliberately simple but representative, and it works well as a concrete illustration when explaining NER mechanics or downstream scoring in a GRPO based setup.

BIO tagging is a common scheme used in information extraction to label sequences of tokens that form named entities. Each entity begins with a B- tag (Begin), continues with one or more I-tags (Inside), and all tokens that are not part of an entity are labeled O (Outside). This structure makes it easy to identify both the boundaries and types of entities in text.

Here, each sentence is then broken down to tokens: each word is on its own row, with associated specific BIO tagged annotations as in the example below.

How does GRPO work?

Here is a simplified overview of GRPO, which aims to introduce the key concepts without going into deep technical detail.

Prompt

This prompt defines a tightly constrained information extraction task, instructing the model to perform token-level Named Entity Recognition using the BIO tagging scheme. It enforces a strict output format with explicit reasoning and answer sections, providing an example and ensuring that every input token aligns to exactly one label and that it is easy to evaluate structural correctness.

Policy

In GRPO, the policy is simply the language model itself viewed as a decision-making system. For any given input (e.g. a sentence for named entity extraction), the policy defines a probability distribution over all possible outputs the model could generate, i.e. how likely the model is to produce one extraction versus another.

During training, the policy is sampled multiple times to produce a group of candidate outputs for the same input. These candidates are then compared against each other using reward functions, and the policy is updated to increase the probability of outputs that score better relative to the group.

Rather than learning from a single correct answer, the policy learns to prefer behaviours that consistently lead to higher rewards, such as producing valid BIO sequences, aligned token labels, and structurally sound extractions.

In short, the policy in GRPO represents the model’s current behaviour, and GRPO improves that behaviour by nudging the policy toward better choices through relative comparison rather than absolute right or wrong feedback.

Reward design and scoring functions

From a reinforcement learning perspective, extracting information is a good fit for GRPO-style training because it lends itself naturally to rule-based and programmatic scoring.

We can evaluate entity extraction along several dimensions

1. Whether the output has the correct structure
2. Whether BIO rules are respected
3. Whether spans align with the input tokens
4. Whether labels are correct.

This allows us to compare multiple candidate extractions for the same input and rank them relative to one another.

The reward setup in this experiment reflects this strategy by breaking the task into several simple signals:

• The reward_format function checks that the model follows the required response structure and includes both the <reasoning> and <answer> sections. This prevents malformed outputs from dominating training and helps stabilise the basic scaffolding early on.
• The reward_token_alignment function focuses on the shape of the output rather than correctness.
• The reward_bio_validity function enforces the BIO tagging rules.
• The reward_token_accuracy function provides a familiar correctness signal by measuring how many predicted labels match the gold data. Here, there is a difference from SFT about how the gold data is used. In SFT, each gold answer is used once to train the model to imitate it. In this GRPO setup, the gold answer is used as a reference to score many different model-generated outputs. The same gold answer can be reused across multiple rollouts and training steps, which means each labeled example gets used many times.
• The reward_reasoning_presence function simply checks that reasoning text exists.

All of these signals are combined in combined_reward_phase1, which heavily prioritises token alignment and BIO validity while treating formatting, reasoning presence, and raw accuracy as secondary concerns.

The goal of this first phase is to teach the model how to produce valid, well-formed extractions before strongly optimising for correctness. Such a staged approach works particularly well with GRPO, where learning is driven by relative comparisons between candidate outputs rather than strict right-or-wrong judgments.

Technology and hardware

The model has been fine-tuned using Low-Rank Adaptation (LoRA), a parameter-efficient technique that freezes the base model and learns a small number of additional weights instead. This significantly reduces memory and compute requirements while still allowing the model’s behaviour to be adapted through reinforcement learning.

To build the training pipeline, we used the Hugging Face ecosystem, leveraging the TRL library to implement the GRPO-style reinforcement learning loop. The metrics were recorded in Weights and Biases.

We attempted the training on various NVIDIA GPU configurations. A consumer GPU, such as a 3090 with 24 G VRAM, proved sufficient to experiment with a small training batch size.

We downloaded the Qwen2.5-3B-Instruct model from Hugging Face, and the saved the fine-tuned model checkpoints back to Hugging Face. This has made the training process fully reproducible and easy to iterate on.

Training pipeline overview

Here is a high-level overview of the training pipeline:

Training process

For the training process, we followed an experimentation-first approach, using iterative refinement to identify what would work best in practice rather than relying on fixed assumptions.

Here is an overview of the process:

• We tested multiple prompt variants, as small changes to the system and user instructions had a noticeable impact on output structure and stability.
• The dataset went through several preprocessing iterations to reach an optimal format that integrated cleanly with the system and user prompts.
• Reward design was also iterative, starting with a small set of simple reward functions and gradually adjusting both the set of rewards and their relative weights. This process is ongoing as behaviour and failure modes become clearer.
• We explored different batch sizes and hardware configurations. If doing a test run with a consumer GPU (e.g. a 3090), tread cautiously, setting the training batch size to 1 or 2 and the number of generations to not more than 4. You would want to monitor memory usage and increase these parameters accordingly. Initially, train for about a 100 steps and check if the metrics are improving even marginally; only then resume training from the checkpoint.
• We saw significant improvement in the post-training evaluation with a training batch size of 8 and 600 steps, trained with an A100 80 GB. This, however, would take around 6 to 7 hours of training.

Before and after

We obtained the experimental results you see below through sequential runs, applying policy updates cumulatively.

Before GRPO fine-tuning, the model already adhered reliably to the required output format and consistently included reasoning, indicating that prompt compliance was well learnt. However, its behaviour on the core extraction task was less stable.

After some GRPO fine-tuning, the model showed clear improvements in structural reliability. Token alignment became far more consistent, BIO validity increased substantially, and token-level accuracy improved as a result of better-formed extractions. Importantly, these gains were achieved without any change to formatting or reasoning compliance, both of which remained stable throughout training.

Jupyter Notebooks with code and evaluations results are available via the links in the table below:

Experiment 1:
Large batch size of 300 steps

Metrics & observations:

Summary (post - pre):
format_mean: 1.0000 → 0.9500 (Δ -0.0500)
format_rate: 1.0000 → 0.9750 (Δ -0.0250)
alignment_mean: 0.7444 → 0.8837 (Δ +0.1393)
alignment_rate: 0.7750 → 0.9000 (Δ +0.1250)
bio_validity_mean: 0.1500 → 0.3000 (Δ +0.1500)
bio_validity_rate: 0.5750 → 0.6500 (Δ +0.0750)
token_accuracy_mean: 0.5754 → 0.6693 (Δ +0.0938)
token_accuracy_exact: 0.1250 → 0.2000 (Δ +0.0750)
reasoning_presence_rate: 1.0000 → 1.0000 (Δ +0.0000)
combined_reward_mean: 2.9908 → 3.8581 (Δ +0.8673)


With larger batch sizes and fewer steps, we saw high oscillations in the training metrics for early steps, which indicated unstable early training.

Large, infrequent steering corrections would mean the updates were more aggressive and could overshoot. However it did take less training time overall to achieve good performance.

Experiment:
Small batch size of 600 steps

Metrics & observations х4

Summary (post - pre):
format_mean: 0.9500 → 1.0000 (Δ +0.0500)
format_rate: 0.9750 → 1.0000 (Δ +0.0250)
alignment_mean: 0.8274 → 0.9122 (Δ +0.0848)
alignment_rate: 0.8500 → 0.9250 (Δ +0.0750)
bio_validity_mean: 0.2500 → 0.4000 (Δ +0.1500)
bio_validity_rate: 0.6250 → 0.7000 (Δ +0.0750)
token_accuracy_mean: 0.6341 → 0.7087 (Δ +0.0747)
token_accuracy_exact: 0.1750 → 0.2500 (Δ +0.0750)
reasoning_presence_rate: 1.0000 → 1.0000 (Δ +0.0000)
combined_reward_mean: 3.5357 → 4.2575 (Δ +0.7218)


With smaller batch sizes, the training process was smoother and more stable, even though it would take more steps (and time) to reach a similar level of performance.

More about metrics

*_mean metrics are the average scores across the evaluation dataset. For example, token_accuracy_mean is the average token-level accuracy over all model outputs in the evaluation dataset. Each sample can contribute a fractional score (e.g. partially correct answers), and _mean tells you how correct the model is on average.

*_rate metrics measure how often something happens, expressed as a proportion.
For example, reasoning_presence_rate is the fraction of outputs that satisfy a condition (e.g. contain a reasoning section). Each output is counted as either yes or ‘no,’ and the rate tells you how frequently the condition is met across the batch.

Finally, the combined_reward_mean acts as the ultimate reward metric. It is the average of the ultimate reward function ie. weighted combination of all individual reward components (token accuracy, exact match, reasoning presence, etc.). It reflects how well the model is doing according to the reward function, not any single evaluation metric in isolation.

Final insights into making LLMs better at extracting information

The results of this experiment show that GRPO effectively improves structural reliability and consistency, but boosting raw extraction accuracy requires additional steps. Once the model reliably produces well-formed BIO sequences and aligned outputs, we can introduce stronger correctness signals without destabilising behaviour. This is a natural next stage in reinforcement learning based on fine-tuning.

A straightforward approach here is to increase the weight of token-level accuracy in the reward function. In the current setup, accuracy was deliberately down-weighted to prioritise structure.

With those foundations in place, gradually amplifying accuracy-focused rewards allows the model to focus more directly on selecting the correct entity labels rather than simply producing valid spans.

Summary

In this article, we explored how the reinforcement learning technique GRPO can be applied to improve information extraction capabilities of LLMs. Using Named Entity Recognition as an use case, we showed how relative, rule-based reward functions can meaningfully improve structural reliability, data validity and overall accuracy, without relying on large amounts of human-labelled data.

The results demonstrate that GRPO is not limited to academic research, but can be repurposed effectively for practical, industry-oriented use cases.

The full training code, reward functions, and experiment setup are available in a public GitHub repository. Use it freely to reproduce the results and adapt the approach to your own domains and datasets.

For more insights into cutting-edge data and AI technology and practices, visit the Infinite Lambda blog.