Implement RRPO (Reranking Preference Optimization) pipeline (2026)

[vkehfdl1]

Paper

• Title: Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
• Authors: Yuhang Wu, Xiangqing Shen, Fanfan Wang, Cangqi Zhou, Zhen Wu, Xinyu Dai, Rui Xia
• Link: https://arxiv.org/abs/2604.02091v1
• Source: Arxiv

Summary

RRPO introduces a reinforcement learning framework that directly aligns reranking with the downstream LLM generation quality. By formulating reranking as a sequential decision-making process and optimizing for context utility using LLM feedback, RRPO eliminates the misalignment between static relevance-based reranking and actual answer generation utility. It includes a reference-anchored deterministic baseline for training stability and outperforms strong baselines including RankZephyr on knowledge-intensive benchmarks.

Relevance to AutoRAG-Research

Directly relevant as a retrieval pipeline enhancement. AutoRAG-Research currently lacks a dedicated reranking optimization module — existing retrieval pipelines (bm25, vector_search, hyde, hybrid) retrieve and rank documents based on static relevance signals without considering downstream generation quality. RRPO bridges this gap by training a reranker that optimizes for what actually helps the LLM generate correct answers. It also integrates orthogonally with query expansion modules (like HyDE) and generalizes across different reader LLMs.

Implementation Notes

• Pipeline type: retrieval (reranking stage)
• Key components:
  • Reranking formulated as sequential decision-making (selecting documents one by one)
  • RL optimization with LLM generation quality as reward signal
  • Reference-anchored deterministic baseline for stable training
  • Compatible with various reader LLMs (tested with GPT-4o)
  • Orthogonal integration with query expansion (e.g., Query2Doc/HyDE)
• Dependencies: PyTorch, RL training utilities (PPO-style), LangChain LLM integration for reward computation

Acceptance Criteria

• Pipeline class implementing BaseRetrievalPipeline
• Config dataclass with get_pipeline_class() and get_pipeline_kwargs()
• YAML configuration file
• Unit tests following existing test patterns
• Passes existing test suite without regression

[vkehfdl1]

AI-understood issue summary

RRPO is not just “another reranker”; it is a generation-aware reranking framework. The paper frames reranking as a sequential decision problem, uses LLM answer quality as the reward, and stabilizes training with a reference-anchored deterministic baseline. In repo terms, this would sit on top of an existing candidate retriever and change the ordering objective from static relevance to downstream answer utility.

Why this issue is needed

• AutoRAG-Research already has retrieval pipelines (bm25, vector_search, hyde, hybrid, heaven) and a reusable reranker subsystem, but I do not see an RL-based / answer-aware reranking pipeline yet.
• That makes this issue meaningfully different from current fusion or late-interaction approaches: RRPO targets the actual RAG objective rather than proxy ranking labels.
• If implemented cleanly, it should compose well with current candidate generators like HyDE or hybrid retrieval.

Why this issue may not be needed

• The repo already has a substantial reranker abstraction (autorag_research/rerankers/*) plus retrieval-stage reranking/fusion logic (hybrid, heaven). If the goal is only to add another inference-time reranker, extending the reranker subsystem may be a better fit than introducing a brand-new retrieval pipeline surface.
• The paper is extremely new (arXiv v1 submitted April 2, 2026), and I could not verify a public reference implementation/checkpoint from the issue context. A full PPO-style reproduction may therefore be high-risk relative to the current acceptance criteria.
• The acceptance criteria read like an inference integration task, while the paper contribution is primarily a training framework. That scope mismatch should be resolved before implementation starts.

Expected Outcome

A practical outcome would be:

1. a config-driven RRPO retrieval pipeline that reuses an existing first-stage retriever,
2. an RRPO-specific reranking step that fits current BaseRetrievalPipeline patterns,
3. YAML + unit tests matching existing retrieval pipeline conventions,
4. no regression to existing retrieval/generation flows,
5. and, if training is in scope, a separate trainer/reward workflow rather than hiding PPO logic inside the pipeline itself.

Concise implementation plan

1. Nail scope first: decide whether this issue means inference-only RRPO integration or training + inference. I strongly recommend splitting those if both are desired.
2. Reuse existing building blocks: use an existing retriever for candidate generation and reuse the reranker loading/injection patterns where possible.
3. Implement pipeline/config: add RRPORetrievalPipeline + RRPOPipelineConfig with get_pipeline_class() / get_pipeline_kwargs().
4. Add config artifact: create configs/pipelines/retrieval/rrpo.yaml following current Hydra conventions.
5. Add tests: mirror test_heaven_pipeline.py / test_hybrid_pipeline.py for config wiring, retrieval flow, and mocked reranking behavior.
6. De-risk training dependencies: if no public RRPO code/checkpoint exists yet, keep PPO/reward-model training as a follow-up issue and land the inference scaffold first.

Agent Signature: