> 1 - Is the document about desks designed for children?
> 2 - Does the document offer products or information on kids’ desks?
> ...
> 9 - Are brands or models of children’s desks listed?
> 10 - Does it cover where to buy or how to choose a desk for kids?
>
> Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance
It seems to me that this is a rediscovery of natural language inference (NLI), which was very active in NLP research pre-LLMs. I have a side hobby of training NLI cross encoders, and I'd bet that many of these use cases could be serviced by a good quality small NLI model.
But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.
EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.
LLMs do well, but are not practical from a cost standpoint. Dougs post utilizes my research across LLMs, decision models, and purpose built rerankers using BEIR datasets.
reranker nDCG@10 p50 / p95 $ per 1k queries
Jev, one call per query 0.768 224ms / 1.8s $0.60
Claude Opus 5, low effort 0.756 5.0s / 7.5s $94.68
gpt-5.6-luna, reasoning off 0.747 3.2s / 5.8s $2.36
Claude Haiku 4.5 0.723 2.9s / 11.6s $13.93
BM25 order, no rerank 0.667
Latency can be an issue with an LLM, but it's not the only shortcoming. That said Luna in particular does pretty well as a reranker (and was the based for OpenAI's decision model).
Importantly though, LLMs can generate questions in parallel, and there can be much fewer questions (q) than segments to evaluate against (n), so the cost of jev on questions (jev(q * n) + llm(q)) vs the cost of putting all the segments in the LLM can be smaller (llm(n)), (llm(n) >> llm(q) + jev(n * q) if n >> q)
The real benchmark that's missing for me is MaxSim on a real embedder from MTEB.
This is a cool idea. It would be interesting to see how sensitive the results are to the strength of the LLM used for query augmentation. They used GPT-5 in their experiments, and generating all those tokens would presumably add a lot of latency in your query pipeline.
Another baseline that would be interesting to see is just pure LLM reranking--is Jev/the query augmentation adding anything? Presumably the idea is that Jev is cheaper, but even with a costly LLM you might get a lot of cache hits as long as the query (the variable portion) goes last. Even setting caching aside, almost the entire request is prefill plus a very small amount of generation for the judgment itself. You may end up generating fewer tokens just using the LLM as a judge directly than you need to generate the rubric for Jev. OTOH if queries can be normalized or are naturally repetitive you may be able to cache the rubric...
I would assume that you could get by with a very small and fast LLM to do the question generation. You’re not asking the model for very much. From what I can tell, Jev is adding the quantitative probability which makes the score not simply a sum of ones and zeros against the questions.
A possible improvement could be generating a sensitivity score for each question. Some are more important than others. For example not mentioning kids at all should cause a huge decrease in relevance, but not mentioning a specific feature should only cause a small decline in score.
Have used decision models the same way: Had LLM generate/tune questions for Clef to label/extract emails with {useful work thing} content
Too early to say how useful this is versus existing techniques, the cool thing is this is the first thing I've found local models genuinely useful for. I think I was able to process 50,000 email questions per hour or so with Clef flash on Strix Halo with a custom engine. A modern gaming GPU would likely be much faster
I use LLMs for ranking and at least for my applications geomean has given me decent scores when the rubric has multiple axes. Raw scores tend to be noisy so either way it helps to take the rough scores and refine by reranking in chunks.
Super interesting. It’s amazing all the applications we are starting to see. Even if not optimal, a lot of people are now thinking about options and workflows that they weren’t before
> 1 - Is the document about desks designed for children?
> 2 - Does the document offer products or information on kids’ desks?
> ...
> 9 - Are brands or models of children’s desks listed?
> 10 - Does it cover where to buy or how to choose a desk for kids?
>
> Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance
It seems to me that this is a rediscovery of natural language inference (NLI), which was very active in NLP research pre-LLMs. I have a side hobby of training NLI cross encoders, and I'd bet that many of these use cases could be serviced by a good quality small NLI model.
See: https://huggingface.co/dleemiller/finecat-nli-v2-l
And also: "Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI" https://www.semanticscholar.org/paper/Less-Annotating%2C-Mor...
He compared techniques that used:
- BM25 only
- Jev only
- LLM + Jev
The latter worked best.
But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.
EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.
LLMs do well, but are not practical from a cost standpoint. Dougs post utilizes my research across LLMs, decision models, and purpose built rerankers using BEIR datasets.
https://hevmind.com/writing/jev-as-a-reranker/I also tested several approaches to reranking in the piece, and added upates for Open AI and Cloudflare models in these follow up posts.
https://hevmind.com/writing/jev-has-company/ https://hevmind.com/writing/openai-makes-four/
https://hevmind.com/research/rerankers/ (sign up required)
A reranker has to be fast. LLMs are not.
Latency can be an issue with an LLM, but it's not the only shortcoming. That said Luna in particular does pretty well as a reranker (and was the based for OpenAI's decision model).
Importantly though, LLMs can generate questions in parallel, and there can be much fewer questions (q) than segments to evaluate against (n), so the cost of jev on questions (jev(q * n) + llm(q)) vs the cost of putting all the segments in the LLM can be smaller (llm(n)), (llm(n) >> llm(q) + jev(n * q) if n >> q)
The real benchmark that's missing for me is MaxSim on a real embedder from MTEB.
This is a cool idea. It would be interesting to see how sensitive the results are to the strength of the LLM used for query augmentation. They used GPT-5 in their experiments, and generating all those tokens would presumably add a lot of latency in your query pipeline.
Another baseline that would be interesting to see is just pure LLM reranking--is Jev/the query augmentation adding anything? Presumably the idea is that Jev is cheaper, but even with a costly LLM you might get a lot of cache hits as long as the query (the variable portion) goes last. Even setting caching aside, almost the entire request is prefill plus a very small amount of generation for the judgment itself. You may end up generating fewer tokens just using the LLM as a judge directly than you need to generate the rubric for Jev. OTOH if queries can be normalized or are naturally repetitive you may be able to cache the rubric...
I would assume that you could get by with a very small and fast LLM to do the question generation. You’re not asking the model for very much. From what I can tell, Jev is adding the quantitative probability which makes the score not simply a sum of ones and zeros against the questions.
A possible improvement could be generating a sensitivity score for each question. Some are more important than others. For example not mentioning kids at all should cause a huge decrease in relevance, but not mentioning a specific feature should only cause a small decline in score.
Have used decision models the same way: Had LLM generate/tune questions for Clef to label/extract emails with {useful work thing} content
Too early to say how useful this is versus existing techniques, the cool thing is this is the first thing I've found local models genuinely useful for. I think I was able to process 50,000 email questions per hour or so with Clef flash on Strix Halo with a custom engine. A modern gaming GPU would likely be much faster
Shouldn't you multiply instead of summing the scores?
Would you show a result with 3 really high scores and one which is really low?
I use LLMs for ranking and at least for my applications geomean has given me decent scores when the rubric has multiple axes. Raw scores tend to be noisy so either way it helps to take the rough scores and refine by reranking in chunks.
There are a bunch of things that could be tried honestly. Like geometric mean, top-N, or even fitting another classifier on the training set.
Super interesting. It’s amazing all the applications we are starting to see. Even if not optimal, a lot of people are now thinking about options and workflows that they weren’t before