Toward Dynamic Evaluation of Factual Numeric Recall in Large Language Models via Next Prime Identification

Abstract

Large Language Models (LLMs) have achieved major breakthroughs in natural language processing; however, their ability to generalize factual abstraction, especially in non-linguistic domains, remains insufficiently explored. This paper introduces Prime Number Prompt Power (PNPP), a dynamic, prompt-based evaluation framework designed to assess LLMs' internalization of numeric facts. It uses prime numbers because their irregular distribution and the absence of a deterministic formula make them an ideal choice for testing generalization. PNPP challenges models to identify the next prime number following a given value across exponentially increasing intervals and defines a quantitative threshold (PNPPx), representing the largest interval range in which a model maintains a specified accuracy level. Experiments across leading LLMs demonstrate PNPP's ability to distinguish numeric recall strengths, revealing differences in generalization patterns under increasing task complexity. As a lightweight, dataset-free, and scalable benchmark, PNPP complements traditional NLP evaluations and offers a principled approach to measuring factual knowledge in LLMs. Empirical results show that PNPP differentiates models based on their numeric recall capabilities, with high-performing models maintaining accuracy over broader intervals. The framework's ability to isolate factual knowledge makes it valuable for benchmarking, model selection, and guiding future training strategies for AI systems. Compared to static or language-oriented benchmarks, PNPP introduces a dynamically scalable approach that directly evaluates factual numeric recall, an ability traditional benchmarks do not capture. By using progressively expanding intervals, PNPP uncovers differences in generalization invisible in conventional evaluations, offering a more robust and contamination-resistant measure of factual knowledge in LLMs.