BlockBench: Contamination-Controlled Evaluation of Large Language Models for Code Analysis
- 1The Aberdeen NLP Research Group, University of Aberdeen, UK
- 2International Institute of Information Technology Hyderabad, India
- 3University of Southern California, USA
- 4Federal University of Technology, Minna, Nigeria
Abstract
How much of large language model (LLM) performance in code analysis reflects memorization of training data rather than genuine reasoning? We introduce BlockBench, a framework that tests sensitivity to possible contamination in smart contract analysis. We evaluate seven models on 362 vulnerable base contracts, 322 variants, and 100 clean contracts. We observe a substantial performance gap across data exposure levels: detection accuracy drops from 86.5% on likely exposed samples to 27.6% on recently disclosed GS contracts, and further to 0.8% on the most recently disclosed subset. Detection rises with public exposure. This pattern is consistent with possible contamination but does not prove its cause. We further identify a pattern matching paradox, where models can often locate security-relevant code regions but fail to explain why they are exploitable, revealing a form of explanation unfaithfulness in code understanding. In contrast, traditional static analyzers show complementary limitations: Slither compiles only 6.6% of GS contracts, while Mythril compiles all inputs but fails to detect vulnerabilities. BlockBench generalizes to code understanding tasks where semantic equivalence can be verified, enabling more reliable evaluation under potential contamination.
1Introduction
Large language model evaluation faces a fundamental challenge: distinguishing genuine capability from training data memorization (Zhang et al., 2024; Zhao et al., 2025). When models achieve high scores on established benchmarks, it remains unclear how much performance reflects true reasoning versus pattern matching against memorized examples. This problem is especially acute for code analysis tasks, where benchmark contamination has been documented across multiple evaluation suites (Jain et al., 2025; Haimes et al., 2024).
We investigate this question in the domain of smart contract vulnerability detection, a setting uniquely suited to contamination-controlled evaluation. Smart contract security offers three properties that make it an ideal testbed: (a) ground truth is objectively verifiable through documented exploits and professional audits, (b) public exposure can be estimated from disclosure dates and documentation, and (c) semantic-preserving transformations are well-defined, enabling systematic removal of surface recognition cues while preserving vulnerability mechanics. The broader cryptocurrency ecosystem also carries substantial financial risk, with more than $3.4 billion stolen in 2025 (Chainalysis, 2025).
We present BlockBench, a contamination-controlled evaluation methodology that tests how performance changes when familiar cues are removed. Our approach evaluates seven frontier models across likely exposed baselines, semantically transformed variants, recent professional audit findings, and audited clean contracts. Performance on likely-contaminated samples reaches 86.5% TDR, but falls to 27.6% on the full GS set and 0.8% on the newest GS slice. This degradation, consistent with contamination patterns documented in adjacent benchmarks (Zhang et al., 2024; Zhao et al., 2025), suggests sensitivity to surface patterns. Our contributions:
(i) A contamination-controlled evaluation methodology using semantic-preserving transformations that progressively strip recognition cues while preserving exploit semantics, showing how performance depends on those cues. The methodology generalizes beyond security to any domain where semantic-preserving perturbations are definable. The transformations and temporal split test performance under distribution shift.
(ii) CodeActs, a taxonomy inspired by speech act theory (Austin, 1962; Searle, 1969) for classifying code segments by security function (e.g., ROOT_CAUSE, PREREQ, DECOY), enabling fine-grained analysis of whether models identify vulnerabilities through causal reasoning or pattern matching.
(iii) BlockBench, a benchmark of 362 vulnerable base contracts, 322 TC variants, and 100 clean contracts.
(iv) Evidence of a temporal performance gradient, showing that detection rates on GS samples rise from 5.9% to 84.7% as public exposure accumulates, consistent with temporal contamination patterns documented in code benchmarks (Jain et al., 2025).
2Related Work
Benchmark Contamination and Memorization: The reliability of LLM evaluation is increasingly questioned due to training data contamination. Zhao et al. (2025) demonstrated that contamination inflates MMLU scores by up to 14.6 percentage points, with GPT-4o dropping from 88.0% to 73.4% on decontaminated items. Zhang et al.
(2024) showed analogous inflation on grade school arithmetic, with models scoring up to 13% lower on matched-difficulty items guaranteed absent from training data. LiveCodeBench (Jain et al., 2025) introduced temporal windowing for code evaluation, revealing that DeepSeek-Instruct’s performance plunges from approximately 60% to near 0% for problems released after its training cutoff. Haimes et al. (2024) formalized inflation measurement through retrospective holdout construction, finding up to 16 percentage points of inflation on TruthfulQA. These findings motivate evaluation methodologies that explicitly control for contamination, the approach we adopt for vulnerability detection. Robustness and Semantic Perturbation: Adversarial robustness evaluation in NLP has established that semantic-preserving perturbations— paraphrases, synonym substitutions, syntactic transformations—can substantially degrade model performance (Ribeiro et al., 2020). In code, Rabin et al. (2021) demonstrated that identifier renaming and other semantics-preserving transformations can substantially affect neural program models, and recent work on code rewriting shows that structural modifications expose memorization reliance in code generation (Zhang et al., 2025). Our transformation suite extends this methodology to vulnerability detection, applying domain-specific semantic-preserving perturbations (sanitization, domain shift, structural obfuscation) that preserve exploit mechanics while systematically removing recognition cues. Explanation Faithfulness in Code: Recent work finds reasoning-unfaithfulness failures concentrated in brittle, convergent domains such as mathematics and code (Han et al., 2026). More broadly, FaithCoT-Bench (Shen et al., 2026) provides a framework for instance-level faithfulness detection. Our CodeActs analysis identifies a domain-specific instance of this phenomenon: models that correctly locate vulnerable code segments cannot articulate why they are exploitable—a gap between pattern localization and causal explanation. We draw on speech act theory (Austin, 1962; Searle, 1969) to formalize this distinction, classifying code segments by their security function (e.g., ROOT_CAUSE vs. DECOY) rather than syntactic category. This functional taxonomy is analogous to dialogue act classification in NLP, where utterances are labeled by communicative function rather than surface form. Traditional Analysis Tools: Early approaches to smart contract vulnerability detection relied on static analysis and symbolic execution. Tools such as Slither (Feist et al., 2019), Mythril (Mueller, 2017), and Securify (Tsankov et al., 2018) demonstrated precision on syntactically well-defined vulnerability classes. Durieux et al. (2020) evaluated nine tools on 69 annotated vulnerable contracts, finding that the tools collectively detected 42% of the vulnerabilities, with Mythril detecting 27%. On a separate dataset of approximately 47,500 real-world contracts, 97% were flagged as vulnerable, suggesting a high false positive rate. We extend this comparison by evaluating both Slither and Mythril on BlockBench, providing direct baseline comparison against LLM performance across all subsets (§5.3). LLM-Based Vulnerability Detection: Initial investigations by Chen et al. (2025) explored prompting strategies for vulnerability detection, achieving detection rates near 40% while noting pronounced sensitivity to superficial features. GPTScan (Sun et al., 2024b) combined GPT-4 with program analysis to achieve 78% precision on logic vulnerabilities. CLeVeR (Li et al., 2025) applied multi-modal contrastive learning for vulnerability code representation at ACL 2025, framing detection as a code representation learning problem. Sun et al. (2024a) demonstrated that decoupling vulnerability descriptions from code context causes catastrophic performance degradation, suggesting reliance on memorized textual associations. Hu et al. (2023) documented output drift across evaluation dates at temperature zero, and Wu et al. (2024) showed systematic failure on counterfactual tasks where familiar patterns are disrupted. Our work extends these observations with a controlled framework that measures performance changes under semantic transformations and temporal separation.
3BlockBench
BlockBench tests whether LLM performance depends on familiar cues. It contains 362 vulnerable base contracts, 322 TC variants, and 100 clean contracts.1 The benchmark spans over 30 vulnerability categories (Appendix E).
Let D+ denote the 362 vulnerable contracts. It is the union of the disjoint DS, TC, and GS subsets. The separate set DNEG contains 100 clean contracts.
Difficulty Stratified. DDS draws from established vulnerability repositories including Smart- Bugs Curated (Ferreira et al., 2020), Trail of Bits’ Not So Smart Contracts (Trail of Bits, 2018), and DeFiVulnLabs (SunWeb3Sec, 2023). Samples are stratified into four difficulty tiers based on detection complexity, with distribution {86, 81, 30, 13} from Tier 1 (basic patterns) through Tier 4 (expert-level vulnerabilities requiring deep protocol knowledge). These public samples form a likely exposed baseline, although their contamination status is unknown.
1The dataset can be explored at https://block-bench. github.io/base/#/inspect.
Temporal Contamination. DTC reconstructs 46 real-world DeFi exploits spanning 2016 to 2024, representing over $1.65 billion in documented losses. Notable incidents include The DAO ($60M, 2016), Nomad Bridge ($190M, 2022), and Curve Vyper ($70M, 2023). These attacks are widely documented, making training exposure plausible but unobservable. We transform them to test sensitivity to surface cues while preserving their semantics. Full details appear in §4.
Gold Standard. DGS comprises 106 professional security audit findings from Code4rena (Code4rena, 2025), Spearbit (Spearbit, 2025), and MixBytes (MixBytes, 2025), disclosed between September 2025 and March 2026. All samples were disclosed after August 2025. This places them after the latest published cutoff among models with provider-reported dates. For models without a precise reported cutoff, temporal separation cannot be confirmed from provider documentation alone. Recent disclosure reduces the opportunity for public exposure but does not establish that the samples were absent from model training. Gold Standard refers to the audit ground truth, not guaranteed absence from training data. The subset spans 11 vulnerability categories including logic errors (48%), arithmetic errors (9%), denial of service (9%), access control (8%), and signature replay (6%), with 24 high-severity and 82 medium-severity findings across 15 independent audit engagements. Thirty-eight samples include multi-file context for cross-contract vulnerability analysis.
Model cutoffs and training data are not fully disclosed (Dodge et al., 2021). We therefore report an association with public exposure, not proof of causation.
Negative Set. DNEG consists of 100 clean, production-deployed Solidity contracts from six of the most extensively audited protocols in decentralized finance: OpenZeppelin v5 (25 contracts), Uniswap v4 (14), Aave v3 (19), Compound v3 (14), Lido (13), and MakerDAO (15). These contracts have undergone multiple professional audits and have been deployed in production with no known vulnerabilities. The negative set enables measurement of false positive rates—a critical dimension absent from prior vulnerability detection benchmarks (Durieux et al., 2020). Contracts span five functional categories: DeFi core (40), utility (20), token implementations (19), infrastructure (10), and governance (7).
Together, these subsets assess detection under different levels of public exposure and changes in surface cues. The negative set grounds evaluation in practical auditing precision.
4Methodology
BlockBench asks whether a model can identify the causal mechanism of a vulnerability after familiar surface cues are removed. Figure 2 summarizes the pipeline: contracts are transformed, annotated with security functions, analyzed by detector models, and judged against ground truth.
Semantic Transformations. For each TC contract c, we construct variants T (c) that preserve vulnerability semantics while changing the cues models may rely on. Concretely, V(T (c)) = V(c) means the same exploit still works for the same underlying reason, even if names, comments, domain vocabulary, or surrounding code have changed. This mirrors standard NLP robustness tests: paraphrases, counterfactual edits, domain shifts, and distractor spans. The transformations and temporal split test performance under distribution shift.
We separate transformations that plausibly occur in production code, including sanitization, comment removal, and renamed forks, from stress tests designed to isolate reasoning failures, including domain shift, Trojan distractors, and misleading attestations. Differential is treated separately because it removes the vulnerability and creates a paired fixed contract. Figure 3 illustrates the original contract and five representative vulnerability-preserving transformations of the Nomad Bridge uninitialized root bypass, with changes highlighted. Full transformation definitions appear in Appendix H.
CodeActs. TDR measures whether a model detects a vulnerability, but not how. Two models may both flag a reentrancy bug: one explains the temporal dependency between an external call and a state update; the other simply flags the external call because it looks suspicious. Standard metrics score them identically. To separate causal reasoning from pattern recognition, we need to know which code segments each model attends to and what security role those segments play.
We introduce CodeActs, an annotation scheme that classifies code segments by their security function. The name draws on speech act theory (Austin, 1962; Searle, 1969): just as speech acts classify utterances by what they do (assert, request, promise) rather than their surface form, CodeActs classify code by what it does in the context of a vulnerability. Each segment receives one of four primary labels: ROOT_CAUSE (code that causally enables exploitation), PREREQ (necessary preconditions), DECOY (suspicious-looking but non-exploitable code, injected via Trojan transformations), and BENIGN (correctly implemented). Figure 4 illustrates this on a classic reentrancy: lines 3–4 constitute the ROOT_CAUSE because their order- ing (external call before state update) creates the exploit window. A model that flags line 3 in isolation without articulating the ordering dependency exhibits pattern matching; one that explains the causal chain demonstrates reasoning. The full 17- operation taxonomy and annotation format are in Appendix F.
Detection Protocol. We evaluate seven frontier models, one per major AI lab: Claude Opus 4.5 (Anthropic, 2025), GPT-5.2 (OpenAI, 2025), Gemini 3 Pro (Google DeepMind, 2025), DeepSeek v3.2 (DeepSeek-AI, 2025), Llama 4 Maverick (Meta AI, 2025), Grok 4 Fast (xAI, 2025), and Qwen3-Coder-Plus (Alibaba Cloud, 2025). DS and TC use a direct zero-shot auditor prompt requesting structured JSON with vulnerability type, location, root cause, attack scenario, and fix. GS additionally uses five prompt settings (Direct, Context, Chain-of-Thought, Naturalistic, Adversarial) to test whether protocol context or framing changes detection. All evaluations use temperature 0; prompt templates are in Appendix I.
Judging and Metrics. Detector outputs are evaluated against ground truth by independent LLM judges: GLM-4.7 (Zhipu AI, 2025), MIMO-v2- Flash (Xiaomi MiMo, 2025), and Mistral-Large- 3 (Mistral AI, 2025). A finding is a TAR- GET_MATCH when it identifies the documented root cause, location, and vulnerability type. Findings that partially match, identify valid issues beyond ground truth, or report fabricated vulnerabilities are classified separately (full classification scheme in Appendix J). Our primary metric is Target Detection Rate (TDR), the proportion of samples where the target vulnerability is correctly identified. We also report false positive rate on clean contracts and explanation-quality scores (RCIR, AVA, FSV) for matched findings (definitions in Appendix L). On GS, GLM-4.7 and MIMO-v2-Flash achieve κ = 0.943 with 97.7% agreement; expert validation and judge failure modes are detailed in Appendix J. The contract is the unit of analysis. DS supports intervals and tests. TC results are descriptive because variants share base contracts. Appendix L gives the procedures.
5Results
We evaluate seven frontier LLMs across 100 sampled DS contracts, all 46 TC contracts with 322 transformation variants, all 106 GS contracts, and 100 clean contracts, yielding over 5,000 model-sample evaluations. Unless noted otherwise, DS/TC scores use majority vote across three judges, while GS uses the two primary judges (GLM-4.7 and MIMO-v2-Flash; κ = 0.943).
5.1Performance on Exposed Samples Does Not Transfer
Table 3 shows the core transfer gap. Claude Opus 4.5 reaches 86.5% TDR on DS but drops to 50.9% across transformed TC variants; Gemini 3 Pro falls from 73.9% to 38.5%, with a 57% relative drop after protocol-identifying names are sanitized. Realistic perturbations (MinS/San/NoC) already reduce performance, while adversarial variants push all models toward a 15–40% floor. These failures occur despite preserved exploit semantics, suggesting dependence on recognition cues rather than invariant vulnerability reasoning.
5.2Detection Rises With Public Exposure
Table 4 and Figure 1 reveal a temporal gradient in GS performance. Samples with longer public exposure (GS-Orig, disclosed 7–8 months before evaluation) average 84.7% TDR, while more recently disclosed samples (GS-New, 2–6 months of exposure) average 0.8%. This pattern is consistent with the possibility that increasing public exposure (through blog posts, GitHub repositories, and educational materials) leads to greater representation of these contracts in model training data, which could contribute to inflated detection scores. However, because training data composition is not observable, the temporal comparison does not establish contamination as the causal mechanism. The combined GS-All score (27.6%) reflects this mixture of exposure levels rather than a stable estimate of model capability on recently disclosed data.
The expanded GS also shows a within-category performance gap. Models detect canonical DS reentrancy and arithmetic bugs, yet average 0% TDR on GS arithmetic errors (n = 10) and reentrancy (n = 4), where the same vulnerability classes appear through novel cross-contract callbacks, TWAP calculation errors, bit-mask corruption, and rounding-direction bugs (full breakdown in Appendix D). Familiar labels do not imply robust reasoning over unfamiliar mechanisms.
5.3Clean Contracts and Tool Baselines
Table 5 addresses two practical gaps in prior evaluation: false positives and comparison to classical analyzers. Clean audited contracts reveal a precision-recall trade-off: Gemini 3 Pro and Grok
4 Fast produce no false positives but have the lowest GS detection, while Claude Opus 4.5 has the best LLM balance (33.0% GS TDR, 9.0% FP). Traditional tools fail differently. Slither compiles only 6.6% of GS contracts because professional audit code often relies on unresolved imports and project-specific dependencies; Mythril processes all GS contracts but detects none, reflecting the mismatch between symbolic-execution patterns and the logic/arithmetic vulnerabilities common in GS. Thus, static tools and LLMs expose complementary limitations rather than a single dominant solution.
5.4Pattern Matching Without Causal Explanation
CodeActs isolates a final failure mode: models can locate security-relevant code without explaining why it is exploitable. On TC Trojan variants, Llama 4 Maverick achieves the highest ROOT_CAUSE match rate (60.9%), while its TDR reaches only 43.5%, a 17.4pp gap between pattern localization and causal explanation. This mirrors explanation-unfaithfulness findings in broader NLP reasoning work (Han et al., 2026): the model’s apparent evidence points to the right region, but the generated security argument does not establish the exploit mechanism. Full quality and CodeActs analyses are in Appendix D, with the SUI sensitivity analysis in Appendix K.
6Discussion
What Does the Degradation Reveal? The DS-to-GS gap may reflect both difficulty and sensitivity to familiar cues. Three findings make surface cue dependence a plausible part of the explanation. First, models detect canonical reentrancy and arithmetic vulnerabilities at high rates on DS but score 0% on GS instances of the same broad categories. The GS cases use less familiar implementations such as cross-contract callbacks, TWAP calculation errors, bit-mask corruption, and rounding errors. This result suggests sensitivity to familiar forms but does not control for complexity. Second, replacing protocol-identifying names with generic equivalents changes no control flow or data dependencies but reduces Gemini 3 Pro performance by 57%. This suggests reliance on identifiers such as “Nomad” or “Curve.” Third, detection rises with public exposure duration. This trend is consistent with possible contamination, but other factors may contribute. It does not prove causation.
Pattern Recognition and Causal Explanation Are Dissociable. The CodeActs analysis reveals that locating vulnerable code and explaining why it is exploitable are separate capabilities. On TC Trojan variants, Llama 4 Maverick identifies ROOT_CAUSE segments at the highest rate
(60.9%), while its TDR reaches only 43.5%. This dissociation matters for deployment: a model that highlights the right lines but provides a generic or incorrect security argument may give practitioners false confidence that a vulnerability has been understood when it has only been flagged. The gap parallels explanation unfaithfulness findings in mathematical reasoning (Han et al., 2026; Shen et al., 2026), but manifests distinctly in code, where vulnerability explanation requires articulating a multi-step causal chain (precondition, trigger, state violation, exploit) rather than a single derivation step. Models that shortcut this chain by matching on the trigger alone produce explanations that appear plausible but omit the causal structure needed for remediation.
The Trojan results add a counterintuitive dimension. DeepSeek v3.2 and Llama 4 Maverick achieve their highest TC scores (43.5%) on variants containing injected decoy code, the opposite of what causal reasoning would predict. One hypothesis is that longer inputs activate retrieval mechanisms that surface relevant training examples, a failure mode where detection improves for the wrong reason.
Implications Beyond Smart Contract Security. The contamination-controlled methodology introduced here applies to any code analysis task where ground truth is verifiable, semantic-preserving perturbations are definable, and temporal separation from training data is achievable. Candidate domains include malware analysis (known families as DS, obfuscated variants as TC, novel samples as GS), software bug detection (using bug-fix pairs from version control), and formal verification (where semantic equivalence is mathematically defined). The temporal gradient we observe also has methodological implications: any code benchmark whose solutions appear in public repositories or documentation should be treated as potentially contaminated, and evaluation claims based on such benchmarks benefit from contamination controls or temporal analysis. The gap between what models appear to detect and what they actually reason about suggests that the field needs evaluation frameworks that go beyond accuracy to probe the causal structure of model explanations.
Future Work. We highlight several promising directions for future work. First, BlockBench can cover other smart contract languages, such as Vyper, Rust, and Move. Second, it can evaluate models that actively use audit tools. Such models could compile contracts, run tests, inspect execution traces, and retrieve similar audit reports before producing a finding. Third, BlockBench can
include LLM-generated rewrites that reflect differences in developer coding style. Each rewrite must be verified to retain the same vulnerability before it is added to the benchmark. More broadly, exploit tests and execution traces can provide stronger evidence that structural transformations preserve the intended behavior. Future evaluations can also compare few-shot settings and different prompt lengths.
7Conclusion
BlockBench tests LLM vulnerability detection on likely exposed, transformed, recent, and clean contracts. Our study includes over 5,000 evaluations across seven frontier models and reveals three principal findings.
First, performance on likely-contaminated samples (86.5% TDR on DS) degrades to 27.6% on the full GS set and 0.8% on the newest GS slice. This gap suggests sensitivity to surface patterns and possible contamination, but it does not prove causation. Related performance gaps have been reported in other NLP benchmarks (Zhao et al., 2025; Zhang et al., 2024; Jain et al., 2025). Second, the pattern matching paradox demonstrates that models can locate vulnerable code without explaining why it is exploitable. On TC Trojan variants, Llama 4 Maverick achieves a 60.9% ROOT_CAUSE match rate but only 43.5% TDR, a 17.4pp gap that parallels explanation unfaithfulness in mathematical reasoning (Han et al., 2026). Third, empirical evidence of a temporal performance gradient, showing that detection rates on GS samples rise from 5.9% to 84.7% with increasing public exposure, consistent with temporal contamination patterns documented in prior work.
These findings carry implications beyond smart contract security. The contamination-controlled methodology—establishing contaminated baselines, applying semantic-preserving transformations, and comparing performance across samples with different disclosure dates—generalizes to any code understanding task where semantic equivalence is verifiable. We release BlockBench as a living benchmark to support evaluation of LLM code reasoning.
Limitations
BlockBench contains 362 vulnerable base contracts, 322 TC variants, and 100 clean contracts. The study yields over 5,000 evaluations across seven frontier models. While this represents a substantial evaluation, several limitations warrant discussion.
Scope. We evaluate exclusively Solidity smart contracts. While Solidity dominates the DeFi ecosystem (accounting for over 90% of deployed contracts by value locked), our findings may not generalize to other smart contract languages such as Vyper, Rust (Solana/Anchor), or Move. The vulnerability types and transformation strategies are specific to EVM-based architectures.
Evaluation Context. We test prompted models rather than complete audit systems. GS includes several prompts and multi-file context. In real audit settings, analysts draw on additional semantic context including protocol specifications, intended invariants, economic assumptions, and cross-protocol interactions. Production audits also use richer documentation and tools.
CodeActs Coverage. CodeActs annotations cover 46 samples across three transformation variants (MinimalSanitized, Trojan, Differential), enabling fine-grained analysis of the pattern matching paradox. Broader annotation coverage across the full dataset would strengthen generalizability of these findings.
Automated Evaluation. Our LLM judge ensemble (GLM-4.7, MIMO-v2-Flash, Mistral-Large) achieves Cohen’s κ=0.943 between primary judges with 97.7% agreement, indicating near-perfect reliability. However, automated evaluation may miss nuanced security reasoning. Disagreements concentrate in access control and logic error categories, where vulnerability boundaries are inherently ambiguous.
Contamination Claims. We sample findings by disclosure date to reduce the opportunity for public exposure, but we cannot guarantee that they were absent from model training. Provider cutoff information is incomplete and uses different definitions. Data mirrors and leaks may introduce recently disclosed material, and model providers do not disclose complete training data composition. Our temporal contamination evidence (§5.2) provides empirical support but not definitive proof.
Ethical Considerations
BlockBench poses dual-use risks: adversarial transformations demonstrate methods that could sup-
press vulnerability detection, while detailed vulnerability documentation may lower barriers for malicious actors. We justify public release on several grounds. First, adversarial robustness represents a fundamental requirement for security tools deployed in production. Second, all samples derive from already-disclosed vulnerabilities and public security audits; no novel exploit information is introduced. Third, responsible disclosure of evaluation limitations enables practitioners to calibrate trust in automated tools appropriately.
The primary positive impact of this work is enabling more informed deployment decisions for LLM-based security tools. The observed gaps may reflect memorization or surface familiarity (§5). This cautions against over-reliance on benchmark scores when evaluating security tools for production use. High scores on exposed samples may create false confidence. We recommend that practitioners treat LLM-based analysis as assistive rather than authoritative, maintaining mandatory expert review for all security-critical assessments.
We follow responsible NLP research practices by releasing all evaluation code, prompts, and data to enable reproducibility. We report confidence intervals and statistical tests throughout, and acknowledge limitations explicitly rather than overstating claims.
Acknowledgments
We thank the anonymous reviewers for their constructive feedback, which substantially improved this work.
References
- Alibaba Cloud. 2025. Qwen3-Coder-Plus model information.
- Anthropic. 2025. Claude Opus 4.5 model documentation.
- J. L. Austin. 1962. How to Do Things with Words. Oxford University Press, Oxford.
- Chainalysis. 2025. North Korea drives record $2 billion crypto theft year, pushing all-time total to $6.75 billion.
- Chong Chen, Jianzhong Su, Jiachi Chen, Yanlin Wang, Tingting Bi, Jianxing Yu, Yanli Wang, Xingwei Lin, Ting Chen, and Zibin Zheng. 2025. When ChatGPT meets smart contract vulnerability detection: How far are we? ACM Transactions on Software Engineering and Methodology, 34(4):1–30.
- Code4rena. 2025. Audit reports. Accessed 26 November 2025.
- DeepSeek-AI. 2025. DeepSeek-V3.2 public summary of training content.
- Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2020. Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE, pages 530–541.
- Josselin Feist, Gustavo Grieco, and Alex Groce. 2019. Slither: A static analysis framework for smart contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain, WETSEB, pages 8–15. IEEE.
- João F. Ferreira, Pedro Cruz, Thomas Durieux, and Rui Abreu. 2020. SmartBugs: A framework to analyze Solidity smart contracts. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE, pages 1349–1352.
- Asem Ghaleb and Karthik Pattabiraman. 2020. How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA, pages 415–427.
- Google DeepMind. 2025. Gemini 3 Pro model card. Technical report, Google DeepMind.
- Jacob Haimes, Cenny Wenner, Kunvar Thaman, Vas-sil Tashev, Clement Neo, Esben Kran, and Jason Schreiber. 2024. Benchmark inflation: Revealing LLM performance gaps using retro-holdouts. arXiv preprint arXiv:2410.09247.
- Yunseok Han, Yejoon Lee, and Jaeyoung Do. 2026. RFEval: Benchmarking reasoning faithfulness under counterfactual reasoning intervention in large reasoning models. In The Fourteenth International Conference on Learning Representations.
- S. M. Mostaq Hossain, Amani Altarawneh, and Jesse Roberts. 2025. Leveraging large language models and machine learning for smart contract vulnerability detection. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference, CCWC, pages 577–583. IEEE.
- Sihao Hu, Tiansheng Huang, Fatih İlhan, Selim Furkan Tekin, and Ling Liu. 2023. Large language model-powered smart contract vulnerability detection: New perspectives. In 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, TPS-ISA, pages 297–306. IEEE.
- Peter Ince, Jiangshan Yu, Joseph K. Liu, Xiaoning Du, and Xiapu Luo. 2026. GenDetect: Generative large language model usage in smart contract vulnerability detection. In Provable and Practical Security, ProvSec 2025, pages 426–445. Springer.
- Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations.
- Jiayuan Li, Lei Cui, Sen Zhao, Yun Yang, Lun Li, and Hongsong Zhu. 2025. CLeVeR: Multi-modal contrastive learning for vulnerability code representation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7940–7951, Vienna, Austria. Association for Computational Linguistics.
- Ye Liu, Yue Xue, Daoyuan Wu, Yuqiang Sun, Yi Li, Miaolei Shi, and Yang Liu. 2025. PropertyGPT: LLM-driven formal verification of smart contracts through retrieval-augmented property generation. In Proceedings of the 32nd Annual Network and Distributed System Security Symposium, NDSS 2025.
- Meta AI. 2025. Llama 4 Maverick model card.
- Mistral AI. 2025. Mistral Large 3.
- MixBytes. 2025. Smart contract security audits. Accessed 26 November 2025.
- Bernhard Mueller. 2017. Mythril: Security analysis tool for Ethereum smart contracts.
- OpenAI. 2025. GPT-5.2 model documentation.
- Md Rafiqul Islam Rabin, Nghi D. Q. Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2021. On the generalizability of neural program models with respect to semantic-preserving program transformations. Information and Software Technology, 135:106552.
- Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguistics.
- Eva Sánchez Salido, Julio Gonzalo, and Guillermo Marco. 2025. None of the others: A general technique to distinguish reasoning from memorization in multiple-choice LLM evaluation benchmarks. arXiv preprint arXiv:2502.12896.
- John R. Searle. 1969. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge, UK.
- Xu Shen, Song Wang, Zhen Tan, Laura Yao, Xinyu Zhao, Kaidi Xu, Xin Wang, and Tianlong Chen. 2026. FaithCoT-Bench: Benchmarking instance-level faithfulness of chain-of-thought reasoning. In The Fourteenth International Conference on Learning Representations.
- Spearbit. 2025. Public portfolio. Accessed 20 November 2025.
- Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, and Yingjiu Li. 2024a. LLM4Vuln: A unified evaluation framework for decoupling and enhancing LLMs’ vulnerability reasoning. arXiv preprint arXiv:2401.16185.
- Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, and Yang Liu. 2024b. GPTScan: Detecting logic vulnerabilities in smart contracts by combining GPT with program analysis. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, pages 1–13, Lisbon, Portugal. ACM.
- SunWeb3Sec. 2023. DeFiVulnLabs: To learn common smart contract vulnerabilities using Foundry. Accessed 11 November 2025.
- Trail of Bits. 2018. (not so) smart contracts: Examples of Solidity security issues. Accessed 20 March 2026; archived 24 February 2023.
- Petar Tsankov, Andrei Dan, Dana Drachsler-Cohen, Arthur Gervais, Florian Bünzli, and Martin Vechev. 2018. Securify: Practical security analysis of smart contracts. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 67–82.
- Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico. Association for Computational Linguistics.
- xAI. 2025. Grok 4 Fast model card.
- Xiaomi MiMo. 2025. MiMo-V2-Flash.
- Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024. A careful examination of large language model performance on grade school arithmetic. In Advances in Neural Information Processing Systems, volume 37, pages 46819–46836.
- Lizhe Zhang, Wentao Chen, Li Zhong, Letian Peng, Zilong Wang, and Jingbo Shang. 2025. Memorize or generalize? evaluating LLM code generation with code rewriting. arXiv preprint arXiv:2503.02296.
- Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, and Furu Wei. 2025. MMLU-CF: A contamination-free multi-task language understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13371–13391, Vienna, Austria. Association for Computational Linguistics.
- Zhipu AI. 2025. GLM-4.7.