Codee Analyzer vs. AI Coding Assistants: A Focus on Correctness in Fortran/C/C++

Authors: The Codee Team — Extracted from the paper “Comprehensive Evaluation of LLMs in HPC Code Performance Optimization” by B. Cui, T. Ramesh and K. Zhou from  George Mason University, and O. Hernandez from Oak Ridge National Laboratory [https://doi.org/10.1145/3750720.3757280].

When it comes to modernizing and optimizing Fortran/C/C++ code, performance and correctness must advance together. Faster modeling and simulations are critical, but in scientific computing, a single wrong answer can invalidate entire weeks of work. The authors of “Comprehensive Evaluation of LLMs in HPC Code Performance Optimization” prove this point, and in this post we quickly summarize their findings. The authors benchmarked Codee against four leading LLMs and found that only Codee consistently produced correct, compilable, and performant code.

The authors of the paper show that Large Language Models (LLMs), such as OpenAI o1, Llama-3.2, Claude-3.5, and HPC-Coder, can suggest useful optimizations and sometimes produce important speedups. They demonstrate cases where LLMs find algorithmic improvements or loop transformations that help performance. However, the authors also emphasize that LLMs frequently fail: “OpenAI o1, Llama-3.2, Claude-3.5, and HPC-Coder yield codes that are partially completed or yield incorrect results”. In many cases, generated code “yields incorrect results, fails to compile, or encounters any runtime errors”. These failures are frequent enough to be a real burden for development teams, underscoring the need for robust, deterministic tools like Codee to ensure the correctness of LLM-generated code.

I ”Codee’s strength lies in automating code optimization with guaranteed correctness; a level of assurance that current LLM-based tools cannot provide.”
— Keren Zhou, PhD | George Mason University

The authors further show that Codee, a compiler-grade static-analysis tool, takes a completely different approach: “Due to heuristics and compiler-based code analysis by Codee, we observed no errors in the optimized code”. Because Codee uses deterministic, compiler-grade analysis, and a curated catalog of performance checks, it consistently generates correct suggestions and deterministic auto-fixes. The authors report that Codee produced correct optimizations across their benchmarks, removing the need for the heavy verification step that LLM suggestions typically require. Remarkably, the paper also shows that Codee’s optimizations achieved measurable speedups on computational kernels, comparable to those obtained with LLMs, confirming that both approaches are capable of producing fast code when applied to HPC workloads.

		Codee	OpenAI o1	Llama-3.2	Claude-3.5	HPC-Coder
Correct		60	54	39	39	14
Incorrect		0	6	21	21	46
	Compilation errors	0	0	1	2	10
	LLM failed to generate the code	0	2	10	7	24
	Incorrect results	0	3	10	11	8
	LLM failed to follow instructions	0	1	0	1	4

The authors also show that the time to apply optimizations differs significantly. This time includes activities such as prompting the model, waiting for responses, implementing necessary code changes, and validating the results. According to their results, Codee required around 2 to 4 minutes per benchmark, while LLMs often took between 2 and 20 minutes depending on the model and experiment. This means that the time LLMs might save during initial code generation is often lost again in verification and debugging. As the authors put it, the verification burden minimizes much of the apparent gain and “Codee is generally faster than LLMs across experiments”.

The authors note that there are key differences in usability as well. Codee and its companion tools like the Codee Formatter are fast and predictable. Codee Formatter alone can handle hundreds of thousands of lines of code in seconds, running locally without sending code to external servers. This makes it practical to integrate Codee into CI pipelines or everyday development workflows. In contrast, LLM-based solutions often depend on cloud inference, introduce privacy concerns, and add unpredictable latency. As the authors point out: “since LLMs (except HPC-Coder) are invoked via remote servers and the models used are large, the process of generating responses can be slow, especially when multiple sequential optimizations are applied”. They report that even local deployment, as with HPC-Coder, “observed high inference latency, which may be due to a poorly optimized inference engine and limitations of the GPU”. These results show that working with LLMs is both slow and unpredictable, unlike Codee which is fast and consistent.

In this paper, the authors recognize ”the strengths of LLMs in understanding human instructions and performing automated code transformations” but, while LLMs show flashes of creativity and can deliver big speedups in some cases, they are also “unpredictable, error-prone, and demand rigorous verification”. In contrast, state-of-the-art compiler based tools like Codee yield “deterministic, correct optimizations”.

In the future, the authors aim to leverage the “potential for integrating LLMs with correctness tools” through an agent system powered by Codee Formatter and Codee Analyzer that performs source code formatting, static code analysis and LLM-generated comments to describe source code written in Fortran/C/C++.

I ”Relying on LLMs alone is insufficient: in our benchmarks, Codee’s compiler-based analysis ‘generated parallel correct code’ while LLMs often produced ‘partially completed or incorrect’ code, yet we also ‘recognize the potential for integrating LLMs with traditional, correctness-oriented tools, to get the best of both worlds.”
— Oscar Hernandez, PhD | Oak Ridge National Laboratory