Instructor:
Wu, Z., Anderson, C., Guha, A.: The Best Programming Language for Tokenmaxxing
Anderson, M.: Which Programming Languages are Most Token Efficient
Does the choice of programming language affect AI coding costs?
AI coding agents work iteratively (each iteration consumes tokens)
4 languages
5 models
Problems
Language changes token cost
Median output token consumption, multipliers relative to Python
Agents continue even when they have a working solution that passes all tests
Results (examples)
Gemma
GLM
Sonnet
Proprietary models tend to submit solution more directly
Problem is often syntax, not algorithms
Languages with little (public) examples fail more often in compilation (stuck)
When compilation fails, next iteration has no additional test feedback
Planning in comments: agents sometimes develop algorithms inside comments before code (comments become temporary workspace, perhaps to avoid information dropping from context)
Solve in Python, then translate: agents sometimes first use a popular language, then attempt to translate (less idiomatic solution in target language)
Second-guessing: agents often question solutions that are functionally correct (are cases missing, efficiency improvement, more elegant implementation?)
Testing and self-examination: agents write additional test cases
Token usage is partly a language problem and partly an agent-control problem
Instead of evaluating the final program, the paper:
Coding agents and LLMs can help solve programming language problems
Can Large Language Models Autoformalize Kinematics?
LLM-Powered Automatic Theorem Proving and Synthesis for Hybrid Games
Read papers, what processes are the agents following?
Read the prompts in the appendix, what elements and themes stick out?
Practical implications when developing agentic solvers:
The best programming language for an AI coding agent is not necessarily the one that produces the shortest code. It is the one in which the agent can reliably reach a correct solution with the least unnecessary reasoning and revision.
Scastie integration for live coding in slides
highlight.js consistent with Scastie (Solarized light theme)
Hidden modal dialogs
Copy code
Scastie embedded Scala editor
Modal dialogs