Evaluating Language Models on Cross-Language Code Functional Equivalence

We introduce PolyHuman, the first dataset for evaluating cross-language code functional equivalence on human-written code (C++/Java/Python, 5,035 problems from CodeContests, 15 sub-tasks), addressing a gap left by prior benchmarks that focus on a single language or synthetically transformed code. A systematic evaluation of 9 open- and closed-source models shows accuracy drops from 0.998 on synthetic code to 0.84 on human-written code; prediction bias is driven by the model rather than the language, and as problem difficulty rises, models increasingly misclassify non-equivalent code as equivalent. Manual analysis of 81 misjudged cases reveals a three-tier failure hierarchy: Knowledge Failures (21%), Abstraction-Level Reasoning Failures (53%), and dataset-labeling issues (19%), motivating complementary multi-model strategies over single-model use.