That's really interesting, indeed I can reproduce this by changing the comment. I also managed to get correct output for this sample by renaming the function.

Is it, though? The major selling point of coding LLMs is that you can use natural language to describe what you want. If minor changes to wording - the ones that would not make any difference with a human - can result in drastically worse results, that feels problematic for real-world scenarios.

The model is small, so it has weaker semantics.

I get that. But they are explicitly comparing it to Codex themselves.

The criticism stands if you have to continue to rewrite your "prompt" until you can coax out the correct desired output.

I agree. Maybe it interpreted it as return the numbers that are more than 10 in the given array of even numbers.

For example, if the instruction says "return person objects that are at least 20 years old", it might be more reasonable to generate:

array.filter(item => item.age >= 20)

as oppose to

array.filter(item => (item instanceof Person) && (item.age >= 20))