CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

1 code implementation • 31 Mar 2024 • Yiqing Xie, Alex Xie, Divyanshu Sheth, PengFei Liu, Daniel Fried, Carolyn Rose

To demonstrate the complexity and solvability of examples in Exec-CSN, we present a human study demonstrating that 81. 3% of the examples can be solved by humans and 61% are rated as "requires effort to solve".

Code Generation Language Modelling +1

It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk

1 code implementation • 2 Oct 2023 • Amanda Bertsch, Alex Xie, Graham Neubig, Matthew R. Gormley

Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lowest risk (expected error) among multiple candidates.