Replication quesons

[jkdufair]

We are trying to replicate some of the results in the paper. There are a few areas discussed in the paper that seem not to be available in the codebase. Maybe there is a newer version that needs to be pushed?

• The paper suggests that the synthesized code is automatically evaluated after it has been generated. I don't see a means to do this in this repo
• There does not seem to be a mechanism in this codebase for generating new problems
• It appears that the context_array items "using sympy" and "using simulations" are not actually used

We don't mind implementing these and other possible omissions if need be, but we'd love to be able to replicate (and eventually expand on) the results if perhaps the code exists but just hasn't been pushed.

Thank you,
Jason Dufair

[Aditya-vardhan13]

Hello @jkdufair have you worked on it, if please can you share it with us, we are trying to develop a similar product.