EVM Compiler Integration

[fubuloubu]

The ethPM spec would be an ideal format for creating the output interface specification for our EVM compilers. Solidity already has a similar JSON output format that has informed the ethPM spec partially, but we have a chance to optimize a common format so that each EVM language compiler stores it's output compiled assets in a format compatible with package managers (to encourage re-use of common contracts using infrastructure instead of recreating common patterns), as well as enabling ecosystem tools requiring compiled assets (testing tools, deployment managers, simulations, formal verifiers, etc.) to make use of this common format for their own purposes.

Optionally, the ethPM spec may be able to formalize the compiler input interface so that any wrappers can leverage this format in order to delegate compilation processes in a common way. An example of this would be how populus would leverage py-solc or vyper's python package directly in order to pass a JSON object representing the input compiler settings and source code to these packages and add to the output.

Came out of discussions with @pipermerriam and @axic

[pipermerriam]

Here is a dump of my thoughts in no specific order or structure.

Overlapping but disparate needs

The needs of package management and the needs of consumers of compilation output are likely heavily overlapping, but not identical needs. I would posture that the compilation output needs are largely a superset of the package management needs.

One example to highlight this would be things like AST. The abstract syntax tree is not something that needs to be present for package management, however it will be an important element for other software which consumes compilation output.

Compilation will normally be part of consuming a package.

Software which consumes packages will likely implement their own compilation step for verification purposes. Thus, were compilation to produce a "package", it's still likely that there would be yet another compilation step which occurs as part of a 3rd party consuming that package.

[pipermerriam]

One thing that is hard from the tooling side is creating minimal packages from compilation output. Consider the following source tree and contracts. (please ignore specific language semantics about inheritance or imports).

• sources/A.file
  • defines contract A
• sources/B.file
  • imports from A.file
  • defines contrat B(A)

Suppose I compile all of this which is common for a framework.

If I wanted to package up contract A, I should only wish to include a minimal source tree necessary to recompile these, so only A.file. If I want to package up contract B, then I need both A.file and B.file.

One thing that a compiler can do easier than a framework is identifying the minimal source tree necessary to compile a specific contract. If compilers generated the absolute minimal package for each contract in their compiler output it would make packaging up the compilation output easier in this way.

[fubuloubu]

So maybe compiler --generate-package contracts/*.file would generate that minimal package (removing extraneous functionality like AST and bytecode->sourcecode mappings) for upload to a package management service?

[pipermerriam]

cc @njgheorghita and @davesque you guys should probably at least be reading along if you aren't already. I'm guessing that we'll be working more and more heavily with the vyper compiler and this is a nice chance to inform compiler output in such a way that makes the tooling we build easier to integrate.

[fubuloubu]

So, maybe expanding a bit on that, the process might look like:

1. compiler contracts/**/*.file > contracts.test.json then run all my testing and simulations using contracts.test.json as an input, which contains all relevant output the compiler would need to provide to do functional/structural/coverage testing, fuzzing, economic and security simulations.
2. ^ Might need to be in multiple stages...
3. compiler contracts/**/*.file > contracts.package.json which contains my minimal package that I would upload to package management service, and/or my front-end UI developer would use to develop the front end with (packaging it with the app or loading it from the package management service)

What do you think of that guys?

[fubuloubu]

I was thinking that populus could manage the package outputs under the hood, with a plugin API our compilers, testing frameworks, coverage testing utilities, fuzzers, model checkers, etc. can link into in order to easily add/remove stages to the process, with the last stage being uploading the package.

Each plugin would use a standard API to register with populus so they can get the input they would need to conduct their procedures, and perhaps some method of running them in parallel

[pipermerriam]

I don't think there's any need to differentiate between testing, in fact, I consider this an anti-pattern. Any testing tools should be able to get what they need from the package itself or the framework should inject the extra information it needs. I'd suggest not adding anything special in this realm for testing.

[fubuloubu]

Under that scheme the testing modules would have to re-compile/re-parse the source code in order to determine the information. This would need to occur in each tool individually (unless a sharing scheme is devised), which increases overhead in the process.

I agree with you that the minimal required members should focus only the compiler outputs (abi, bytecode, runtime) and that additional compiler information should be optional in the final spec. I do however think that describing what structure the additions might take should be described in the spec, so that extensibility can be built in a bit.