[Re]Design ASCII benchmark file format

[rrnewton]

Whether or not this is a good idea is debatable.

• Con: no config file format is as expressive as actual Haskell code. Exposing a library rather than an executable is a tried and true technique.
• Pro: hsbencher-fusion and friends pull in a huge number of dependencies. It's a pain to build them inside Travis or Jenkins jobs, and its also an extra source of failures. It would be easy from a system config perspective to just have an "hsbencher" executable on the evaluation platforms, which reads in a benchmark script from a file.
• Pro: These aren't mutually exclusive -- providing an executable does not preclude falling back to using the library directly when it becomes necessary.
• Pro: an executable could actually run some Haskell code (with GHCI), and thus offer more expressive configuration options WITHOUT actually linking hsbencher and friends.
• Pro: if the benchlist format is good enough, it could make hsbencher accessible to folks who don't write Haskell code.

Design Options:

Designing a format is an ugly process. The old ASCII format for benchmark lists was terribly limited. Here are some early ideas of what might work in the future:

• .cabal style sections, possibly with some blocks being indentation-quoted blocks of other languages, such as executable shell scripts
  • header delimited blocks would make up the file
  • one block type would be benchmarks that lists where the build targets are, and connects benchmarks to parameter spaces
  • another top-level block/declaration would be vary -- named declarations that would set up parameter spaces.
• Yaml format -- Yaml might be a better replacement for the above. I don't know YAML well enough, but it seems to work for travis to include shell script chunks within .travis.yml without huge quotation/anti-quotation headaches.

Declarative vs operational

As with all configuration file design there's a tradeoff between having passive data and an active script. An example challenge problem in the HSBencher case is varying the number of threads. We don't set the number of threads to a fixed value, e.g. [1,2..16]. Rather, it's based on reading the number of cores on the current machine, which requires running code!

This is why the hsbencher benchmark config will probably need to include some kind of script component. I think we basically need small script chunks that are evaluated to a string and bound to a top-level variable. These can be arbitrary "shebang" scripts, so they could be Haskell, bash, python whatever.

Thus, a typical script that varies the number of threads will have a mostly declarative config for where benchmarks live and what parameters to vary. But the bit that computes the domain for THREADS will be a small script.

[rrnewton]

@eholk -- if we proceed with this I'd like some feedback from someone who has tried to use hsbencher but does not write Haskell code all the time. The ideal would be to require little to no Haskell knowledge.