Testing destructive changes in isolated environments

[jjl]

One of the things I'm not satisfied about with the current test suite is the fact that we only test with the one configuration - we proclaim that you can choose hstore as the default, and while we test roundtripping through hstore, we don't actually have the ability to test whether when it's set as the default, it actually does what it says on the tin.

Because of the protocolly nature of this, I'm not entirely sure how to go about this. My current estimate is that we'll need to spin up isolated JVMs in the way that boot does, for which there are a couple of libraries of which I'm aware. Any other ideas I'd be interested in hearing, but I'd definitely like better test coverage in the wake of a conversation I had on github yesterday where they were experiencing some odd bugs and their "fix" was to add a catch-all case that just passed the input through unmodified. There are clearly some holes in our abstraction, and I want to flush them out.

[JuneKelly]

Sounds good.

Is the basic problem that because of protocols, once we've set up mpg in a certain way, inside a JVM instance, we can't adjust it and test another configuration?

Perhaps a top-level set of test scripts which fire up separate test processes to run isolated tests? then the top-level test runner reports back on passes/failures?

[jjl]

Yes, bang on with your description there. Going for separate scripts has the advantage of ensuring that the JVM will never get OOMkilled (this has happened to me on travis before...), but it means multiplying the testing time by the number of cases we want to test.

It would be quicker if we were to use boot's pod isolation mechanism (my preference, you'll notice all my projects use boot instead of leiningen :) ) or clojail or something. Then again, one might say that given the bulk of the time in travis is setting up the environment for a miniscule test runtime, maybe it's not so bad after all.

But boot is still totally awesome and we should totally use it :)

[JuneKelly]

I'd be fine with switching to boot or whatever, if it makes the job easier.

(not a huge fan of leiningen, personally)

[jjl]

See my 'boot' branch, where the tests already pass (although they won't on travis because i haven't doctored the travis file). Don't merge it in yet, I want to play with the pods stuff first.