ZMQStream mocking to reduce flakiness

[fahhem]

https://github.com/chainreactionmfg/cara/blob/master/tests/cara_pseud_test.py

To reduce flakiness, I took the mocking route and mocked out ZMQStream and ZMQContext. The downside was that I had to implement the ROUTER behavior in the mocked ZMQStream, but now I don't even open a port to test my code and so I can run it a hundred times (in parallel) without any failures. Of course, 100 parallel py.test instances take 40 seconds to return, but none fail now when at least 20% failed before.

Another benefit is I can check the packets that go by for the correct sender, receiver, and data. If you're interested, I can look into putting the fixture into pseud or into a separate shared project like pytest-zmq?

[ticosax]

I like the idea of pytest-zmq or (pytest-pyzmq), but I'm wondering if I'd used mocking from the beginning I would probably never be able to use pyzmq correctly. My concern is that I'm trying to stay as close as possible to prod environment in my tests. It brings me better confidence that it will work. When I introduced curve into pseud, I certainly succeed because I used real sockets.
So I think It might be a good idea to mock sockets for some tests, that but not for all of them. We can think of splitting tests into unit tests and integration tests categories where we allow mocking only in unit tests category.

[fahhem]

Yes, there should be a separation. Unit tests should be plenty, but fast,
since they test the logic in question, while integration tests should be
fewer but mock as little as possible. Minimal mocking should be allowed as
some things are very difficult to test otherwise, such as reactions to
certain dates require mocking datetime.

On Wed, Feb 11, 2015 at 1:09 AM, Nicolas Delaby notifications@github.com
wrote:

I like the idea of pytest-zmq or (pytest-pyzmq), but I'm wondering if I'd
used mocking from the beginning I would probably never be able to use pyzmq
correctly. My concern is that I'm trying to stay as close as possible to
prod environment in my tests. It brings me better confidence that it will
work. When I introduced curve into pseud, I certainly succeed because I
used real sockets.
So I think It might be a good idea to mock sockets for some tests, that
but not for all of them. We can think of splitting tests into unit tests
and integration tests categories where we allow mocking only in unit tests
category.

—
Reply to this email directly or view it on GitHub
#81 (comment).