Revamp testing infrastructure

[martijnbastiaan]

Back when I introduced clash-protocols and Df, I realized that many of the circuits I was writing had a pure equivalent. For example, fifo :: Circuit (Df dom a) (Df dom a) ~ id. I figured I'd generalize this construct into propWithModel and friends. For the places where it has worked, it has been pretty great, but it has a bunch of problems:

• It assumes quite a bit about the structure of your circuit.
• It has complicated type class machinery (Test / Simulate / etc.)
• It tries to generalize all possible protocols into a single concept (e.g., it assumes a single backpressure channel, Why there is no Drivable instance for AXI)
• It doesn't work all that well with Hedgehog, relying on a fork of it.
• It evaluates channels one-by-one, leading to massive memory use for longer running simulations.

So all in all, we can do better. For round two, I think running the circuit should be separate from building the test bench. For example, a FIFO test harness would look like:

dut ::
  DriverOpts ->
  ExpectOpts ->
  [a] ->
  Circuit () (Vec n DriverSamples)
dut driverOpts expectOpts dat = circuit $ do
  fifoIn <- driver clk rst driverOpts dat
  fifoOut <- fifo clk rst -< fifoIn
  results <- expect "fifo" clk rst expectOpts dat -< fifoOut
  idC -< [results]
 where
  clk = clockGen @XilinxSystem
  rst = noReset

The Hedgehog test would look like:

prop_dut :: Property
prop_dut = property $ do
  driverOpts <- forAll genDriverOpts
  expectOpts <- forAll genExpectOpts
  dat <- forAll genDat
  testResult <- liftIO $
    runTest
      (testOptions{generateVcd=True})
      (dut driverOpts expectOpts dat)

where

runTest ::
  (1 <= n, KnownNat n) =>
  TestOptions ->
  Circuit () (Vec n DriverSamples) ->
  IO TestResult

-- | A 'DriverSamples' is a collection of samples and their timestamps, as well as the
-- status of the driver. It is used to record traces of signals in a simulation.
--
-- TODO: Support different active edges.
data DriverSamples = DriverSamples
  { name :: Text
  -- ^ Name of the driver
  , traces :: [Samples]
  -- ^ (Optionally) traces associated with this driver
  , period :: Maybe Femtoseconds
  -- ^ Period of the clock. Set if dealing with classical clocks (i.e., non-dynamic).
  , statuses :: Infinite (Femtoseconds, Status)
  -- ^ Statuses with their timestamps
  }

data Sample = Sample
  { value :: !Natural
  , mask :: !Natural
  }
  deriving (Eq, Show)

data Samples = Samples
  { name :: Text
  -- ^ Name of the signal
  , bitSize :: Natural
  -- ^ Size of the signal element in bits
  , period :: Maybe Femtoseconds
  -- ^ Period of the clock. Set if dealing with classical clocks (i.e., non-dynamic).
  , samples :: Infinite (Femtoseconds, Sample)
  -- ^ Samples with their timestamps
  }

data Status
  = -- | Do not shut down simulation.
    PreventStop
  | -- | Error condition. Driver is free to go to any other state after
    --  asserting this.
    Error Text
  | -- | Shutting down simulation is OK. Driver may still be monitoring
    -- for errors.
    StopOk
  deriving (Eq, Show)

Though there is now more boilerplate to construct a test, it doesn't suffer from any of the issues propWithModel and friends suffer from:

• It does not assume anything about the design under test
• It does not rely on any type classes to be implemented
• It does not try to generalize protocols, instead relying on protocol-specific drivers and expect circuits.
• It moves all "fuzzing" to a user-written test, making it work well with Hedgehog again.
• It can step through the simulation in constant memory. All Streamss contain timestamps of their samples, allowing runTest to evaluate all samples in time-based order.

As a bonus, runTest can also access traces, paving the way for generating waveforms of failed tests. As a cherry-on-top, circuits can export traces too making test failures much easier to debug.

TODO

There is already a bunch of code in martijn/revamp-test-infrastructure.

• Implement drivers and expecters for Df
• Replace propWithModel tests in testsuite of clash-protocols with structure mentioned here.

Future

• Expand support to all protocols in clash-protocols
• Implement "monitors": circuits that are only there for waveform purposes.
• Extend with scoreboards: tracking functional coverage across multiple hedgehog tests
• Optimize evaluation strategy: we could group signals per clock (domain), dropping the need to calculate periods for every sample/trace.
• Explore whether Samples should be a tree. That way we could collect a bunch of traces in the design and have waveform viewers reflect the structure of the design.