Thoughts on unit tests

[AlexDaniel]

So as I was working on tests there were some things that I figured out, but as the time went on it seems that this work won't be quite finished in a way that was intended. But here are my thoughts on how things can work.

As for the framework, http://vunit.github.io/ can be used. Seems to be a solid framework, although a bit simplistic in my opinion. Anyway, a simple test file can then look like this (a real working example for xbar):

`include "vunit_defines.svh"

module xbar_tb;
   parameter DATA_WIDTH = 8;

   logic [DATA_WIDTH-1:0] North_in = 42;
   logic [DATA_WIDTH-1:0] East_in  = 250;
   logic [DATA_WIDTH-1:0] West_in  = 1;
   logic [DATA_WIDTH-1:0] South_in = 27;
   logic [DATA_WIDTH-1:0] Local_in = 182;
   logic [4:0]            sel      = 0;
   wire  [DATA_WIDTH-1:0] Data_out;

   xbar #(.DATA_WIDTH(8)) xbar(.*);
   `TEST_SUITE begin

      `TEST_CASE("Select Local") begin
         sel = 'b00001; #1ns `CHECK_EQUAL(Data_out, Local_in);
      end
      `TEST_CASE("Select South") begin
         sel = 'b00010; #1ns `CHECK_EQUAL(Data_out, South_in);
      end
      `TEST_CASE("Select West") begin
         sel = 'b00100; #1ns `CHECK_EQUAL(Data_out, West_in);
      end
      `TEST_CASE("Select East") begin
         sel = 'b01000; #1ns `CHECK_EQUAL(Data_out, East_in);
      end
      `TEST_CASE("Select North") begin
         sel = 'b10000; #1ns `CHECK_EQUAL(Data_out, North_in);
      end
      `TEST_CASE("Other sel inputs besides proper one-hot values") begin
         for (integer i = 0; i < 1 << $size(sel); i++) begin
            sel = i; #1ns;
            // skip almost all one-hot values
            if (i == 'b1 || i == 'b10 || i == 'b100 || i == 'b1000) begin
               automatic logic not_north = Data_out != North_in;
              `CHECK_EQUAL(not_north, 1);
            end else begin
              `CHECK_EQUAL(Data_out, North_in); // defaults to North
            end
         end
      end
   end
endmodule

One issue that I noticed at the time was that each TEST_CASE takes quite some time, so it seems that you'd want to minimize the number of test cases in general. That's a bit weird and I haven't really found any justification for it, and to me it seems like the slowness comes from the startup time of modelsim or something. The natural thing to try in cases like this is running the same stuff with a free/open-source simulator, but it turns out that VUnit does not support any existing free/open-source simulator. Recently there was some progress on it and I was thinking about jumping back to the task, but the thing is still not finished. See this ticket for progress on that part: VUnit/vunit#188

Anyway, even if it's a second or two between test cases, it is alright. But then I was stuck with the organizational issues, like:

• Should these tests be written for Verilog or VHDL designs? In theory, it should be possible to swap the designs underneath the tests and it will work exactly the same, but that would complicate running those tests… which brings us to the next point
• Would be nice to have some sort of Makefile to handle all these things. But given that everything is extremely configurable (just look at the options for simulate.py!), the Makefile has to be kind of transparent in terms of additional options. Which means that…
• Perhaps instead of using makefiles, a short script can be written in any other language. Similarly to what simulate.py does, actually. And that should be fine…
• Except that when I continued writing tests, I started seeing more things that could be improved in the designs themselves. For example, it was easier to write arbiter_inout_tb.sv file instead of doing it twice, and that's about when I started creating issues like https://github.com/Project-Bonfire/Bonfire/issues/23.

And that's when my mind bogged down with the organizational complexity of the project and how things would rather be improved by cleaning stuff up rather than adding even more. :)

Anyway, hope that this is somewhat useful and the next person stumbling on this would have more info to proceed correctly. Although, I wasn't able to keep up with the changes so maybe some of this is done already.