Performance seems to be limited by Cassandra write times

[nsoft]

threw 33 million 1k docs at a fairly simple ingest and the threads for our steps don't seem to be doing much

There is no thread that seems to be pegged, but several cassandra related threads are busier than any of ours:

One possible place to look for performance is to avoid the index on the status column

[nsoft]

This is in no way disk bound...

----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
 12   2  85   0   0|  15M  280k|5011B 7419B|   0     0 |  85k  160k
  8   2  89   0   0|1223k   12k|6385B 7844B|   0     0 |  81k  153k
  8   2  90   0   0| 264k    0 |1238B  132B|   0     0 |  78k  148k
  8   2  89   0   0|  16k   69M|1490B  264B|   0     0 |  64k  112k
  8   2  90   0   0|  11M  592k|1238B  132B|   0     0 |  61k  114k
  5   2  92   0   0|   0  4099B|1540B  264B|   0     0 |  46k   86k
  8   2  89   0   0|  22M    0 |1770B  517B|   0     0 |  87k  164k
  9   2  88   0   0| 308k 8192B|2735B  638B|   0     0 |  82k  155k
  7   2  90   0   0|  68k   14M|1962B 1392B|   0     0 |  78k  149k
  8   2  89   0   0|2314k 1277k|  88k 7438B|   0     0 |  68k  127k
  7   2  90   0   0|4636k   96k|  82k 5367B|   0     0 |  66k  124k
  7   2  91   0   0|  32k 4096B|  11k 6511B|   0     0 |  44k   81k
  9   2  88   0   0|1293k   21M|1365B  198B|   0     0 |  83k  156k
  8   2  89   0   0|2321k   18M|1989B  264B|   0     0 |  93k  156k
 10   2  86   0   0|  13M   21M|2675B  414B|   0     0 |  77k  144k
 11   4  84   0   0| 230M  313M|3883B 2534B|   0     0 |  77k  140k
 14   2  84   0   0|1505k   12M|3439B 3169B|   0     0 |  76k  117k
  9   1  89   0   0|   0    76k|8863B 7833B|   0     0 |  47k   88k
 13   2  84   0   0|3000k    0 |5047B 3674B|   0     0 |  78k  145k
 13   2  84   0   0|2816k  428k|1238B  132B|   0     0 |  86k  161k
 12   2  86   0   0|3512k   12M|1956B  759B|   0     0 |  87k  147k
 11   2  85   0   0|5472k 4349k|3373B 2134B|   0     0 |  78k  142k
 11   2  86   0   0|5307k 8197B|2364B  198B|   0     0 |  60k  110k
 10   2  89   0   0|  96k   54M|6594B 6039B|   0     0 |  49k   92k
 11   2  87   0   0| 972k 1492k|  83k 5886B|   0     0 |  64k  120k
 12   2  85   0   0|   0   236k|8340B 5196B|   0     0 |  89k  166k
 12   2  85   0   0|5287k    0 | 101k 9195B|   0     0 |  81k  152k
 11   3  85   0   0|3652k 8193B|4677B 2939B|   0     0 |  77k  144k
 12   2  86   0   0|5414k   14M|1238B  132B|   0     0 |  58k  109k
  9   1  89   0   0| 168k  496k|1238B  132B|   0     0 |  58k  107k

[nsoft]

Supportive of the idea that the index is limiting us:

2023-04-21T18:35:42,563 WARN CompactionExecutor:24 org.apache.cassandra.io.sstable.format.big.BigTableWriter Writing large partition jj_8fea9a05291baa6da40e00c3475582d3/jj_output_step_status.jj_ft_idx_step_status:BATCHED (1.004GiB) to sstable /home/gus/.jj/wikidocs/cassandra/data/data/jj_8fea9a05291baa6da40e00c3475582d3/jj_output_step_status-621ce490dfec11ed849fa590da16ca1a/.jj_ft_idx_step_status/nb-259-big-Data.db