sayuri with the TensorRT backend

[MAOmao000]

We evaluated sayuri with the TensorRT backend, and found a speedup of over 1.5x when using fp16.

Since it uses C++17 features, a C++ version of 17 or higher is required.

https://github.com/MAOmao000/Sayuri-TensorRT

I hope this will be helpful for future development.

[CGLemon]

Looks great! I think TensorRT could be a suitable new backend. But I’m away from home these days, so I’ll review the TensorRT code later.

[CGLemon]

After testing, the performance has indeed become much faster, and there are no issues with compilation. However, the results of network inference seem to be incorrect. The errors appear to be minor and might be related to details such as bias addition or SE module. I will investigate the exact cause. In addition, I would like to ask:

1. I noticed that some functions and files under the neural/cuda path appear to have been removed. Could you explain the reason? Was it because they were unused, or were there other issues?
2. Is most of the code directly ported from KataGo?

[MAOmao000]

The activation was missing from some of the value header processing, so the following source code was modified.

• cuda_forward_pipe.cc
• version.cc

Some of the source code in the cuda folder was removed because the cuda backend processing has been replaced with TensorRT backend processing. To coexist with the cuda backend processing, add a folder for the TensorRT backend separately from the cuda folder and copy this source code there.
Also, cuda_sha2.cc and cuda_sha2.h were used as they were in KataGo.
cuda_forward_pipe.cc and forward_pipe.h were newly created based on KataGo's trtbackend.cpp.
DENABLE_FP16 and RTX GPU

• sayuri-cuda: black win 2, white win 1, total 3, 2559 n/s
• sayuri-TensorRT: black win 4, white win 3, total 7, 4821 n/s

[CGLemon]

t still looks slightly different. Below is a comparison of the output results, and some outputs differ by about 1%. I’m not sure if this is due to TensorRT itself. In any case, an error of around 1% is already acceptable for self-play.

Sayuri-cuda> sayuri-raw_nn avg
= stm winrate: 0.507551
wdl winrate: 0.509208
win probability: 0.482262
draw probability: 0.0538904
loss probability: 0.463847
final score: 0.0707824
q error: 6.50347e-05
score error: 0.00196111
probabilities: 
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.001973  0.000213  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000641  0.000000  0.000000  0.000668  0.001119  0.001453  0.001036  0.000138  0.000320  0.000170  0.000000  0.000000  0.000233  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000028  0.000002  0.000000  0.000000  0.000000  0.326359  0.656569  0.000487  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000001  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000003  0.000001  0.000002  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000007  0.000001  0.000008  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000005  0.000001  0.000006  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000002  0.000004  0.000001  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000014  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000001  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000018  0.000000  0.000000
  0.000000  0.000000  0.001316  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.001365  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000000  0.000000
  0.000000  0.000000  0.001852  0.000000  0.000000  0.001059  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.001058  0.000000  0.000001  0.001798  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
pass probabilities: 0.000001
ownership: 
  0.142763  0.091326 -0.008726 -0.091040 -0.160249 -0.199829 -0.180481 -0.171605 -0.174805 -0.193526 -0.177754 -0.125470 -0.052897  0.063287  0.140304  0.403313  0.448880  0.549233  0.595308
  0.210232  0.151672  0.037147 -0.100615 -0.058246 -0.226738 -0.175206 -0.125228 -0.165445 -0.182325 -0.115035 -0.065554 -0.018632  0.009089  0.127515  0.283914  0.513395  0.576179  0.598720
  0.260405  0.271912  0.299842  0.220146  0.316014 -0.505140 -0.068670  0.053964 -0.062925 -0.176573 -0.081335 -0.008523 -0.018926 -0.033937 -0.449580  0.948998  0.702018  0.626212  0.584124
  0.353868  0.403060  0.446939  0.727793  0.165223  0.061804 -0.002552  0.040621  0.023615 -0.033273  0.021816  0.003100  0.000168  0.019449  0.491340  0.890835  0.583754  0.578939  0.569747
  0.412913  0.477787  0.457510  0.285708  0.156027  0.031431  0.033303  0.038182  0.043663  0.037221  0.021687  0.016965  0.022878  0.067380  0.253239  0.511978  0.524447  0.440387  0.456207
  0.387739  0.482111  0.744310  0.262617  0.105185  0.053265  0.022416  0.027409  0.037061  0.034690  0.030043  0.030285  0.036245  0.125500  0.182188  0.723859  0.391532  0.344480  0.339480
  0.305088  0.335241  0.189238  0.004194  0.033138  0.009360  0.000552  0.008337  0.021144  0.011690  0.026436  0.025241  0.028392  0.069958  0.102725  0.137517  0.254945  0.227085  0.208222
  0.185080  0.142807  0.108519  0.020723 -0.011497 -0.015027 -0.012926  0.001065  0.011030  0.028461  0.019707  0.029472  0.012811  0.025068  0.043690  0.110339  0.028593  0.072640  0.087383
  0.080445  0.069816  0.054425  0.006208 -0.021739 -0.026035 -0.018303  0.013205  0.045227  0.039702  0.042169  0.020543  0.004896 -0.007760  0.000193  0.032370  0.028403  0.011538  0.006590
  0.023752  0.025071  0.035448 -0.007240 -0.032753 -0.031338 -0.019580  0.021491  0.024908  0.023423  0.026520  0.027019 -0.005556 -0.009719 -0.009451  0.012034  0.010298 -0.027399 -0.033456
  0.003592  0.009786  0.014345 -0.021555 -0.033878 -0.031203 -0.019428  0.010309  0.030436  0.030020  0.029362  0.012121 -0.014999 -0.022664 -0.019778 -0.017419 -0.003241 -0.030987 -0.037532
 -0.000680  0.003807 -0.026745 -0.034132 -0.050563 -0.040744 -0.024401 -0.006531  0.026095  0.002378  0.018636 -0.006199 -0.021512 -0.036782 -0.042753 -0.036209 -0.032424 -0.020678 -0.019702
 -0.015927  0.016654 -0.021877 -0.064079 -0.084877 -0.061668 -0.043849 -0.033199 -0.035168 -0.023338 -0.035116 -0.032204 -0.043208 -0.060852 -0.082506 -0.066147 -0.024847  0.007729 -0.021811
 -0.013895 -0.000380 -0.173642 -0.171355 -0.153297 -0.099294 -0.077306 -0.068727 -0.066582 -0.086074 -0.064630 -0.067472 -0.076298 -0.099415 -0.154924 -0.169727 -0.181898  0.001803 -0.015072
 -0.023157 -0.074006 -0.113250 -0.408638 -0.242550 -0.186567 -0.137322 -0.117851 -0.112844 -0.121784 -0.111861 -0.116496 -0.135759 -0.186181 -0.244178 -0.410982 -0.103950 -0.067365 -0.016990
 -0.019861 -0.031580 -0.287987 -0.852080 -0.442597 -0.273708 -0.176653 -0.155267 -0.173295 -0.203132 -0.172988 -0.153801 -0.173388 -0.266841 -0.442971 -0.856618 -0.284320 -0.017959 -0.011780
  0.034641  0.027368  0.237963 -0.312811 -0.169575 -0.291272 -0.160539 -0.170234 -0.171944 -0.201728 -0.171214 -0.168527 -0.154315 -0.286896 -0.161343 -0.308131  0.261770  0.040013  0.045536
  0.031034  0.066930  0.017847 -0.056492 -0.118928 -0.089483 -0.087501 -0.118185 -0.140159 -0.152392 -0.138997 -0.115525 -0.082469 -0.081572 -0.109926 -0.042169  0.029832  0.080407  0.042461
 -0.010100  0.020780  0.015046 -0.050150 -0.067305 -0.081335 -0.094200 -0.099260 -0.112537 -0.121783 -0.112199 -0.097956 -0.091133 -0.077293 -0.061542 -0.042934  0.025801  0.032756  0.001395

Sayuri-TesnorRT> sayuri-raw_nn avg
= stm winrate: 0.507625
wdl winrate: 0.509284
win probability: 0.482351
draw probability: 0.0538656
loss probability: 0.463783
final score: 0.0689347
q error: 6.48171e-05
score error: 0.00195356
probabilities: 
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.001973  0.000212  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000640  0.000000  0.000000  0.000668  0.001118  0.001453  0.001037  0.000139  0.000323  0.000170  0.000000  0.000000  0.000233  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000028  0.000002  0.000000  0.000000  0.000000  0.326398  0.656535  0.000486  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000001  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000003  0.000001  0.000002  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000008  0.000001  0.000008  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000005  0.000001  0.000006  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000002  0.000004  0.000001  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000014  0.000000  0.000000  0.000000  0.000000  0.000001  0.000001  0.000001  0.000001  0.000001  0.000000  0.000000  0.000000  0.000000  0.000019  0.000000  0.000000
  0.000000  0.000000  0.001317  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.001365  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000001  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000001  0.000000  0.000000
  0.000000  0.000000  0.001849  0.000000  0.000000  0.001059  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.001057  0.000000  0.000001  0.001795  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
pass probabilities: 0.000000
ownership: 
  0.142525  0.090994 -0.009140 -0.091341 -0.160923 -0.200317 -0.181082 -0.172425 -0.175674 -0.194538 -0.178673 -0.126032 -0.053401  0.062566  0.139646  0.402909  0.448235  0.548920  0.594954
  0.210022  0.151454  0.036803 -0.100780 -0.058590 -0.227061 -0.175611 -0.126189 -0.166736 -0.183662 -0.116228 -0.066358 -0.019244  0.008477  0.127371  0.283498  0.512946  0.575728  0.598217
  0.260000  0.271796  0.299495  0.220033  0.315477 -0.505697 -0.069185  0.053294 -0.064082 -0.178196 -0.082789 -0.009127 -0.019466 -0.034497 -0.449882  0.948839  0.701332  0.625687  0.583593
  0.353665  0.402983  0.446684  0.727610  0.164730  0.061429 -0.002966  0.039845  0.022607 -0.034572  0.020948  0.002578 -0.000385  0.018518  0.490928  0.890777  0.583437  0.578401  0.569141
  0.412585  0.477473  0.457169  0.285529  0.155759  0.031341  0.033020  0.037603  0.043030  0.036415  0.021243  0.016441  0.022494  0.066955  0.252784  0.511305  0.523958  0.439803  0.455751
  0.387256  0.481846  0.744089  0.262259  0.104822  0.052896  0.022009  0.026820  0.036537  0.034082  0.029539  0.029881  0.035918  0.125074  0.181556  0.723538  0.391026  0.343985  0.339039
  0.304786  0.334825  0.188786  0.003980  0.032836  0.009160  0.000003  0.007687  0.020853  0.011155  0.025796  0.024882  0.027890  0.069617  0.102264  0.136754  0.254567  0.226570  0.207827
  0.184715  0.142524  0.108382  0.020409 -0.011839 -0.015426 -0.013267  0.000646  0.010968  0.027821  0.019292  0.029135  0.012553  0.024655  0.043302  0.110079  0.028163  0.072465  0.087195
  0.079970  0.069506  0.054035  0.005907 -0.022099 -0.026285 -0.018789  0.012811  0.044950  0.040168  0.042187  0.019984  0.004459 -0.008096 -0.000271  0.031979  0.028066  0.011212  0.006155
  0.023117  0.024550  0.035039 -0.007633 -0.033164 -0.031618 -0.020144  0.021222  0.025381  0.023677  0.027003  0.026669 -0.006035 -0.010122 -0.009840  0.011445  0.009882 -0.028046 -0.033951
  0.003371  0.009370  0.013686 -0.022134 -0.034337 -0.031826 -0.019888  0.009862  0.030752  0.029889  0.029565  0.011893 -0.015640 -0.023151 -0.020419 -0.018006 -0.003812 -0.031380 -0.038056
 -0.001367  0.003177 -0.027420 -0.034742 -0.051108 -0.041123 -0.024842 -0.006999  0.025684  0.002828  0.018102 -0.006687 -0.022170 -0.037132 -0.043473 -0.036918 -0.033189 -0.021416 -0.020409
 -0.016759  0.015841 -0.022615 -0.064703 -0.085447 -0.062221 -0.044243 -0.033556 -0.035469 -0.023654 -0.035424 -0.032372 -0.043747 -0.061465 -0.083218 -0.066705 -0.025517  0.006923 -0.022731
 -0.014621 -0.001120 -0.174137 -0.172048 -0.153901 -0.099793 -0.077805 -0.069349 -0.067086 -0.086470 -0.065074 -0.068104 -0.076884 -0.099972 -0.155376 -0.170403 -0.182207  0.001082 -0.015500
 -0.023896 -0.074769 -0.114179 -0.409499 -0.242956 -0.186827 -0.137755 -0.118573 -0.113595 -0.122298 -0.112160 -0.116869 -0.136176 -0.186851 -0.244530 -0.411511 -0.104931 -0.067944 -0.017633
 -0.020560 -0.032164 -0.288480 -0.852402 -0.443086 -0.274209 -0.177330 -0.155680 -0.173875 -0.203715 -0.173654 -0.154374 -0.174202 -0.267373 -0.443423 -0.856854 -0.284832 -0.018441 -0.012221
  0.034150  0.026902  0.237201 -0.313494 -0.170464 -0.291674 -0.161269 -0.171108 -0.172561 -0.202182 -0.171763 -0.169199 -0.155044 -0.287351 -0.162345 -0.308739  0.260911  0.039457  0.044965
  0.030518  0.066532  0.017490 -0.057091 -0.119559 -0.090070 -0.088160 -0.118783 -0.140720 -0.153084 -0.139535 -0.116247 -0.083273 -0.082099 -0.110458 -0.042555  0.029414  0.080227  0.042027
 -0.010773  0.020326  0.014590 -0.050856 -0.067939 -0.081931 -0.094684 -0.099669 -0.113029 -0.122221 -0.112789 -0.098570 -0.091840 -0.077812 -0.062194 -0.043613  0.025290  0.032156  0.000799
  0.034150  0.026902  0.237201 -0.313494 -0.170464 -0.291674 -0.161269 -0.171108 -0.172561 -0.202182 -0.171763 -0.169199 -0.155044 -0.287351 -0.162345 -0.308739  0.260911  0.039457  0.044965
  0.030518  0.066532  0.017490 -0.057091 -0.119559 -0.090070 -0.088160 -0.118783 -0.140720 -0.153084 -0.139535 -0.116247 -0.083273 -0.082099 -0.110458 -0.042555  0.029414  0.080227  0.042027
 -0.010773  0.020326  0.014590 -0.050856 -0.067939 -0.081931 -0.094684 -0.099669 -0.113029 -0.122221 -0.112789 -0.098570 -0.091840 -0.077812 -0.062194 -0.043613  0.025290  0.032156  0.000799

[CGLemon]

I plan to start working on porting the TensorRT backend over the weekend. At that time, I’ll be copying a significant amount of code from Sayuri-TensorRT. Would you mind if I did it this way, or would you prefer to submit a PR yourself?

[MAOmao000]

I don't have a full grasp of the details other than the backend processing, so I'll leave all the modifications to the main body to you.

[CGLemon]

Hi, while porting the program, I noticed that the current TRT network doesn’t seem to support playing games of different board sizes simultaneously — I mean the multiple-board-sizes-via-masking. Is my understanding correct?

[MAOmao000]

I apologize for the late reply. I will answer again. The source code I provided last time was without masking. This time, I will provide the code with masking added. I have also added plugin processing, but this is for verification purposes only, so there is no need to import it.
https://github.com/MAOmao000/Sayuri-TensorRT

[CGLemon]

Thanks for your help.

[MAOmao000]

We've revised the timing of activation and masking.
See the training results for a simple model with ResidualBlock. We achieved accuracy comparable to that of the CUDA backend.
https://github.com/MAOmao000/Sayuri-TensorRT

[CGLemon]

@MAOmao000

Thank you for your assistance! It has noticeably improved the overall speed. Due to being busy recently, the progress of the porting work has been a bit slower. Once the project is completed, I will include your name in the acknowledgments. I truly appreciate your contribution to the project.

[MAOmao000]

There was an error in the TensorRT model definition of the mixer block, so I fixed it. Please make the same correction.
https://github.com/MAOmao000/Sayuri-TensorRT

[CGLemon]

@MAOmao000
Thank you for your contribution! I’ve added the link to the version you developed into the README.