Default worker configs result in huge slow-down when running esprint check

[powersjcb]

Followup based on what's referenced in the ticket #122

Using the default number of workers on a MacBook pro results in the system grinding to a halt. 100% CPU/Memory utilization for a ~10k LOC project.

With defaults:

time TIMING=true npm run lint

>lint /Users/jpowers/workspace/project/server
> esprint check


Rule | Time (ms) | Relative
:----|----------:|--------:
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2405.702 |    69.4%
@typescript-eslint/await-thenable                |   813.312 |    23.5%
import/order                                     |   119.299 |     3.4%
@typescript-eslint/no-misused-promises           |    25.982 |     0.7%
@typescript-eslint/no-unused-vars                |    20.091 |     0.6%
quote-props                                      |    16.679 |     0.5%
@typescript-eslint/no-unnecessary-type-assertion |    13.511 |     0.4%
one-var                                          |     7.027 |     0.2%
@typescript-eslint/type-annotation-spacing       |     4.930 |     0.1%
@typescript-eslint/prefer-function-type          |     4.844 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1815.848 |    59.0%
@typescript-eslint/await-thenable                |   646.832 |    21.0%
@typescript-eslint/no-unnecessary-type-assertion |   287.133 |     9.3%
import/order                                     |   178.274 |     5.8%
@typescript-eslint/no-misused-promises           |    43.354 |     1.4%
@typescript-eslint/no-unused-vars                |    24.754 |     0.8%
quote-props                                      |    15.292 |     0.5%
no-undef-init                                    |     8.347 |     0.3%
@typescript-eslint/unified-signatures            |     6.827 |     0.2%
constructor-super                                |     5.520 |     0.2%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1954.114 |    50.7%
@typescript-eslint/await-thenable                |   977.483 |    25.3%
@typRule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2357.169 |    60.2%
@typescript-eslint/await-thenable                |   634.642 |    16.2%
@typescript-eslint/no-unnecessary-type-assertion |   379.759 |     9.7%
@typescript-eslint/no-misused-promises           |   297.357 |     7.6%
import/order                                     |   124.817 |     3.2%
@typescript-eslint/no-unused-vars                |    37.274 |     1.0%
quote-props                                      |    35.337 |     0.9%
@typescript-eslint/type-annotation-spacing       |     5.664 |     0.1%
no-undef-init                                    |     5.323 |     0.1%
no-multiple-empty-lines                          |     3.612 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1934.117 |    50.6%
@typescript-eslint/await-thenable                |  1615.115 |    42.2%
import/order                                     |   151.284 |     4.0%
@typescript-eslint/no-unnecessary-type-assertion |    22.628 |     0.6%
@typescript-eslint/no-unused-vars                |    20.450 |     0.5%
quote-props                                      |    13.902 |     0.4%
@typescript-eslint/no-misused-promises           |     7.386 |     0.2%
@typescript-eslint/type-annotation-spacing       |     7.214 |     0.2%
one-var                                          |     6.300 |     0.2%
@typescript-eslint/triple-slash-reference        |     6.231 |     0.2%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2083.999 |    70.9%
@typescript-eslint/await-thenable                |   545.413 |    18.5%
import/order                                     |   108.218 |     3.7%
@typescript-eslint/no-misused-promises           |    95.606 |     3.3%
@typescript-eslint/no-unused-vars                |    40.991 |     1.4%
quote-props                                      |    10.765 |     0.4%
@typescript-eslint/no-unnecessary-type-assertion |     8.610 |     0.3%
@typescript-eslint/triple-slash-reference        |     5.154 |     0.2%
object-shorthand                                 |     4.409 |     0.1%
prefer-const                                     |     3.790 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1828.159 |    55.2%
@typescript-eslint/await-thenable                |   791.248 |    23.9%
@typescript-eslint/no-unnecessary-type-assertion |   374.754 |    11.3%
import/order                                     |   160.821 |     4.9%
@typesescript-eslint/no-unnecessary-type-assertion |   647.662 |    16.8%
import/order                                     |   115.840 |     3.0%
@typescript-eslint/no-misused-promises           |    53.599 |     1.4%
@typescript-eslint/no-unused-vars                |    38.657 |     1.0%
@typescript-eslint/type-annotation-spacing       |    10.788 |     0.3%
no-eval                                          |    10.223 |     0.3%
@typescript-eslint/triple-slash-reference        |     7.961 |     0.2%
no-undef-init                                    |     7.169 |     0.2%
cript-eslint/no-misused-promises           |    83.192 |     2.5%
@typescript-eslint/no-unused-vars                |    16.090 |     0.5%
quote-props                                      |    15.594 |     0.5%
object-shorthand                                 |     3.882 |     0.1%
@typescript-eslint/triple-slash-reference        |     3.781 |     0.1%
no-undef-init                                    |     3.481 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1968.008 |    71.4%
@typescript-eslint/await-thenable                |   353.530 |    12.8%
@typescript-eslint/no-unnecessary-type-assertion |   193.051 |     7.0%
import/order                                     |   107.403 |     3.9%
@typescript-eslint/no-misused-promises           |    34.862 |     1.3%
@typescript-eslint/no-unused-vars                |    27.230 |     1.0%
quote-props                                      |    23.326 |     0.8%
@typescript-eslint/type-annotation-spacing       |     6.214 |     0.2%
object-shorthand                                 |     5.387 |     0.2%
no-unsafe-finally                                |     3.939 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1906.192 |    69.6%
@typescript-eslint/await-thenable                |   673.993 |    24.6%
import/order                                     |    67.404 |     2.5%
@typescript-eslint/no-unused-vars                |    24.295 |     0.9%
@typescript-eslint/no-misused-promises           |    21.142 |     0.8%
@typescript-eslint/no-unnecessary-type-assertion |     8.853 |     0.3%
quote-props                                      |     4.491 |     0.2%
one-var                                          |     4.461 |     0.2%
@typescript-eslint/triple-slash-reference        |     3.839 |     0.1%
@typescript-eslint/type-annotation-spacing       |     3.559 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2052.226 |    76.5%
@typescript-eslint/await-thenable                |   249.864 |     9.3%
@typescript-eslint/no-misused-promises           |   214.061 |     8.0%
import/order                                     |    70.266 |     2.6%
@typescript-eslint/no-unnecessary-type-assertion |    37.768 |     1.4%
@typescript-eslint/no-unused-vars                |    12.217 |     0.5%
quote-props                                      |    11.349 |     0.4%
no-trailing-spaces                               |     8.992 |     0.3%
constructor-super                                |     5.879 |     0.2%
object-shorthand                                 |     4.490 |     0.2%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2671.810 |    81.2%
@typescript-eslint/await-thenable                |   436.855 |    13.3%
import/order                                     |    85.942 |     2.6%
@typescript-eslint/no-unnecessary-type-assertion |    25.398 |     0.8%
@typescript-eslint/no-unused-vars                |    12.903 |     0.4%
@typescript-eslint/no-misused-promises           |     8.031 |     0.2%
quote-props                                      |     5.421 |     0.2%
@typescript-eslint/ban-ts-ignore                 |     4.919 |     0.1%
no-undef-init                                    |     3.972 |     0.1%
one-var                                          |     3.893 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prRule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1787.264 |    58.8%
@typescript-eslint/await-thenable                |   488.108 |    16.1%
@typescript-eslint/no-unnecessary-type-assertion |   418.249 |    13.8%
@typescript-eslint/no-misused-promises           |   157.097 |     5.2%
import/order                                     |   102.093 |     3.4%
quote-props                                      |    23.257 |     0.8%
@typescript-eslint/no-unused-vars                |    15.112 |     0.5%
no-console                                       |     8.238 |     0.3%
@typescript-eslint/type-annotation-spacing       |     5.045 |     0.2%
@typescript-eslint/triple-slash-reference        |     4.734 |     0.2%
Rule                                       | Time (ms) | Relative
:------------------------------------------|----------:|--------:
prettier/prettier                          |  2219.086 |    84.4%
@typescript-eslint/await-thenable          |   317.094 |    12.1%
import/order                               |    45.439 |     1.7%
@typescript-eslint/no-unused-vars          |    13.673 |     0.5%
eol-last                                   |     6.244 |     0.2%
@typescript-eslint/triple-slash-reference  |     3.785 |     0.1%
object-shorthand                           |     3.248 |     0.1%
@typescript-eslint/no-misused-promises     |     2.678 |     0.1%
@typescript-eslint/type-annotation-spacing |     1.825 |     0.1%
no-eval                                    |     1.711 |     0.1%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2624.948 |    69.5%
@typescript-eslint/await-thenable                |   675.807 |    17.9%
import/order                                     |   222.969 |     5.9%
@typescript-eslint/no-misused-promises           |   109.236 |     2.9%
@typescript-eslint/no-unnecessary-type-assertion |    57.721 |     1.5%
@typescript-eslint/no-unused-vars                |    33.332 |     0.9%
quote-props                                      |    16.972 |     0.4%
@typescript-eslint/type-annotation-spacing       |     3.856 |     0.1%
constructor-super                                |     3.192 |     0.1%
prefer-const                                     |     2.962 |     0.1%
ettier/prettier                                |  2147.697 |    77.3%
@typescript-eslint/await-thenable                |   355.747 |    12.8%
@typescript-eslint/no-unnecessary-type-assertion |    99.543 |     3.6%
import/order                                     |    93.347 |     3.4%
quote-props                                      |    16.467 |     0.6%
@typeRule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  2068.428 |    57.6%
@typescript-eslint/await-thenable                |  1082.621 |    30.2%
import/order                                     |   170.699 |     4.8%
@typescript-eslint/no-unnecessary-type-assertion |   143.324 |     4.0%
@typescript-eslint/no-misused-promises           |    49.256 |     1.4%
@typescript-eslint/no-unused-vars                |    17.106 |     0.5%
quote-props                                      |    10.092 |     0.3%
prefer-const                                     |     8.666 |     0.2%
@typescript-eslint/type-annotation-spacing       |     8.607 |     0.2%
one-var                                          |     5.899 |     0.2%
script-eslint/no-misused-promises           |    16.311 |     0.6%
@typescript-eslint/no-unused-vars                |    13.947 |     0.5%
no-undef-init                                    |     4.402 |     0.2%
@typescript-eslint/triple-slash-reference        |     4.317 |     0.2%
object-shorthand                                 |     3.775 |     0.1%
TIMING=true npm run lint  1.14s user 0.50s system 2% cpu 56.361 total

With --workers=2:

$ time TIMING=true npm run lint

> /Users/jpowers/workspace/project/server
> esprint --workers=2 check


Rule | Time (ms) | Relative
:----|----------:|--------:
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1389.799 |    64.3%
@typescript-eslint/await-thenable                |   480.108 |    22.2%
import/order                                     |   121.040 |     5.6%
@typescript-eslint/no-unnecessary-type-assertion |    55.078 |     2.5%
@typescript-eslint/no-unused-vars                |    36.622 |     1.7%
@typescript-eslint/no-misused-promises           |    26.087 |     1.2%
object-shorthand                                 |     6.845 |     0.3%
no-undef-init                                    |     5.523 |     0.3%
@typescript-eslint/type-annotation-spacing       |     5.049 |     0.2%
one-var                                          |     4.668 |     0.2%
Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
prettier/prettier                                |  1235.131 |    64.8%
@typescript-eslint/await-thenable                |   402.469 |    21.1%
import/order                                     |   121.505 |     6.4%
@typescript-eslint/no-unused-vars                |    35.038 |     1.8%
@typescript-eslint/no-unnecessary-type-assertion |    34.936 |     1.8%
@typescript-eslint/no-misused-promises           |    34.211 |     1.8%
object-shorthand                                 |     5.669 |     0.3%
quote-props                                      |     4.658 |     0.2%
no-undef-init                                    |     4.370 |     0.2%
one-var                                          |     3.521 |     0.2%
TIMING=true npm run lint  0.85s user 0.19s system 7% cpu 14.785 total

Environment:

$ esprint --version
0.7.0

$ eslint --version
v6.8.0

[powersjcb]

Anyone have any clues why there is such a small performance improvement on this project from changing the number of worker at the low end?

My initial thought is that there would be close to a doubling of performance when adding the first few workers. :-\

$ time esprint --workers=1 check
esprint --workers=1 check  0.62s user 0.12s system 5% cpu 14.778 total
$ time esprint --workers=1 check
esprint --workers=1 check  0.66s user 0.12s system 5% cpu 15.227 total
$ time esprint --workers=1 check
esprint --workers=1 check  0.68s user 0.13s system 5% cpu 15.402 total

$ time esprint --workers=2 check
esprint --workers=2 check  0.67s user 0.13s system 5% cpu 13.913 total
$ time esprint --workers=2 check
esprint --workers=2 check  0.65s user 0.13s system 5% cpu 14.041 total
time esprint --workers=3 check
esprint --workers=3 check  0.65s user 0.14s system 5% cpu 14.136 total
$ time esprint --workers=3 check
esprint --workers=3 check  0.65s user 0.13s system 5% cpu 14.062 total

[Muritavo]

According to my inspection of the execution process, I had a huge performance loss because the startup of the eslint engine takes a good amount of time, not really the linting process.
For example, I'm using the typescript parser on lint but when it's used on multiple workers, the initilization of the typescript watcher takes a HUGE amount of time.