HN h-index Javascript slowness guess

[abetusk]

This is more of a guess than an issue but I suspect the reason why the Javascript version is so much slower is because it's single threaded. Even though the Promise.all(...) execution queues everything in parallel, it's still only being processed serially since it's single core.

Contrast this with the aiohttp.TCPConnector(limit=1000) which really will, presumably, create 1000 concurrent processes.

One way to speed things up is to put the Javascript requests into a Web worker to actually create concurrent executions.

Again, this is an educated guess so I could be totally off base.

[alexmolas]

hi @abetusk I just saw your comment, sorry for the late reply. To be honest I don't know almost anything about JavaScript, so I don't know how to implement what you propose. I'll try later with cursor to see if I can speed up the calculations. Thanks!!

[abetusk]

By Cursor, you mean a Copilot like LLM programming assistant? For what its worth, my experience with programming tasks that are even slightly novel haven't met with a lot of success. I would be interested to know if you have any success with it.

I have a rough idea of how to do this but it falls just outside of my interest to annoyance threshold. I could maybe be persuaded to work on it if you think its interesting enough.

Regardless, I can give a short preview of how one would go about doing this. I'm using the MDN documentation on "Using Web Workers" as the basis for the following.

So, the idea would be to create a web worker that does the bulk request of items. Call this hn_item_wrkr.js:

// WARNING: COMPLETELY UNTESTED!!!!
//
var BASE_URL = "https://hacker-news.firebaseio.com/v0/item/";
self.addEventListener('message', function(ev) {
  let url = BASE_URL + ev.data + ".json";
  fetch(url).then( function(p_resp) {
    p_resp.json().then( function(j_data) {
      self.postMessage(j_data);
    });
  });
});

In the main thread (app.js), something like the following:

  ...
  let worker_pool = [];
  for (let pool_idx=0; pool_idx<100; pool_idx++) {
    worker_pool.push(new Worker("hn_item_wrkr.js");
    worker_pool[pool_idx].onmessage = function(ev) {
      // process results with ev.data as HN item object result
    }
  }
  ...
  let batch = submittedItems.slice(i, i+100);
  for (let pool_idx = 0; pool_idx < 100; pool_idx++) {
    worker_pool[pool_idx].postMessage(batch[pool_idx]);
  }
  ...

All the above is completely untested so caveat emptor.

I don't have a deep understanding but my shallow understanding is that the new Worker(...) will create a separate process thread that will run concurrently and in parallel to the main thread. The other assumption is that the main bottleneck is the HTTP fetch request/response and not the data parsing, so making sure multiple HTTP requests can be sent in parallel rather than sequentially, would help with the JS runtime.