App Crash Incidents

[xinbenlv]

It has been pretty regular that the WikiLoop DoubleCheck crashes.
For example: https://uptime.betterstack.com/team/116381/incidents/428892595
So far the root cause seems to be memory leak

One example of crash log:
https://logs.betterstack.com/team/184467/tail?rf=1690628460000000&rt=1690628580000000&a=1690628546696516.55031612

#crash

The frequency:

[xinbenlv]

https://logs.betterstack.com/team/184467/tail?rf=1690628460000000&q=Error&rt=1690628580000000&a=1690628522998101.44004241

Error due to

{
    "app_name": "app",
    "dt": "2023-07-29T11:02:02.998101Z",
    "host": "host",
    "message": "FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory",
    "platform": "Heroku",
    "proc_id": "web.1",
    "source_type": "heroku_logs"
}

{
    "app_name": "app",
    "dt": "2023-07-29T11:02:03.000902Z",
    "host": "host",
    "message": " 2: 0x9f13b2 node::OnFatalError(char const*, char const*) [node]",
    "platform": "Heroku",
    "proc_id": "web.1",
    "source_type": "heroku_logs"
}

---

Start of crash:

https://logs.betterstack.com/team/184467/tail?rf=1690628460000000&rt=1690628580000000&a=1690628546696495.55031609

https://logs.betterstack.com/team/184467/tail?rf=1690628460000000&rt=1690628580000000&a=1690628546696516.55031612

[xinbenlv]

2023-07-26 16:38:42.639 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=XjzVcdHXNRm7QPHHsxAhJpg-IxNcNGDu level=info message="PERF        15ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71085109&pageId=70746657&revId=1941160148&ts_crawled=1690421276&ts_sendout=1690421922&ts_expire=1693013922&priority_score=-0.398277
2023-07-26 16:38:43.430 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=kf_BBFkLyG2MIGZOpzNOaI_40gtj0F-E level=info message="PERF        13ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71086140&pageId=70747685&revId=1941160155&ts_crawled=1690421282&ts_sendout=1690421923&ts_expire=1693013923&priority_score=-0.398277
2023-07-26 16:38:43.430 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71086140&pageId=70747685&revId=1941160155&ts_crawled=1690421282&ts_sendout=1690421923&ts_expire=1693013923&priority_score=-0.398277 66.249.66.206 at=info method=GET path="/api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71086140&pageId=70747685&revId=1941160155&ts_crawled=1690421282&ts_sendout=1690421923&ts_expire=1693013923&priority_score=-0.398277" host=doublecheck.wikiloop.org request_id=5a501dd2-2c06-451f-a871-de45df21561b fwd="66.249.66.206" dyno=web.1 connect=0ms service=13ms status=200 bytes=391 protocol=https
2023-07-26 16:38:44.170 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /socket.io/?EIO=3&transport=polling&t=OcKfImD 66.249.66.206 at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=OcKfImD" host=doublecheck.wikiloop.org request_id=053798c4-9512-4275-8eab-3e7e60f9b6f6 fwd="66.249.66.206" dyno=web.1 connect=0ms service=4ms status=200 bytes=272 protocol=https
2023-07-26 16:38:44.170 [wikiloop_doublecheck_prod] web.1 level=info message="A socket client connected. Socket id = C6rkpo1lBdwpjHiBAAAd. Total connections =
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 <--- Last few GCs --->
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 [26:0x3fa77b0] 28284665 ms: Mark-sweep 248.2 (258.5) -> 248.2 (258.2) MB, 173.3 / 0.1 ms  (average mu = 0.542, current mu = 0.005) last resort GC in old space requested


2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 [26:0x3fa77b0] 28284814 ms: Mark-sweep 248.2 (258.2) -> 248.2 (258.2) MB, 149.3 / 0.0 ms  (average mu = 0.381, current mu = 0.000) last resort GC in old space requested
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1


2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 <--- JS stacktrace --->
"app_name":"app"
"dt":"2023-07-27T01:38:46.970963Z"
"host":"host"
"message":"<--- JS stacktrace --->"
"platform":"Heroku"
"proc_id":"web.1"
"source_type":"heroku_logs"
[Expand all](https://logs.betterstack.com/team/184467/tail?rf=1682906210493000&a=1690421926970963-108004892#)
[Copy JSON](https://logs.betterstack.com/team/184467/tail?rf=1682906210493000&a=1690421926970963-108004892#)
Cancel
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:40.325 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075558&pageId=70737163&revId=1941160124&ts_crawled=1690421275&ts_sendout=1690421920&ts_expire=1693013920&priority_score=-0.398277 66.249.66.163 at=info method=GET path="/api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075558&pageId=70737163&revId=1941160124&ts_crawled=1690421275&ts_sendout=1690421920&ts_expire=1693013920&priority_score=-0.398277" host=doublecheck.wikiloop.org request_id=c2cea245-055d-4fbf-8fdd-aec3e476f782 fwd="66.249.66.163" dyno=web.1 connect=0ms service=15ms status=200 bytes=392 protocol=https
2023-07-26 16:38:40.326 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=HlfX9Y0ynVgYtowrW04eEko4y0qfFtw8 level=info message="PERF        15ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075558&pageId=70737163&revId=1941160124&ts_crawled=1690421275&ts_sendout=1690421920&ts_expire=1693013920&priority_score=-0.398277
2023-07-26 16:38:40.549 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075555&pageId=70737160&revId=1941160118&ts_crawled=1690421275&ts_sendout=1690421920&ts_expire=1693013920&priority_score=-0.398277 66.249.66.133 at=info method=GET path="/api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075555&pageId=70737160&revId=1941160118&ts_crawled=1690421275&ts_sendout=1690421920&ts_expire=1693013920&priority_score=-0.398277" host=doublecheck.wikiloop.org request_id=bd396cf9-307b-46c9-824c-956d252f472b fwd="66.249.66.133" dyno=web.1 connect=0ms service=16ms status=200 bytes=392 protocol=https
2023-07-26 16:38:40.550 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=GJe57UNzdxXe0h8qG_YYzJtic3HVjDpb level=info message="PERF        16ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075555&pageId=70737160&revId=1941160118&ts_crawled=1690421275&ts_sendout=1690421920&ts_expire=1693013920&priority_score=-0.398277
2023-07-26 16:38:41.310 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=nddarmro8vlmQ-XHdrnkhNohITDVWvXJ level=info message="PERF        15ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q72281219&pageId=71934679&revId=1941160135&ts_crawled=1690421275&ts_sendout=1690421921&ts_expire=1695605921&priority_score=0.930467
2023-07-26 16:38:41.318 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q72281219&pageId=71934679&revId=1941160135&ts_crawled=1690421275&ts_sendout=1690421921&ts_expire=1695605921&priority_score=0.930467 66.249.66.206 at=info method=GET path="/api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q72281219&pageId=71934679&revId=1941160135&ts_crawled=1690421275&ts_sendout=1690421921&ts_expire=1695605921&priority_score=0.930467" host=doublecheck.wikiloop.org request_id=ce9db5db-2497-4412-8e1c-a1970c31b66a fwd="66.249.66.206" dyno=web.1 connect=1ms service=16ms status=200 bytes=394 protocol=https
2023-07-26 16:38:41.578 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=iknADyI0owrhk5G4TbstXzGIch15a1z2 level=info message="PERF        25ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075974&pageId=70737579&revId=1941160136&ts_crawled=1690421277&ts_sendout=1690421921&ts_expire=1693013921&priority_score=-0.398277
2023-07-26 16:38:41.586 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075974&pageId=70737579&revId=1941160136&ts_crawled=1690421277&ts_sendout=1690421921&ts_expire=1693013921&priority_score=-0.398277 66.249.66.69 at=info method=GET path="/api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075974&pageId=70737579&revId=1941160136&ts_crawled=1690421277&ts_sendout=1690421921&ts_expire=1693013921&priority_score=-0.398277" host=doublecheck.wikiloop.org request_id=a03fa471-0e3c-4d81-a1dd-6b80f524730d fwd="66.249.66.69" dyno=web.1 connect=0ms service=26ms status=200 bytes=390 protocol=https
2023-07-26 16:38:41.844 [wikiloop_doublecheck_prod] web.1 nuxtServerInit start
2023-07-26 16:38:41.850 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=-LwTMgHGYsPIDiMj-iavS2hMG7B3DZAg level=info message="PERF         5ms  200 GET /api/flags
2023-07-26 16:38:41.856 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=cENcvtoGipLyQQjcXgv1v3ZXxI_1yMjn level=info message="PERF         5ms  200 GET /api/version
2023-07-26 16:38:42.052 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=1t2VtEx_dMUOzWqetIZDcxPYJ8106FTg level=info message="PERF       195ms  200 GET /api/metrics
2023-07-26 16:38:42.052 [wikiloop_doublecheck_prod] web.1 nuxtServerInit req.session.id 3XgsrRszVzWaiUI3tXQiArtk1i4hRgBV
2023-07-26 16:38:42.052 [wikiloop_doublecheck_prod] web.1 nuxtServerInit store state clearProfile because req.user is not defined
2023-07-26 16:38:42.052 [wikiloop_doublecheck_prod] web.1 nuxtServerInit done
2023-07-26 16:38:42.077 [wikiloop_doublecheck_prod] router web.1 200 GET doublecheck.wikiloop.org /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075687&pageId=70737292&revId=1941160130&ts_crawled=1690421277&ts_sendout=1690421921&ts_expire=1693013921&priority_score=-0.398277 66.249.66.133 at=info method=GET path="/api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075687&pageId=70737292&revId=1941160130&ts_crawled=1690421277&ts_sendout=1690421921&ts_expire=1693013921&priority_score=-0.398277" host=doublecheck.wikiloop.org request_id=972abb99-134d-4e6e-bbdf-7da340862bf6 fwd="66.249.66.133" dyno=web.1 connect=0ms service=14ms status=200 bytes=394 protocol=https
2023-07-26 16:38:42.078 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=ZVcqvky6azIiLX0AlurGIdlMvNayh9G7 level=info message="PERF        14ms  200 GET /api/feed/wikitrust/revision?feed=wikitrust&wiki=wikidatawiki&title=Q71075687&pageId=70737292&revId=1941160130&ts_crawled=1690421277&ts_sendout=1690421921&ts_expire=1693013921&priority_score=-0.398277
2023-07-26 16:38:42.244 [wikiloop_doublecheck_prod] web.1 ga_id= session_id=ryQcZb9M7IcuiOCjrNzknCS_4MagTDPG level=info message="PERF       191ms  200 GET /api/revision/thwiki:9809740
2023-07-26 16:38:42.402 [wikiloop_doublecheck_prod] web.1 heroku.130323585.b88357d3-be78-4b34-b5f2-1d181b8814e8 source=web.1 dyno=heroku.130323585.b88357d3-be78-4b34-b5f2-1d181b8814e8 sample#load_avg_1m=0.00 sample#load_avg_5m=0.01 sample#load_avg_15m=0.02
2023-07-26 16:38:42.423 [wikiloop_doublecheck_prod] web.1 heroku.130323585.b88357d3-be78-4b34-b5f2-1d181b8814e8 source=web.1 dyno=heroku.130323585.b88357d3-be78-4b34-b5f2-1d181b8814e8 sample#memory_total=312.89MB sample#memory_rss=301.84MB sample#memory_cache=0.20MB sample#memory_swap=10.85MB sample#memory_pgpgin=5767929pages sample#memory_pgpgout=5692651pages sample#memory_quota=512.00MB
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 ==== JS stack trace =========================================
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1     0: ExitFrame [pc: 0x13a71b9]
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 Security context: 0x0ae98a0008d1 <JSObject>
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1     1: assign(aka assign) [0xae98a0011f9](this=0x3034fe5004b1 <undefined>,0x332069dfffb9 <Object map = 0x2dac1b471b29>,0x1a0a272a4d49 <Object map = 0x2dac1b471b29>,0x2011e94fffd1 <Object map = 0xd08c66a4889>)
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1     2: format [0x35b909411bd1] [/app/node_modules/heroku-logger/lib/index.js:190] [bytecode=0x1f573d2f0551 offset=134](this=0x1b6b73447ce1 <Logger map = 0x2...
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:46.970 [wikiloop_doublecheck_prod] web.1 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
2023-07-26 16:38:46.976 [wikiloop_doublecheck_prod] web.1  1: 0x9ef190 node::Abort() [node]
2023-07-26 16:38:46.976 [wikiloop_doublecheck_prod] web.1  2: 0x9f13b2 node::OnFatalError(char const*, char const*) [node]
2023-07-26 16:38:46.976 [wikiloop_doublecheck_prod] web.1  3: 0xb5da9e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
2023-07-26 16:38:46.977 [wikiloop_doublecheck_prod] web.1  4: 0xb5de19 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
2023-07-26 16:38:46.977 [wikiloop_doublecheck_prod] web.1  5: 0xd0a765  [node]
2023-07-26 16:38:46.978 [wikiloop_doublecheck_prod] web.1  6: 0xd1b068 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
2023-07-26 16:38:46.978 [wikiloop_doublecheck_prod] web.1  7: 0xce1d2d v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::RootIndex, int, v8::internal::Object, v8::internal::AllocationType) [node]
2023-07-26 16:38:46.979 [wikiloop_doublecheck_prod] web.1  8: 0xce1e20 v8::internal::Handle<v8::internal::FixedArray> v8::internal::Factory::NewFixedArrayWithMap<v8::internal::FixedArray>(v8::internal::RootIndex, int, v8::internal::AllocationType) [node]
2023-07-26 16:38:46.979 [wikiloop_doublecheck_prod] web.1  9: 0xf00609 v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New(v8::internal::Isolate*, int, v8::internal::AllocationType, v8::internal::MinimumCapacity) [node]
2023-07-26 16:38:46.980 [wikiloop_doublecheck_prod] web.1 10: 0xf0ae39 v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, int) [node]
2023-07-26 16:38:46.980 [wikiloop_doublecheck_prod] web.1 11: 0xf0af55 v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [node]
2023-07-26 16:38:46.981 [wikiloop_doublecheck_prod] web.1 12: 0xf0b089 v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::AddNoUpdateNextEnumerationIndex(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [node]
2023-07-26 16:38:46.981 [wikiloop_doublecheck_prod] web.1 13: 0xf0b0c5 v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [node]
2023-07-26 16:38:46.982 [wikiloop_doublecheck_prod] web.1 14: 0xed45af v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>) [node]
2023-07-26 16:38:46.982 [wikiloop_doublecheck_prod] web.1 15: 0xf05571 v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [node]
2023-07-26 16:38:46.983 [wikiloop_doublecheck_prod] web.1 16: 0xf06157 v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [node]
2023-07-26 16:38:46.983 [wikiloop_doublecheck_prod] web.1 17: 0x103b424 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [node]
2023-07-26 16:38:46.984 [wikiloop_doublecheck_prod] web.1 18: 0xeb804f v8::internal::JSReceiver::SetOrCopyDataProperties(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Object>, v8::internal::ScopedVector<v8::internal::Handle<v8::internal::Object> > const*, bool) [node]
2023-07-26 16:38:46.984 [wikiloop_doublecheck_prod] web.1 19: 0x103de33 v8::internal::Runtime_SetDataProperties(int, unsigned long*, v8::internal::Isolate*) [node]
2023-07-26 16:38:46.985 [wikiloop_doublecheck_prod] web.1 20: 0x13a71b9  [node]
2023-07-26 16:38:47.012 [wikiloop_doublecheck_prod] router web.1 503 GET doublecheck.wikiloop.org /api/interaction/beta/thwiki:9809740 66.249.66.69 at=error code=H13 desc="Connection closed without response" method=GET path="/api/interaction/beta/thwiki:9809740" host=doublecheck.wikiloop.org request_id=dddc64b1-7239-48ec-8cbc-8d80d3af9870 fwd="66.249.66.69" dyno=web.1 connect=0ms service=2353ms status=503 bytes=0 protocol=https
2023-07-26 16:38:47.012 [wikiloop_doublecheck_prod] router web.1 503 GET doublecheck.wikiloop.org /api/score/thwiki:9809740 66.249.66.69 at=error code=H13 desc="Connection closed without response" method=GET path="/api/score/thwiki:9809740" host=doublecheck.wikiloop.org request_id=109ec61a-ccb8-43d9-8c04-eb697b763b72 fwd="66.249.66.69" dyno=web.1 connect=0ms service=2623ms status=503 bytes=0 protocol=https
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 npm ERR! code ELIFECYCLE
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 npm ERR! errno 1
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 npm ERR! wikiloop-doublecheck@4.1.2 start: `cross-env NODE_ENV=production npx ts-node -r tsconfig-paths/register --project tsconfig.json server/index.ts`
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 1 npm ERR! Exit status 1
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 npm ERR! 
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 npm ERR! Failed at the wikiloop-doublecheck@4.1.2 start script.
2023-07-26 16:38:47.103 [wikiloop_doublecheck_prod] web.1 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2023-07-26 16:38:47.110 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:47.110 [wikiloop_doublecheck_prod] web.1 npm ERR! A complete log of this run can be found in:
2023-07-26 16:38:47.110 [wikiloop_doublecheck_prod] web.1 npm ERR!     /app/.npm/_logs/2023-07-27T01_38_47_104Z-debug.log
2023-07-26 16:38:47.347 [wikiloop_doublecheck_prod] web.1 1 Process exited with status 1
2023-07-26 16:38:47.378 [wikiloop_doublecheck_prod] web.1 State changed from up to crashed
2023-07-26 16:38:47.385 [wikiloop_doublecheck_prod] web.1 State changed from crashed to starting
2023-07-26 16:38:56.079 [wikiloop_doublecheck_prod] web.1 Starting process with command `npm run start`
2023-07-26 16:38:57.159 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:57.159 [wikiloop_doublecheck_prod] web.1 > wikiloop-doublecheck@4.1.2 start /app
2023-07-26 16:38:57.159 [wikiloop_doublecheck_prod] web.1 > cross-env NODE_ENV=production npx ts-node -r tsconfig-paths/register --project tsconfig.json server/index.ts
2023-07-26 16:38:57.159 [wikiloop_doublecheck_prod] web.1
2023-07-26 16:38:59.273 [wikiloop_doublecheck_prod] web.1 DotEnv envPath =  template.env  if you want to change it, restart and set DOTENV_PATH
2023-07-26 16:39:01.485 [wikiloop_doublecheck_prod] web.1 Loading locale file af.yml, as af
2023-07-26 16:39:01.495 [wikiloop_doublecheck_prod] web.1 Loading locale file ar.yml, as ar
2023-07-26 16:39:01.499 [wikiloop_doublecheck_prod] web.1 Loading locale file bg.yml, as bg

[fredster33]

Looks like https://doublecheck.wikiloop.org/ has been down for a while.