Statsig Startup Performance Problems

[bromano-oai]

Background

We are observing Statsig client initialization contributing to a significant portion of startup latency. Our setup is that we create two Statsig Clients on startup, one for device-level experiments (e.g. pre-login experimentation) and one for user-level experiments. Both of which have different parameterizations.

Today, we perform initialization synchronously within Activity#onCreate which blocks the main thread for a significant amount of time.

This is only expected to get worse as the number of feature gates and experiments increases over time.

Observations

Attached below is screenshots of what Statsig client initialization looks like from Perfetto traces and sampling profiler data.

Shared Preferences

The initial shared preference read ends up blocking the main thread for a significant amount of time (~150ms on a Samsung Galaxy A15 which is a top device in Brazil and popular low-end device in the US).

The way Shared Preferences works is that it acquires process-level lock to create a new thread, spins up the thread, reads shared preferences XML bytes from disk, and then passes that back to the original thread. The rationale for the additional thread is to ensure serialization of reads and writes; but, this is something that can be achieved piggybacking off of standard coroutine dispatchers with limited concurrency instead of an entirely new thread. Secondly, Shared Preferences is backed by XML, which adds additional overhead both on size and requiring an additional round of deserialization on top of JSON deserialization for no gain.

In general, it's critical to minimize disk I/O as lower-end devices have cheaper eMMC flash memory. It's worth auditing whether there are opportunities to trim unnecessary data from the JSON payload or using more efficient formats such as proto to minimize the total size of the file. As-is, the underlying shared prefs xml file is close to ~1.6MB (!), which is substantial. Granted, some of this size will come from customer's keys & value strings which are not easily mitigable from platform perspective.

Gson Reprocessing

When initializing two statsig clients, the Gson needs to be parsed again. As-is, there does not seem to be a mechanism to re-use the previously parsed Gson payload from the prior instance. I haven't looked at the implementation deeply, but it seems there may be some possibility to re-use the parsed payload across both instances.

Preloading disk I/O

Disclaimer: I'm not sure this is a good idea

It may be beneficial to provide an API to more aggressively preload portions of the Statsig client so the disk I/O is not perceivable on the critical path. Notably, the Statsig client initialization could be triggered and ran on a background thread before Application create using the androidx.startup library. This (1) enables statsig configs to be used for much earlier experimentation and feature gating on notification flows (which start pre-Application) and (2) allows developers to continue using the synchronous API and worry less about managing asynchronous initialization.

The rationale being that the asynchronous flow can lead to unexpected issues where the client is used before the proper configs are ready, leading to unexpected application states (e.g. kill-switch implemented as a feature gate not being loaded yet causing a feature to get initialized). It's preferable to have strong guarantees on config states even it means blocking.

Screenshots

Perfetto trace of startup with Shared Preference blocking

Time-Ordered Samples for main thread

Time-Ordered Samples of Shared Preferences thread