[Quest/Malawi Core] Technical Proposal: Performance improvements for large facilities

[calmwalija]

Fixes #116

Checklist

• I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the project's style guide
• I have built and run the fhircore app to verify my change fixes the issue and/or does not break the app
• I have added any strings visible on UI components to the strings.xml file

[calmwalija]

@KhumboLihonga
@evance-mose
@sevenreup

During initialization the app fetches the Binary file which has the configurations. I have added another block of configurations to determin if the organization uses offline first.

Below is sample code for per organization configuration for offline first enabled tingathe-saa-prod

"organizationSyncConfig": {
  "items": [
    {
      "id": "9365a7f3-dc63-4972-9dc4-c290cebb9ce2",
      "name": "Chagunda Organization",
      "offlineFirst": true
    },
    {
      "id": "c97798b6-9c12-4b0a-bb66-3cd63237206f",
      "name": "Salima DHO Organization",
      "offlineFirst": false
    },
  ]
}

When a CHW logs in, the app checks for this configuration in this using this function

 fun getPerOrgSyncConfigs(): OrganizationSyncConfig? =
  applicationConfiguration.value?.organizationSyncConfig

this helper function can be called anywhere to get this config

fun syncConfigOfflineFirst(
  configurationRegistry: ConfigurationRegistry,
  sharedPreferencesHelper: SharedPreferencesHelper,
): Boolean {
  val configs =
    onPerOrgSyncConfigItem(configurationRegistry, sharedPreferencesHelper) ?: return false
  return !configs.offlineFirst
}

after a successful anthentication, the app goes calls runSync as usual but before the actual syncing, there app checks if the offlineFirst is enabled or not on this function

private fun downloadWorkManager(): DownloadWorkManager = runBlocking {
    syncConfigOfflineFirst(configurationRegistry, preference)
      .takeIf { it }
      ?.let {
        getIdentifiers(engine)?.let { item ->
          return@runBlocking IdentifierSyncParams(item.data, tagSystem) {
            runBlocking {
              onSendBroadcast(broadcaster, it)
              syncStrategyCacheDao.insert(it.logicalId.toEntity())
              if (it.patientPositionAt == item.data.size) {
                purgeListResource()
              }
            }
          }
        }
      }

    val logicalIds = logicalIds(syncStrategyCacheDao)

    return@runBlocking if (logicalIds.isNotEmpty()) {
      LogicalIdSyncParamsBased(logicalIds) {
        Timber.e("${it.patientPositionAt} of ${logicalIds.size}")
        runBlocking {
          it.logicalId
            .toEntity()
            .map { catchEntity -> catchEntity.copy(shouldSync = true) }
            .also { syncStrategyCacheDao.upsert(it.map { it.logicalId }) }
          saveLastUpdatedTimestamp(dataStore)
          onSendBroadcast(broadcaster, it)
        }
      }
    } else {
      defaultDownloadManager()
    }
  }

If this is true, for the very first sync, the app will go straight to the else case which calls defaultDownloadManager which will call syncParams

  private fun syncParams(): Map<ResourceType, Map<String, String>> {
    if (
      syncConfigOfflineFirst(
          configurationRegistry,
          preference,
        )
        .not()
    ) {
      return syncListenerManager.loadSyncParams()
    }
    return when {
      hasCompletedInitialSync(preference) -> SyncParamStrategy(preference).syncParams()
      else -> syncListenerManager.loadSyncParams()
    }
  }

The flow of code at point will go straight to hasCompletedInitialSync(preference) -> SyncParamStrategy(preference).syncParams().

  fun syncParams(): Map<ResourceType, Map<String, String>> {
    listOf(
        ResourceType.Questionnaire,
        ResourceType.StructureMap,
        ResourceType.Patient,
        ResourceType.List,
      )
      .forEach {
        when (it) {
          ResourceType.Patient ->
            syncParams.add(
              MutablePair(
                  it,
                  mutableMapOf(
                    "_tag1" to tagMeta,
                    "_tag2" to tagSystem,
                    "_count" to "500",
                  ),
                )
                .toPair(),
            )
          ResourceType.List ->
            syncParams.add(MutablePair(it, mutableMapOf("code" to identifierCoding)).toPair())
          else -> syncParams.add(MutablePair(it, mutableMapOf("_count" to "500")).toPair())
        }
      }
    return syncParams.toMap()
  }

At this point, the app will sync these resources

1. ResourceType.Questionnaire
2. ResourceType.StructureMap
3. ResourceType.Patient
4. ResourceType.List

For ListResource the app appends the tag http://smartregister.org/fhir/patient-identifier-list%7Cded72f19-49be-41bc-9979-965604d50199 in order to get a specific List which has the Patient Identifiers for the Guardians

When sync is complete, there is a listener in the PatientRegisterViewModel that listens for successful sync, this success sync, which is also the FIRST sync at this time will update SharedPreferenceKey.SYNC_STATUS to CompletedInitialSync and calls another sync.

The app goes through the same path by calling AppSyncworker

At this point, the app will proceed with syncConfigOfflineFirst(configurationRegistry, preference) which will be evaluate to true. Next, private fun downloadWorkManager() returns a DownloadWorkManager, this is the actual implementation of FHIR SDK DownloadManager, syncConfigOfflineFirst(configurationRegistry, preference) check if offlineFirst is enabled, if that is true, it will proceed to getIdentifiers

suspend fun getIdentifiers(fhirEngine: FhirEngine): ListResourceItem? {
  val listResource: ListResource = getListResource(fhirEngine) ?: return null
  return ListResourceItem(
    data = listResource.entry.mapNotNull { entry -> entry.item.display.toInt() }.toList(),
    idPart = listResource.idPart,
  )
}

this function query the List Resource which contains the Identifiers of Patient Guardians the programs team shared for the exposed infants for Salima DHO. If this list is not null, it continues to call IdentifierSyncParams which is class that implements a DownloadManager, this takes a list of Identifiers, tagSystem which is appended when querying the Patient by Identifier ie https://fhir.d-tree.org/fhir/Patient/?identifier=102&_tag=http://smartregister.org/fhir/organization-tag|99e04ce0-b130-44a2-919a-c2b08ae5c34d & a callback

data class ParamSyncStatus(
  val logicalId: List<String>,
  val idsTotal: Int,
  val patientPositionAt: Int,
) : Serializable

This will first create a bundleand then make a request passing the bundle to DownloadRequest.of(bundle = ids.bundleOf().also { patientPosition += ids.size }), then this patientPosition += ids.size keep track of the current identifier it is working on at that moment. For every request the a callback if invoked and gets passed the actual patient logicalId. To keep track of the logicalIds, a separate table SyncStrategyCacheEntity catches all the ids, this improves performance by keeping track of the logicalId status using shouldSync when app wants to retry an error. For every callback, a broadcast is sent to show a toast to the UI with the current progress. This could change but in the meantime the best way I found to send a message from the sync worker to the UI

@Entity
data class SyncStrategyCacheEntity(
  @PrimaryKey val logicalId: String,
  val shouldSync: Boolean = false,
  val timestamp: Long = System.currentTimeMillis(),
)

when this is done purgeListResource is called to delete the list resource, this is purged to avoid going through this process for no reason. At this point the downloadWorkManager returns IdentifierSyncParams and runSync using the parameters explained above.

Next up is querying the SyncStrategyCacheEntity where shouldSync = true here. This returns a list of all logicalIds what needs to be synced, if this is not empty, it will call LogicalIdSyncParamsBased and get passed the logicalIds. Just like IdentifierSyncParams, the process is done here, the only difference is how the bundle is created, for every successful download of resource, the callback is invoked and at this point, the status for shouldSync is changed to false the updates the SyncStrategyCacheEntity. This ensures that if something went wrong, the app is able to determine which logicalIds should download their resources. When thisis done, that is sync is complete, for a subsequent sync, the app checks for SyncStrategyCacheEntity if it has some logicalIds what shouldSync is true and goes through the same process. If this returns an empty list, the app uses defaultDownloadManager and at this point, the syncParams return syncListenerManager.loadSyncParams() which are the default params.

Here is the flow diagram