How to reduce "app:patch DexDebug" task execution time

[lcm92]

The project I'm working on takes a lot of time with each build.
Each time Dexpatcher will rewrite the dex in the directory '..\app\build\intermediates\dexpatcher\patched-dex\debug' and it took me almost 5 minutes for each such build.
Do you have any solution?

I am using:

• Android Gradle plugin 3.4.3
• Dexpatcher 2.1.0

[Lanchon]

hi,

i don't know what dexpatcher you are using, start by telling me that. yes, i know you are using dxp-gradle 2.1.0 (which you should not because it is prerelease, but anyway). the version of dxp-tool is specified in your gradle build files.

things you can do:

• paste your gradle.build files, redacting all sensitive info first.
• or at least inform version of dxp-tool used.
• round-trip with apktool (no need to sign the apk, better if you skip resource processing) and inform decode and build times.
• paste the build log using --quiet --stats on dxp-tool (dxp-gradle has options verbosity = QUIET and logStats = true for that)
• are you using bytecode transforms? (if you do not know what they are, you are not)

we can start debugging with that info.

thanks!

[lcm92]

project build.gradle

buildscript {
    repositories {
        google()
        jcenter()
    }
    dependencies {
//        classpath 'com.android.tools.build:gradle:4.1.3'
        classpath 'com.android.tools.build:gradle:3.4.2'
    }
}

buildscript {
    repositories {
        gradlePluginPortal()
        flatDir dirs: rootProject.file('dexpatcher-gradle/libs')
    }
    dependencies {
//        classpath 'com.github.lanchon.dexpatcher:dexpatcher-gradle:2.1.3'
        classpath 'com.github.lanchon.dexpatcher:dexpatcher-gradle:2.1.0'
    }
}

allprojects {
    repositories {
        google()
        jcenter()
    }
}

allprojects {
    repositories {
        maven { url 'https://dexpatcher.github.io/dexpatcher-repo/m2' }
    }
    dependencies {
        configurations.all {
            switch (name) {
                case 'apktool':
                    add name, 'dexpatcher-repo.ibotpeaches.apktool:apktool:2.5.0'
                    break
                case 'dex2jar':
                    add name, 'dexpatcher-repo.dexpatcher.dex2jar:dex-tools:2.1-20190905-lanchon@zip'
                    break
                case 'dexpatcher':
                    add name, 'dexpatcher-repo.dexpatcher.dexpatcher-tool:dexpatcher:1.8.0-beta1'
                    break
                case 'dexpatcherAnnotation':
                    break
                case 'aapt2':
                    break
            }
        }
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

app build.gradle

//apply plugin: 'com.github.lanchon.dexpatcher.patched-application'
apply plugin: 'io.github.dexpatcher.patched-application'

dexpatcherConfig {
    patchedApplication {
        printAppInfo = true
        importSymbols = true
        disableResourceValidation = false
        useAapt2BundledWithApktool = true
    }

    dexpatcher {
        verbosity = QUIET
        logStats = true
        extraArgs.addAll([
                '--verbose',
                '--map-source', '--unmap-output', '--map', 'mapping.txt',
                '--deanon-source', '--reanon-output',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--no-reanon-errors',
                '--decode-patches', '--no-decode-errors',
        ])
    }

    apktool {
        frameworkInDir = layout.projectDirectory.dir('framework')
        keepBrokenResources = true
    }

    dex2jar {
        handleExceptions = true
        translateCode = true
    }
}

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.3"
    defaultConfig {
        minSdkVersion 23
        targetSdkVersion 29
        applicationId 'xxx.xxx.xxx.xxx'
        versionCode 100
        versionName '1.0.0'
        multiDexEnabled true
    }

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

    buildTypes {
        debug {
            debuggable true
            minifyEnabled false
            useProguard false
            proguardFiles getDefaultProguardFile('proguard-android.txt')
            signingConfig signingConfigs.debug
            jniDebuggable false
        }
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    sourceApk fileTree(dir: 'apkSrc', include: ['*.apk', '*.jar', '*.zip'])
    implementation fileTree(dir: 'libs', include: ['*.jar'])
//    implementation 'androidx.constraintlayout:constraintlayout:2.0.4'
//    implementation 'androidx.appcompat:appcompat:1.3.0-rc01'
//    implementation 'androidx.cardview:cardview:1.0.0'
//    implementation 'com.google.android.material:material:1.3.0'
}

build log

> Task :app:patchDexDebug
DexPatcher version 1.8.0-beta1 by Lanchon (https://dexpatcher.github.io/)
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app': 43801 types, 169 ms, 4 us/type
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug': 75 types, 1 ms, 13 us/type
stats: patch process: 43876 types, 336 ms, 8 us/type
stats: transform output: 43827 types, 16924 ms, 386 us/type
info: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug'
stats: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug': 43827 types, 279698 ms, 6382 us/type
stats: total process: 43876 types, 297159 ms, 6773 us/type
0 error(s), 0 warning(s)

And the original apk has 11 dex.

[Lanchon]

thanks

remove the '--verbose' item, it is handled by verbosity = ...

im worried by your use of '--no-reanon-errors' and '--no-decode-errors', why are they necessary?

i see you got how to using the advanced features in the latest beta from reading the very sparse docs, congrats.

i haven't used the new transform features (no time) on any real hacks, that's why it is all still in beta, i haven't tested. there could be issues of course. but i aim for correctness first, then deal with optimization if needed, so i haven't got to study performance issues other than at the design phase.

0.34 secs are used to patch the code.

17 s are wasted on the 'transform output' step. you can recover most of that by setting --pre-transform dry (so output code will be pretransformed only of not written -ie, if a dxp error happens). otherwise transform and write back will happen simultaneously. caveats: transform logs will be intermixed with write logs. dxp will produce an (invalid) output if errors happen during output transforms! (given that write starts before transforms are executed) this causes no issues with dxp-gradle though.

280 s are used to write back the dex, there is not much that can be done here. this is an issue with dexlib2 (smali): it is not written for performance.

but there is a way to multi-thread the write back phase: if you have a quad core machine you can have a significant speed gains in this phase. take a look here: https://github.com/DexPatcher/dexpatcher-tool/releases/tag/v1.2.0-beta2

note that multi-dex multi-threading is on by default when using dxp-gradle under certain conditions on debug builds!

how to check if you are using MT mdex? set verbosity = VERBOSE and re run build with logStats = true.

then only paste here the end of the build log like this:

info: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug'
stats: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug': 43827 types, 279698 ms, 6382 us/type
stats: total process: 43876 types, 297159 ms, 6773 us/type
0 error(s), 0 warning(s)

but this time you should see info for each dex file of the output multidex.

try it please, thanks.

[Lanchon]

oh, and please tell me what is the CPU model you use. thanks.

[lcm92]

AMD Ryzen 5 1600

[Lanchon]

ok. after you check what i asked, for that CPU id recommend you use multiDexJobs = n, with n ranging from 6 to 12. currently dxp-tool limits itself to 4 jobs due to java programs not being able to distinguish between CPU cores and hyperthreads. this of course requieres MT mdex being used.

[lcm92]

Thanks for the help, I'll give it a try

[lcm92]

 patchedApplication {
        printAppInfo = true
        importSymbols = true
        multiDexThreadedForMultiDexDebugBuilds = true
        multiDexThreadedForAllDebugBuilds = true
        disableResourceValidation = false
        useAapt2BundledWithApktool = true
    }

    dexpatcher {
        verbosity = VERBOSE
        logStats = true
        multiDex = true
        multiDexThreaded = true
        multiDexJobs = 12
        extraArgs.addAll([
                '--verbose',
                '--pre-transform', 'dry',
                '--map-source', '--unmap-output', '--map', 'mapping.txt',
                '--deanon-source', '--reanon-output',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--decode-patches',
        ])
    }

DexPatcher version 1.8.0-beta1 by Lanchon (https://dexpatcher.github.io/)
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app': 43801 types, 178 ms, 4 us/type
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug': 75 types, 1 ms, 12 us/type
stats: patch process: 43876 types, 328 ms, 7 us/type
info: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug'
stats: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug': 43827 types, 254646 ms, 5810 us/type
stats: total process: 43876 types, 255184 ms, 5816 us/type
0 error(s), 0 warning(s)

Does it work?

[Lanchon]

hey im sorry! its been a year for me since i touched any of this

i think you need more logging. is there a verbosity = DEBUG? try that instead. sorry :(

[lcm92]

I think you might want to check out my project.

[Lanchon]

dexpatcher {
        verbosity = VERBOSE
        logStats = true
        multiDex = true // THIS SHOULD BE AUTO TRUE IF THE APP MINSDK>=ANDROID 5 I THINK
        multiDexThreaded = true // THIS SHOULD BE AUTO TRUE FOR DEBUG BUILDS ONLY (RECOMMENDED) IF ABOVE CONDITION IS MET
        multiDexJobs = 12 // IT IS PROBABLY BETTER FOR THIS SETTING TO BE SET AT 6 FOR YOUR CPU
        extraArgs.addAll([
                '--verbose', // REMOVE THIS, THIS IS SET BY DXP-GRADLE BASED ON THE verbosity SETTING ABOVE
                '--pre-transform', 'dry', // THIS MADE THE TRANSFORM OUTPUT PHASE GO AWAY, WITH CAVEATS SAID EARLIER
                '--map-source', '--unmap-output', '--map', 'mapping.txt',
                '--deanon-source', '--reanon-output',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--decode-patches',
        ])
    }

[Lanchon]

from your latest times:

• transform output phase went away due to --pre-transform dry saving 17 s, but producing worse quality logs.
• i guess you are using MT mdex already. you can only increase jobs from default 4 to 6 then.
• you could pre transfrom the source to save some processing, but it's tricky with current dxp-gradle.
• instead of '--map-source', '--unmap-output', '--map', 'mapping.txt', you can inverse-map the patch; it will run faster and produce the same output. CAVEAT: all build logs will refer to the cryptic unmapped items!!!

you can try this:

• round-trip with apktool (no need to sign the apk, better if you skip resource processing) and inform decode and build times.

probably you'll find that smali/dexlib2 is just slow with your big app.

[lcm92]

Right now I cannot swallow such a mass of knowledge :D. I think I need to do more research. Thank you very much, your DexPatcher project is very good.

[Lanchon]

i just tried an apktool roundtrip on your apk (without resource decoding). it took 1m 24s, so there is ample space to increase performance. i suppose transforms are to blame here. i'll check options later when i can.

[lcm92]

I've done many projects with DexPatcher and this is the first one where I had such a problem. Thanks for the support.