A lot of failure here, but also a lot of learning... #3

randytarampi · 2019-04-23T05:36:32Z

Lots of tiny changes, and a couple notes about failures with Travis and Perfecto.

Running headless Android on cloud CI infrastructure is definitely not the way to go you just can't run them.

The biggest change is is the addition of a Travis CI pipeline in 796a557, and the usage of jest-circus in 5ff010e.

Meat

796a557, 2341c2b – Detail where I gave up on Travis, but add a basic CI pipeline

If you want to run this on your own @hadnazzar, you'll need to replace the openssl aes-256-cbc -K $encrypted_78b6aab41b54_key -iv $encrypted_78b6aab41b54_iv -in android/app/my-release-key.keystore.enc -out android/app/my-release-key.keystore -d line with your own by adding your own keystore and following these instructions
- Or you could just commit the keystore, since the credentials are already in there... 😂

Various friends and I made a bit of a drinking game of guessing which combination of ANDROID_EMULATOR_API_VERSION, ANDROID_EMULATOR_ABI, and ANDROID_EMULATOR_PLATFORM_VERSION. You can give it a shot by running a custom build (or replacing .travis.yml) with the following configuration:

notifications:
  email: false
os: linux
dist: trusty # NOTE-RT: This needs to be `trusty`, android builds just hang on `xenial` :\
env:
  matrix:
    - ANDROID_EMULATOR_API_VERSION=25 ANDROID_EMULATOR_ABI=arm64-v8a ANDROID_EMULATOR_PLATFORM_VERSION=7.1
    #    - ANDROID_EMULATOR_API_VERSION=24 ANDROID_EMULATOR_ABI=arm64-v8a ANDROID_EMULATOR_PLATFORM_VERSION=7.0
    - ANDROID_EMULATOR_API_VERSION=25 ANDROID_EMULATOR_ABI=armeabi-v7a ANDROID_EMULATOR_PLATFORM_VERSION=7.1
    #    - ANDROID_EMULATOR_API_VERSION=24 ANDROID_EMULATOR_ABI=armeabi-v7a ANDROID_EMULATOR_PLATFORM_VERSION=7.0
    - ANDROID_EMULATOR_API_VERSION=23 ANDROID_EMULATOR_ABI=armeabi-v7a ANDROID_EMULATOR_PLATFORM_VERSION=6.0
    - ANDROID_EMULATOR_API_VERSION=22 ANDROID_EMULATOR_ABI=armeabi-v7a ANDROID_EMULATOR_PLATFORM_VERSION=5.1
    #    - ANDROID_EMULATOR_API_VERSION=21 ANDROID_EMULATOR_ABI=armeabi-v7a ANDROID_EMULATOR_PLATFORM_VERSION=5.0
    - ANDROID_EMULATOR_API_VERSION=28 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=9
    #    - ANDROID_EMULATOR_API_VERSION=27 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=8.1
    #    - ANDROID_EMULATOR_API_VERSION=26 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=8.0
    - ANDROID_EMULATOR_API_VERSION=25 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=7.1
    #    - ANDROID_EMULATOR_API_VERSION=24 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=7.0
    - ANDROID_EMULATOR_API_VERSION=23 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=6.0
    - ANDROID_EMULATOR_API_VERSION=22 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=5.1
  #    - ANDROID_EMULATOR_API_VERSION=21 ANDROID_EMULATOR_ABI=x86 ANDROID_EMULATOR_PLATFORM_VERSION=5.0
  global:
    - ANDROID_PLATFORM_VERSION=9
    - ANDROID_API_VERSION=28
    - ANDROID_TOOLS_BUILD_VERSION=4333796 # NOTE-RT: To match `ANDROID_API_VERSION`
    - ANDROID_BUILD_TOOLS_VERSION=28.0.3
#    - ANDROID_EMULATOR_API_VERSION=28
#    - ANDROID_EMULATOR_ABI=x86
#language: node_js
#node_js: 10
#cache: yarn
#jobs:
#  include:
#    - name: Unit tests
#    - name: E2E tests (android)
sudo: required
language: android
jdk: oraclejdk8
cache: yarn
android:
  components:
#    - tools
#    - android-$ANDROID_EMULATOR_API_VERSION # NOTE-RT: Per https://medium.com/@nocnoc/android-emulators-in-thcloud-f39e11c15bfa
#    - emulator
    - tools
    - platform-tools
#    - build-tools-$ANDROID_BUILD_TOOLS_VERSION
#    - android-$ANDROID_API_VERSION
#    - extra-google-google_play_services
#    - extra-google-m2repository
#    - extra-android-m2repository
#    - addon-google_apis-google-$ANDROID_API_VERSION
#    - sys-img-$ANDROID_EMULATOR_ABI-google_apis-$ANDROID_EMULATOR_API_VERSION # NOTE-RT: Per https://medium.com/@nocnoc/android-emulators-in-thcloud-f39e11c15bfa
before_install:
  - export ANDROID_HOME=~/android-sdk
  - export ANDROID_SDK_ROOT=$ANDROID_HOME
  - |
    if [ $TRAVIS_OS_NAME == "linux" ]; then
      export DISPLAY=:99.0;
      sh -e /etc/init.d/xvfb start;
      sleep 3; # give xvfb some time to start
    elif [ $TRAVIS_OS_NAME == "osx" ]; then
      export HOMEBREW_NO_AUTO_UPDATE=1;
      brew cask uninstall java; brew tap caskroom/versions; brew cask install java8;
      sudo ln -sf /usr/share/zoneinfo/US/Pacific /etc/localtime;
    fi
  - |
    if [ $TRAVIS_OS_NAME == "linux" ]; then
      wget -q "https://dl.google.com/android/repository/sdk-tools-linux-$ANDROID_TOOLS_BUILD_VERSION.zip" -O android-sdk-tools.zip;
    elif [ $TRAVIS_OS_NAME == "osx" ]; then
      wget -q "https://dl.google.com/android/repository/sdk-tools-darwin-$ANDROID_TOOLS_BUILD_VERSION.zip" -O android-sdk-tools.zip;
    fi
  - unzip -q android-sdk-tools.zip -d ${ANDROID_HOME}
  - rm android-sdk-tools.zip
  - PATH=${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools
  - mkdir -p ~/.android
  - touch ~/.android/repositories.cfg
  - yes | $ANDROID_HOME/tools/bin/sdkmanager --licenses
  - $ANDROID_HOME/tools/bin/sdkmanager \
    "tools" \
    "platform-tools" \
    "emulator" \
    "build-tools;$ANDROID_BUILD_TOOLS_VERSION" \
    "system-images;android-$ANDROID_EMULATOR_API_VERSION;google_apis;$ANDROID_EMULATOR_ABI" \
    "patcher;v4" \
    "platforms;android-$ANDROID_API_VERSION" \
    "platforms;android-$ANDROID_EMULATOR_API_VERSION" \
    "extras;google;google_play_services" \
    "extras;google;m2repository" \
    "extras;android;m2repository" > /dev/null
  - openssl aes-256-cbc -K $encrypted_78b6aab41b54_key -iv $encrypted_78b6aab41b54_iv -in android/app/my-release-key.keystore.enc -out android/app/my-release-key.keystore -d
  - export NODE_VERSION=${TRAVIS_NODE_VERSION:=10}
  - nvm install $NODE_VERSION
  - npm install -g yarn
install:
  - yarn install
  - npm run build:android
  - ls -al android/app/build/outputs/apk/release/app-release.apk
before_script:
  - echo no | $ANDROID_HOME/tools/bin/avdmanager create avd -n $(node -p 'require("./package.json").name') -k "system-images;android-$ANDROID_EMULATOR_API_VERSION;google_apis;$ANDROID_EMULATOR_ABI" # NOTE-RT: This should really be `yarn run emulator:android:create`
  # NOTE-RT: The following here should really be `npm run pree2e`
#  - EMU_DEBUG="-debug all -logcat *:e -verbose -show-kernel"
  - EMU_ADDITIONAL_OPTIONS="-no-boot-anim -no-audio -memory 2048"
  - |
    if [[ $ANDROID_EMULATOR_ABI == "x86" ]]; then
      EMU_PARAMS="-no-accel -no-window -gpu swiftshader"
    elif [[ "${ANDROID_EMULATOR_API_VERSION}${ANDROID_EMULATOR_ABI}" == "25armeabi-v7a" || "${ANDROID_EMULATOR_API_VERSION}${ANDROID_EMULATOR_ABI}" == "24armeabi-v7a" ]]; then
      EMU_PARAMS="-no-window -gpu off"
    else
      EMU_PARAMS="-no-window -gpu off"
    fi
  - $ANDROID_HOME/emulator/emulator -avd $(node -p 'require("./package.json").name') $EMU_PARAMS $EMU_ADDITIONAL_OPTIONS ${EMU_DEBUG} &  echo $! > /tmp/$(node -p 'require("./package.json").name').emulator.android.pid; echo "Started Android emulator (`cat /tmp/$(node -p 'require("./package.json").name').emulator.android.pid`) from /tmp/$(node -p 'require("./package.json").name').emulator.android.pid"
  - ./bin/android-wait-for-emulator
  - $ANDROID_HOME/platform-tools/adb devices
  - $ANDROID_HOME/platform-tools/adb shell input keyevent 82 &
  - npm run appium:kill
  - npm run appium:start
script:
  - ANDROID_PLATFORM_VERSION=$ANDROID_EMULATOR_PLATFORM_VERSION JEST_TIMEOUT=900000 DEVICE_TIMEOUT=300000 npm run appium:test
after_script:
  - npm run poste2e # NOTE-RT: Make sure we kill things and clean up
  - npm run emulator:android:delete

5ce8b32 – Detail how far I got with Perfecto

Sides

cb6d810 – npm run e2e starts & stops an android emulator
46ec27a, 741d133, 71fedc9, 4cc3387 – Add a bunch of android specific Appium capabilities.
0ee6df7 – npm run emulator:android:create should be more general
9a83559, 733cf1f – Add a DRIVER_INITIALIZATION_TIMEOUT that sleeps the tests for some time (to allow the emulator to start the app)
d19f7fa – Don't default IOS_DEVICE_NAME or IOS_PLATFORM_VERSION.

Dessert

5ff010e – Set jest's testRunner to jest-circus/runner.

Android specific for now, but we'll cross the iOS bridge when we get there...

Just for slow environments.

It's not totally necessary.

…ut` are actually in seconds, not milliseconds.

Per jestjs/jest#2713 (comment)

…or longer before tests.

@hadnazzar

* Try hooking this up all up in Travis. Two jobs, one to run the unit tests and one to run the E2E ones headlessly on Travis infrastructure. Tempted to add jobs to build and run against BrowserStack, Sauce Labs and TestObject, but one thing at a time first... @hadnazzar already committed the dummy credentials in `android/gradle.properties` though, so it's not like having Travis decrypt this committed keystore will be more secure or anything. If anything, it's worse 😂 * Just install Android SDK dependencies ourselves. Travis is still using the deprecated `android` command to install things per travis-ci/travis-ci#8874 (comment), which might be behind this issue (and others to come). * Actually, just prefer letting Travis install these (properly). Copy/pasta problems coming from the `sdkmanager` command notation - `android` should still be able to find these versions no? * Try for `ANDROID_BUILD_TOOLS_VERSION` at 28.0.3. * `npm install -g yarn` `before_install` on `E2E tests (android)`. * Use the given `nvm` to install `$TRAVIS_NODE_VERSION`. Travis still sets this right? I mean, the version is in the build name... * Defer to `TRAVIS_NODE_VERSION` if it exists, otherwise use `10`. * Explicitly say `yes` to all `sdkmanager --licenses`. * Just explicitly install the Android SDK per travis-ci/travis-ci#8874 (comment). Why wasn't this license already accepted? * Explicitly call `$ANDROID_HOME/tools/bin/sdkmanager`. And accept all licenses. * Remove extra slash. * `${ANDROID_HOME}` is what needs to be evaluated, not `$ANDROID_HOMEtools` * This wanted to be a space, not a `"`. * Whitelist all the licenses, even though it shouldn't matter... * Set `supportLibVersion` to be the same as `buildToolsVersion`. * Move `node_js` language/version selection to the `Unit tests` job. * Revert "Set `supportLibVersion` to be the same as `buildToolsVersion`." Yeah. No. This reverts commit 6fd8f25. * Just `echo` in the license manually... * Run `build:android`, not `bs:android`. * Actually specify the correct `ANDROID_API_VERSION`. And use `android create avd` and `android delete avd` instead of `avdmanager`, which doesn't seem to get installed properly. * `pree2e` script should `adb wait-for-device get-serialno`. * Note the closest I got to getting Travis to run the app on an emulator. I'm pretty sure I'm just running into a race condition somewhere installing the app, but I can't quite tell exactly where, or what to wait for. * Just bring our own Android environment per https://medium.com/@nocnoc/android-emulators-in-thcloud-f39e11c15bfa. It boots the emulators, but getting Appium to install the emulators is a whole other story... * Fix missing `ANDROID_TOOLS_BUILD_VERSION` and incorrect `ANDROID_EMULATOR_PLATFORM_VERSION`. * Define `ANDROID_HOME` and `ANDROID_SDK_ROOT` in `before_install`. Not `before_script`. * Try testing with `android-23` on `armeabi-v7a`. * Try and let `appium` boot and initialize the emulator on its own. * Only `sleep` for at most 60 seconds `beforeAll` `wd` E2E tests. * Set Travis timeouts to be 10 minutes. Worth a shot I guess? * Revert "Only `sleep` for at most 60 seconds `beforeAll` `wd` E2E tests." Just defer to the `DEVICE_TIMEOUT` here. This reverts commit 4f1416c. * Don't bother caching `gradle` artifacts. * Just boot the emulator ourselves. This commit reverts (conceptually) 4ae60c0. * `sdkmanager` installs `tools` on its own, to avoid `sdkmanager` errors. I don't know why it can't find `tools` sometimes, but this seems to help. * I think the whitespace here is tripping Travis up. It finds `tools` just fine when I copy/paste this file in... * Travis allows the local E2E Android tests to fail. * Add a separate `E2E` stage.

Let's not use these guys. Their examples are over two years old and their documentation is all kinds of contradictory and confusing – just look at all my notes inline. Looks like the below _should_ work, but it doesn't, and their API doesn't return anything useful. I'll link their support to this issue and maybe they can point us in the right direction I'm already not feeling them. ```json { "user": "randy.tarampi@smunch.co", "password": <REDACTED>, "platformName": "Android", "automationName": "UIAutomator2", "platformVersion": "9", "app": "PRIVATE:apktest.apk", "automationInfrastructure": "UIAutomator2", "appPackage": "com.apktest", "appActivity": ".MainActivity" } ```

I don't know why they want this specifically, but let's give it to them.

Since it looks like this is our new pick for a webdriver.

randytarampi and others added 17 commits April 22, 2019 13:58

Automatically start (and kill) an android emulator on npm run e2e.

cb6d810

Android specific for now, but we'll cross the iOS bridge when we get there...

Specify a adbExecTimeout

46ec27a

Add a bunch of android specific timeouts. And make them long.

741d133

Just for slow environments.

emulator:android:create script shouldn't specify a device.

0ee6df7

It's not totally necessary.

Add deviceReadyTimeout and note how it and `androidDeviceReadyTimeo…

71fedc9

…ut` are actually in seconds, not milliseconds.

disableWindowAnimation on Android clients in CI or HEADLESS envs.

4cc3387

Prefer jest-circus/runner so we bail out when beforeAll explodes.

5ff010e

Per jestjs/jest#2713 (comment)

emulator:android:create shouldn't require any input.

8c21d70

Give appium some more time to install the application by sleeping f…

b78cf78

…or longer before tests.

Add a separate DRIVER_INITIALIZATION_TIMEOUT.

9a83559

Set DRIVER_INITIALIZATION_TIMEOUT on Android (local) to 5 minutes.

733cf1f

Explicitly allow the E2E Android (local) job to fail.

2341c2b

Don't provide any default IOS_DEVICE_NAME or IOS_PLATFORM_VERSION.

d19f7fa

chore: Set BrowserStack appium_version properly.

da188e2

I don't know why they want this specifically, but let's give it to them.

chore: Reinstate WDIO tests.

4521a3a

Since it looks like this is our new pick for a webdriver.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A lot of failure here, but also a lot of learning... #3

A lot of failure here, but also a lot of learning... #3

Uh oh!

randytarampi commented Apr 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

A lot of failure here, but also a lot of learning... #3

Are you sure you want to change the base?

A lot of failure here, but also a lot of learning... #3

Uh oh!

Conversation

randytarampi commented Apr 23, 2019

Meat

Sides

Dessert

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant