Enable Browserstack iOS Tests

[karunkop]

Close #2127

Changes Made:

I have completely replaced the exisiting webdriverio infrastructure paired up with jest and browserstack-local lib with Webdriverio own test runner @wdio/cli which does all the configuration, framework and service such as browserstack management. Since wdio handles everything, i.e starting tunnel before tests, automatic stops after completion, we do not require manual tunnel setup like before. Also wdio is built specifically for browser/app automation (unlike vitest/jest which is for unit testing). wdio executes test suites with native WebDriver support via @wdio/mocha-framework and @wdio/browserstack-service which is responsible for automatic tunnel, session, and credential management. wdio also provides lifecycle hooks that are very helpful for initiating a session more effeciently.

The configuration files are now divided as wdio.base.conf.ts which contains common settings for ios safari testing, wdio.browserstack.conf.ts using @wdio/browserstack-service that creates a automatic tunnel with cloud browserstack service on the go, and wdio.local.conf.ts for local Appium testing which is only used incase we need to run tests on local IOS simulator.

Here is the relevant links to wdio documentation:

• https://v6.webdriver.io/docs/clioptions.html
• https://v6.webdriver.io/docs/configurationfile.html
• https://v6.webdriver.io/docs/browserstack-service.html

Running Tests:

Local Development

Run on BrowserStack
yarn run test:ios:browserstack

Run on local Appium (requires local setup)
yarn run test:ios:local

Prerequisites

Environment Variables:

.env.test.local
BROWSERSTACK_USERNAME=your_username
BROWSERSTACK_ACCESS_KEY=your_access_key

Start the Dev Server:

yarn start  # Runs on port 3000

Helpers Update:

tap.ts and gesture.ts:

Now updated to use performActions directly to avoid the automatic releaseActions call. Safari/XCUITest doesn't support the DELETE /actions endpoint (releaseActions) now which WebDriverIO's action().perform() calls automatically after performing. This was one of the main issues which was causing tests to fail. Minor adjustments/updates to some other helpers are also done along the way as per wdio browser API.

Note: There are still 5 tests that are skipped in caret.ts. These tests are needed to be studied in much more detail and debug accordingly.

As of now, running tests in CI is pretty straightforward. Although we need to do it manually for time-being as it is temporarily disabled expect when manually triggered. We can always check for builds in https://automation.browserstack.com/projects/em/builds. Local build and CI build are clearly identified because of their identifier. We can also walthrough the entire automation via https://automation.browserstack.com/projects/em/builds/CI/2?tab=tests&testListView=spec&details=2566272669 which was super helpful for debugging purpose :)

Note: I also did manage to restore ios tests in our old setup with vitest as well. main...karunkop:em:restore-browserstack-ios-tests. But I think it would be a good idea to upgrade the architecture with native webdriverio cli as it is very convenient and up to date.

[raineorshine]

Thanks! This looks fantastic!

Although we need to do it manually for time-being as it is temporarily disabled expect when manually triggered.

What's the plan for adding this to to GitHub Actions workflow? That’s pretty critical.

[karunkop]

Thanks! This looks fantastic!

Although we need to do it manually for time-being as it is temporarily disabled expect when manually triggered.

What's the plan for adding this to to GitHub Actions workflow? That’s pretty critical.

https://github.com/karunkop/em/blob/71bdda4b60bfad783a7daf6b6aae41f3594c6ddd/.github/workflows/ios.yml, I suppose running IOS tests in CI is pretty similar to how we run tests locally. We will need to comment out the disabled 'on: push' event in the yml file and remove the manual workflow_dispatch event. For now since the overall architecture and changes need to be reviewed I haven't enable automatic triggering of the workflow.

[raineorshine]

I pulled the PR, installed dependencies, and ran npm run test:ios:browserstack. The caret tests pass with enough retries, but it looks like the split and home tests are timing out. Can you see my build output in browserstack?

https://automate.browserstack.com/projects/em/builds/Local+rainerevere/3

[raineorshine]

Overall looks good! Just need your help getting them to pass on my end.

Please merge main and resolve merge conflicts when you get a chance.

[raineorshine]

This might be a bit slow since it has to cross the API boundary twice. Does the global browser have a native way to wait for a selector? Otherwise we should probably do the polling in the inner await.

We can come back to this later as an optimization but I wanted to point it out.

[raineorshine]

It might not be necessary to send the keys individually. After all, the native input event can even send multiple characters in a single event when typing quickly.

[raineorshine]

We can migrate this to directly imported helpers in another PR as we did with Puppeteer.

[raineorshine]

Great, thanks for updating this to match our Puppeteer helper.

[raineorshine]

TODO: Change to wait until

[raineorshine]

I'm also seeing a few log files which maybe need to be added to the gitignore to exclude them from source control:

• browserstack.log
• log/
• logs/