One million dollar mistake in the instrumental tests

Photo by Jp Valery on Unsplash

Creator of a flaky UI test should bring a cake for the team during the week duration at least

I love instrumental tests recently. With the robot pattern, we gain high maintainability. With the build scripts for emulator creation+setup and the additional tests restart if failed, we gain around 99% stability. However, it was not always like this.

The setup for one million dollar mistake

We mock the network calls with OfflineProxy class and unfortunate it is kind of singleton set up in our tests. We configure network responses in the test setup, usually like:

@Before
fun setUp() {
    OfflineProxy.recordedResponses([json file]+)
}

We also have the test rule to mock some essential dependencies and OfflineProxy as well. As well we made clean up of the OfflineProxy after the activity is finished.

open class MockServerTestRule<T : Activity>(
        activityClass: Class<T>,
        initialTouchMode: Boolean = false,
        launch: Boolean = true,
        private val pollInterval: Long = 100
) : ActivityTestRule<T>(activityClass, initialTouchMode, launch) {
    @CallSuper
    override fun beforeActivityLaunched() {...}
    @CallSuper
    override fun afterActivityFinished() {
        OfflineProxy.cleanup()
    }
}

If you haven’t spotted the mistake, please stay tuned.

The mistake explanation

The lifecycle of the activity doesn’t correspond with the lifecycle of the test! Sweet!

The setup of the next test might be called before the activity finishes for the previous one.

The proper solution is not to have a shared state between tests if you can’t don’t rely on the order of the two methods above.

Thank you for reading! Clap if you like it.