On a production support kanban development team, a process dilema came up. In the case where something needs to be tested by a tester:
- Should the tester perform the testing first in a development environment, then in a production-like environment after the thing-under-test has been packaged and deployed? Note: in this case, the package/deploy process is handled semi-manually by two separate teams, so there is a delay.
- Or, should the tester perform all the testing in a production-like environment after the thing-under-test has been packaged and deployed?
Advantage of scenario 1 above:
- Dev environment testing shortens the feedback loop. This would be deep testing. If problems surface they would be quicker and less risky to fix. The post-package testing would be shallow testing, answering questions like: did the stuff I deep tested get deployed properly?
Advantage of scenario 2 above:
- Knock out the testing in one environment. The deep testing will indirectly cover the package/deployment testing.
From the surface, scenario 2 looks better because it only requires one testing chunk, NOT two chunks separated by a lengthy gap. But what happens if a problem surfaces in scenario 2? Now we must go through two lengthy gaps. How about a third problem? Three gaps. And so on.
My conclusion: Scenario 1 is better unless this type of thing-under-test is easy and has a history of zero problems.
2 comments:
Subscribe to:
Post Comments (Atom)
RSS
Where I am now, we test only in development environments, ones we entirely control. A part of me feels uneasy about it, as it just feels better to me to test in a production-like environment. But our developers deliver pretty solid work to us, and we can fail fast in Production, and developers and testers at least do pause together to discuss higher risk changes before they're built. A couple times that's led us to use one of our staging environments, which are more production-like, to test something. We keep monitoring this approach for effectiveness and should it slide south we will reevaluate and adjust.
I've worked on projects where we had to pro-actively stop testing because of release management issues. The code to be tested was developed on existing code in the development environment that was not in line with what was in production.
Based on these experiences I would go with Scenario 2 and deep-test in both environments. I control the test environment and make sure that it's in line with what is in production. This helps me make sure that what has been developed has been deployed the right way. And it helps to know that the right code has been deployed!