I find it belittling…the notion that everything must be tested by a tester before it goes to production. It means we test because of a procedure rather than to provide information that is valuable to somebody.
This morning our customers submitted a large job to one of our software products for processing. The processed solution was too large for our product’s output. So the users called support saying they were dead in the water and on the verge of missing a critical deadline. We had one hour to deliver the fix to production.
The fix, itself, was the easy part. A parameter needed its value increased. The developer performed said fix then whipped up a quick programmatic test to ensure the new parameter value would support the users’ large job. Per our process, the next stop was supposed to be QA. Given the following information I attempted to bypass QA and release the change straight to production:
- Testers would not be able to generate a large enough job, resembling that in production, in the available time given.
- There was no QA environment mirroring production bits and data at this time. It would have been impossible to stand one up before the one hour deadline.
- The risk of us breaking production by increasing said parameter was insignificant because production was already non-usable (i.e., it would be nearly impossible for this patch to make production worse than it already was).
Even with the above considerations, some on the team reacted with horror…”What? No Testing?”. When I mentioned it had been tested by a developer and I was comfortable with said test, the response was still “A tester needs to test it”.
After convincing the process hawks it was not feasible for a tester to test, our next bottleneck was deployment. Some on the team insisted the bits go to a QA environment first, even though it would not be tested. This was to keep the bits in sync across environments. I agree with keeping the bits in sync, but how about worrying about that once we get our users safely through their crisis!
As I watched the email thread explode with process commentary and waited for the fix to jump through the hoops, I also listened to people who were in touch with the users. The users were escalating the severity of their crisis and reminding us of its urgency.
I believe those who insist everything must be tested by a tester do us a dis-service by making our job a thoughtless process instead of a sapient service.
RSS
Great post. Couldn't agree more.
Process and "having a tester test it" makes some people feel safer. As if a tester can sprinkle magic safety dust over code. I like your analysis and conclusion that this fix should go right to production.
This is an interesting post. At my job we lack process and many times production fixes are implemented and QA is bypassed. One this to make note of here is that QA was NOT bypassed here. QA made the decision that the fix did not need to be tested before release to production given the constraints. Often times at my company production fixes are implemented without QA involvement at all. We usually find out about this "fixes" when defect are created for changes we were not expecting. I would love to have your problem with following process too strictly than not to have a process at all.
"You have 1 hour, go!"
So you have an hour to debug the problem, invent a solution, test it, and deliver it to production.
And it sounds like your usual test scenario would use more time than you had at your disposal.
So you made a decision to do the best you could in the time available to reduce the risk. And you kept people informed that you were (of necessity) deviating from the norm.
Sounds to me like you did exactly the right thing, at the right time.
Well done! Crisis handled!
Now, I'd expect a follow-up to decide if you should test more thoroughly, even though the fix is already in production, in case new problems are now looming.
And I'd expect a quick analysis to determine if this sort of 1-hour time frame is something you need to be prepared to handle on a regular basis - perhaps requiring a new, quicker test method - or not.
Overall - a job well done in my book! And an excellent example of when it's not only appropriate, but necessary, to deviate from the "best practices/standards/norms".
This is a typical disregard for qa process which is being viewed as a bottleneck. I have heard devs on many occasions claim a safe and targeted fix, only to cause regression in a previously unrelated operation. I am sure that the customer would not want to request a second hotfix to address the gaps from the first emergency. Perhaps investing in an automated acceptance test as a way to qualify the change and thereby establish some level of quality would be the proper remedy. If the qa operations are too rigid, then they cannot react to customer priorities.
Since it's a report, and reports are read-only you could also have changed the connection string to point to Prod temporarily in the QA env. to see if the report would work correctly if the issue is with time constraints of getting Prod data refreshed into a non-Prod environment.
formaturface,
This was not a report. It was on a project you don't work on. A project named after a bird.
However, I like your suggestion.
BTW - I always figured it was you who added "format your face" 5 years ago.
Process should never come before common sense
Yaniv,
BAM! You summed it up well.