“Golden Master”, it sounds like the bad guy in a James Bond movie. I first heard the term used by Doug Hoffman at STPCon Spring 2012 during his Exploratory Test Automation workshop. Lately, I’ve been writing automated golden master tests that check hundreds of things with very little test code.
I think Golden-Master-Based testing is super powerful, especially when paired with automation.
A golden master is simply a known good version of something from your product-under-test. It might be a:
- web page
- reference table
- grid populated with values
- report
- or some other file output by your product
Production is an excellent place to find golden masters because if users are using it, it’s probably correct. But golden masters can also be fabricated by a tester.
Let’s say your product outputs an invoice file. Here’s a powerful regression test in three steps:
- Capture a known good invoice file from production (or a QA environment). This file is your golden master.
- Using the same parameters that were used to create the golden master, re-create the invoice file on the new code under test.
- Programmatically compare the new invoice to the golden master using your favorite diff tool or code.
Tips and Ideas:
- Make sure the risky business logic code you want to test is being exercised.
- If you expand on this test, and fully automate it, account for differences you don’t care about (e.g., the invoice generated date in the footer, new features you are expecting to not yet be in production).
- Make it a data-driven test. Pass in a list of orders and customers, retrieve production golden masters and compare them to dynamically generated versions based on the new code.
- Use interesting dates and customers. Iterate through thousands of scenarios using that same automation code.
- Use examples from the past that may not be subject to changes after capturing the golden master.
- Structure your tests assertions to help interpret failures. The first assertion on the invoice file might be, does the item line count match? The second might be, do each line’s values match?
- Get creative. Golden masters can be nearly anything.
Who else uses this approach? I would love to hear your examples.
RSS
This also works well for applications that output XML data or captures events in XML format. You can diff the XML output with the golden file. You can even make your own diff tool where you can give it a list of XML tags to ignore. For example timestamps, date, time are commonly ignored. But sales totals, number of requests, account names, error type, etc would be compared.
Moving this into CI then is fairly easy. The golden masters are checked into source code control just like the code for the application under test. If the new version of the application changes the XML, a new version of the XML can be checked in as well after the differences are approved by QA.
If you are testing an application that updates a database. At the end of a series of tests(sanity, smoke, regression, etc), the database can be exported to XML and this can be used as a golden master as well.
Big Data can use this method as well. The output of a Hadoop job, once verified by QA, can be used as a golden master file.
Big fan of "golden data" here. You have to make sure that your new code will work with what's set up in prod (at least). It's also helpful for setup and tear down for automation and experimentation.
Big fan of "golden data" here. You have to make sure that your new code will work with what's set up in prod (at least). It's also helpful for setup and tear down for automation and experimentation.
Yep, using a "Baseline" for the A vs. B comparison (actual vs. expected) is an ideal thing to do. Nothing new. Been doing it for years.
Sorry, I'm just an olde fart who thinks the renaming/rebranding of terms is just insane right now in the testing world.
As a keynote presenter at a conference I attended years ago said "I know 21 languages! All of them English."
Good post Eric, sorry if you take this to mean I am slamming on you. I'm not, just pointing out that the vocabulary of testing is out of whack.
Regards,
Jim Hazen
I would think at a granular level, this could apply to unit, API, integration tests that are more developer centric as well?
This can also be used at the UI level with imaged based comparison. It is possible to do pixel based comparison with a golden master. A good way to catch visual regressions. You can have your automated UI test take a screen shots and you can identify your golden master. Then you can have your automated test run, take snapshot and your comparison logic can compare them and fail if the threshold for pixel difference is crossed. There are a few production ready tools that can do this. I have actually written a tool and it was easier than I thought.
So, the golden master is a very valuable and efficient was to to quickly test regressions in a variety of types of automated tests.