How to Make Sure Others Don’t Use Our Test Data

Written By Paul Merrill

Hi Paul, thanks for replying. I was wondering how to approach the test data in a Playwright script, I test an ecomm shop and I buy products. The app is connected to ERP as well so the products must exist there and be in stock. My current test is now failing since the product data was changed. The price got discounted and this test isn't meant to take that into consideration.

So, I was wondering, what is the best approach here? I usually take the first or second product from the list in a certain category. The env for automation is the same that manual testers use (including me). I don't know what would be the best practice to make sure test data is not tampered with in any way? If you have any kind of advice, I would appreciate it. Thanks! - Marijana

In 2016, I presented my first major conference talk at StarWest. The presentation was on Test Data Strategies. Since then, I’ve written and presented about it at other conferences, on webinars, as an invited guest on podcasts. Find links to those at the end of this post.

Let’s answer this question…

“The app is connected to an ERP… so the products must exist there and be in stock.”

The app is the system under test. It relies on the ERP (Enterprise Resource Planning, examples include: MS Dynamics 365, SAP, or Oracle NetSuite).

If you haven’t worked with ERP systems, you may not understand their complexity. ERP systems are often extremely complicated containing complex relationships between data. That means duplicating or generating data for tests is often difficult to impossible.

It is likely, this app can’t function without the ERP system. Further, it is likely the app can’t function without organic data. Chances are, this team is working with a pre-production QA system that uses a snapshot of data from production. These are a few of my assumptions. If I were working with this team, I would verify my assumptions before making a recommendation on what to do. In the rest of this post, I’ll make recommendations based on these assumptions and leave it to Marijana’s team to validate or invalidate them.

I try to take a simple approach to things. I love the concept of “The simplest thing that could possibly work”. When creating my first tests in this system, I would have done exactly as Marijana did. I would have used data from the system under test.

Further, I would have made the test use queried data, just as Marijana has done. While she queried through the user interface, I prefer to query the database directly, but either way, querying data is the “simplest thing that could possibly work” to get the test working.

 

“My current test is now failing since the product data was changed. The price got discounted and this test isn't meant to take that into consideration … the env for automation is the same that manual testers use (including me).”

No surprise, we have shared data. Someone or something is not sharing nicely. Likely, another user innocently changed the data.

I’d ask Marijana if the price is important for what the test is intended to verify. Often, tests check attributes that are irrelevant and do not help the test toward its purpose. Automated tests should ONLY check what they need to, this keeps tests small and independent making them more robust and easier to maintain (download my “5 Secrets of Test Automation”).

Who or what is changing the data?

Our approach to solving Marijana’s issue depends on the answer. If a system is changing it, we likely need a systematic solution. If people are changing it, we would prefer a systematic solution (preferably, a programmatic solution), but if that doesn’t work, we’ll need a convention, process, or rule-based approach. I prefer lightweight solutions over heavyweight solutions.

 

“what would be the best practice to make sure test data is not tampered with in any way?”

We can’t “make sure” of this. But you can mitigate the likelihood and impact of tampering.

We need a way for the test automation to use pristine data. We can either implement a solution using:

  1. People – Set up conventions, processes or rules for the teams OR

  2. Code – Set up data in a way that only test automation uses it.

Option 1 – Implement with people

If we’re going to use people as a solution, support them with the clearest possible communication and simplest process. Too often, we create complexity for complexity’s sake.

Option 1 is easier to put in place and takes less time to implement, but has all the shortcomings humans offer. People make mistakes, conventions and rules are thwarted, and implementation of rules is error-prone.

Option 2 – Implement with code

Option 2 would be ideal. If we can program a solution we get the benefits of:

  • traceability;

  • reliability;

  • troubleshooting of code, rather than people; and

  • iterative improvement of the solution.

 

Solutions

Whenever possible, I like to generate data for use with test automation. Generating data for test is the most successful test data strategy. When each test run can work with its own data, conflicts are rare.

In this case, however, the ERP system makes generating good, useable data very difficult if not impossible.

Since we must use existing data, the next best solution is to divvy up the data and assign it for use.

Divvying up data

We need to identify types of data in the failed tests – what is common in the data? Ask:

  • What do we call this data? Does what we call it tell us what the group of data is?

  • What aspects or attributes of the data are important? Is it the same for each test?

  • How many pieces of data are like this (broadly, a lot of a little)?

We should start to see groups of data emerging. Are they both products that are on sale? Are they both products that must be ordered online? If there is commonality and there are a lot of them, then we need to decide how many of these each of us needs.

When we identify the type or group of the data, we will also begin to identify the meaningful part of the queries we’ll use to get the data. If the queries for automated and manual tests return distinctly different results, then we don’t need to worry about divvying up the data. If they return overlapping results, then we may need to modify one of the queries to get distinct data for it. If they return the same results, then we need to divvy up the data.

A few ideas for divvying up data:

  • Count them and split. Use an identifier like a sku or product id. For example, we may decide items above sku 5555555 are for manual, items below are for automation;

  • Use another attribute that is orderable, like date created and split up. One group gets recently created, one gets old.

This will all be recorded in the coded query used by the test automation. So, automation will follow the divvying up agreement. It’s more difficult with testers. When executing a test manually, we will need to remember and be consistent with what data we use.

We can help testers by partnering with test leaders to create a cheat sheet or simple listing of the ranges of records that can be used for automation and manual.

Because it’s so easy for us to use the wrong data, I like to make it as dummy-proof as possible. Adding a clear visual indicator to the data can help, like signing data.

Assignment

Once we divvy up data, we need to assign it to someone or something for use. For some teams, the act of divvying up the data will automatically assign the data to someone. For instance, items 0-9999 are for the pricing team’s manual testing and items 10000-19999 are for the pricing team’s automation.

For some teams it won’t be that easy. This will depend on a number of factors including how the data is shared, modified, created or pulled. It will also depend on the way automation is created and the type of testing being done.

There are several options for assigning data:

  • One-time assignment

  • Runtime assignment with checkout

  • Assignment by convention

    • Marked

    • Signed

 

One-Time Assignment

When a team divvies up data and assigns it to those who run tests in one fell-swoop, we call it “one-time assignment”. The teams decide which data is used by which teams, as described in the first paragraph of the “assignment” section above.

Runtime Assignment with Checkout

An alternative to one-time assignment is runtime assignment and checkout. At runtime the person manually executing a test case “checks out” the data in a checkout system. If this is an automated test case, the test itself checks out the data.

This means we would need to create a checkout system. The checkout system must be programmatic for automation to be able to use it.

The downsides to this approach far outweigh the upside. Development of the checkout system and maintenance of it alone are extremely costly for most teams. Solving the problem of “locking” the data by checking it out is non-trivial. Many folks have spent years studying race conditions, which is what we will fight when implementing this system. We will get more control over the problem, but we have to ask, at what cost?

Having tried this, I would make every effort to dissuade almost everyone from attempting to create a checkout system for test data.

Assignment by Convention – Marked Data

I do NOT recommend this solution for Marijana’s system. Generally, modifying the schema of an ERP system is a non-starter, but I’m including this approach for completeness.

With this approach we mark data in the database with a flag to denote it as test data. The idea is, that the schema includes one or more fields for each record. These fields are flags indicating this data is used for testing.

While a simple flag may work, other fields may help with troubleshooting and debugging such as: is_test_data, last_used time, user, and test_execution_id.

 

Assignment by Convention – Signed Data

We can "sign” the data with a recognizable signature. Signatures are recognizable by the automated tests and by testers.

Usually, there is an attribute that we can use to denote that the data is assigned. It could be as easy as a product name or sku prefixed with “AT-” (for Automated Testing) or “MT-“ (for Manual Testing). It could be modifying a serial number or product description with a known prefix or postfix.

I call this a signature, because it should be unique, and it should be easily identifiable and queriable.

The uniqueness of the signature depends on the problem. Some teams will need a unique identifier for automated tests vs manual tests. This can be done each time the data is refreshed from production. Some teams will need uniqueness for the tester. Some may even benefit from having unique test execution ids with a timestamp.

In the case of a system like Marijana’s, where data is pulled from production periodically, we need the team pulling the data to modify the data with signatures every time data is pulled. This should be done with a script and should be automatically triggered when data is pulled.

We should expect to incrementally improve this script over time. We will not get it exactly right the first time, it is not a one-and-done work item. We don’t know all the data we need to assign. We don’t know all the impacts our changes may have. We don’t know if the data we assigned is all the data we will ever need to assign. We don’t know if the balance we decided on during divvying up data was correct. We need to remain open to learning and adapting our scripts as we go.

Some teams will even add discussions on shared data into their Agile retrospectives with a standard question like “did we identify any new shared data we need to modify or handle?”

This solution doesn’t keep any human from using the data, but because we are persisting decisions in code we can control, fix, and develop our decisions as we learn.

By taking this approach we’ve systematically mitigated the risks of sharing data. We’ve also put ourselves in the position to log what the data we use in testing which will allow us to better troubleshoot issues in the future.

Testing your tests

I don’t create automated tests for automate tests. But I do test the testing infrastructure I’ve built. Some tests that may be helpful:

  • A sanity test to make sure the data in the system is “ready” for tests to use (have our scripts run?),

  • Tests for any data checkin/checkout system we’ve created,

  • Tests for scripts that modify production data, so that we know when they have changed,

  • Sanity tests for signatures – do they look right?

 

Finally

In Marijana’s situation, I would recommend trying the following approaches and use the first one that works:

  1. Divvying up data with a one-time assignment; OR

  2. Assignment by convention with signed data

Shared data is the single biggest culprit in failed test execution after defects, poor test automation code, bad assumptions, and changes to the SUT. This post will help us handle it responsibly.

Happy automating!

 

References:

Paul Merrill https://dpaulmerrill.com
Previous

5 WAYS TO SIMPLIFY YOUR AUTOMATED TEST CASES
Next

Test Data Definitions