Quality Assurance hints for Testing Database and Data Synchronization
Most likely you'll be looking at a subset of these items. And I've noticed that several areas may not be popular concerns, but generally I feel it's better for a QA Engineer to be mindful of all of these, regardless of whether they are encouraged to consider them. And the terms I picked are not necessarily standardized - though I tried to make it obvious as to what I was referring to. This is meant to be helpful to whomever needs it. Happy Holidays!
- Positive Test Cases
- Should be relatively straight-forward to understand. Generally consists of variations/combinations on Data Inserts, Data Updates and Data Deletes.
- Negative Test Cases
- Again, should be relatively straight-forward to understand. Generally consists of variations/combinations on Data Inserts, Data Updates and Data Deletes. But will generally involve playing (incorrectly) with Key combinations as well (to force a negative test scenario).
- Border Test Cases (0 diffs to Sync, 1 diff to Sync, minimum - 1, minimum, minimum + 1, maximum - 1, maximum, maximum + 1, maximum + (increase in number of digits/characters)
- The minimum value will typically be defined by the frontend/UI. However, the maximum may be defined either by the UI or by the underlying database.
- It may be good enough to look at a subset of these test cases. For instance, 0 and 1 diff may not be necessary. However, even if they are not it may help to use them as some sort of baseline test case.
- Exception Test Cases (Functional/API failures returning Errors)
- Of less concern if there is no public API. However, similar to Negative test cases you can most likely trigger a Database Exception by playing with the database schema.
- Scalability Test Cases (both Application-based and data based)
- Usually will be defined at the Server level (if you are testing a 1-many Data Sync model, then Scalability of your primary database schema should be verified against whatever Server Applications are used.)
- Data Sync Rules Verification
- Whether your model is Server Wins/Client Wins or First in First out, there will be some scenarios you can define that must function in a well-known manner based on the Data Sync rule being used. Of course, the Data Sync Rules may also be affected by Business Logic, so it may not be a simple model to follow.
- Platform Dependency Test Cases (Win32, Linux, Win64, Mac, Droid, iPhone, WInMobile, Symbian, Palm)
- Fill in your favorite platform. Some general things to keep in mind, though, are that data definitions may not be shared across platforms. INTEGER may not exist in all platforms - though some replacement will exist. In this case you need to verify the replacement functions correctly, and can Synchronize data bi-directionally (if this is supported.)
- Another good scenario to keep in mind is to verify data translation from Platform1 to Platform2 and back to Platform1. In this case, the final set of data on Platform1 should be identical to the original set of data on Platform1.
- Data Distribution Model Test Cases (1 to 1, 1 to many, many to 1, many to many, bi-directional)
- This Test Case is dependent on your Data Sync model. Generally your Data Model will be either complete functional Syncs between full-fledged databases, or Data Sync from one primary database to multiple small-footprint databases (with minimal feature-sets).
- Scalability/performance becomes an issue based on your Model.
- Data Sync Rules also become an issue of varying importance based on which Distribution Model your Sync follows.
- Timing Test Cases (batching/ordering of commands)
- The easiest way to describe is to provide this example (how is this scenario covered through Data Sync):
- Insert 1 record
- Update the same record
- Delete the same record
- The easiest way to describe is to provide this example (how is this scenario covered through Data Sync):
- Life Cycle Test Cases
- Usually Life Cycle test cases concern themselves with Functional Objects (Users, Groups, etc.) However, the same scenarios can apply to individual records within a database.
- In other words, you can track the state of individual records throughout an applications use. Some data you would expect to be cleaned up once the application completes. Some you would expect to be saved.
- Cross-Server Test Cases (in environments that rely on multiple Servers functioning in a cohesive unit)
- Has become much more important with the sharing of specific Servers through different applications. Some examples include Apache, Postgresql, Oracle, Java.
- In these cases you may want to consider Test Cases that span multiple Servers (as well as give you the ability to trigger functionality on each Server during the Test Case.)
- This also overlaps with Timing Test Cases
- Unit Test Cases (API)
- Should be straight-forward to understand. But keep in mind that typically the API that you eventually end up with does not match the original Development Specification. So if it is imperative that it does, you may want to define Test Cases that directly match up to the original Spec.
- In other instances, though, Dev Specs are looked at more as a general guideline to be refined later.
- Environment State Test Cases (individual Servers being shutdown in realtime)
- Touches on Cross-Server Test Cases.
- This can also become important in environments where mulitple Databases function as a single entity (fail-over, for example.)
- This can also be touched on in a Cloud Computer Model - where a database schema is distributed across multiple databases (not for fail-over, but rather because the database model has been implemented this way.)
- Language Translation Test Cases (byte size changes)
- Becomes important when the data maximum size is in danger of being reached. But can also be relevant if there is a translation engine running during Data Sync.
- Use Cases (auto-create, or from Product Management)
- Auto-creation of Use Cases can be an extremely useful idea to implement, though the difficulty is generally in getting development to implement it. Basically, to include a debug mode that logs all API/Data Sync Requests to an external file (without extraneous debug information).
- Even without the ability to playback this log directly, it gives the QA engineer a quick way to automate the same scenario. So (for instance), the QA engineer can use the UI tool to trigger some Data Sync calls. These calls will be recorded in a log file, which can then be converted into a JUnit Test Case
- Plus, if gives the QA engineer a quick way to fill in Details in Defect Tracking.
- Security/Privacy Test Cases (primarily clear-text inclusion of Personally Identifiable Information)
- This was discussed in a prior Blog entry. But the basic idea is to verify that PII and Secure Data is not included in a readable format via a Data Sync request.
- Hacker Functionality Test Cases (db password manipulation, port access, access to non-shared data)
- Usually is more important to verify in a Server/Client environment that relies on 3rd-party applications. Basically, the standard security holes should be verified as closed to external manipulation. For example, authentication on the Data Sync port
- Upgrade Test Cases (specifically future-proof upgrade scenarios)
- Not exactly a Data Sync Test Case - though Application Upgrade is a form of automated Data Sync. But it can become important in Applications that support future upgrades. How are you supposed to verify that an upgrade will work without having an application to upgrade?
- Generally this can be done by playing with the Application/File version (to fake the environment into believing there is a valid upgrade available, and then verifying the application functions correctly after the upgrade completes.
- Resource Test Cases (hard drive, memory, minimum requirements)
- One important scenario that sometimes gets missed is to assume that your database exists on some other drive that the primary drive.
- Generally you will want to verify that Data Sync either completes or rolls back to prior to the Data Sync (it is generally worse to have some sort of in-between state).
- Performance Test Cases (of individual Clients, of Server parsing of multiple Clients simultaneously)
- There are basically 3 types of performance you should be concerned with in Data Sync. And most likely there will be confusion over which of these is being discussed with regards to Performance Testing.
- Client Data Access - I'm still not sure whether this matters in a Data Sync environment. Specifically, the bottleneck may be the resources available on a Client (not the actual Database/Data Sync Model.)
- Server Data Access- Can become important if you're talking about large numbers of Clients attempting to Sync simultaneously
- Data Sync (especially in an environment containing large numbers of Clients). Can become an issue if there is a timing concern. Can also be severely impacted by the Data Sync engine model (how Data Packets are handled)
- There are basically 3 types of performance you should be concerned with in Data Sync. And most likely there will be confusion over which of these is being discussed with regards to Performance Testing.
- Documented/Expected Behavior (to verify that the expected response matches the actual response)
- As mentioned before, the expected behavior won't necessarily be correct once the application is developed. So you'll have to figure out whether this is a major problem, or whether you are expected to adapt to the changes as they occur. I can't really say one way is correct.
- For Data Sync, this usually will be an issue in defining the final Data Sync Rules
- Debug Cleanup (verify removal of debug code)
- Not much more to say. Debug code is sometimes left in production by accident. Similarly, on occasion place-holder Error Messages may be left in by accident. It's a good idea (if you can) to setup some scenario/time to verify this type of code has been finalized.
- Add-on Functionality (JDBC Driver)
- Usually add-on functionality is not considered of primary importance. However, the example given here is considered standard at this point (so I don't know that it would still be reasonable to ignore testing of a JDBC Driver included in your application.)
- You also may need to look at Datatype Translations between different drivers.
- Undocumented Functional Test Case (undocumented logical functional steps)
- Sometimes there are scenarios that become obvious while developing an Application. Therefore, they are probably not included in the original Functional Spec.
- Duplication of Steps/Functions (verify that duplication of functionality is minimized)
- While it is generally is encouraged in Development to provide multiple ways of accomplishing the same task, from a QA perspective this will decrease the Quality of the application. To put it another way, you are adding unnecessary lines of code - which increases the likelyhood of defects.
- There should be a trade-off (what is wanted versus what is needed), but generally QA is not included in this decision
- American Disabilities Act
- There are websites that describe the requirements.
- One thing to keep in mind is that by following the ADA your UI may be easier to test automatically. Because the ADA can encourage applications to provide keyboard shortcuts, you can take advantage of the same shortcuts to automate testing. Generally, UI automation has the hardest time with simulating mouse clicks (internal object names always seem to screw things up.)
- Following of Standards
- Sometimes you can find 3rd-party applications that can be modified/used to validate that Standards are followed. So while testing Standards can greatly increase the amount of work you need to do, in some cases it can be easy to test to the standards
- On the other hand, if you are forced to use a test application developed by someone else you may be forced to work in a development language you aren't comfortable working in. And you probably won't get the support you need to work in it. So be careful about defining the Test Effort up front.
- UI/API Disconnect (functionality only available through one or the other)
- Generally I feel that all UI functionality should be available through APIs. However, not all API functionality need be available through the UI.
- UI Generic Functionality Test Cases (resizing of windows)
- The goods news is this functionality is not limited to your application. So it may be worth researching generic UI functionality to verify your application is working correctly.
- For that matter, your application may have functions that are meant to simulate generic functionality (without actually being generic.) A primary example are applications that include a Login window - while the window may appear to be a generic windows dialog box, it may be a customized window meant to appear to be a dialog box.
- Resource Growth (for example, database does not resize after users are deleted)
- This is a pretty common problem (generally I think the issue is lack of effort in cleanup versus effort in creation.) But it should give you some ideas of areas you can test that fall into similar categories.
- Install Test Cases
- Not much more to say
- Minimized Credentials (limit the number of credential requests)
- From a functional perspective, there shouldn't be a need to authenticate a user more than once per session (expect for well-know exception cases).
- Minimize Functional Steps (assign priorities to Functions, and define a maximum number of steps to achieve the function from any screen)
- I tend to take the view that any important function within an application should be available to the End User within a certain number of clicks (regardless of what they are doing.) Generally this is not considered in developing an Application, but I try to keep it in mind while testing (and maybe bring it up with Product Management if I have a good rapport with them.)
- Randomized Execution Plans (discussed elsewhere in this Blog)
- Not much more to say, except that it can get complicated quickly. You need to define some subset of functions that will cover a majority of all functions. Once you do that, you can play with grouping of those functions.
- For example, from a database/data sync perspective this means determining which Datatypes will cover other datatypes.
- Code Coverage
- We tend to use EMMA (http://emma.sourceforge.net/). But it really doesn't matter as long as you have a good understanding of the product you choose
- Connection Leaks
- You might want to consider a tool like FindBugs (http://findbugs.sourceforge.net/)
That's about it for now. I started out focusing on Data Sync and Databases and probably went beyond those. But I'm always hopeful someone will find this information useful.





Comments