Category Archives: IT

The future of the test manager in IT projects

Now that 2016 is coming to an end, I share with you some observations I made in the past year concerning testing in IT projects. At first glance these observations seem disconnected, but I started to realise that they were forebodes of general developments in the IT world and will also have their impact on IT testing and the role of the test manager. Connecting the observations also shed some light for me on how a test manager can stay afloat in a rapidly changing IT landscape.

The first observation was that during meetings participants very often mentioned they were busy testing and measuring by the number of times they mentioned it, these testing activities seemed to take a considerable amount of their time and effort and could be considered an integral part of their daily job activities. Yet these testing activities were not (always) part of their job description or project role. It could be that these testing activities were not mentioned or recognized in a test plan, let alone be known in advance to the test manager.

The second observation was that a lot of focus in IT testing is geared towards aligning testing activities with developments around continuous delivery and iterative development. The result of these new developments is that more and more energy within companies is geared towards working ‘agile’ and subsequently test automation becomes a hot and interesting subject, partly because a lot of the testing work now directly comes into the hands of the developers.

The test manager should have or develop a fine radar to pick up the signals about project members running their own tests. Not in a negative manner as if the test manager missed out on something in his planning phase but by understanding that testing is a very natural way of how humans make progress and how the human mind works. So testing is a basic human quality that enables us to get forward with IT projects and the test manager will never (nor has to) be able to catch all the testing activities during the IT project or in advance during the planning phase.

The tester on the other hand might not always be completely aware of the significance of the testing activities. He or she may think that the result of the test is not essential for the progress of the project (other stakeholders might hold a different opinion). Then there are two other effects that the test manager needs to be aware of. First, if the test was performed to see if the idea (or read: service, product, piece of code) works and the test is negative, then the tester might get shy to communicate the failure of the tests. It becomes personal and therefore not an experience to share broadly and be proud of.

If the test result was positive the tester could boast about the results, while the test might have a not too big relevance for the final outcome of the IT project. This is where the importance of the test manager is at stake. He is one of the few project members with the necessary objectivity who can valuate these ‘under the radar’ tests within the project and determine the attention they need as well as the level and range in which these test results should be communicated.

Gone are the days when the test manager could write his impressive test plan (that only few would read from beginning to end) and gone are the days when the test manager could claim time slots to execute with his team the carefully designed and well prepared test cases.

So what is the connection between the first and the second observation? With the current developments in IT projects, the test manager will no longer be able to completely influence or comprehend all test activities within an IT project before the project starts. For a test manager to be successful within a project, the test manager will have to develop a sensitivity to the signals about on going and intended test activities.

The number of implicit tests within IT projects appeal to elementary human qualities that are essential for progress in general and in our case progress within the IT project. They stimulate the participation of the tester, yet there needs to be an independent source who eventually can judge, evaluate and inform about these tests. The outcome of this process will determine how the test manager can support the testers (with necessary facilities and resources) and how information about the test results could be distributed among the stakeholders. These will be important features for the test manager to survive in an IT industry dominated by rapid changes in methodologies and views concerning the deliverance of IT projects. He can rely less and less on his own plans and methods but will have to bring the information from test results in the projects to the surface.

IT Testing: Production Acceptance Test (PAT) -updated-

One of the more problematic aspects of acceptance testing for new IT systems is usually labelled the Operational Acceptance Test (OAT) or Production Acceptance Test (the abbreviation PAT will be used here after in this article). The goal of a PAT is to validate whether the system meets the requirements for the daily operation after a new IT system will be implemented in a live environment. With the PAT the quality concerns and acceptance criteria of system administrators and application managers can be addressed. The range, scope and objects of testing are depending on the release of the IT system to be implemented, typically you might see a mixing of dynamic and static testing. Typical examples of areas of testing  are validation of the backup/restore procedure, disaster recovery, maintenance tasks, system monitoring and logging, data exchange (on a technical level) and the technical implementation of web services, which all require dynamic testing. Usually the acceptors also perform static testing by checking if all deliverables (for example installation guides and other technical documents) have been received.

Similar discussions and question are being raised when discussing the PAT during a project: “in which environment should the PAT take place?” and “In what phase of the project should the PAT take place?”. Usually project people involved have already made their own presumptions as what the answers to these questions should be. The PAT will take place in the production (live) environment because the name PAT already makes clear what environment to use: Production Acceptance Test, so this test takes place in the live environment. To the second question as to when the PAT should take place the answer will most likely be combined with the answer to the first question: “In the live environment just before the system will go live” and in some instances you will have suggestions to perform the PAT after the go live of the new IT system.

There are multiple and urgent reasons why these answers above are unfavourable and could present the ones involved with testing with lots of issues. First of all testing in a live environment is in general a bad idea with any kind of testing, possibly leading to a multitude of issues. The most prominent one is that testing of any type needs data and test data will most probably lead to pollution of the live system. If not communicated you are prone to cause stress and confusion to users  especially when they suddenly are confronted by test data popping up  among their ‘real’ live data. Of course you might find yourself in the circumstance that there is a representative number of ‘real’ live data available so you can perform your test while entering live data possibly in combination with a freeze of the live environment.

If the PAT is not possible in another staging phase or non-live environment it means almost certainly there is a deviation between the live and non-live environment. Some times this is insurmountable, for example in situations where there is no equivalent of an interface between an internal and externally hosted, third party system and there is no alternative available like a simulated interface (a stub). In those cases it might also be unavoidable to perform tests in a live environment as this is the first opportunity to validate these type of interfaces. Also financial issues might come into play, when the additional expense of components like hardware or licenses can not justify the expenditure of an environment besides the live environment. Despite these exceptional situations the non-live environment should be a good representative (preferably a clone) of the live environment so all PAT related testing tasks can be performed before the go live of the IT system.

Another issue related to the PAT is that it more than often muffled into the end of the project or project phase. Usually it is planned somewhere after the User Acceptance Test (UAT) and before the go live date. The UAT usually will be the main focus during all test activities.  If the testers of the UAT give a ‘no go’ on base of their test results that means the project is delayed. Any delay in that phase means that the available time for the PAT is diminished and not unlikely completely skipped. For some reason the PAT testers have less power to halt the go live.

Actually this leads to strange situations where for the users a system can perform nicely on first sight more than often based on a limited set of test data. While in fact there is a risky situation: the procedures and tasks ‘behind the scene’ are not in place because the PAT could not count on a same level of priority as the UAT. Soon after go live the first problems usually arise because the IT system does not work as expected and I am not even going into cases like a system crash mercilessly shows that there is no back up procedure in place and important data is lost. Usually to tackle the problems from a badly or non performed PAT resources previously dedicated to the project are called back to give assistance to the system administrators, taking away these resources for new project activities.

When the users and business is generating test data in the non live environment it is a good idea to create test scenarios that can be used both by UAT and PAT testers and which will allow testers to validate their test scenarios. Examples are trying to generate logging when performing unauthorised transactions or mutations or thinking out specific test cases in consultation with the testers of the UAT that trigger aspects recognised important for the PAT.

The PAT needs a similar level of attention as the UAT that is usually considered of greater importance. The PAT needs careful and conscious planning when the over all planning for the project or project phase starts. This means the testers who will perform the PAT need to be involved early on in the project to allow them to see how the development and realisation of the IT system so they will be able to come to grips with the new system and carefully generate their test scenarios, supported by the test manager and his test team. Many aspects of the PAT do not have to wait until the final phase of the project as the non-live environment almost always is earlier in place than the live environment. The more conventional and basic tasks of future operation could be tested long before the go live date, relieving the stress from all activities that usually come together at the implementation of the new IT system.

Ewald Kegel, The Hague, April 11th 2016
(updated April 16th 2016)