Testing – Rules of engagement

Testing is depending on a lot of other areas within an ICT project. Ignoring these dependencies and not following the principal rules can have devastating results on the testing. Here you find some come ground rules (principles) which contributes to better test results.

  • Requirements / Statement of Work (SOW). Have (very) clear requirements in the Statement of Work (SOW)
    • Clear out an informational message in the SOW such as header information which only introduces a series of functional requirements;
    • break down a requirement with a long descriptive text into small pieces that can be tested on;
    • when the requirement is referring to a list, figure of other document, make exactly clear what this reference is all about;
    • do a thorough analysis of the requirements and separate them into groups;
    • the ones important are the functional requirements and get rid of descriptive text such as life cycle requirements, folder requirements and so on
    • make notes of all requirements referencing to domains which have a mutual effect on your and their test cases to avoid overlaps in testing and making sure that the dependencies are set correctly.
  • Design documentation. Everybody, but especially the builders/engineers, depend on the design documentation. Not the low level design documents because that is the engineers responsibility. So, design documentation MUST be ready and approved. And even more important, the overlap/intertwine between all the infrastructure components and services should be well discussed and designed. To be short; the dependencies are very crucial to the project. But an additional word here is important. Do not trust blindly the design since experience show that the designs are not always reflecting the reality!
  • Build documentation and attached documents. The “as-build” documentation should be written by all the engineers (SME’s) in an approved and common format and must be released on regular basis and available in an easy digital way. The engineers must be trained and monitored at all times since it is a fact that most engineers do not want or possess the quality to write solid build documents. Reality shows that many times there are a lot of (small) deviations with the design documents. This is due to better insights and many changes through the Change Advisory Board are applied without proper documentation. This is ” killing” for thorough testing, especially security testing.
  • Other documentation. All other documentation which are referred at such as security policies, asset management and configuration documentation, flows and so on should be available too, at the same time as the “As build documentation” is released. Test cases can only be properly written when the documentation package is ready.
  • Work packages. Work packages must be written before the build starts in order to translate the design documents into actionable steps for the management, builders, auditors and of course the testers. These work packages give direction to the test plan and test cases. Regrettably I miss work packages too many times and guidance is lacking.
  • Start testing. When building a new infrastructure and services including management, make sure that the actual execution of the testing is only done after an official approved first build of the infrastructure, services et cetera including the implementation of major changes that are coming out from the first build. It is useless to do a test if the build is not ready. But also the build itself and the infrastructure that the build is depending on, should be ready, to allow integrated testing. An additional problem arises here. The problem of integration testing. Unfortunately, integration testing is yet the hardest part. Reason for that is that the engineers build their environment without taking notice and being involved in what engineers in other areas are building. Strong technical leadership on senior level is needed here.
  • Test knowledge: In order to do proper testing the testers must have a thorough understanding of
    • Testing tools to administer and track all the test cases;
    • test procedure and test organisation in order to organise the official Service Certification tests;
    • have a very good understanding of the topics they want to test, especially about the infrastructure and security;
    • make sure the requirement can be tested. E.g the following requirement is a poorly written requirement “ The product will meet the best practice”. This cannot be tested and is open for any interpretation.
  • Test tools. Make sure that not only the tool for administering and tracking the tests are in place, but also the other tools for testing like performance testing should be available at the time that testing starts.

To sum up the aforementioned issues: If the requirements are unclear (lets say 30%) then the foundation for a ” Jelly pudding”  is laid out. If the designs are based on these requirements and not completed on time then the project already is falling apart. If the builders are not take ownership for creating their build documentation or do not deliver it on time,  then you really have a big problem. And if the project hasn’t a proper change management process then almost certainly the project will severely be delayed due to many incidents.

Since testing and testers are fully depending on what I have mentioned above, you can imagine that the quality of the test cases and test steps are a reflection of the requirements, design, build and change management. The value of testing reduces to a minimum and worse, the tests brings only a lot of defects on the surface that must be resolved. But the builders are still building the infrastructure. What is more important? Continue the build or resolving the defects? The management kicks in and set the priorities thereby frustrating the builders even more and the ” train comes almost to a full stop” . Then the management is bringing new resources in and the focus is on tight planning and a rigide change management process with daily meetings. Instead of building, processes becomes more important. But still the problems keeps coming in and there is no end to this. Do you recognise all of this?

So to avoid an upcoming disaster, the above mentioned principles must be obeyed although they seem so trivial! But imagine that you need to test the whole infrastructure based on 596 pages of requirements and over 4000’s test cases and many more test steps! To give you an impression about the magnitude of the testing of a project I was involved in. My test area was based on 116 pages of test cases and test steps regarding Cross Domain Gateways and an additional 45 pages for testing the Network Acces Control (NAC) and VPN.

Herman Rensink, Infrastructure – and Security Architect at Greater BV, Doorn Netherlands.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s