Testing a Windows 10 Upgrade Programme

Date: March 2018

What exactly is involved in testing a Windows upgrade programme within the Legal sector? Is it as simple as pushing the power button on a shiny new laptop, pausing a moment to enjoy the artsy new login screen, then accepting without question that because your login was successful the upgrade was a resounding success, the tests passed with flying colours? To certain individuals, perhaps the answer is yes. But there are many, many more things to consider.

Windows 10 is comprised of FIFTY MWindows 10 Upgrade - [Testing a Windows 10 Upgrade Programme]ILLION lines of code – 25% more than in Windows 7. That all makes for one impressive user experience, but what is it that enables this newer, fresher, larger operating system to continue to deliver everything its users expect? Has anything in the new infrastructure – undoubtedly changed in size and complexity since its predecessor – caused something to stop working along the way? If the upgrade has stripped confusing customisation within a system’s applications, where is the guarantee that those changes haven’t broken any existing integration? These sorts of concerns are exactly why we test.

Planning a Windows 10 upgrade

The first question to ask when planning a Windows 10 upgrade is to assess what it will entail, with the answers driving the considerations for an approach. Does the business wish to enhance their security? Is application customisation being redesigned, to reduce compatibility? Is an upgraded document management system being integrated? It is how testers interpret this information, translating it into tests that will appease the business need without detriment to the user experience, that gauge the successfulness of delivering an upgrade in this type of enterprise.

It is not as simple as flicking an Off switch for Windows 7 at the same time as flicking an On switch for Windows 10. The nature of the approach is another hefty consideration requiring careful attention and planning: which departments within the business have greater need, how can a rollout be phased to ensure minimal fee-earner disruption, how will user training fit into the rollout timelines, where will devices be sourced from, how will those devices be configured, what changes have been made to the business applications and, crucially, where should the testers begin with all this?

First and foremost: the build itself. Having devices to deliver, having trainers to train, having managers to manage and testers to test means nothing if there is no build to do any of it on in the first place. The packaging and subsequent organisation of the applications into user collections, the updates to user and machine policy groups, architectural reviews; all of this must be considered when putting together a build, because all these factors dictate its stability. No one wants a build that deploys but doesn’t meet any compliance rules, that doesn’t allow the users to use their time reporting tools, that crashes as soon as too many Word documents are opened.

Careful attention must be applied: the build is the backbone to which applications are integrated, updates applied and, functionality enabled. Having a stable base build opens the possibilities for ongoing activities and enhancements of which testing is but one: there is change control, build release, application updates and integration, security upgrades, automation. But the central focus through it all must be the user experience. An upgrade is, ultimately, for the benefit of the users.

Windows 10 Upgrade - [Planning a Windows 10 upgrade]

Scoping the test requirements for a Windows 10 roll out

With a build progressing under the watchful eye of deployment and packaging teams, testers can move onto their next consideration: the scope of the adjustments within the new operating system. Requirements elicitation can offer a view as to the scope of testing required, and analysis of existing system use via tracking tools and software can help to generate some form of priority list for application delivery. Working with the business to establish owners for system applications themselves can help drive this elicitation forward, and the resulting collaboration unifies the delivery team, helping to set expectations and smooth over issues more quickly.

There must be established traceability between what is expected of the upgraded system, what it is realistically going to be able to deliver, and what it actually does. Perhaps, in a particular iteration, it will not be feasible – and in some cases, possible – to continue the use or support of certain applications or functionality, because an OS upgrade has caused an incompatibility that cannot easily be solved. It is in these instances that consideration of the user experience is even more crucial – users tend to complain when something they are fond of or familiar with is taken away from them or changed, regardless of the reason. Expectations need to be set.

There should be an agreement with the business on the overall test scope. There is little point running an exhaustive test suite for an application used by one person for five minutes a day; similarly, it is not acceptable to run just a couple of sanity checks on an Office suite application.

Managing BAU during a Windows upgrade

Existing processes within the business also play a major part in the success of an upgrade rollout. There needs to be a strong problem management team, who can liaise with both the business users and the testing teams to align on issues, changes and fixes. But they also need to integrate this information with the change control management – business users, particularly in globally distributed offices, will be working at different times – how are updates going to be delivered to them?

The BAU team needs to continue Windows 7 support until a firm-wide migration to a new OS is complete – what should their focus be during this time, and how will it shift when they find themselves suddenly working with a new system? This particular team should have some re-usable test collateral – useful for test planning in an upgrade programme – but they need to be familiar with the changes that are taking place outside of their own remit.

What if there are business processes which are missing, or not up to scratch? There needs to be a set defect management process, focussing on triaging issues, that is shared, understood and enforced. The application packaging and build processes need to run to a tight schedule to allow enough time for testing. Release and deployment processes should be carefully planned allowing time for all of this, but also aligned with training schedules, so users are best prepared for their migration. Testers should prepare their tests in advance of deployments, with time allocated for change requests arriving via problem management. Automation can be considered if it is appropriate, and there is time and budget available to implement it.

Mobile working –  compatibility testing

Windows 10 Upgrade - [Mobile working -  compatibility testing]

Another key consideration in the test planning is identifying the range of devices that are used across the firm. Test devices (as many as will be used within the business) must be made available, and configured to the required specs. It will be necessary to understand if policies need to be applied or administrative accounts set up to enable the devices to best replicate a complete end-user view, for time reporting tools and full application functionality and so on, to be used?

Maturity through automated testing

With a signed off build and set of applications, testers would seem finished and ready for their next project. Things are not so simple in the new security conscious world. With the need for patching on all systems to be up to date and quick -to -release, and Microsoft pitching Windows 10 as the last “version” of Windows (as it is now Windows-as-a-service),  your build will be updated frequently. Changes applied may or may not have an impact on your configurations and application, how can you be confident without testing prior to release?

With so many changes scheduled through the life of the build and an urgency around releasing the security updates there will be a need to perform regression testing quickly and frequently.

The foundations to perform this testing at the required cadence and speed are a robust test suite with good traceability. Automation can build upon this foundation to reduce the manual effort required on each execution of the suite.

Engagement and user experience

Ultimately, again, the primary focus, particularly during testing, must always be the user experience. Engagement between testing and business users is vital to fully understand and balance both the user needs of the clients, and with the functional, operational expectations. In order to deliver a successful Windows 10 rollout, employing the best practises for testing together with strong collaboration will be key.



We are a proven and trusted partner to the legal sector; providing a comprehensive range of testing solutions for a number of leading law firms. Services provided include; strategic test consultancy, functional and non-functional testing as well as test automation.

We bring with us in-depth, sector-specific knowledge of the common technology platforms, applications and integration requirements. Through our dedicated Legal Sector Team we can provide you with strategic advice on these technical challenges and your future roadmap, as well as offering you the opportunity to knowledge-share with peers through our regular Legal Sector Roundtable events.

Our experienced Legal Sector Delivery Team provides you with the expertise to successfully setup and execute testing solutions that give you the assurance that your systems are fit-for-purpose.

To find out more click here.

If you have any further questions regarding testing a Windows 10 upgrade programme then please complete our short form to request a call back.