How To Test Cloud Software

By Jim Coyle, Product Test Manager at Sage

Testing Cloud Software You test software to find bugs. The bugs fixed, your shiny new software gets released along with a flurry of congratulatory inter-departmental emails, and everyone is happy. Everyone, that is, apart from the test team who worry they may have missed something. And if that happens, failure is very public, and no more so than in the world of cloud software, running 24/ 7, where everyone with an Internet browser can choose to log in and see the problem for themselves.

With true Cloud (aka Software as a Service) software, the risk of failure has increased over traditional desktop software because the customer relies on you more than ever before:

You manage your customers’ data
You host the service, around the clock
You update the software when it’s necessary
You assume the role previously held by the customer’s old IT department

And if there’s one thing a room full of testers will agree on (and there aren’t many), it’s that if the risk of failure increases, you need to respond by doing more, and ever smarter, testing.

What extra tests do we need to conduct? What challenges does cloud software bring to a test team that previously tested ‘traditional’ PC installed software. Here are some points to ponder:

The user’s configuration is different. They may not be sitting at a PC anymore with an installation CD in their hand. Their window (pun intended) into your software is the browser – so focus on the Internet Browser and not the Operating System. Are you going to support Safari as well as Chrome? How about testing on an iPad? Or an Android tablet? How about testing on IE7 as well as IE9? And then there’s testing for broadband latency? What about Portrait mode versus Landscape? Test the swipe / pinch / flick / double-tap etc movements peculiar to mobile devices.
Data migration testing during software upgrades. You manage one database with all of your customers’ data on it. If there’s a schema change – it has to work. Database tools will help.
Performance. True SaaS is scalable. With more users logging on, will the service slow down? Stress the system beyond what the spec says and see what happens. A good system will handle this gracefully.
Security. Customers trust you to keep their data secure. Test for security gaps. If you don’t know how – pay someone to do it for you.
Usability. Wow factor. Customers expect cloud services to be slick. They’re right – what’s the point in launching a mediocre release of software to the world? Heuristic evaluations can help here. Look up a guy called Jakob Nielsen.
Testing after you go live. Sounds strange, I know. This topic makes everyone sit straight. What’s the point in testing after we go live, people ask – but how do you know your service is working unless you test it in the live environment, and keep testing it, and test some more, and test smarter.
DevOps is a new term and I think this one is here to stay. Traditionally, the Development (Dev) teams worked independently of the Operational (Ops) teams. Dev teams developed the software, and then handed it to the Ops team who looked after the CD distribution. In the new cloud world, these lines of separation are blurring because the Ops teams upgrade and maintain a hosted service.