Performance testing, a non-functional testing technique performed to determine the system parameters in terms of responsiveness and stability under various workload. Performance testing measures the quality attributes of the system, such as scalability, reliability, and resource usage. In laymen terms, performance tests ensures an application meets the service levels expected in production, as well as deliver a positive user experience. As application performance is a key determinant of adoption, success, and productivity.

The main objectives of performance testing are:

• Is to eliminate performance congestion.
• Is to make software rapid.
• Is to make software stable and reliable.

Do all these points sound like the same? Indeed, yes! the same old wine that you’re very much aware of. So what are the questions that have altered the landscape of performance testing? before we dig deeper about the dynamics that altered performance testing let’s understand the landscape of modern software application or service.

We’re living in an era where almost everything has been provided as a service (like SaaS, PaaS, DBaaS, DaaS, FaaS and a few more adding into the list occasionally). At this juncture let’s thank the advent raise of technologies making modern software applications more robust and distributed across various geometrics in architectures (like microservices, nano-services, server-less).

A decade ago most of the software applications was hosted or deployed in a couple of private data centers and most of the time one data center will be serving the active production traffic and the other staying almost idle just because these firms need to have a fallback if the primary data center goes down, but today most of the applications are deployed in cloud with multiple regions active at the same time serving requests to end users and scales up and down on its own based on the load that the application is receiving.

In fact, cloud technologies also added another dimension to deployment for software engineers, which is none other than the machine type/instance type, most of the public cloud vendors have tweaked their machine offerings in categories like general purpose, Compute optimized, Memory optimized, accelerated computing, Storage optimized and few more, each of these categories in turn have a plethora of options which at times result in paradox of choice.

Rewinding back to application deployments, today most of the companies have moved away from the waterfall model and adopted agile delivery of software applications.

With CI/CD in place, a few companies like Meta (Facebook) have deployments designed in efficiently in a way, that any feature deployment push can be triggered or initiated at any point in day without or with restrictions like deployments targets users from certain geography, user group (like beta users), country specific and so on.

It’s not just the technologies, that are making things complex! Sometimes, regulations related to data protection also has impact to modern software applications to certain extent! These regulations mandates firms to store the data within certain geography locations, some regulations allow firms to process the data outside but stored only in certain vicinity. When there is a liberty firms choose to go with limited number of zones / regions as much as possible to have a hold on the cost.

Last but not the least that I wanted to highlight about modern software application is the data intense and dependency on data pipelines as enterprise firm have understood that data is new oil in although data and data pipeline is not new to industry (AT&T long-distance have used their call volumes couple of decades back to identify their prospective business customers who are not part of their network), these data pipelines were sidekick projects/applications but today, these pipelines are very much integral part of applications. For example, product suggestion in Amazon or song suggestion in Spotify is very much dynamic and they’re refreshed every now and then with the help of these data pipeline.

Dynamics of software applications development and deployments have changed, and does it had an impact on performance testing? The core dynamics remains intact, but the following questions have altered the equations while running performance tests:

• With auto-scaling turned on in my cloud deployments will the performance of application stay intact and meet the application SLA?
• What would be the performance of the application if it’s deployed in multiple region / zones in cloud?
• What if the application is going to use multiple datastore that’s been deployed across the globe to meet the regulations?
• Will the performance improve if it’s deployed in a different instance type?
• What would be the impact of application when I use a managed service offering for my data layer?
• What would be the cost impact if the application consumes third party service while running performance analysis, which is integral part of the functionality?
• What would be infrastructure cost and maintenance cost to run this analysis?
• How would I compare performance analysis findings with the past analysis?
• What could be frequency of analysis execution? If the analysis is going to be performed on each version of application in master branch, won’t it exceed the budget?
• While generating the load to analyze the application, should I distribute, or it can be centrally generated?
• Should I consider the network latency while doing analysis or should I ignore the network latency? How can I measure response time if I need to ignore the network latency?
• While running the performance tests what would be impact to my analysis when the browser cache by turning on/off.
• How to balance the performance of the application between the server side and the client side?
• Is the performance of my application being consistent across operating system?
• What is the performance of my application on different device or on a different network?
• How much memory is being used by mobile application in various operating system?
• How much data is being consumed while the app is in use?
• How many times does the application screen freezes?

The above are subset of questions that we ask while doing performance testing. If you’re planning to run performance tests for your application do write to us.