The Green Sheet Online Edition
September 9, 2013 • 13:09:02
Evaluating payment gateway performance
From internal stakeholders to clients, everyone involved with payment gateways expects maximum performance, stability and uptime. But how does one quantify this performance? How do we know if the gateway is indeed performing at the most optimal level? To provide answers, this article ventures into aspects of payment gateway testing from a performance perspective.
Testing challenges
Performance testing for payment gateways has multiple objectives. The most obvious is to ascertain that the gateway will perform as expected under certain volumes anticipated. Other objectives to address include:
- Business growth: To manage business growth, payment gateway owners want to know what their hardware requirements will be in the near future. Growth may be organic expansion or a result of mergers and acquisitions. It may also result from the launch of new services and products or similar activities.
- Stability: Performance testing can also help ascertain the stability of the gateway's code base. All gateways undergo periodic upgrades. Payment gateways are 24/7 systems and require a high degree of uptime. It is difficult to quantify the impact of minor and major changes to the code base from a stability point of view. Performance testing offers a way to do that.
- Sustainability: Performance testing can validate the impact of code changes on the customer experience. Payment gateway code changes may influence the transaction processing flow, look-ups, logs and many other aspects that can slow down the system.
Also, code changes are often done to speed up and optimize transaction processing. To measure the effect of such changes, performance testing is the most commonly used and reliable tool. Comparing before and after results gives a clear view of the impact code changes have on the overall system.
Testing metrics
A fundamental question asked is, How many transactions per second does the payment gateway handle? This is an indication of the number of concurrent transactions the gateway can process without reaching a break point. The real question is, What is the gateway's break point?
Generally, the break point for a payment gateway is not the moment when the gateway software stops, freezes or crashes. A well-built gateway is unlikely to reach that stage. The point where gateway performance is too slow and becomes unacceptable is the break point.
The slowdown occurs when transactions are taking too long to process inside a gateway. Some relevant questions to ask during a performance testing exercise are:
- How many concurrent transactions can the gateway handle?
- How long does it take to process a message from one end to the other?
- How long does it take for a gateway to receive a request and return its response?
- In case of a failure, how many transactions are dropped or lost? How quickly can the backup system resume function?
- What are the queue sizes, and are they growing?
- What is the highest transaction volume as a spike versus the transaction volumes that can be sustained for a time, say, for an hour or so?
Test-case solutions
Performance test-case formation is a complex task involving many variables. Here are major points to consider during this process:
- Transaction mix
Payment gateways process varied sets of transactions and might handle each differently. To make the performance test reflect the reality of the production environment, the test must include a transaction mix that is similar to the production environment's mix. This includes testing specifics such as ratios of transaction types, bank identification number ratios, routing scenarios, etc.
- Single node versus multiple nodes
While designing the test, it is necessary to determine if the test needs to run from one node or multiple nodes. Both scenarios are important and give performance data from two different angles. Single node offers understanding of scalability from a single point of acquisition; multiple nodes stress the gateway's ability to handle many points of acquisition simultaneously. An ideal performance test suite will contain test cases for both single and multiple nodes.
- Stub, local loopback, processor
Performance test cases can employ a local loopback inside the gateway, a stub representing a processor or an actual processor. The most realistic scenario is to run the test cases with a processor, but that isn't always possible. Many processors throttle the performance in their test systems to enable all gateways to test without performance degradation.
An approach to resolve this issue is to use a processor stub that will use the same network connectivity, accept incoming transactions and send back a response. Advanced stubs can even vary the response at random or on a predefined basis – denying every fifth transaction, for example.
If writing a stub isn't an option, some developers use a local loopback method via which a transaction response is generated inside the gateway itself. While this method is simplest to implement, it does not reflect the reality of transaction processing, except for stand-in processing scenarios.
- Failover scenarios Failover handling puts an additional burden on the payment gateway. During such times, some transactions are either slowed down or dropped, resulting in timeouts. While these are important scenarios, they are not daily events. It's essential to run performance tests that include and exclude failover scenarios and understand the results in each case.
- Stress testing The most common reason for performance testing is to know the platform's capacity during stress situations. It is important to know how the gateway is performing under high load conditions, as well as how the hardware handles disk usage, queuing, memory usage and swap, network utilization, etc. A stress test allows for future capacity planning both in terms of software tuneup and hardware upgrades.
Statistical concepts
While performance testing can generate valuable data, interpreting that data requires some knowledge of statistical concepts. Arithmetic mean is useful to find out average values such as transactions per second, but a mean does not reflect the spread of the data points.
The spread of data points is better reflected in variance or standard deviation. This reveals the spread of the data. An example is the system's total throughput. If the data points have low variance, it will imply that most transactions have similar performance.
For most performance-related measurements, the value should remain within a narrow band; that is, low standard deviation. For example, it may be acceptable that 95 percent of the transactions complete their round-trip journey inside the gateway within one second. Though 5 percent do not fall in this boundary, it does show a high degree of gateway performance.
Statistical mode represents the most common value in the collection of data points. This is the most likely value that occurs, and hence most data in the collection have this value. For example, if the mode of transaction round trip is 0.8 seconds in a collection of values for transaction round trip, it indicates most merchant transactions are handled in 0.8 seconds. Some transactions may be handled with even better performance; others may be worse.
Testing strategy recommendations
A test platform built for payment gateway performance testing should capture quite a few metrics, some of which I have just been mentioned. While test strategy is governed by the overall expectations and goals, here are four best practices to follow:
- Generally, do a performance test repeatedly for comparison purposes. To accommodate this, I suggest creating an automated test harness at the beginning and maintaining it with every change to the code base, updating as needed. Test cases must be predefined and upgraded from time to time to reflect changing business scenarios.
- A clear understanding of the transaction mix is essential. Design the performance test so that it reflects the target transaction mix.
- Capture and store multiple data points for future comparisons. Raw data points must be retained so that future calculations can be handled.
- Apply appropriate statistical calculation to the data to draw accurate conclusions from the testing.
Gateways are complex systems, and businesses have high expectations for them. Evaluating performance requires precision. There are a number of moving parts in the evaluation process, test methods and analysis of the methods. A repeatable, automated performance test suite should be a mandate for any quality assurance team responsible for payment gateway testing. A planned approach to performance testing with clarity on strategy, goals and methods can lead to enhanced payment gateway quality and stability.
Chandan Mukherjee is the co-founder of PayCube Inc., which is a San Francisco Bay Area, Calif.-based payment consulting and IT services company providing custom software solutions and custom gateways for acquirers, ISOs, retailers and varied organizations in the world of payments and consumer transaction acquiring and management, including prepaid and gift card program, loyalty and promotion, payment start-up, POS solution, mobile payment and e-commerce players. For more information, email cm@paycubeinc.com, call 510-545-6854 or visit www.paycubeinc.com.
Whether you want to upgrade your POS offerings, find a payment gateway partner, bone up on fintech regs or PCI requirements, find an upcoming trade show, read about faster payments, or discover the latest innovations in merchant acquiring, The Green Sheet is the resource for you. Since 1983, we've helped empower and connect payments professionals, starting with the merchant level salespeople who bring tailored payment acceptance and digital commerce tools, along with a host of other business services to merchants across the globe. The Green Sheet Inc. is also a proud affiliate of Bankcard Life, a premier community that provides industry-leading training and resources for payment professionals.
Notice to readers: These are archived articles. Contact information, links and other details may be out of date. We regret any inconvenience.
