Real-time Analysis of Payment Gateway Failures in E-Commerce

Introduction

The 21st century has seen the growth of numerous E-commerce companies whose business model involves buying and selling of products and services over electronic systems such as the internet and computer networks. A major function of E-commerce is handling business transactions and processing of payments. Processing of payments is a crucial task as it involves tens of thousands of transaction requests every second. It is also crucial for revenue generation to ensure that any hardware or software issues that might crop up are identified and handled immediately to avoid customer churn.

Real-time analytics of Payment patterns in E-Commerce involves a sophisticated methodology wherein we use a Real-time Processing Engine, a temporary persistence layer and a permanent persistence layer to help business users gain insights which was never before possible. This technique gives the business user the power by giving him information regarding the transactions and payments that has happened till that second. He can perform various complex analysis on this data and see how well is he doing compared to another point in time, predict what his revenue would be for the coming days. He can identify his major clients/customers and make sure that all the offers and discounts are conveyed to them. He would be given the power wherein he can address payment gateway failures as soon as they crop up, identify the cause and fix it in real-time before it drastically affects his business or revenue. He can also segregate this clients/customers into various categories and provide category specific services and discounts. Payment Gateway trend analysis will be of much use in the coming age as transactions and banking methods shift more and more into the virtual environment. The growth of PaaS (Payment of a Service ) is a very good example of this trend, where E-commerce companies are actually outsourcing their entire payments to Payment as a Service Providers.

Various Stages of Online Payments

There are various issues with respect to processing of online transactions. The entire payment processing takes around 2-3 seconds in general and involves various stages:

1. Customer places the order on website by pressing the 'Submit Order'.
2. The customer's web browser encrypts the information to be sent between the browser and the merchant's web server. The merchant then forwards the data to the payment gateway. The in between steps are maybe done via  SSL (Secure Socket Layer) encryption. Data could also be directly sent from the customer's browser to the gateway.
3. The payment processor forwards the transaction information to the card association (e.g., Visa/MasterCard/American Express).
4. The credit card issuing bank receives the authorization request and does fraud and credit or debit checks and then sends a response back to the processor.
5. Processor forwards the authorization response to the payment gateway.
6. The payment gateway receives the response, and forwards it on to the website.

Characteristics of Payment Gateways

1) Compatibility: The payment gateway's technology platform should be compatible with the shopping cart's technology.

2) Security: Ability to detect fraud in a moving window.

3) Price: The fees charged for using the service.

4) Simplicity: Ease to use and the end user's need.

Problems faces during Payment stage of an E-commerce transaction are many. They arise due to many factors like:

1. One time Password send in SMS mode
2. Poor network
3. Device and browser combinations.
4. Eight server hops for successful payment, anything could go wrong in between, each stage adds its own complication.

Problems Faced by E-Commerce Merchants

The various problems that lead revenue loss and loss of transactions with respect to an E-commerce company are the following:

1. Drop Offs: From the time the shopper clicks on the 'Pay' button on the merchant's site till the time the entire transaction is processed, the request goes through various servers. Some payment gateways have 8 hops in between during which the connection could drop off. There are also some providers who have less number of hops and enable direct payment, though in these cases success rates are much higher drop offs are inevitable.
2. Multiple Payment Failure: Multiple payment failure is another crucial scenario wherein a loyal customer is trying to make payments more than once and failed more than once. This is a very serious issue, since it could lead to customer churn especially of loyal customers, they will opt for competitor sites.
3. Payment Success but the confirmation message is not shown: In some cases payment is successful but, if the confirmation messages aren't shown it could lead to confusion and customers might misinterpret their payment status.

Present Approach

In most cases the merchant sites are unaware of the reason why their purchase to search ratio for a product/service is dropping. They know that the user hasn't completed his purchase, but the reason is still unknown and they are thus unable to rectify the issue.

Payments are so crucial to the functioning of an E-commerce site that any sort of approach to improve payment drop offs is welcome, a decrease of even 1% is considered enough to improve revenue by millions of dollars.

1. Scheduled Batch Reports: The reports done my batch jobs give indications of payment failures being the reason for low sales for a given time period. While this method might help fix similar issues in the future, it cannot help get back drop offs or customer churn during the analysis interval.
2. PaaS: Many E-commerce companies are now outsourcing their work to PaaS providers (Payment as a Service). This helps prevent drop offs by means of reduced number of hops and one click payments, etc. These services come with huge monetary costs.

Real-time Analysis of Payment Patterns in E-commerce

Architecture

This approach aims at giving E-commerce companies a 360 view of the payments happening in their site in real-time. The proposed architecture consists of:

1. Real-time Processing Engine: A real-time engine like storm serves the function of ingesting the traffic across the website on a real-time basis, processing the data, based on certain complex software algorithms and forwarding the processed results either to a storage layer or reporting layer.

5) A Temporary Persistence Layer: A temporary persistence layer, a key value storage like Redis would help serve for temporary storage and look ups which are necessary to identify customers , unique apache cookies, on a real-time basis. It is preferred over a time consuming permanent persistence layer since storage and retrieval are much more efficient.

6) A permanent persistence layer: A permanent persistence layer stores the processed data that comes from the real-time processing engine. This can be any distributed database like Hbase or Cassandra. They will be able to scale and handle the huge volumes of E-commerce traffic.

7) User Interface: This gives the personnel at the E-commerce real-time insights into their payments data and traffic.

Solutions

Drop-Offs

According to this method, Payment-drop offs can be tackled real-time. The real time counts can be got for each stage of the funnel. The drop offs at each stage can be identified. The number of people in the 1st few stages will always be many times more than those at the lower stages. It reduces stage by stage to a very low number in the last stage. Figure shows a typical e-commerce book/buy funnel.

The algorithm for the storm code :

1. Set store the entire message as the key of a map when "payment id"( unique id for each payment) exists and "payment status" is INITIAZE (making connection). The value will be the details of the message.
2. If a message comes in with the "payment status" is SUCCESS or FAILURE, clear the map content with that particular "payment id". It means that the particular individual with that unique payment id has either succeeded or failed in making the transaction and his connection has not dropped off in any in between stage. We can therefore clear his information from the map.
3. .The elements in the map are now filtered out by unique "payment id" where "payment status" is INITIAZE. This final map gives us details of those payment messages where payment has dropped off in between while trying to contact the bank or gateway, ie. those payment id's which have not generated a SUCESS/FAIL status, and therefore have got dropped off in between.
4. These data is now shifted out to a Permanent Persistence layer where it is stored and the various details of the message are separated out, eg: reason for drop off can be got from the "Payment gateway response" field, the various other details leading to drop offs can also be identified as we can save the browser name , version , device type, etc.
5. The data stored in the database can be used for analysis.
6. For immediate action we can send customer details to the Call-centre for immediate call backs. This data could also be pushed to an e-mail or SMS queue to prevent customer churn.

Multiple Payment Failures

There are scenarios where customers face payment failure multiple times, it leads to customer churn and loss of loyalty as customer who face such issues will be unsatisfied with the website. This is a difficult scenario that has to be tackled in real-time before the customers choose other alternative sites. This use case can be leveraged to provide customer specific services and call backs, based on the value of that customer to organisation.

The proposed approach :

• We use a temporary persistence layer ( Redis) which stores look-up data for all high value customers, their user id and other details.
• As and when a payment message with a "Fail" status is received, a look-up is done with the High Value Customers list in Redis.
• If the user is High value, his details are pushed to the call centre channel for immediate action.
• If he is a non-high value user , his details are pushed into a temporary map in Redis with payment id as the key and the count of failures.
• If the number of failures are greater than 2 in the given interval of time, his details are published onto an E-mail/SMS channel.

No Confirmation Message

There is another case when customers who have successfully completed transaction do not receive a success message, this in turn leads to a lot of confusion and might lead to customers attempting to pay again.

The proposed approach:

• Whenever a Payment message comes in with status "Success" , it is pushed into a map in Redis (with fixed life time)." Payment id" is the key and message the value.
• Whenever the funnel data reports completion of transaction, ie. the "confirm order id" is generated, storm checks up if data exists in the Redis map which has the same payment id . In case it exists, that map entry is removed.
• At the end of the life time of the map elements in Redis, it pushed to an SMS.E-mail queue. This is a list of users who have not received a confirmation message.

PERFORMANCE EVALUATION

The traditional approach followed in most E-commerce companies, which involve using batch processing to record and analyze payment gateway failures were compared with the Real-time approach this paper proposes from both an efficiency point of view and time saving:

• Drop Off :

The tradition approach of using scheduled batch jobs took around 300ms to query the payment gateway drop offs for a given period, the job was scheduled every 5minutes.

The real-time approach took around 20 seconds for the same query, data was recorded every 30 seconds.

• Multiple Payment Failures & No confirmation Message Shown : T

These issues which are hardly handled in the traditional approach. Even if they were, it was only much later, after the customer left the site. These kind of actions are usually taken at intervals of 24hours or 12 hours.

The real-time approach was tried and payment failures were tackled on a real-time basis, it was also possible to contact the customer facing problems immediately, as soon as a failure was reported. The response time was a matter of few seconds or minutes at the most.

advantages

• Scalability: The size of data doesn't matter
• Real-time Action: Real-time pattern capture, analysis and actions.
• Increased reporting performance: The absence of any pre-computation in the backend of Reports and Dashboards, leads to extremely fast and efficient reporting.
• Ease of implementing custom rules: Not limited by the offerings of a proprietary software. Custom rules can be created and deployed in no time.

CONCLUSION

Addressing various issues with respect to payment gateways is one of the major concerns of all e-commerce companies nowadays. The traditional approach to identify drop offs, multiple payment gateway failures were using batch processing. This methodology involves scheduling a job which runs at regular intervals, collecting past payment data, analyzing this data and understanding the reason for payment failures. Since online payments lead to real-time customer churn and loss of loyalty, it should be tackled on a real-time basis. The proposed approach involves using storm, redis and a permanent storage layer to tackle payment failures.

The major advantage of this approach is being able to communicate with the customer and provide him customer specific services. This could directly translate into a huge increase in revenue, as payment gateway failures are the major cause for revenue loss in e-commerce and has a huge potential to improve based on the proposed approach. This real-time customer service could lead to increased customer loyalty and resultant increased traffic on the merchant’s site.