The problem

The marketing business function of a leading US bank wanted to acquire customers into their premium credit card lines and have a higher conversion line. With the current collaborative filtering technique, the current infrastructure was failing to scale nation-wide on its marketing strategy. There existed several shortcomings in using many textual features from customer data and computationally intensive processing.

The problem was actually two-faced – a need for moving current data into a more big-data friendly environment for scaling; and improving the performance of the recommendations from processing and model performance at scale.

The cognitive solution

Once the problem was identified as a two-phase problem, without disrupting BAU operations, the first task was to migrate the data into a Hadoop environment. The second one was to optimize and enhance the current recommendation engine for scale.

1. As-is migration to Hadoop

• Migrated all current data into a Hadoop framework with an automated lineage from source to destination
• UAT tests on data quality to ensure source and Hadoop data match

2. Optimization and model enhancement

• Modularized the entire algorithm using current best processing technologies
• Using parallelism frameworks for faster read writes and data handling for memory-intensive operations
• Improving on similarity index scores to improve the relevance

Key insights

The entire process led to improvements in both scaling and process improvements –

• Pre-enhancement metrics of processing 2.5MM customers in 20 hours scaled to 14MM customer is just over 2 hours
• Data migration to Hadoop improved many volume reduction opportunities and helped override severe data expansions in the process
• A complete graphical map from W’s of customers to W’s of the merchants enhanced the entire recommendation engine for micro-personalization