Machine Learning and AI: The Future Possibilities in Business

20 Jun 2018 | , , ,
Share on:

At Kx25, the international kdb+ user conference held in New York City on May 18th, Alan Rozet, from Kx partner, gave a presentation (available on the Kx Youtube channel here) about how machine learning (ML) is being used in business today. Alan is a Principal Machine Learning Engineer at Capital One with wide ranging experience in ML including in the financial, health care and consumer sectors.

Brainpool is an academic collective of over 250 experienced data scientists with doctorates and masters degrees from top universities around the world whose goal is to bridge the gap between academia and industry. They are partnering with Kx on a number of ML consulting projects and are involved in creating internal machine learning training material.

The need for ML talent by businesses is far outstripping the rate at which trained specialists are entering the workforce. In his talk, Alan cites figures from the McKinsey Global Institute, which estimates that there will be a 50% to 60% gap between the number of trained data scientists and the requisite demand for data scientists in 2018. At the same time, the Artificial Intelligence market is estimated to grow from $640M in 2016 to $37B in 2025, at a compound annual growth rate of 50%.

In his presentation, Alan presented several real world business uses for ML, including for credit card spend forecasting. In this case, ML was used to manage risk and prevent credit card fraud. To begin, the developers of the application asked themselves, ‘Given a customer with an historical transaction record, how does one predict their aggregate spend month to month?’

Assuming that a single customer might have 50 to 100 transactions per month, aggregated across millions, or tens of millions of customers, and across entire organizations, this quickly became a Big Data problem.  A large part of the project focused on how to take advantage of technologies for fast batch processing and other back-end resources for model training that allowed the data scientists to use many instances, or servers, at once for faster training. This supervised learning case predicted aggregate monthly spend per customer.

Alan also looked at the same problem in a slightly different way, in terms of anomalous credit card spend detection. He pointed out that anomaly detection is very much a problem where you need a close interplay between humans and the machine learning model. The data scientist needs to consider what kinds of signaling is needed in anticipation of an anomalous event, and once one has happened, who should be alerted, and how might that decision change depending on the consumer segment.

For example, if a new consumer item that is enormously popular, like a ‘Tickle Me Elmo’ toy comes out and suddenly tons of consumers are purchasing it, there will be widespread new random spends of $100 — is that truly an anomaly that should generate an alert? Versus a small business customer who typical spends $10K a month, and suddenly has charges that jump to $100K or $1M. What should that alerting look like? Which department should be notified and how should the credit card company let the customer know?

In addition to demonstrating how forecasting and anomaly detection can be tied to the proactive case in credit card fraud detection with ML, Alan described a number of other ML use cases in other industries in his talk. For further insights into how businesses are using ML watch Alan’s full Kx25 presentation here on the Kx Youtube channel.

For more information about ML at Kx, please write to us at


Kx extends relationship with NASA Frontier Development Lab and the SETI Institute

Detection of Exoplanets at NASA FDL with kdb+

13 Dec 2018 | , , , ,

Kx data scientist Espe Aguilera explains a NASA FDL mission to improve the accuracy of finding new exoplanets using machine learning models. The data for the project will come from the Transiting Exoplanet Survey Satellite (TESS), which was launched in April 2018, with the objective of discovering new exoplanets in orbit around the brightest stars in the solar neighborhood.

Kx extends relationship with NASA Frontier Development Lab and the SETI Institute

The Exploration of Space Weather at NASA FDL with kdb+

4 Dec 2018 | , , , ,

Our society is dependent on GNSS services for navigation in everyday life, so it is critically important to know when signal disruptions might occur. Physical models have struggled to predict astronomic scintillation events. One method for making predictions is to use machine learning (ML) techniques. This article describes how kdb+ and embedPy were used in the ML application.

kdb+ Machine Learning embedPy

Machine learning: Using embedPy to apply LASSO regression

23 Oct 2018 | , , ,

By Samantha Gallagher   The use of kdb+ for machine learning in financial technology and other industries is expanding following the release by Kx of the powerful embedPy interface, which allows the kdb+ interpreter to manipulate Python objects, call Python functions, and load Python libraries. Now Python and kdb+ developers can fuse both technologies together, […]