What is Machine Learning?
Machine learning (ML) is an application of artificial intelligence (AI) that allows computer systems to use algorithms to adapt to enable better outcomes, based on data rather than explicit programming. ML is based on the concept that systems can be taught to learn from data, identify and recognize patterns and then make decisions with minimal human input. Self-driving cars, social media tracking, next best offers and online recommendations, cyber-security and fraud detection, along with predicting sensor failure in industrial processes are some of the more familiar examples of machine learning applications.
The increasing volumes and availability of different types of data combined with cheaper and more powerful processing capabilities along with reduced data storage costs continues to drive the interest in machine learning and its applications. As a result of these factors, it’s even easier now to build models that can automatically analyze bigger and more complex data, on a much larger scale, and deliver more accurate and faster results.
Why Kx for Machine Learning?
Kx, incorporating the market-leading in-memory time-series database kdb+, is able to rapidly process vast quantities of data using less computing resource than competing technologies and is therefore ideally placed to enable adoption of ML across multiple industries and use cases. Our core technology is particularly well-suited to a number of machine learning tasks:
- Kx’s ability to manage, ingest, store and analyze huge datasets, makes it the ideal engine to feed deep neural networks – where enormous volumes of data are required for effective training
- the streaming analytic capabilities of q, its integrated programming language, enable online training of models and real-time prediction
- the time-series operations and array processing capabilities of qSQL, its querying language are perfect for the feature engineering steps of sampling, aggregating and joining datasets
Kx clients and partners, being data-centric by nature, were among the earliest adopters of machine learning techniques and continue to develop ML applications such as alpha generation for algorithmic trading, portfolio optimization and trade surveillance for financial applications. In other sectors, our ML team has developed models for predictive maintenance in manufacturing and network optimization models for telcos and utilities.
How can Kx help?
Machine-learning capabilities are at the heart of future technology development at Kx and the aim of the Kx machine learning team is to centralize research and development in order to provide users with tools and libraries to help them gain insights from their data. To discover more about our machine learning toolkit that contains libraries and scripts that provide kdb+/q users with general-use functions and procedures to perform machine-learning tasks on a wide variety of datasets here or for more information on how to set up kdb+ to create a machine-learning environment using either Anaconda, Docker or a manual build go to our Machine Learning toolkit.
Widely deployed in the data science community, tools such as Analyst for Kx take much of the mundane effort out of visualization, normalization, cleansing and data munging. Widely used native statistical libraries sit alongside the tightly coupled tools such as TensorFlow, Theano and Keras. Even your favorite python libraries can be called directly from Kx code.
For research, feature engineering and real-time classification and regression, Kx technology is ideal for predictive analytics across any velocity of data; real-time, mini-batch, through to petabytes of deep history are all equally supported, and highly performant.
To meet the expected demand for ML and AI consultancy, Kx has a number of consultants available to deliver the benefits of ML to the Group’s customers:
- Domain knowledge of the business opportunities or problems that machine learning can address
- An understanding of ML algorithms and techniques
- A technology platform capable of dealing with the volume and velocity of data that is involved
The basis for Kx Technology is a unique integrated platform which includes a high-performance historical time-series columnar database called kdb+, an in-memory compute engine, and a real-time streaming processor all unified with an expressive query and programming language called q. Designed from the start for extreme scale, and running on industry standard servers, the kdb+ database has been proven to solve complex problems faster than any of its competitors.LEARN MORE