By Thomas Kearney
Collecting and analyzing big data in order to differentiate services in a competitive marketplace has been a common theme for many years among Western developed nations. In recent years, emerging markets have seen double-digit growth rates from similar provision of cheaper, faster telco services and from digitizing and optimizing their delivery chain. Now, as these markets mature, competition is forcing businesses to rapidly innovate in order to further improve their delivery and reduce costs.
This drive for innovation was evident in major themes at ConnecTech Asia 2018 in Singapore in June. ConnecTech Asia brought together practitioners from leading businesses in the information and communications technology (ICT) industry working in the telecommunications, manufacturing, retail and transport sectors across the Asia Pacific. Some of those themes included:
- Increasing IoT data volumes across all industries
- Population growth in Asia leading to major data management challenges
- B2B and digitized business value chain integration challenges
Increasing IoT data volumes
While individual IoT devices may send and receive relatively small amounts of data, collectively they are generating huge data volumes that incumbent systems struggle to cope with and derive value from. In telecommunications, for example, most operators working towards 5G are completely re-architecting their network infrastructures and data management systems to deal with the resulting data deluge. Current network nodes are not designed to handle the number of users and devices that will need to be connected and will be transmitting or receiving messages in real time. The amount of data flowing through networks is growing at between 25 and 50 percent a year and this is expected to continue until 2030 at least. With the evolution of new network technologies to increase capacity, there comes the pressing need to address the processing and data management limitations of legacy systems in order to derive actionable insights from the oceans of data becoming available.
Data Challenges from Population Growth
From discussions with companies who visited our booth at the NeXt generation Tech arena (NXT), it was clear that many companies are facing similar data volume challenges as population growth inevitably swells customer numbers. A critical insight was how sticking unthinkingly with the old approaches will almost certainly lead to scaling issues in the future. Telco customer analytics provided a good example; technology that may be applicable in Singapore, Hong Kong or Tokyo with populations of 5-15 million will struggle to serve the volumes of markets like India with more than 1.1 billion subscribers. Having spoken to several SIs across Asia at the event, this came across as a common theme which most technology players are grappling with.
B2B Collaborations for digitized value chains
In addition to the challenges of increased data volumes are the challenges of increased data diversity – dealing with the variety of formats, schemas and frequencies in which data floods in from different devices. The problem is compounded by organisations forming B2B partnerships to digitize the value chain and thereby introducing even further complexity in the data fabric. Organizations are therefore seeking technical solutions that, in addition to offering complex event processing to extract actionable insights from both streaming data and historical data, also offer comprehensive integration and publishing capabilities to enable flexible consumption of data and sharing of results throughout the value chain. As one of the keynote speakers remarked
“We have an Amazon of smart devices flooding the market with every other vendor claiming to have the ultimate magic wand that can future proof your business. However, ask yourselves what impact and value are you adding and how are you really differentiating yourselves while making a sustainable impact on economic and social growth.’
The key data themes expressed during the conference highlight the rapidly growing volume, variety and complexity of data. KX technology provides a flexible platform to address these challenges and for advanced data management and analytics at unparalleled performance. From backend data capture, storage and complex event processing to front-end websockets, permissions and reporting, Kdb+ can provide a full stack, or augment legacy systems to supercharge the bottlenecks faced by other technologies.
For more information on how we can help you with your big data needs please visit us at kx.com.
