Research in kdb+/q: AQuery

29 Sep 2016 | , , ,
Share on:

Dr. Dennis Shasha, of the Courant Institute of Mathematical Sciences at NYU, and José Pablo Cambronero, a PhD student at MIT, recently presented the results of their research on AQuery and q/kdb+ at the latest Kx Community NYC Meetup.

AQuery is a simple extension to SQL that makes joins, moving averages, correlations, and other such operations, fast on terabyte-scale data. AQuery unifies the declarative syntax of SQL 92 with the addition of an ASSUMING ORDER clause, moving aggregates, flatten etc to achieve the semantics of kdb+. The system includes an optimizer. AQuery compiles to q.

There are two aspects of this project that Dennis and José presented: (i) AQuery, the language and the optimizer; and (ii) using q/kdb+ as the execution engine for new languages. They provided a series of helpful dos and don’ts.

Check out the video on the Kx Systems’ Youtube channel [here].


Python vs kdb+ for data analytics

A comparison of Python and q for data analysis

21 Aug 2018 | , , , ,

Guest blogger Ferenc Bodon illustrates using Python, SQL and kdb+ for data analytics in this blog. He takes an example that goes just one step beyond the simplest use cases by performing some aggregation based on multiple columns. Anybody who analyzes data tables will bump into this type of problem, probably on the third day.

Kx Insights: Machine learning subject matter experts in semiconductor manufacturing

9 Jul 2018 | , ,

Subject matter experts are needed for ML projects since generalist data scientists cannot be expected to be fully conversant with the context, details, and specifics of problems across all industries. The challenges are often domain-specific and require considerable industry background to fully contextualize and address. For that reason, successful projects are typically those that adopt a teamwork approach bringing together the strengths of data scientists and subject matter experts. Where data scientists bring generic analytics and coding capabilities, Subject matter experts provide specialized insights in three crucial areas: identifying the right problem, using the right data, and getting the right answers.