Don Orth
This manual begins with two sections summarizing kdb+ data and functionality that is covered in this manual. The rest of the manual contains the details.
The table below summarizes the types in kdb+. Many of them were introduced in kdb+, but there is more to be said. For example, you will see in the table that there are two ways to designate a type: a char and a short int. Shorts are used to functionally determine the types of objects and cast objects from one type to another. You will also see the null value for each type in the table; see More about Nulls for more information.
Varchars are interned in a hash table. An atom has a negative type value, say -n. A simple list of those atoms has type value n. For example,
Enumerations are familiar from kdb+ as foreign keys. They are like atomic types, in that individual enumerated values can be atoms or simple lists, atoms have negative datatype values, and simple lists have positive values. See Enumeration for more.
The table below summarizes the primitive functions discussed in this manual. Note the entry for the function Deltas; this function is just one of several defined in Uniform Functions. Unlike the Datatypes table above, this table does not include the primitive functions defined in the kdb+ manual.
Syntactically, q has verbs and functions. Verbs are primitive symbols and names that can be evaluated with infix notation, as in
A function expression appears more than once below if the expression defines more than one distinct function (depending on the arguments). The examples in the table are representative and do not necessarily indicate the full scope of the definitions.
You may have noticed that the various forms of list notation in kdb+ do not provide a way to create a one-item list. Creating one-item lists comes up from time to time, and there is a monadic primitive function called Enlist that does it. For example,
is a one-item simple float list whose item is 34.5 If you enter this expression in a kdb+ console you will seen the following.
As you can see, a one-item list is displayed with a leading comma. In general, Enlist applies to all data objects; the result for any data object is a one-item list whose item is that object. For example,
These constructs all share the same syntactic form and the same functional form. We'll start with the syntactic form, which is denoted by square brackets ([]). Simple assignment of a value to a name with : has been used since early in the kdb+ manual. There is also the matter of both simple assignment and index assignment within defined functions; see Localization.
Indices start at 0. That is, the index of the first item of a list is 0. For example,
Lists of indices produce lists of selected items.
The index 0 is located at the first, fourth and fifth positions of the index list
We say that the result is congruent to the index. In general, the index and result are conforming lists in the sense of atom functions. That is, the structures of the two match up in certain ways. Congruency means that equal values in the index correspond to equal values in the result (but not necessarily vice versa).
Congruency applies to any index list, not just to simple lists. For example,
The index list
Index assignment replaces specified items of a list with new values. For example,
Assignment results are not displayed; the cursor (blinking _) is waiting for the next input.
Index assignment into simple lists is strict with regard to the type of the new value. The replacement atoms must have the same datatype as those that are replaced. For example,
A type error is reported in the console
Duplicate indices are permitted. For example,
The replacement rule for a simple list of indices (0 1 0 2 in this case) is that the indices and corresponding values are used in index order. That is, the following individual steps give the same result as above.
Finally, you can replace multiple items with a single atomic value.
The result of index assignment is the value to the right of the colon, not the new value of the identifier. For example,
If the result of the assignment was the new value of b, which is
Items of a list can be modified with a combination of indexing and index assignment. For example,
This can also be expressed more compactly, and often executed more efficiently, as follows.
The value of the right does not have to be an atom. The rule is that the value on the right must conform to the index of the left, in the way of atom functions.
As before, modifying simple lists is strict; the new items must be the same type as the old.
Any dyadic primitive atom function can be used in place of +.
The syntactic forms for function evaluation use square brackets in the same way as indexing. For example,
Indexing and function evaluation are closely related. In fact, indexing is a form of function evaluation. For example, the following list of three items,
denotes a function whose range consists of the three items
The basic form for evaluating a function f with two arguments is
Indexing of the forms
Functions with no arguments are evaluated by
The following list has several levels. Example results may be displayed on multiple lines to aid comparisons with the indices that created them.
You can select a top-level item, as in
You can select an item of an item and an item of an item of an item.
If the indices are lists then the result is a cross-section of d.
Note that the selected items are arranged in the result in the same order as the atoms in (2 0;0 1). There is no limit on the structure of the indices.
An example of an elided index is the item index expression d[]. There are more interesting situations when indices in an index at depth expression are elided. For example, d[;0] denotes the 0th item of each item of d.
The situation is more complicated when an index other than the top-level index is elided. For example, in d[0 2;;1 0] , the elided index corresponds to 0 1 when indexing d[0], but to 0 1 2 when indexing d[1].
Function arguments can also be elided, as in f[a;;b]. The result is another function, monadic in this case; see Projections for more.
Function evaluation in kdb+ is commonly done with juxtaposition, as in
The kdb+ manual contains instances of function evaluation that uses square brackets and juxtaposition simultaneously, for example
The insert function is dyadic. The first argument (the one on the left) the name of a table. The second is a list of data items to be appended to the table as a new row. This function can be evaluated with square brackets as follows:
The expression
All indexing and function application can be done with the verb denoted by Dot. Here are some examples from earlier in this section.
You can see from the examples that the ith item in the index selects from the ith level of the list. You can also see that to index at the top level only, you must use a one-item list. For convenience, there is another verb denoted by @ for indexing at the top level only. For example,
Analogously, a function f with three arguments a, b and c can be evaluated by
Dot is particularly useful when executing functions of varying valence (argument count) under program control. Both Dot and @ are referred to as Index or Apply, depending on the context.
There are functional forms of Amend based on @ and Dot. The @ form,
Unlike the bracket form of Amend, s is not modified; the result is a copy of s with the specified modifications. This form of Amend is more like select and update expressions. And like select and update, the modifications are applied directly to s if the first argument is `s.
The functional form is also more general than the bracket form because the function f can be any dyadic function, not only primitive atom functions.
The functional form using dot is
Uniform functions extend the concept of atom functions. Like an atom function, a uniform function applies to lists, and the count of the argument list equals the count of the corresponding result list. Unlike an atom function, an item of a uniform function result is not solely dependent on the corresponding item of the argument. The relationship between the argument and result is more general. A typical example is the function named sums. Let's look at an example.
This function is also called Running Sums. The relationship of result items to an argument list is that the nth result item is the sum of the first n items of the argument. For example, the third item, 6.00, is the sum of the first three items of the argument. Similarly, the fourth result item, 2.00, is the sum of the first four items of the argument.
Another example is the deltas function, which produces the differences of neighboring items in its argument. For example,
The ith item of the result of
In SQL, update statements modify existing tables. This is also true in q, but q has also extended the traditional definition to include the creation of new columns. It also permits update expressions to produce new tables instead of modifying the source tables in place. In effect, update expressions are an alternative form of select expressions. Here is an example comparing an aggregation within a select expression to an aggregation within an update expression.
If you compare the ap columns of the two results you will see that the aggregations in the select result are distributed in the aggregation column of the update result according to the sym column. For example, the aggregation value for `aaa, which is 74.44, appears in every row of the update result where the sym value is `aaa.
Update expressions that include aggregations are called update aggregations, or update-by expressions.
Uniform functions are effective in update aggregations because of the way aggregation results are distributed. For example, the following aggregation computes the percent change in prices for each sym.
The pct value for the first or only occurrence of a sym is 100.00. Otherwise, for example, the value of pct in the fourth row is the percent change of the prices of the sym `aaa in the second and fourth rows.
Null values are used in tables to represent missing values. Some null values are exceptional values, such as the IEEE Nan for floats. Others are actual values, such as 0b for bools. Nulls for all datatypes can be found in Datatypes Summary. Indexing a list with an out-of-range index produces the null value for the datatype of the list. For example,
There are two primitive functions for processing null values. The dyadic function ^(fill) replaces null values in the right argument with the atomic left argument. For example,
Integer left arguments are promoted when the right argument is float or real. For example,
The monadic function
This function is useful when replacing null values in a way other than that defined by Fill. For example, suppose you want to replace the null values with their previous values. The Where function (Where and Distinct), used with Null, gives the indices of the null values.
The replacement can then be done as follows:
There are two cases not handled by this expression. The first case is when the first item in v is null, and the second is when there are successive null values. The first case must be handled separately, depending on circumstances. For example, if you know the simple list will always be a float list and that a null first item should be replaced with 0.0, then the following expression will do.
See Associations, Xkey, Key and Value for a more general example. The second case can be handled in a While loop or with the kdb+ construct called Over (see Over for Monadic). The while loop is used here. Let's first do all the iterative steps explicitly.
The test we need for a While statement is whether or not any item of v is a null; if so, another pass through the loop must be made. Again, if we know that the simple list is a float list, then the test can be done with
(Also see While Statements for a more general test). Here is the While statement.
The Type function is a monadic function that gives the numeric datatype value of its argument ( see the Datatype Value column of the table in Datatypes Summary). For example,
Note that values of the Type function are shorts. These examples illustrate a general rule. For datatypes that come as both atoms and simple lists, the type value of an atom is the negative of the type value for a simple list.
Cast (
We saw in Temporal Arithmetic and Comparison Functions that an int d, representing a day count, can be added to a date to give the date d days ahead or -d days previous. The concept of a day count is also associated with a date. For example, casting a date as an int gives the day count for that date; casting an int as a date gives the date for that day count.
Cast is an atomic function. For example,
(0x0064c8;0xff)
The positive value of the Type Value column of the table in Datatypes Summary can also be used as a left argument to Cast. For example,
The negative type values for atoms is not used in Cast.
Casting simple char lists to varchars is a very important operation. It is used, for example, for importing text (varchar) data into tables. There is a special case of Cast that does it, where the left argument is the empty sym `. For example,
This form of Cast applies to right argument lists differently than the forms in the preceding section. Those forms, with left argument "x" or "d" or 6, apply independently to all atoms in the right argument, giving an atom result for each one. However, when the left argument is `, Cast applies independently to all simple char lists in the right argument, giving an atom for each one.
Note that, as these examples show, all varchars are displayed in the console with back-quote followed by the varchar's contents.
Casting with datatype values is particularly useful when associated with computed type values. For example, replacing an item in a simple list is strict, in that the datatype of the replacement value (short, float, etc.) must be the same as the simple list. There may be situations in which more latitude can be allowed. For example, if a simple list has type int, a replacement of type boolean, byte, short or int may be acceptable. A general expression that accepts all such values is
We'll continue this example in Protected Evaluation.
kdb+ provides primitive functions for extracting data from text files, but sometimes you have to do it yourself. A text file is read into the workspace as a list of simple char lists. The simple char lists are processed by partitioning them into smaller lists containing the text of individual values. If the values can be extracted with a special form of casting. For example,
Note in this example that the left argument of Cast is upper case "S" instead of the lower case type name "s". Other conversions can also be done this way.
Casting with "D" will also extract dates in other standard formats.
The primitive function named String produces simple char lists from atomic data. For example,
This function is used to produce char strings for display or export to text files. Consequently, the results contain none of the special notation that distinguishes types in q. For instance, in the above examples the resulting string for the short 23h drops the type name "h". So does the result for long value 4294967296j. You must use Cast with the appropriate left argument to retrieve the q values. For example,
Like an atom function, the string function applies independently to every atom in a list argument. However, unlike an atom function, the result when applied to an atom is a simple char list, not an atom. For example,
Note that the result of
The join of two data objects a and b is denoted by a,b. Any two atoms or lists can be joined (concatenated). For example,
Join provides another important example of Casting with Computed Types. When two simple lists are joined, or an atom and a simple list, the result is a general list unless the types of the two operands are the same. For example,
As when replacing items in a simple list, there may be situations in which more latitude is required. For example, if the left argument is a simple short list then a right argument of type int, boolean or byte may also be acceptable .
Tables can be joined.
Tables can be joined sideways, i.e. each record to each record.
Join for key tables is strict; both the key and data columns must match in names and datatypes.
The process of enumeration is a form of cast. The general form is
where d is the name of an existing data object. There are two possibilities for d, either a key table or a simple list of unique symbols. The object v is either an atom whose value is a key value or item of d, respectively, or a list of those atoms. We say that the projection
If `d names a key table then the enumeration defines a foreign key of that table.
Individual enumerations are distinguished datatypes. Datatype values are assigned in the order in which the enumerations are defined, starting at +-20. (As for the primitive atomic types, an enumerated atom has a negative datatype value and an enumerated simple list has a positive datatype value. For example,
If another enumeration is subsequently defined it will have datatype value +-22, and so on.
Enumerating simple varchar lists with large item counts and relatively few distinct items is an important optimization technique. Kdb+ data is self-describing. For example, a simple int list is stored as a few bytes of descriptive information followed by the int atoms in a contiguous array. When saved to disk this list has essentially the same format. When read from disk it is simply mapped into memory, a nearly instantaneous operation.
Varchars are different because their contents are text, which means that their storage requirements vary. When a simple varchar list is saved to disk, the contents of its items are written end-to-end. When read from disk the individual varchar atoms are recreated. This is not a problem when files are seldom read, say only when an application is initialized. But select expressions can be applied directly to stored tables, which can cause performance problems due to repeated creation of the varchar atoms in varchar columns.
The performance problem is overcome by enumerating varchar columns; a column c would be replaced by the enumeration
where u is a list of the unique items in c, e.g.
Enumerated values can be used in the same ways as non-enumerated ones. For example, compare the following results for c and e.
Internally, an enumerated value is a pair of objects, the name of the domain, say `d, and an int list k of indices into d; the enumerated value is identical to
Associations are the only primitive datatype that has no syntactic form; associations are created with the primitive dyadic function denoted by ! and called Xkey. Associations are associative lists; the items of the left argument to Xkey are the indices, or keys, and the items of the right argument are the values. The left and right arguments must have the same length. The arguments can be any lists, although duplicate items in left argument lists are ineffective. An association whose left argument is a simple varchar list is called a dictionary. See More about Tables for the relationships between tables and dictionaries.
For example, the following associations relate type values to type names.
An example in Processing Nulls uses the following expression to replace a leading null with another value. The example assumes that v is a simple float list.
Associations provide an easy way to extend this expression to handle more than one datatype. For example, the following association provides initial values for simple lists of type char, varchar, int and float.
The replacement expression then becomes
which applies to simple lists of those four types. For example,
This association is easily extended to apply to all lists.
The Key function gives a list of the keys and the Value function gives a list of the values, in the same order as the keys. For example,
The following association has a duplicate key (3).
The first value item keyed by 3 (`c) is accessible by key indexing; the other one (`e) is not.
The following example illustrates indexing at depth in associations.
The result of selecting the value of an association for a non-existent key is the null of the value list. For example,
Here are examples of primitive functions applied to associations.
Unlike non-associative lists, new items can be appended to an association with indexing. For example, the byte datatype can be added to the n2v and v2n associations as follows.
Note that, as with other datatypes, the console display of an association is an executable form from which the data can be recreated.
The Til function also applies to a positive integer n, giving the simple list of all ints from 0 to n-1, in order. In particular, if n is the count of a list then
This section deals with the relationships between dictionaries and tables. We start with the fact that tables are lists of dictionaries, which in this context are called records.
The dictionaries that make up a table are called records. Since tables are lists, structural primitives such as indexing apply to them. Here are some examples.
An out-of-range index gives a record of the null values for each column. For example,
Indexing at depth applies to tables
A simple list of indices gives a sub-table.
The last line shows the console display of a table, as well as the functional form for defining tables. We are now in a position to discuss this form.
In the last result display in the previous section, you can see that everything to the right of the leading + is the definition of a dictionary. That dictionary has the column names of the trade table as its keys and the contents of the columns as its value list.
The leading + denotes the Flip function, or transpose. In this manual we use the Flip function to create tables, not +.
Flip is a monadic primitive that applies to lists and associations. The effect is to interchange the top two levels of it argument. That is, if
Each item of b has 2 items, and therefore can be flipped.
The list c has 2 items because each item of b has 2 items; each item of c has 3 items because b has three items.
Note that if you apply flip twice in succession you get back where you started, which means that the flip of a table is a dictionary. For example,
Since d is a dictionary it can be indexed with items from its key list. In particular,
Equivalently,
Since d is the flip of trade, we can use the same indices of trade, but reversed.
When creating a table in kdb+, we used the syntactic form to initialize the table with empty columns and then populated it with applications of the insert function. For example,
We followed that path because it is the one used in SQL, but we could also have initialized and populated a table at the same time, as follows.
Even so, this is usually done only for small tables. Large amounts of data are often read from files and organized in lists. For example, the data for the table t might have been created as a list,
from which t would be created with
Of course this can also be done syntactically with
One last point: even though a table is the flip of a dictionary, the flip is never carried out. In particular, if the value items of the dictionary are simple lists, so are the columns of the table. There is some indication of this in the console displays. The console display of a flipped list shows the effect of the flip, while the console display of a table does not show a list of records, but simply indicates the flip with a leading +.
A key table is not a table, but a pair of tables instead. The primary key columns form one table and the data columns the other. For example, the key table s has the primary key s and data columns name, status and city. Those two tables are
The console display of s is
You can't see all of s, but you can see enough. To the left is the table of primary keys in parentheses, followed by the Xkey function (!) followed by data table. Since s is a dictionary its primary key table can be extracted with the Key function and its data table, with the Value function.
We can now see how dot notation works. For example, consider the query
Column s in sp is the following enumeration.
which is a list of indices into the key table s. More precisely, it is a list of indices into both the key s and value s tables. Consequently, the column in the above select result denoted by s.name is
Arithmetic is defined for associations with numeric value lists. For example,
You can see that the keys of the sum are the union of the keys of the arguments. Also, if a key is in both arguments then the value for that key in the result is the sum of the values in the arguments. Otherwise, the value in the sum for that key is the value for that key in whichever argument it appears.
kdb+ provides a convenient way to generate random test data, which is very useful when experimenting with the language or modeling applications. The primitive function called Rand produces random sequences of ints and floats. For example, the expression
If the right argument is a float, say 4.5, then the result is a simple float list whose items are random floats between 0.0 and 4.5. For example,
Sequences of random selections from a specific set of values can also be generated. To do that, first form a list of the values, say
The ints 0, 1 and 2 are the valid indices of this list. Consequently, 10?3 is a list of random indices into this list. A list of random selections from this list is generated as follows.
The function named Find is a dyadic function whose left argument is a list and right argument is any data object. The result is the lowest index for which the right argument matches (using the Match function) an item of the left argument (the so-called "first occurrence"). If there is no match the result is the count of the left argument. For example,
Find is type-specific relative to the left argument. In the case of a simple list left argument, the right argument can also be a list whose atoms are all the same type as the left argument. The result corresponds to the left argument item-by-item. For example,
If the left argument is a list of lists and the right argument is simple list, then items of the left argument are matched with the entire right argument. For example,
However, if the right argument is a general list then items of the left argument are matched with items of the right argument. For example,
In this case Find matches items of the left argument with
If the left argument is a table then right argument must be a compatible record (dictionary or list) or table. That is, each column of the left argument, paired with the corresponding item of the right argument, must be valid arguments of Find. For example,
Reverse applies to lists and gives their items in the opposite order. For example,
Reverse comes into play when it is more efficient to work with a list in its reverse order. For example, suppose that an application calls for many joins to the front of a long list l, as in
This expression causes a list of size
then it is possible to do it so that (usually) only b is copied onto the end of a. This is due to the fact that the chunk of memory allocated for a list usually
has unused space at the end. To specify this behavior use the form of amend that modifies l in place. The computation would proceed something like this:
See More about Amend.
The primitives in this section provide the basic functionality for aggregation queries of lists other than tables. They also provide the potential for calculations not easily done as queries. As an example, first form a list of the data in the sp table.
The Where primitive function gives a list of indices where a simple boolean list has the value 1b. For example,
This function serves the same purpose for selecting sub-lists as "where phrases" in select expressions for selecting sub-tables. For example,
or, since sp is a list,
We have seen Distinct before (Enumeration; it gives a list of the distinct items in a list (including tables). For example,
The Take and Drop primitives are useful for selecting from the front or back of a list. Cut, which is an extension of Drop, is used to partition a list into sublists. For example,
Cut is often used with Where. For example, suppose we want to cut the following simple char list at tabs ("\t").
There are two things to notice in this result. First of all, the tabs are still there, one in the front of each segment of the result. But there is also a missing segment,
Now it also typical that after the cut, the character which determines where the cuts take place is dropped from the segments. The first segment is different from the others because it has no leading tab, and so we don't want to drop its first character. The "1 Drop" function should be applied only to each of the other 3 items. There are two ways to express the "1 Drop" function, either as a function projection or a defined function. It is largely a matter of taste. For example,
or
Apply the "1 Drop" function to all but the first item of the above result r, as follows (Verbs and Adverbs).
or
Note that "all but the first item of r" is also expressed with the "1 Drop" function. Now that we see what to do, adjust r as follows:
All three of these primitives apply to tables. For example,
So far we have discussed only one empty list, the one denoted by (). In addition to this, every atomic datatype has a empty list. The thing that distinguishes these lists is their datatypes. Each of the empty lists can be created by Cast. For example,
The empty char list is also "".
Tables in the kdb+ manual are created as empty tables and then populated with the insert function. For example, here is the definition of the trade table from that manual.
Since all simple lists are also general lists, any type of data can be inserted in this empty table. Here is the first insert.
Now the column datatypes are more specific.
From this point on the data that can be inserted is limited in type; the new data for a column is cast to the datatype of the column or an error is reported if that is not possible. We have seen in this manual that tables can be created and populated all at once, but if you want to start from empty tables it is may be best initialize each column with the appropriate specific empty list. You can use null values, as in
Syntactically, kdb+ has nouns, verbs and adverbs. All data objects are nouns, as are all functions. Verbs are primitive symbols and names that can be evaluated with infix notation, as in
Adverbs modify dyadic functions and verbs to produce new, related verbs. The functions produced by adverbs are called derived functions or derived verbs, depending on context.
The adverb Each, denoted by quote ('), modifies dyadic functions and verbs to apply to the items of lists instead of the lists themselves. For example,
The arguments of a dyadic function derived from Each must be lists of the same length, or either argument can be an atom. Actually, both arguments can be atoms, but then each has no effect. For example,
It's common to need to join a list, not an atom, to every item in another list. For example, suppose, in the last example, instead of appending comma to each item, you wanted to append comma-blank, as in ", ". To do that with Each, a 3-item list of comma-blanks must first be created, as follows.
There are two variants of Each for dyadic functions called Each-Left and Each-Right that simplify cases like this. Here is Each-Left (\:) used to append comma-blank to each item of v.
Each-Right is analogous.
There is a form of Each for monadic functions that uses the keyword "each". For example,
Usually the biggest cost in execution time for evaluating an aggregation, particularly for large tables, is the grouping caused by the "by phrase". Moreover, that cost may occur repeatedly because the same grouping may be done for many different aggregations. In q it is possible to precompute the grouping and save it in a separate table to which aggregations are applied later. The aggregations must be computed differently than in an ordinary select-by expression, but a few examples will show you what to do.
In the kdb+ manual we restricted out attention to tables whose columns are simple lists, which conforms to the SQL domain. However, q tables allow any lists as columns, as long as the columns all have the same length. In particular, the following q statement, which groups but does not aggregate, is valid and produces a q table.
You can see that the items of the price and size columns are simple lists, not atoms. For example, compare the row of t for the sym `aaa to the result of the select expression that follows.
You can see that the price column in this result is identical to the price item in the row of table t where sym is `aaa. Similarly, the size column in this result is identical to the size item in t. The table t is an example of grouping without aggregating. Columns price and size of t are called partitioned columns.
Now the question is, how are queries formulated for table t? We'll show two representative examples. Here is the first.
The equivalent query for t is as follows.
Without the Each modifier, the query would average the simple price list for the sym `aaa with the simple price list for the sym `ccc, resulting in another simple list if those two had the same count, or a length error if not. What we want, however, is to average the items in each list separately, which is what each[avg] does.
Note that the only difference in the two results is that sym is a primary key of r1, but not r2.
The second example is an aggregation where part of the "where phrase" in the trade query applies to a partitioned column in t.
The restriction
where the <exp> to be applied to items from price and size is
That is, the expression must be applied to the price and size items for the sym `aaa and separately to those items for the sym `ccc. To do that we define
and then
The difference between r3 and r4 is the same as r1 and r2.
So far the identifiers (names) used in the manuals are composed of alphanumeric characters, with the condition that the first character must be alphabetic, for example, abc and a0B. Kdb+ also permits dots (.) in names; the first character can be a dot and if so, one other character can also be a dot. For example, .c0w and .c0w.abc are valid names, but a.b and .a.b.c are not.
The kdb+ workspace is logically partitioned into directories, or contexts, according to dots in names. For example, start a new kdb+ session and then do
The objects whose names have no dots are said to be in the default context. Up to now all objects created in these manuals are in the default context. The objects named .cx.b and .cx.c are in the .cx context. The object named .cx.c is also referred to as the object c in the .cx context, or the entry c in the .cx directory.
Contexts are used to organize applications. For example, you could put the functions that perform mathematical calculations in the .math context, those that manage files in the .files context, and so on. There are reserved names for managing connections and other things in the reserved context named .z. All .z entries are discussed below. There are other reserved contexts as well. You should consider all contexts with single character names to be reserved.
Contexts are data; they are dictionaries. For example, the Key function gives a list of objects in .cx context.
The default context:
You can work directly in a context. The working context, or current context, is specified with the directory command \d. For example,
You can find the name of the working context with \d and its contents with \v.
New objects can be defined in the working context with unqualified names.
Return to the default context.
The following illustrates the rules for referencing unqualified global names in defined functions.
That is, the value of an unqualified global name in a defined function is the one in the context where the function was defined, not the one where the function resides. This rule for referenced global names works for any two contexts; it is not necessary that one is the default context. It takes time to get used to this rule, and you can avoid it by always using fully qualified names. However, that can be tiresome in complex applications. It is also easier to reorganize scripts when names are unqualified.
The general form of defined functions is
The argument list is a sequence of names separated by semi-colons. The body is a list of expressions and control statements separated by semi-colons. The items in the body are executed strictly from left to right, as are items within control statements. For example, a function that computes the amount of interest paid in terms of principal, rate and time can be defined as follows:
This function has 3 arguments, principal, rate and time. The first statement, somewhat contrived for the example, tests whether rate has an integer part. If so, it assumes that rate is given as a percent and converts it to a fraction. The expression that follows does the evaluation. For example,
The argument list for a monadic function is one name between square brackets (no semi-colons). It is possible to define a function with no arguments by specifying the empty list, []. The empty argument list can be omitted, but if so, default arguments may come into play.
The argument list can be omitted when default arguments are used. The default arguments are x, y and z. If any of these names appears in a function definition that has no argument list then that name is a function argument. Moreover, z is always the third argument, whether or not x or y is present in the function definition. Similarly, y is always the second argument and x is always the first. For example,
Whether or not you use default arguments is a matter of taste. In general, they are useful in general utility functions whose arguments can't be named in meaningful ways.
The value of the right-most expression in a function body is the default result if there is no semi-colon between it and the closing }. However, functions always have some kind of results. Even in a case such as
where nothing is displayed when the function is evaluated,
the expression
will not fail. Consequently, we say that this function has no explicit result.
All unqualified names in a function definition in which there is at least one simple assignment of the form
For functions of valence at least two, when any of the argument positions are left blank in an evaluation expression, i.e. when those arguments are not specified, the effect is to create a new function in which the specified arguments have those fixed values. The new function is called a projection onto those fixed values. For example:
The monadic function g is the projection of f onto its first and third arguments. The function g is formed by fixing the first and third arguments of f to be 1 and 3, respectively. That leaves one argument unspecified, which is the argument of the function g. Continuing,
Trailing arguments that are left unspecified can be left out of the projection expression. For example,
The function h, which has valence 2, is equivalent to f[1;;]
Projections of projections are equivalent to projections of the original function. For example,
which has valence 1, is equivalent to f[1;;3].
Function projections retain their definitions even when their source functions are redefined. For example,
It is also possible to project onto the left argument of a verb. For example, 3+ is a monadic function.
In this example 10 is the argument to the monadic function 3+; function evaluation is by juxtaposition, which in this case is equivalent to (3+)[10]. It is not possible to project on the right argument of + in infix form, but it can done if function evaluation notation can be used for +.
Projections are useful when many different evaluations of a function have some number of common arguments. For example, in kdb+ the trade table was constructed with a series of table inserts of the form
Insert is a dyadic function whose first argument is the table that receives the data in the second argument. That is, the following is an equivalent expression.
In the first expression, the subexpression
Select, update and delete expressions can be evaluated in defined functions. Every name except column names can be a function argument, local variable or global variable (think of column names as local within the phrases). For example,
Default arguments cannot be used in parameterized queries.
If a defined function fails you will see a console display of the failed primitive function and it's argument(S). The console prompt becomes a "> ". You can examine the arguments and local variables of the function that failed. For example,
The underbar in the last line indicates a blinking cursor. You can see from the console display that the failure is a type error of the function +, applied to the arguments `x and 8. In this simple example you can see immediately what the problem is, but it's not always that easy. When it's not, you can examine the arguments and locals in the function. For example,
Enter a back-slash to abort the execution. You will then see the default cursor.
Let's look at a slightly more complicated example.
Again, we see the definition of the failed function. We can examine its arguments.
The problem is clearly the argument b. We can signal up with quote to see where this function is called.
We can examine the argument and the local variable in g.
We can now see that the local d is the problem. Enter \ to abort or ' to signal up and out and fix the definition of function g.
Kdb+ scripts are text files with file extensions q or k. For example, the scripts sp.q and q.k are kdb+ product distribution scripts. Scripts with the file extension q support the level of the q language described in this manual and kdb+.
Scripts can contain any expressions or commands, which are executed from top to bottom when the script is loaded. A script can be named in the startup command, or loaded from a kdb+ console with a load command (\l), or loaded by another script with a load command.
Here are a few lines from sp.q.
The first thing of interest is that the definition of the table sp occupies more than one line. The rules for multi-line definitions are as follows.
* The first line cannot be indented.
* All other lines must be indented.
Function definitions are like table definitions:
can be put on multiple lines at follows: The display shows all possible line breaks.
There are two lines below the definition of sp in the first display. The first line is a comment and the second line is a q expression that will be executed whenever the script is loaded. The commented line contains the equivalent SQL statement. The "s)" in that line indicates SQL.
The simplest form of Conditional evaluation is denoted by
Using a function in this example was simply a matter of convenience. The conditional expression can have any atomic integer value (e.g., boolean or int); truexp is executed if that value is
There are two extended forms of conditional evaluation. In one case there can be more than one conditional expression. The general form is
For example,
More generally, any "true expression" or "false expression" in the above forms can be replaced a block of expressions enclosed in square brackets. For example,
The individual expressions in a bracketed set such as
It is possible to execute a function in such a way that an error does not halt execution. This is particularly useful when executing user input, such as ad hoc queries. Like Amend, there is a Dot version and an At version.
The primitive Value function is useful for executing user input. Here is an example of a protected execution of Value.
The example in Casting with Computed Types shows casting atomic values into the type of a given list c. The examples use atoms that are known to be convertible to the type of c. Protected evaluation can be used when the atoms are not known to be convertible. The general form of the previous expression was
A protected evaluation version is
If the cast fails, the result of the protected evaluation will be the result of an out-of-range indexing selection from c, which is the null value for the type of c. Repeating the previous example,
Continuing with an atom that cannot be cast to an int,
The first Do statement executes the expression count times. The second do statement executes the expression list count times. The expressions in the expression list are executed left to right. If count is an expression then it is executed first and that value determines the number of times the other expressions are executed. Do statements do not have explicit results.
The conditional expression is like the one in Conditional Evaluation. If its result is not 0 then the expression or expression list is executed. In the second case the expressions are executed left to right. If statements do not have explicit results.
The conditional expression is like the one in Conditional Evaluation. As long as its value remains not equal to 0, the expression or expression list is executed. When there are more than one expressions they are executed left to right. While statements do not have explicit results.
An example in Processing Nulls provides an example of a While statement. In that section, the function
was applied repeatedly to a list to replace all nulls with the nearest non-null value to the left. The example list was
The interation in Processing Nulls can then be carried out as follows.
There are two primitive functions that alter the default execution flow in a defined function by exiting, either with a result value or an error signal. The first is Return, which is denoted by
The second function, called Signal, signals an error instead. This function is denoted by
Signal is useful for debugging and processing user input. In the above example, the reason for the signal can be analyzed by the techniques in Debugging. The function f could also be called with user input under protected execution by another function. The signal from f would indicate to the caller that the message "other" was an error message, which could be reported to the user.
Commands are special statements for interacting with the programming environment. All commands have a leading back-slash. Here is a list.
The brackets indicate optional content. If the content is omitted then the command displays its current value.
We have seen \d and \v in The Organization of the Kdb+ Workspace.
This command is used to specify the current dictionary; see The Organization of the Kdb+ Workspace for examples.
This command displays the names of all global variables in the specified dictionary; see The Organization of the Kdb+ Workspace for examples.
This command loads the named script into the current k session. In most examples in these manuals the script to be loaded is given in the startup command. However, it is also possible to load a script from within the kdb+ console. For example,
This is useful when testing changes to a script because changes can be made and repeatedly loaded without exiting from the kdb+ process. Also, commands can be included in scripts. Consequently applications can be partitioned into sets of scripts, all of which are loaded by the top-level script given in the command line.
Set the listening port. As with script loading, in this manual we have set the port with the -p command line option. It can also be set (or changed) in a console, e.g.
Set the timer to the number of milliseconds given by the integer represented by digits, or display the current setting if digits is not present. If the setting is a positive integer n then the function .z.ts is called every n seconds. No calls are made if the setting is 0 (the default). The function .z.ts is user defined and has no arguments.
The execution time of the expression, in milliseconds, will be printed out when execution completes. It often happens that the result is 0, in which case you can execute the expression a sufficient number of times to get a positive result, with which you can compute the average number of milliseconds required for one execution. For example,
The last result shows that, on average, execution of
This command displays 3 longs summarizing the memory usage of the current kdb+ session. For example,
The middle number is the number of bytes allocated by the kdb+ process. The first number is the number of bytes actually in use at the present time. The third number is the size of the current mapped file space.
End the kdb+ session.
When the text following the back-slash is not the text of one of the above commands, it is passed to the operating system for evaluation.
Interrupt.
A Kdb+ process is a server. The port is specified with the -p option in the startup command. For example, the following process will have the port 5001.
The port can also be specified in a script or in the console with the command \p; see Commands.
We have already used the port in kdb+ for executing q queries and displaying results in a web browser. We'll now look at Kdb+ data clients.
A kdb+ client can connect to a kdb+ server process on the same computer, on the same network, or remotely. The client uses the Hopen function to connect. For example, suppose the server listening on port 5001 is named srvr and is on the same network as the client. In that case the client connects to the server as follows.
The symbol `:srvr:5001 is called the communication handle; the int h is called the connection handle. There is no need to identify the server if the client is running on the same machine. The communication handle in that case can simply be `::5001. A remote server is identified by its IP address, say 66.108.242.81, in which case the communication handle is `:66.108.242.81:5001.
An open connection is closed with the hclose function, as in
There are two ways to send messages to a server, asynchronously and synchronously. An asynchronous message does not return a result. The expression that sends an asynchronous message completes as soon as the expression is executed (which may be a little sooner than when the message is sent). Asynchronous messages are also called set messages because typically they cause changes in the state of the server. For example, a delete or insert statement can be sent asynchronously.
A synchronous message expects a response from the server. The expression that sends a synchronous message waits for the response and returns it as its result. For example, a select statement must be sent as a synchronous message. Synchronous messages are also called get messages, since they expect a response.
There are two forms of messages. One is a character string holding an executable expression. For example,
The second form of an message is a list. The first item of the list is a char string or sym atom holding the name of a function (i.e., stored procedure) to be executed on the server. The other items are the arguments of the function call, in order. For example, the above insert message can also be phrased as a remote procedure call, as follows.
A synchronous message is sent by executing the expression
and an asynchronous message is sent by
The following examples send synchronous character string messages to the communication partner.
Since the insert message was sent synchronously the result (the name of the modified table) is returned. This confirms that the insert was sucessful. However, if the client doesn't require a response, the insert message can be sent asynchronously, as in the follow example.
Execution of this message returns immediately with no result.
The function-arguments list form of this insert message is
In most realistic situations the data to be inserted is not constant, but is either generated algorithmically or received from an external source. Consequently, the function-arguments message format is the more generally useful one because it does not require formatting the data into char strings.
The default behavior of a kdb+ server is to apply the Value function to incoming messages. This behavior can be modified with message filters. Message filters are monadic, user-defined functions with reserved names .z.ps for asynchronous (set) messages and .z.pg for synchronous (get) messages. The reason for two filters is that the asynchronous message filter is not expected to return a result - and even it did, the result would not be sent to the sender - while the synchronous filter is expected to return a result. The argument to each function is the complete message received from the sender.
There are many different uses for message filters. In a server that processes ad hoc queries the filters can use protected execution to avoid stopping on query errors. In so-called gateway servers, message filters route client messages to other servers. For example, select statements with a date specification in the where phrase can be routed to a real-time server or historical server. (How can the message filter know the date? The message can be in a stored procedure call, with a parameterized query and the date as arguments). Message filters can also maintain state while a sequence of related messages is processed.
Whenever a message is received the variable .z.w is automatically given the value of the server's connection handle to the sender. This value can be referenced in the message filters. The value can be used for various purposes. For example, if you maintain a list of connected partners and the current value of .z.w is not in the list then you know this is a new user, in which case authorization and initialization functions can be executed. For example,
The primitive Value function can evaluate either form of message. It applies to a simple char list holding a valid kdb+ expression, evaluates the expression as if it had been entered at a console, and returns the result of the evaluation, if any. For example,
Either communicating partner can close a connection. A message indicating the close is printed in the console of the other partner unless that partner has a close handler. A close handler is a monadic function with the name .z.pc that is automatically called with the connection handle to the partner that closed the connection.
If a client closes a connection then the server's .z.pc can use the argument to remove that client from its client list. If the server unexpectedly closes a connection (the server crashes) then the client may try to reconnect. Typically this is done with a timer so that reconnection is attempted repeatedly until successful, or perhaps up to some maximum number of attempts. In the following example, .z.pc resets the connection handle to 0 and sets the timer to 1 second. Note that the way to execute a command within a defined function is to apply the Value function to a char string holding the command text. The function .z.ts will then be called automatically every second. On each call the client attempts to reconnect. If successful, the connection handle will be assigned a positive int value. In addition, the if statement in .z.ts resets the timer to 0 if the reconnection attempt is successful.
There is also a message filter for HTTP messages, with the reserved name .z.ph. HTTP messages are always synchronous, so there is only one filter. This filter, which manages the Kdb+ Web Viewer, is the only one with a default definition. It can be replaced with a customized version.
Kdb+ distinguishes three types of files, kdb+ data files, text files and all others, which we will call binary files. Files are identified by file handles, which are sym atoms of the form
For example, `:c:/kdbtestdata is the file handle. File handles, like communication handles, are sym atoms that begin with the character ":".
Data in the Kdb+ workspace is self-describing, which is accomplished by appending the descriptive information to the actual data. A kdb+ data file is a binary file that is the image of a kdb+ workspace data object, including the descriptive information.
Kdb+ data can be written to kdb+ data files with Dot Amend. For example,
It is also possible to append to an existing file with Amend. For example,
Note the comma (Join) in place of colon (Assign). The file can be read by applying the Value function to the file handle, as follows.
Each of these operations opens (or creates) the file, applies the operation, and closes the file. It is also possible to explicitly open a file, in which case it stays open until you close it. A file is opened and an open file handle is created in the same way as a connection handle. For example,
Data can be appended to the open file with the same syntax for sending messages, i.e. either
The returned value, which may be different when you repeat this example, is the value of h. This is analogous to the above Amend statement returning the file handle as its result. Any number of appends can be done with this open file handle, which can then be closed with the Hclose function.
Kdb+ data files are used for transaction logging. When a transaction is appended to the log file with Amend, the file write is immediately synchronized to disk because the file is closed after the write. If an open file handle is used then the file write is not synchronized, which permits many more logging operations per second.
The above Amend expression, when applied to tables, writes an entire table as one file. It is also possible to write the table to disk as a directory whose file contents are the individual columns. That is, there is one file per column in the directory. This format of a table on disk is called splayed format. The Amend statement for writing a splayed table differs from the one above only in the file handle.
It is also possible to write the entire contents of a database context to one file. In particular, the default context can be written to file namedcurrentdb as follows.
Text files are not Kdb+ data files and therefore the above applications of Amend do not apply. The primitive verb denoted by 0: takes a file handle as its left argument and a list of char strings as it right argument and writes the list to the text file with that handle. For example,
As in the Amend cases, the result is the file handle. New rows can be appended to an open text file by using the negative handle.
The monadic primitive function read0 is used to read a text file.
Binary files are managed in the same way as text files, using the verb 1: instead of 0:. From the Kdb+ point of view, binary files are lists of byte items. For example,
creates the binary file f.bin with contents
The following reads the file.
Text files can also be read and written with read1 and 1: and managed as binary or char data. For example,
There are two primitives for doing this, one for text files (0:) and one for binary files (1:). The field types are designated by letters, like the q datatypes. The letters are same for text files as q data, only capitalized. For binary files, the letters are lower case.
For example,
A text file with fixed-width fields is read by
and a binary file is read by
The right argument f is either a file name as a sym atom or a 3-item list of the form
The result in both cases is a general list of lists with an item for each field.
Variable width, delimited text files can also be read. The left argument in this case is a pair of the form
For example, suppose that the following display are the rows of a csv file named test.csv.
If this file read by the expression
then the result is the list
You can see that the text "abc" on the first row of the file becomes the int 0N, "def" becomes the float 0n, and "jkl" becomes the null char " " (because it has more than one character). However, if the file is read with an enlisted delimiter,
the result is the table