- Quick Overview
- Series ID
- Fields
- Points
- Field Values
- Alternate value types
- Templates
- Standard field names reference
- Meta Fields
- Dynamic Fields
Quick Overview
Series are the basic Shooju data structure.
Series are composed of:
- sid: a unique identifier called a Series ID (sid). Example:
Weather\Japan\Tokyo
. - Fields: an association between names and values. Example:
{city: "Tokyo", aspect: "Temperature"}
. - Points (optional): a list of dates and numbers. Example:
[[2020-06-01, 25.6], [2020-12-02, -3.0]]
.
Every time series are written, past versions are stored in history. Any version of the series can be accessed using the @asof
operator.
Series ID
A unique identifier (sid) is used when information is called from the Shooju database and needs to be distinguished from other information in the database. The sid is a tree-like structure separated by \
. The following structured query is an example:
sid=Weather\Japan\Tokyo
Fields
A field is an area in Series Explorer that accepts input from the user.
A Shooju series may have 0 or more fields. Each must have two key elements: the field name, and its value. The field name must be at least 4 characters long and may only contain numbers, underscores (except as the first character), and lower-case letters. The fields of a series may look like this:
field | value | notes |
city | Tokyo | |
aspect | Temperature | |
description | Tokyo, Japan Temperature | The description field is often used as a human-readable description, but it's not required. |
Fields may be one of three types:
- User-defined: These are specified by users when writing series, they are the only fields that can be controlled.
- Meta fields: These are specific to each series, and hold metadata about the series like the number of points, update date, and others. They are prefixed by
meta.
- Dynamic fields: These are associated with a series, but they change depending on the query used to find the series. They are prefixed by
_
.
Points
A Shooju series may have 0 or more points. Each point has two key elements: a date/time, and a numeric value. Dates with a time element are stored in UTC. The points of a series may look like this:
date | value |
2012-01-01 | 6 |
2012-01-02 | 42 |
2012-01-07 | -7 |
2012-01-12 | -6 |
Note that there is no rule to how the dates are sequenced (hourly, daily, ad-hoc, etc). However, there may only be one value for one date.
Some series have different collections of points associated with different report dates. If available for a certain series, you can retrieve points as they looked in a different report date with @reppdate
operators.
Field Values
All field values are saved in the submitted json-style format (boolean, numeric, string, object, and arrays).
By default, field values are processed as text for purposes of writing queries and perform facetting.
Alternate value types
To enable type-specific processing and type checking, use the field suffixes below
Field Suffix | Meaning | Example Values | notes |
_date | Date | 2015-01-01
2015-01-01T12:42
1642204799000 | ISO-formatted (as YYYY-MM-DD or YYYY-MM-DDThh:mm ) or milliseconds since epoch (same as native Javascript format).
Note: does not support localization; only stores and retrieves one date representation. |
_geo | GeoJSON | {“type”: “point”, “coordinates”: [10, 50]} | Fully supports GeoJSON. The only difference is that the type is lower case. |
_num | Number | 42.67 | Any numeric value (stored as a float). |
_obj | Object | {“hello”:”world”} | The keys of the object become normal Shooju fields and are accessible through the dot-notation (e.g. me_obj.hello). |
_tree | Tree | this\is\a\tree | \\-delimited text with special tree-like query abilities. |
Note: These are suffixes, so they ought to be appended to the end of a field name.
Templates
Field templates can be used to generate new field values on the fly when retrieving fields from existing series. Field values can be combined, transformed, etc. using a jinja2 template.
- For example, if we have a series with two fields:
first_name: "John"
andlast_name: "Smith"
, we can combine these and request a field generated on the fly: <first_name> <last_name>. The example below will return the value"John Smith"
.
={{ first_name }} {{ last_name }}
- We can also transform existing field values and add static text. Consider a field that contains a datetime value
future_date: 2020-01-01T00:00:00
. We can shorten this to only show the date portion and add some descriptive static text. The following example will return the valueFuture Date: 2020-01-01
.
=Future Date: {{ future_date[:10] }}
- Note that when passing a template field, inner quotes will sometimes be required. The inner quotes must be escaped with a backslash. Example:
"\"={{ first_name|replace('_', '') }}\""
Standard field names reference
Meta Fields
Meta fields cannot be edited directly by the user and they are prefixed with meta. Below is the list of meta fields. Note that they are all referenced by meta.<meta field name>
, like meta.updated_by
.
Field | Example value | Notes | deprecated names (do not use) |
meta.points_count | 42 | Number of Points | points_count |
meta.points_min | 6.7 | Minimum value across all points | points_min |
meta.points_max | 7.6 | Maximum value across all points | points_max |
meta.updated_by | jane.doe | Username who last updated the series | updated_by |
meta.updated_at | 1595348051000 | Timestamp of when then the last update to the series ocurred, either to points or to fields, returned as milli since epoch. Note that for XPR series the timestamp only reflects changes to fields, as points are generated dynamically. | updated_at |
meta.freq | H | One-letter description of the time interval: Yearly, Quarterly, Daily, Weekly, Hourly, and X for less than Hourly) | n/a |
meta.processor | weather | ID of the Processors that last updated this series. | n/a |
meta.upload_preset | mydata | ID of the Uploader that last updated this series. | |
meta.dates_min
meta.dates_max | 2015-01-01... | Minimum and maximum point date. | dates_min
dates_max |
meta.reppdates_min
meta.reppdates_max | 2015-01-01... | Minimum and maximum reported point dates. | meta.repdates_min
meta.repdates_max |
meta.repfdates_min
meta.repfdates_max | 2015-01-01... | Minimum and maximum reported field dates. | n/a |
meta.job | [129394, 129749, …] | Array of job IDs that processed the series. | job |
meta.job_created | 190 | ID of job that created the series. | job_created |
meta.job_changed | [192839, 234140, …] | Array of job IDs that modified the series | job_changed |
meta.job_addedto | [192839, 325151, …] | Array of job IDs that added to the series (fields or points) | job_addedto |
meta.job_delfrom | [172859, 291231, …] | Array of job IDs that deleted from the series (fields or points) | job_delfrom |
meta.levels | 3 | Number of levels in the sid. A value of 3 means sid=level1\level2\level3 | levels |
In addition to these, there are also two special meta fields: sid
and l<x>
. These do not use the meta.
prefix, but are also always defined for any series.
field | example | notes | deprecated names (do not use) |
sid | ‘some\sample\sid’ | String containing the current sid. | series_id |
l1
l2
l3 … | l1=some
l2=sample
l3=sid | Contains the level identifier of the sid, | idterms used to contain an array [ l1 , l2 , l3 ], but this behavior will be removed completely. |
Dynamic Fields
They are the only fields whose name may begin with an underscore _
.
Field | Example value | Notes |
_query | sid=Weather\Virginia\Arlington | This returns the queries used to obtain the series. |
_score | 317.59604 | Relative importance of a given series as the result of the query in comparison with other series. There is no defined range, it may go from negative to very big numbers. The score may be affected by custom score functions. |
_repdate | 1624445100000 | This returns the operator UTC timestamp that was used to obtain the series point/field values.
Note: This isn't supported on expressions or XPR series. |
_asofdate | 1532631290998 | Associated with @asof operators. Returns the UTC timestamp of the job immediately preceding the specified date in the operator. |