Skip to main content

Time Series Data

Time-series data is the recorded history of a Parameter — every temperature reading, every energy meter value — kept so a user can look at a chart instead of a single current number. In ESP RainMaker Neo the Node decides what to record and publishes it on a dedicated topic; the cloud stores each sample and keeps running rollups per hour, day, week and month.

Recording is opt-in per Parameter. Nothing is recorded unless firmware asks for it.

What the user sees​

A history view for the Parameters your firmware marked as time-series — typically a line or bar chart with a time range selector.

Three shapes of query are available:

  • Raw — the individual samples, newest first, over a time range.
  • Latest — the single most recent sample.
  • Aggregates — precomputed rollups over hourly, daily, weekly or monthly windows.

Each aggregate window returns a fixed set of fields: count, sum, min, max, average, first and last value, a cumulative value, the window bounds and the timestamp of the last sample in it. There are no aggregate functions to choose from — the rollups are computed as data arrives and you read whichever window you want.

Day and week boundaries are computed in the timezone the Node stamped on the sample, not the server's, so "yesterday's energy use" means yesterday where the device is. A sample with no timezone falls back to UTC.

Simple vs cumulative series​

The distinction is about what the rollups mean, and it is chosen in firmware.

Simple — the value is aggregated directly. A temperature sensor reporting 21.5, 22.0, 21.8 gives you the min, max and average of those numbers.

Cumulative — the value is a monotonically rising total, like an energy meter, and the rollups aggregate consumption rather than the reading. The first sample in a window is only a baseline and contributes nothing; each later sample adds the delta from the one before. If a reading comes in lower than its predecessor — a meter reset — that reading is itself treated as the step's consumption. At a window boundary the last reading carries forward as the next window's baseline.

Pick cumulative for anything that counts upwards and never resets in normal use. Pick simple for anything you would want an average of.

What the firmware must do​

  • Set PROP_FLAG_TIME_SERIES on the Parameter for a simple series, or PROP_FLAG_TS_CUMULATIVE for a cumulative one. With either flag set, values are queued for time-series publishing when the Parameter is updated.
  • Also list time_series in the Parameter's properties in the reported Configuration. The cloud reads that list when clearing a Node's history, so a Parameter missing from it is not cleaned up.
  • Enable time sync. Every sample carries its own timestamp, so a Node with no clock produces useless history.
  • Tune the publish path if your data rate is high: CONFIG_RMAKER_TIMESERIES_DATA_QUEUE_LENGTH (default 100 items) and the publish delay range. A full queue drops incoming samples silently.

See Time series data for the firmware side, and the Data collection spec.

What the operator must do​

Almost nothing. There is no per-deployment switch for time-series: the ingest rule is created enabled, and no Lambda sits on the hot path — samples go from the MQTT broker straight into DynamoDB, so ingest scales with IoT Core and DynamoDB rather than with Lambda concurrency.

The one setting that exists is week alignment. A timeseries_config.json object read at startup takes a single week_start field of monday or sunday, defaulting to Monday.

Two tables hold the data — one for raw samples and one for the rollups — and point-in-time recovery is enabled on both. They are the only tables in the platform with PITR turned on; do not assume it elsewhere.

week_start does not affect stored data

The ingest path always buckets weeks from Monday. week_start is consulted only when reading a single historical weekly window, so setting it to sunday does not change how weekly aggregates are computed or stored. Treat weekly aggregates as Monday-aligned until this is fixed.

Limits and caveats​

Retention is unlimited, and that is the cost problem. Neither table sets a TTL. Raw samples and rollups are kept until a Node's history is explicitly deleted. Storage grows without bound, fastest on the raw table, so enable time-series only on the Parameters a user will actually look at — every flagged Parameter is a write and a stored row on every update.

You must know what to query. A query requires both the Parameter key and its data type. There is no discovery API that lists which Parameters on a Node have history, because that would need an extra index.

Only four window sizes. hourly, daily, weekly, monthly. Nothing custom, nothing below an hour.

Timezone is per sample and sticks. The first timezone seen for a Parameter is frozen into the in-progress window. Changing a Node's timezone mid-stream closes that window early, leaving one misaligned window at the switch. Raw and latest queries are unaffected.

Values must be coercible to a number. Numbers pass through, numeric strings are parsed, booleans become 1.0 and 0.0. Anything else fails aggregation for that sample.

Access follows the Group. A caller needs access to the Group and read permission on the Node. The stream processor that builds the rollups runs with system privilege and bypasses those checks, by design.

Query page-size and span limits unconfirmed

The default page size is 20 and no maximum page size or queryable time span is documented, so a very wide range may be limited only by the API Gateway timeout. If you need very wide queries, verify the practical ceiling against your deployment.