If you're running Zenoh, whether as rmw_zenoh in ROS 2 or as the transport between a robot, a gateway, and the cloud, you'll eventually run into the same question:

"Where does this data actually go? I need to replay yesterday's run, debug a fault from a remote robot, and pull a few thousand frames for training."

There's plenty of material on the communication layer itself: pub/sub primitives, how Zenoh compares to DDS, multi-node setups down to a Raspberry Pi. rmw_zenoh reached Tier-1 status starting with ROS 2 Kilted Kaiju, and it's getting real attention at events like ROSCon.

What's harder to find is the next layer: what happens to a sample after it's published. Connecting a live Zenoh network to something persistent and queryable usually means working around the limits of Zenoh's existing storage options.

This post shows another way. By the end you'll have a Python publisher sending data over Zenoh, a storage backend persisting it automatically, and a query client pulling exact time ranges back out, all without a custom backend or a bridge process.

What the existing storage backends leave out​

Zenoh ships official storage backends: memory, S3, InfluxDB, RocksDB, and filesystem via the zenoh-plugin-storage-manager. By default, queries return the latest stored value for each matching key. The InfluxDB backend goes further with a _time selector for historical queries:

robot/alpha/**?_time=[now(-5d)..now()]

That works for structured numeric telemetry. The problem is that InfluxDB stores binary payloads as base64-encoded strings, wildcard queries only work with v1.x, and sample attachments don't map to InfluxDB tags. The memory, RocksDB, filesystem, and S3 backends don't have a time-range query mechanism at all.

What was missing was a backend built for binary time-series data: one that stores every sample as-is, indexes it by time, and fits Zenoh's data model without a translation layer. That's what led to building a native Zenoh API in ReductStore.

A storage backend that speaks Zenoh natively​

Starting with version 1.19, ReductStore (an open source time series object store built for robotics and edge data) ships a native Zenoh API. It keeps every sample, not just the latest, indexed by time, so the queries above just work. A ReductStore instance joins your Zenoh network as a regular peer: it registers a subscriber for writes and a queryable for reads, talking directly to the storage engine. No bridge process, no extra protocol, no custom backend to maintain.

The mapping between Zenoh and ReductStore's data model is close to one-to-one:

A Zenoh key like robot/alpha/camera becomes an entry in a bucket. The attachment, Zenoh's mechanism for attaching metadata to a sample, becomes a set of labels you can later filter on. Everything else (HTTP API, SDKs, web console, retention policies, replication) works on that same data, regardless of whether it arrived via Zenoh or HTTP.

Setting it up​

The full runnable example, two simulated robots, a Zenoh router, ReductStore, and a query client, is on GitHub at reductstore/zenoh-example. Clone it and docker compose up if you want to follow along directly. Below is the minimal setup, piece by piece.

1. Run ReductStore as a Zenoh peer​

The entire integration is configuration, with no code changes to ReductStore itself. Add this service to your docker-compose.yml:

services:
  zenoh-router:
    image: eclipse/zenoh:latest
    ports:
      - "7447:7447"

  reductstore:
    image: reduct/store:latest # v1.19+ has a native Zenoh API
    environment:
      RS_API_TOKEN: "reductstore"
      RS_BUCKET_0_NAME: "fleet"
      RS_BUCKET_0_QUOTA_TYPE: "FIFO"
      RS_BUCKET_0_QUOTA_SIZE: "1 GB"
      RS_ZENOH_ENABLED: "true"
      RS_ZENOH_CONFIG: "mode=client;connect/endpoints=[tcp/zenoh-router:7447]"
      RS_ZENOH_BUCKET: "fleet"
      RS_ZENOH_SUB_KEYEXPRS: "robot/**"
      RS_ZENOH_QUERY_KEYEXPRS: "robot/**"
      RS_ZENOH_QUERY_LOCALITY: "Any"
    ports:
      - "8383:8383"
    depends_on:
      - zenoh-router

RS_ZENOH_SUB_KEYEXPRS and RS_ZENOH_QUERY_KEYEXPRS are key expressions, wildcard patterns like robot/**, that define which keys ReductStore subscribes to and answers queries for. The fleet bucket is created automatically on startup, and the quota settings make it a rolling store that keeps the most recent 1 GB of data.

2. Publish from your ROS 2 / Zenoh node​

Any node publishing on a matching key gets persisted automatically. No SDK calls, no extra client, just publish the way you already do, with timestamps and an optional attachment for metadata:

import json
import zenoh

ROBOT_ID = "alpha"
KEY = f"robot/{ROBOT_ID}/telemetry"

with zenoh.open(zenoh.Config()) as session:
    pub = session.declare_publisher(KEY)

    telemetry = {"battery": 87, "status": "ok", "speed": 1.2}
    labels = {"robot": ROBOT_ID, "status": telemetry["status"]}

    pub.put(
        json.dumps(telemetry).encode(),
        encoding=zenoh.Encoding.APPLICATION_JSON,
        attachment=json.dumps(labels).encode(),
    )

The JSON attachment becomes record labels in ReductStore. ReductStore also adds two labels automatically to every record: zenoh_source_id (which node published it, useful for tracing data provenance across a fleet) and zenoh_ts_ntp64 (the original Zenoh HLC timestamp at full precision).

3. Query a time range back out over Zenoh​

This is the part that's usually missing: querying isn't bolted on as a separate HTTP step. It's the same Zenoh session, using get() with a selector that carries a time range:

import zenoh

KEY = "robot/alpha/telemetry"
start_us = 1_700_000_000_000_000
stop_us = 1_700_000_600_000_000

with zenoh.open(zenoh.Config()) as session:
    replies = session.get(
        f"{KEY}?start={start_us};stop={stop_us}",
        consolidation=zenoh.ConsolidationMode.NONE,
    )
    for reply in replies:
        if reply.ok:
            print(reply.ok.payload.to_bytes())

ConsolidationMode.NONE matters here: without it, Zenoh may consolidate replies down to a single sample per key. For a time-series query you want every record in the range, so consolidation has to be disabled.

4. Filter by label while you're at it​

Because the JSON attachment becomes labels, you can filter server-side using the same conditional query language ReductStore's HTTP API uses, with no client-side filtering required:

import json
import zenoh

attachment = json.dumps({"when": {"&status": {"$eq": "warn"}}}).encode()

with zenoh.open(zenoh.Config()) as session:
    replies = session.get(
        f"robot/alpha/camera?start={start_us};stop={stop_us}",
        attachment=attachment,
        consolidation=zenoh.ConsolidationMode.NONE,
    )
    for reply in replies:
        if reply.ok:
            print(reply.ok.payload.to_bytes())

"Every camera frame from the last ten minutes where the robot was in a warn state" is one get() call. The when syntax supports comparison, logical, arithmetic, and aggregation operators. See the Conditional Query Reference for the full list.

5. Replay and inspect with the Python SDK​

For replay, debugging, or building a training set, you don't have to stay on the Zenoh API. The same data is available over HTTP via the reduct-py SDK (or the JS, Go, Rust, and C++ SDKs):

import asyncio
from reduct import Client, BucketSettings, QuotaType

async def main():
    async with Client("http://localhost:8383", api_token="my-token") as client:
        bucket = await client.get_bucket("fleet")

        async for record in bucket.query(
            "robot/alpha/camera",
            start="2026-06-15T00:00:00Z",
            stop="2026-06-15T01:00:00Z",
            when={"&status": {"$eq": "warn"}},
        ):
            print(record.timestamp, record.size)
            frame = await record.read_all()
            # feed `frame` into your replay tool, notebook, or training pipeline

asyncio.run(main())

Live ingestion via Zenoh and historical queries via HTTP (or Zenoh) both read from the same bucket, so there's no separate export or sync step.

The same query works from anywhere on the network​

This is Zenoh's location transparency at work: it routes a query to whoever answers for a key, so the same get() call works whether ReductStore is running next to the robot, on a gateway, or in the cloud:

# on the gateway
python query_zenoh.py --robot alpha --last 60

# from a laptop on a different network
python query_zenoh.py --robot alpha --last 60 \
    --endpoint tcp/gateway.example.com:7447

RS_ZENOH_QUERY_LOCALITY controls whether a given ReductStore instance answers local queries, remote queries, or both, useful if you want an edge instance to serve replay requests only from the robot itself, while a cloud instance serves everyone else.

Retention and replication, for free​

Once data is flowing in, two problems show up fast: disks fill up, and you want a long-term copy of some of it in the cloud.

For retention, ReductStore buckets support a size-based FIFO quota: set a limit, and the oldest blocks are dropped to make room:

environment:
  RS_BUCKET_0_NAME: "fleet"
  RS_BUCKET_0_QUOTA_TYPE: "FIFO"
  RS_BUCKET_0_QUOTA_SIZE: "1 GB"

This is more robust at the edge than a time-based "keep the last 24 hours" policy: if a robot goes offline for a few days, a time-based rule would silently wipe everything by the time it reconnects. A size-based policy just keeps the most recent 1 GB, however long that takes to fill.

For replication, you can forward a filtered subset to a cloud instance using the same conditional query syntax, for example, only frames and telemetry around a "warn" event:

environment:
  RS_REPLICATION_1_NAME: "warn_to_cloud"
  RS_REPLICATION_1_SRC_BUCKET: "fleet"
  RS_REPLICATION_1_DST_BUCKET: "fleet"
  RS_REPLICATION_1_DST_HOST: "https://reduct.example-cloud.com"
  RS_REPLICATION_1_DST_TOKEN: "${CLOUD_TOKEN}"
  RS_REPLICATION_1_ENTRIES: "robot/*/camera,robot/*/telemetry"
  RS_REPLICATION_1_WHEN: '{"&status": {"$eq": "warn"}}'

The label you attach at ingest, status: warn, is the same label that drives the read query, the retention policy, and the replication rule. One concept, three jobs.

Try it yourself​

The runnable version of everything above (two simulated robots, a router, ReductStore, and a query client) is on GitHub:

git clone https://github.com/reductstore/zenoh-example
cd zenoh-example
docker compose up --build

Once it's running, open the ReductStore web console at http://localhost:8383 (API token: reductstore) to browse the fleet bucket, inspect labels, and preview frames directly in the browser. From a separate terminal, query over Zenoh:

cd query && pip install -r requirements.txt

# latest 60 seconds of telemetry for robot alpha
python query_zenoh.py --robot alpha --last 60

# last 10 minutes, only records where status was "warn"
python query_zenoh.py --robot alpha --last 600 --only-warn

Next steps​

If you're running rmw_zenoh in production or evaluating it for a fleet, the gap between "messages are flowing" and "I can replay, debug, and train on this data" is smaller than it looks. It's a Docker Compose service and a handful of environment variables, not a new component to build and maintain.

For the full configuration reference, including all RS_ZENOH_* settings, key expression syntax, and current limitations (single-bucket mode, no deletion via the Zenoh API), see the ReductStore Zenoh integration docs. For label-based filtering across queries, retention, and replication, see the Conditional Query Reference and the Data Replication guide.

If you have any questions or comments, feel free to use the ReductStore Community Forum.

ReductStore 1.20.0 is now available. This release adds lifecycle policies for automated retention and compression, expands Protobuf support across the ingestion and query ecosystem, and brings SQL and Parquet workflows to the ReductSelect extension.

To download the latest release, visit the Download Page.

What's new in 1.20.0?​

The first major change in v1.20 is lifecycle policies. ReductStore can now run background tasks that delete or compress old records based on age, entry names, and optional label conditions. This makes retention and storage-efficiency rules part of the database configuration instead of an external cleanup script.

The second major change is Protobuf support across the ecosystem. ReductBridge can ingest Protobuf payloads from MQTT, extract labels from message fields, and store schema metadata as a $schema entry attachment. ReductSelect can then use the same attachment to decode records and run SQL over the decoded fields.

The release also extends ReductSelect with SQL processing for structured records and Parquet support for analytics-oriented pipelines. You can query CSV, JSON, Protobuf, and Parquet records with the same ENTRY() table function and export selected results as Parquet files.

Lifecycle Policies​

Lifecycle policies are background tasks that automatically manage old records in a bucket. A policy can delete records or compress persisted blocks with zstd, and it can be scoped by entry names, record age, and, for delete policies, conditional expressions.

This is useful when data has different operational and historical value over time. For example, recent telemetry can stay uncompressed for frequent access, older telemetry can be compressed to reduce storage usage, and very old data can be deleted later by a separate policy.

Policies support three modes: enabled, dry_run, and disabled. Every run writes diagnostics to the $system/lifecycle system entry, so operators can inspect policy behavior before and after enabling automatic actions.

Here is a provisioning example for deleting temperature records older than 30 days from selected sensor entries:

version: "3"
services:
  reductstore:
    image: reduct/store:v1.20.0
    ports:
      - "8383:8383"
    volumes:
      - ./data:/data
    environment:
      RS_API_TOKEN: my-api-token
      RS_BUCKET_1_NAME: telemetry
      RS_LIFECYCLE_1_NAME: purge-sensors-30d
      RS_LIFECYCLE_1_BUCKET: telemetry
      RS_LIFECYCLE_1_TYPE: delete
      RS_LIFECYCLE_1_OLDER_THAN: 30d
      RS_LIFECYCLE_1_INTERVAL: 10m
      RS_LIFECYCLE_1_ENTRIES: "sensor-1,sensor-2"
      RS_LIFECYCLE_1_WHEN: |
        {
          "&sensor_type": { "$eq": "temperature" }
        }

For the full configuration reference and SDK examples, see the Lifecycle Policies guide.

Protobuf Support in the Ecosystem​

v1.20 connects Protobuf ingestion, storage, and querying through a shared schema-attachment convention. This makes binary telemetry easier to keep compact at ingest time while preserving the metadata required for downstream processing.

ReductSelect Extension​

ReductSelect processes Protobuf records with content types such as application/protobuf, application/x-protobuf, and application/vnd.google.protobuf. By default, it reads the message descriptor from the entry's $schema attachment and exposes decoded fields to SQL.

The attachment stores the payload encoding, message type, and base64-encoded FileDescriptorSet:

{
  "encoding": "protobuf",
  "schema_name": "pkg.SensorReading",
  "schema": "<base64-encoded FileDescriptorSet>"
}

With the schema attached to the entry, a ReductSelect query can stay small:

{
  "protobuf": {},
  "sql": "SELECT device_id, temperature FROM ENTRY() WHERE temperature > 20"
}

For lightweight cases, ReductSelect can also extract top-level scalar fields directly by Protobuf field number without a full descriptor. See the ReductSelect Protobuf documentation for both modes.

ReductBridge​

ReductBridge's MQTT input can now ingest Protobuf payloads and extract labels from message fields. In schema mode, the bridge stores the descriptor as a $schema attachment so the same data can later be decoded by ReductSelect or exported by ReductROS.

[[inputs.mqtt.main.topics]]
name = "factory/electrical/+/power"
entry_name = "power"
content_type = "application/protobuf"
schema_path = "./factory.desc"
schema_name = "factory.PowerReading"
labels = [
  { field = "device_id", label = "meter_id" },
  { field = "panel", label = "panel" }
]

If a pipeline only needs a few top-level scalar fields, ReductBridge can also extract labels from raw wire fields without a schema. For details, see the MQTT input documentation.

SQL and Parquet Support in ReductSelect​

ReductSelect is now better suited for structured data processing directly inside ReductStore queries. The extension exposes each queried record through the ENTRY() table function, allowing SQL selection, filtering, aggregation, and computed output columns over supported structured payloads.

Parquet records are processed when the record content type is application/vnd.apache.parquet or when the query explicitly enables Parquet parsing. The same extension can export query results as Parquet, with batching controlled by row count or duration.

{
  "parquet": {},
  "sql": "SELECT name, value FROM ENTRY() WHERE value > 10",
  "export": {
    "format": "parquet",
    "rows": 100
  }
}

SQL result columns can also be written back as computed labels, which is useful when a pipeline extracts metadata from payloads and wants later queries or replication rules to use it as regular ReductStore labels.

For query parameters and examples, see the ReductSelect extension documentation.

What's Next​

The next area of work is making ReductStore's integration ecosystem easier to operate in production. We plan to keep improving ReductBridge pipelines, schema-aware tooling, and examples that connect MQTT, ROS, Zenoh, and analytics workflows around a shared storage layer.

We are also continuing to improve the extension model so data can be transformed, queried, and exported closer to where it is stored, without forcing every workflow to move raw payloads into separate processing systems first.

Compatibility and Migration​

This release is expected to be compatible with existing v1.19 deployments. Lifecycle policies are opt-in, and ReductSelect and ReductBridge additions only affect deployments that enable those workflows.

The $audit bucket is deprecated in v1.20. Audit events are now written to the $system/audit entry, together with other system events. Persistence of system events is enabled by default and can be disabled with RS_SYSTEM_EVENTS_ENABLED=false; quota and replication-related options are documented in the System Event Settings section.

If you have questions or feedback, join the ReductStore Community forum.

Thanks for using ReductStore!

This is a predictable consequence of crossing a scaling threshold that almost every AMR operator hits.

The Inflection Point​

A single AMR collecting camera frames, LiDAR scans, and control logs generates somewhere between 30 and 100 GB per shift, depending on sensor configuration and logging frequency. At ten robots, that's manageable. A custom script, a shared drive, maybe an S3 bucket — it works well enough that nobody questions it.

At one hundred robots, you're collecting 3–10 TB per shift. The custom script doesn't scale. The shared drive is full. The S3 bucket costs more than expected. And none of these tools were designed for the actual shape of the data: binary sensor payloads that are too large for a time-series database and too time-sensitive for a generic object store.

The failures start slowly and accumulate over time.

Some teams sidestep this entirely by processing at the edge by extracting metrics, downsampling, discarding the raw frames before they ever leave the robot. That keeps storage small and transfer costs low. The tradeoff is irreversibility: once the raw record is gone, it is gone. When a model needs retraining on a different feature representation, when an incident investigation requires the original sensor data, or when a regulator asks for the record that predates the processed output, there is nothing to retrieve.

The infrastructure problem described below is the cost of doing something valuable: keeping the raw data that makes your fleet improvable over time.

Four Problems That Grow With the Fleet​

Storage overflow at the edge. Most AMR deployments use some form of local storage on the robot itself to buffer data before it transfers to a central server or cloud. When that buffer fills and there is no intelligent retention policy, robots start dropping records. The newest data overwrites the oldest without any selection logic. By the time an incident occurs and someone asks for the sensor data from the previous three shifts, it is gone.

Cloud transfer cost explosion. Object storage is cheap per GB. What is not cheap is the API call volume that high-frequency sensor data generates. A fleet writing thousands of small records per second to S3 can generate millions of PUT requests per day. At standard pricing, that cost compounds fast and it scales with fleet size in a way that storage cost alone does not.

Model drift without a feedback loop. Retraining a perception model requires labelled examples of edge cases: near-misses, unusual lighting, unexpected obstacles, novel environments. Those examples only exist in operational sensor data. If the data pipeline doesn't preserve the right records — specifically, the records around anomalous events — retraining becomes either impossible or biased toward common cases. The model drifts. Failure rates climb slowly enough that the connection to data quality is rarely made until much later.

Audit gaps when incidents happen. Regulatory and insurance requirements for autonomous systems are tightening, particularly in the EU under the Cyber Resilience Act and related frameworks. An incident investigation requires a continuous, tamper-evident record: what the robot saw, what it decided, what happened next. If records were dropped because local storage filled, or if the audit trail has gaps because replication wasn't reliable, that investigation stalls — or produces a conclusion that is hard to defend.

What the Infrastructure Actually Needs to Do​

A requirements framework for AMR data infrastructure has three layers, and most off-the-shelf tools cover at most one of them.

Edge durability with intelligent retention. The robot's local storage needs a FIFO-by-volume quota: when the disk fills, the oldest records roll off automatically, keeping the system stable regardless of how long a mission runs or how long connectivity is interrupted. Critically, high-priority records — fault windows, flagged events, anything tagged for retraining — need to be preserved ahead of routine telemetry. This is not what generic object stores do.

Selective replication, not full sync. Sending everything to the cloud is how you generate the cost explosion described above. The alternative is conditional replication: a filter that runs on labels attached to records at ingest time, and sends only the subset that matches. Fault events go through. Routine navigation frames from uneventful shifts stay on the edge until the FIFO rolls them off. The bandwidth bill reflects the operational value of the data, not its raw volume.

Time-indexed auditability across tiers. Every record needs a precise timestamp and an immutable chain from the sensor to wherever it ends up. When an incident investigation asks "what did robot 47 see at 14:23:07 on Thursday," the answer should be retrievable in seconds — not reconstructed from fragmented logs across three storage systems.

A Self-Assessment for Operations Teams​

Before evaluating vendors, run these checks against your current setup:

• If a robot loses connectivity for 8 hours, does it continue logging without dropping records? What happens when it reconnects?
• Can you retrieve the sensor data from a specific 30-second window on a specific robot from 6 weeks ago?
• Do you know which records from last month's fleet operation are currently stored in the cloud, and why those ones specifically?
• If a model retraining job requires 500 labelled fault examples from the past 90 days, how long does it take to locate and export them?
• In the event of a fleet incident, what is your maximum gap in the sensor record?

If any of these questions produces a slow or uncertain answer, the infrastructure has a gap. The question is whether that gap is large enough to matter at your current fleet size and whether it will still be acceptable when the fleet doubles again.

If you want to walk through these questions against your current setup, we offer a free architecture review.

What Good Looks Like​

A well-designed AMR data stack has the same shape at 10 robots as at 1,000, because the architecture scales rather than requiring a rebuild at each inflection point.

On each robot: a lightweight storage engine with FIFO-by-volume quotas, label-based filtering at ingest, and continuous local buffering. When the robot is connected, it replicates only the records that match a pre-configured conditional query — fault windows, flagged events, training candidates. When it's not connected, it keeps logging.

At the fleet server tier: aggregated data from all robots, with the same query interface. Engineers can pull a specific time window from a specific robot without knowing which physical server holds it.

In the cloud: a cost-efficient long-term store using batched writes to object storage, with lifecycle-aware tiering that moves infrequently accessed data to cold storage automatically. The same query interface applies.

Tools like ReductStore are built specifically for this architecture: binary payloads stored with microsecond-precision timestamps and labels, replicated conditionally across tiers, queryable by time window and label combination from any tier. The same model from edge to cloud, with no translation layer between them.

The Board Framing​

If you need to explain this problem upward, the frame is straightforward: the data your fleet generates is the feedstock for every improvement you will make to it — model updates, route optimization, incident response, regulatory compliance. Infrastructure that drops records, transfers data indiscriminately, or lacks a continuous audit trail is eroding the value of the fleet as it scales, silently and in advance of any visible failure.

The cost of fixing it before the fleet grows is a fraction of the cost of rebuilding the pipeline after it breaks. And rebuilding under pressure, after an incident, is when mistakes get made.

MQTT is a common choice for the communication stack in IoT and robotics applications because it is lightweight and easy to integrate. But many of those applications do not send only small JSON telemetry messages. They also publish JPEG frames, vibration waveforms, audio clips, protobuf messages, and other binary payloads that need to be stored and queried later.

This is where a regular MQTT broker or a traditional time-series database starts to fall short. Brokers are designed for message delivery, not long-term historical storage, and many databases either expect structured numeric fields or make it hard to keep large binary records tied to accurate timestamps.

In this tutorial, we will use ReductBridge to subscribe to MQTT topics and write the raw binary payloads into ReductStore with a time index. This lets you keep camera frames and sensor payloads as they are, while still querying them by time range, labels, and entry name for replay, debugging, and offline analysis.

Why Existing Stacks Fail for Binary MQTT Payloads​

Most MQTT storage solutions work well for small structured telemetry: temperature, pressure, status values, or JSON events. They start to break down when the same system needs to store camera frames, audio chunks, vibration buffers, protobuf messages, or other binary payloads. These records are larger, often opaque to the storage layer, and still need to be queried later by time, device, topic, and process metadata.

Traditional SQL databases can store binary objects, but they are not a good default for this workload. Large blobs quickly make tables heavy, backups slow, and queries expensive. They also do not give you a natural way to keep an append-only history of records by topic and timestamp. You can build this model yourself, but then you need to manage schema, indexes, retention, and cleanup logic on top of the database.

Time-series databases are closer to the problem, but they are usually designed for metrics. They work best when each record is a small set of numeric fields with tags. Binary MQTT payloads do not fit that shape. If you store only a file path in the time-series database and put the binary data somewhere else, you now have two systems that must stay in sync.

Object storage and local files solve the blob part, but not the history part. They are good at storing large objects, but they do not provide a proper time index for range queries. Putting timestamps into file names or object keys works for a demo, but it becomes fragile when you need to ask practical questions such as: give me all frames from this camera between two timestamps, only for this production line, and only when this label was present.

Disk usage is another problem. Binary data can fill a disk very quickly, and the rate is not always predictable. A camera may switch to a higher resolution, a sensor may start publishing faster, or a fault may produce more events than usual. Time-series databases usually have retention based on time, but that is not always enough when the data volume can spike. That can lead to out-of-disk errors, data loss, and downtime.

FIFO (first-in-first-out) quota is a better fit for this workload because it removes the oldest data when the disk budget is reached and guarantees that the newest data is always accepted and stored. This way, you can keep a rolling history of the most recent data without worrying about disk management

Data replication also needs a different model. For MQTT binary data at the edge, copying the whole database or mirroring all files is usually too expensive. Most teams need to stream only new records that match metadata: a site, a device, a topic, a camera, an event type, or an error label. That way, important data can be moved to another node or to the cloud, while raw high-volume data can stay close to the device.

So the storage layer needs to handle three things together: raw binary payloads, a time index, and metadata for filtering. Without all three, the system becomes a collection of workarounds instead of a queryable history of MQTT data.

The Timestamp Problem: MQTT Does Not Timestamp Messages​

MQTT is good at moving messages between devices and services, but it does not define when the data inside a message was created. A broker may know when it received a publish packet, and a subscriber may know when it processed the message. For historical storage, however, you usually need a different timestamp: when the camera captured the frame, when the waveform was sampled, or when the sensor event happened.

If you store MQTT data by arrival time, the history can look correct but still be shifted from reality. Network delay, broker queues, reconnects, QoS retries, batching, and slow consumers can all move the stored timestamp away from the capture time. This may be acceptable for simple monitoring, but it becomes a problem when you need to replay data or compare several streams.

For example, you may need to match a camera frame with a PLC state, a vibration buffer with a machine alarm, or an audio clip with a robot event. If each stream uses a different idea of time, the correlation becomes unreliable.

For binary MQTT payloads, the timestamp should usually come from the producer and stay outside the payload. You should not need to decode a JPEG frame, waveform buffer, or protobuf message only to find its time. It is better to pass the timestamp as metadata, for example in an MQTT user property, a topic convention, or another field that can be mapped to the stored record.

When the producer cannot provide a timestamp, using ingestion time is still useful. But it should be treated as the time when the bridge received the message, not necessarily the time when the data was captured.

Architecture Overview​

In this tutorial, we use MQTT as the transport layer and ReductStore as the historical storage layer. ReductBridge connects them together and converts MQTT messages into time-indexed records.

The data flow looks like this:

Several producers can publish different types of data to the same broker. A camera may publish JPEG frames, a vibration sensor may publish waveform buffers, and an automation system may publish the current machine state or events.

We use MQTT v5 because it can attach user properties to every message. The binary payload stays unchanged, while metadata travels next to it. For example, a camera frame can include properties such as device_id, line, content_type, and event_time.

The event_time property contains the original capture timestamp as Unix time with microsecond resolution. ReductBridge reads this property and uses it as the timestamp of the ReductStore record. Other MQTT user properties are mapped to ReductStore labels, so the data can be filtered later without decoding the binary payload.

ReductBridge can also use metadata from one topic to enrich data from another topic. For example, an automation topic may publish the current machine state. The bridge can remember this label and apply it to camera frames or sensor buffers, so later you can query data by both time and process state.

ReductStore stores the resulting records as append-only time-indexed data. It keeps the original binary payload, stores labels for filtering, applies FIFO quota rules to control disk usage, and can optionally replicate selected data to another storage node.

Install ReductStore and ReductBridge​

This tutorial assumes that your MQTT v5 broker is already running and that your producers already publish binary payloads to it. The broker can be Mosquitto, EMQX, HiveMQ, or another MQTT v5-compatible broker. We only need to add two components: ReductStore for storage and ReductBridge for moving MQTT messages into the store.

The quickest way to start ReductStore is with Docker:

mkdir -p ./reduct-data
sudo chown -R 10001:10001 ./reduct-data

docker run -d --name reductstore \
  -p 8383:8383 \
  -e RS_API_TOKEN="my-token" \
  -v "$PWD/reduct-data:/data" \
  reduct/store:latest

ReductStore will listen on http://127.0.0.1:8383 and store data in ./reduct-data. The RS_API_TOKEN value is used later by ReductBridge and reduct-cli.

For ReductBridge, use the iot build because it includes the MQTT input:

docker pull reduct/bridge:latest-iot

After creating the configuration file in the next section, run the bridge with:

docker run --rm --network host \
  -v "$PWD/bridge.toml:/etc/reduct-bridge/config.toml:ro" \
  reduct/bridge:latest-iot reduct-bridge /etc/reduct-bridge/config.toml

The --network host option keeps the example short because the container can reach 127.0.0.1:8383 and a broker running on the host. In production, you can run ReductStore and ReductBridge as systemd services or in your own container stack.

If you do not use Docker, download a ReductStore binary from the ReductStore releases and the iot ReductBridge build from the ReductBridge releases. Then run them with the same data path, API token, and config file:

RS_DATA_PATH=./reduct-data RS_API_TOKEN="my-token" ./reductstore
reduct-bridge ./bridge.toml

Configure ReductBridge for Binary Payloads​

Create bridge.toml with the MQTT input, a pipeline, and a ReductStore remote:

# MQTT input named "factory". It connects to the existing broker and
# subscribes to the topics declared below.
[inputs.mqtt.factory]
broker = "mqtt://127.0.0.1:1883"
client_id = "reduct-bridge-mqtt-binary"
version = "v5"
qos = 1

# Camera frames are binary JPEG payloads. The MQTT topic becomes the source,
# while entry_name controls where records are stored in ReductStore.
[[inputs.mqtt.factory.topics]]
name = "factory/camera/+"
entry_name = "camera_frames"
content_type = "image/jpeg"
timestamp = { property = "event_time", format = "unix_us" }
labels = [
  { property = "device_id", label = "device_id" },
  { property = "line", label = "line" },
  { static = { source = "mqtt", kind = "camera" } }
]

# Vibration messages are also stored as raw bytes. Only MQTT v5 properties are
# read for timestamp and labels, so the waveform payload is not decoded.
[[inputs.mqtt.factory.topics]]
name = "factory/vibration/+"
entry_name = "vibration"
content_type = "application/octet-stream"
timestamp = { property = "event_time", format = "unix_us" }
labels = [
  { property = "device_id", label = "device_id" },
  { property = "line", label = "line" },
  { static = { source = "mqtt", kind = "vibration" } }
]

# Automation state is a lightweight JSON stream. The state label is extracted
# from the JSON payload and can be copied to other entries by the pipeline.
[[inputs.mqtt.factory.topics]]
name = "factory/automation/state"
entry_name = "automation_state"
content_type = "application/json"
timestamp = { property = "event_time", format = "unix_us" }
labels = [
  { property = "line", label = "line" },
  { field = "state", label = "state" },
  { static = { source = "mqtt", kind = "automation" } }
]

# Pipeline routes records from the MQTT input to the ReductStore remote.
# The label rule copies the latest JSON-derived automation state to records.
[pipelines.mqtt_to_store]
remote = "local"
inputs = ["factory"]
labels = [
  { from = "automation_state", labels = ["state"], to = "*" },
]

# ReductStore remote. All records are written to this bucket, and the prefix is
# prepended to each entry name, for example mqtt/camera_frames.
[remotes.reduct.local]
url = "http://127.0.0.1:8383"
token_api = "my-token"
bucket = "mqtt-binary"
prefix = "mqtt/"

# Create the bucket on startup with a FIFO quota so the edge node keeps the
# newest data when the storage budget is reached.
[remotes.reduct.local.create_bucket]
quota_type = "FIFO"
quota_size = "20GB"

The camera and vibration topics are treated as binary streams. ReductBridge does not decode their payloads. It writes the MQTT payload bytes as the ReductStore record body and uses metadata from MQTT v5 properties.

The important part is the timestamp line:

timestamp = { property = "event_time", format = "unix_us" }

This tells ReductBridge to read the MQTT v5 user property event_time and use it as the ReductStore record timestamp. The value must be Unix time in microseconds. Other supported formats include unix_s, unix_ms, unix_ns, iso8601, and ros_stamp. If the property is missing or cannot be parsed, ReductBridge falls back to ingest time.

The labels arrays copy metadata into ReductStore labels. For binary topics, labels come from MQTT v5 properties so the payload remains untouched. For the automation topic, state comes from the JSON payload field:

{ field = "state", label = "state" }

For example, a camera publisher can send these user properties with each JPEG frame:

event_time=1780582163840000
device_id=cam-01
line=line-a

ReductBridge stores that frame in the mqtt/camera_frames entry because the remote adds the mqtt/ prefix. It also stores labels such as device_id=cam-01, line=line-a, and kind=camera. The record content type is handled separately by the topic-level content_type = "image/jpeg" setting.

The pipeline-level labels rules show the enrichment pattern from the architecture section. When the automation_state entry receives a state label extracted from JSON, the pipeline can copy the latest value to later records. That lets you query camera frames not only by time and device, but also by process state:

labels = [
  { from = "automation_state", labels = ["state"], to = "*" },
]

The remote section creates the mqtt-binary bucket automatically with a FIFO quota. With this setting, ReductStore keeps accepting new records and removes the oldest data when the bucket reaches 20GB. Adjust this value to the disk budget of your edge node:

[remotes.reduct.local.create_bucket]
quota_type = "FIFO"
quota_size = "20GB"

Query Data by Time Range​

After ReductBridge is running and messages are arriving, use the Python SDK to verify the entries and query records by time range and labels:

import asyncio
from pathlib import Path

from reduct import Client


async def main():
    async with Client("http://127.0.0.1:8383", api_token="my-token") as client:
        bucket = await client.get_bucket("mqtt-binary")

        output_dir = Path("records")
        output_dir.mkdir(exist_ok=True)

        async for record in bucket.query(
            "mqtt/*",
            start="2026-06-04T10:00:00Z",
            stop="2026-06-04T10:05:00Z",
            when={
                "&state": {"$eq": "running"},
            },
        ):
            payload = await record.read_all()
            entry = record.entry.replace("/", "_")
            extension = {
                "image/jpeg": ".jpg",
                "application/json": ".json",
            }.get(record.content_type, ".bin")
            filename = output_dir / f"{entry}-{record.timestamp}{extension}"
            with open(filename, "wb") as output:
                output.write(payload)
            print(filename, record.content_type, record.labels)


if __name__ == "__main__":
    asyncio.run(main())

You should see entries such as mqtt/camera_frames, mqtt/vibration, and mqtt/automation_state. The timestamps should match the producer-provided event_time property when it is present. The query uses the mqtt/* wildcard to select all entries, then filters records by time range and the copied state=running label. Each payload is saved as-is, with the file extension selected from the record content type where possible.

Next Steps​

This tutorial focused on the bridge configuration because the MQTT broker and data sources were assumed to exist. From here, you can adapt the same pattern to your own topics, labels, and retention policy.

• Learn how to store MQTT data with the Python MQTT tutorial.
• See a Rust implementation in the Rust MQTT tutorial.
• Compare storage choices in How to Choose the Right MQTT Database.
• Review the ReductBridge documentation and the MQTT input reference.
• Read about ReductStore data querying, bucket quotas, and data replication.

I hope you found this article helpful. If you have any questions or feedback, don't hesitate to use the ReductStore Community forum.

Thanks for reading!

The EU Cyber Resilience Act (Regulation (EU) 2024/2847) is the “GDPR for connected products.” It entered into force on 10 December 2024, with critical milestones approaching fast:

• 11 September 2026: Mandatory reporting of actively exploited vulnerabilities and severe incidents (24-hour early warning, 72-hour full notification).
• 11 December 2027: Full compliance — Security by Design, lifecycle support (minimum 5 years), technical documentation, and CE marking.

For robotics fleets (AMRs, cobots, autonomous systems, and ROS 2-based platforms) the stakes are particularly high. These systems are “products with digital elements” (often Class II or critical), generating massive multimodal data streams (camera feeds, LiDAR, IMU, logs, ROS bags) under real production constraints: intermittent connectivity, edge hardware limits, and high physical safety risks.

Generic storage solutions force painful trade-offs: either accept data loss and compliance gaps, or accept exploding costs and slow performance. ReductStore eliminates this trade-off.

Why Common Storage Solutions Fall Short for CRA in Production Robotics​

Most robotics teams still rely on a patchwork of rosbag2, MinIO/S3, InfluxDB, or TimescaleDB. While these tools can technically meet some basic CRA requirements with significant custom development, they create substantial challenges when applied to real-world ROS production fleets:

• rosbag2 is excellent for local single-robot recording, but it is not designed for fleet-scale edge-to-cloud architectures, selective replication, or long-term auditability. It lacks native mechanisms for data minimisation, granular access control, and automated compliance reporting.
• MinIO/S3 and general-purpose object stores provide scalable blob storage, but they offer no built-in time-series indexing, label-based filtering, or efficient selective replication. Achieving CRA-compliant audit trails and data minimisation usually requires complex additional layers and custom glue code.
• InfluxDB and TimescaleDB are strong for structured metrics, but they have limited native support for large binary objects (images, LiDAR, ROS bags) and are not optimized for high-throughput multimodal ingestion or production-grade edge-to-cloud workflows under intermittent connectivity.

In practice, these solutions force teams to invest heavily in custom engineering to achieve CRA compliance — often at the expense of performance, simplicity, and cost efficiency. Most importantly, this custom-built approach increases the risk of documentation gaps during a CRA conformity assessment.

The result? Many operators risk non-compliance, higher insurance premiums, or blocked EU market access after 2027.

ReductStore's CRA-Native Architecture for ROS Fleets​

ReductStore is purpose-built as a time-series blob storage engine for multimodal robotics data. It turns compliance from a burden into a competitive advantage.

Key CRA-Aligned Features:

1. Token-Based Authentication & Granular Access Control Every device, service, or user receives scoped tokens. Access is logged at Bucket/Entry/Record level with timestamps. Perfect for proving “who accessed what data and when” — a core CRA requirement.

2. Label-Based Integrity Protection Attach cryptographic labels or hashes to records. The engine supports automatic corruption detection and reporting. Write-once-append-only semantics ensure data cannot be silently altered.

3. Selective Edge-to-Cloud Replication with Data Minimisation Replicate only records matching specific labels or conditions (e.g., “security-critical”, “anomaly detected”, or specific ROS topics). This dramatically reduces attack surface and cloud costs while meeting CRA’s data-minimisation principle.

4. Encryption at Rest and in Transit ReductStore enables strong data confidentiality through multiple proven approaches. Sensitive payloads can be encrypted client-side before ingestion and server-side (e.g., via S3 backend options), while preserving its high-performance batching architecture. All network communication is secured via HTTPS/TLS. The system is fully compatible with ROS 2 Security (SROS2). Native built-in client- and server-side encryption is on the near-term roadmap.

5. Extended Audit Logging & Grafana Integration Detailed logs of all read/write/replication events. Visualize compliance status directly in Grafana dashboards — ready for technical documentation and conformity assessment.

6. SBOM Export & Versioned Updates Generate Software Bill of Materials for ReductStore itself. Security updates are delivered via dedicated channels without disrupting production data flows.

These capabilities are already running in 100+ production deployments managing over 1 PB of multimodal robotics data.

Real-World Results from Production Fleets​

Operators using ReductStore report:

• “We have forgotten about disk overrun problems on our edge devices.” — Ingo Kaiser, PANDA GmbH
• “ReductStore handles terabytes of unstructured data in a production environment.” — Michael Welsch, Metric Space UG
• “We can stop worrying about data collection and focus entirely on building our robots and autonomous layers.” — Victor Massagué Respall, INSAION

In one fleet deployment, selective replication reduced cloud data volume by approximately 85% while maintaining full auditability for CRA preparedness.

Free CRA Checklist for Robotics Operators (Focus: Data Integrity & Storage Layer)​

Download our practical checklist covering:

• Immediate actions before September 2026 reporting deadline
• Mapping ReductStore features to specific CRA Annex I requirements
• Step-by-step migration guide for existing ROS fleets
• Risk assessment template for multimodal robotics data

Download CRA Checklist for ROS Fleets (PDF)

Ready to Make Your Data Streaming And Storage CRA-Ready Without Compromising Performance?​

The September 2026 reporting deadline is approaching fast. Don’t wait until conformity assessment bodies are fully operational in mid-2026.

Book a 20-minute production fleet audit with our team. We will:

• Analyze your current data pipeline against CRA requirements
• Run a live benchmark with your own ROS data
• Show exactly how ReductStore delivers compliance + 10x performance at lower cost

→ Schedule Your Free Fleet Audit

Next up in this series: Physical AI Training Data at Scale: Why Foundation Models Need Specialized Edge-to-Cloud Storage

Robots generate massive amounts of data, and managing it well is harder than it looks. Storage fills up fast, cloud transfer gets expensive, and real time ingestion is unforgiving when you're running cameras and sensors at high frequency.

This article covers practical strategies for handling robotic data, introduces ReductStore, and walks through a hands on example. Along the way, we cover native ROS integration, Grafana dashboards, MCAP export for Foxglove, a Zenoh API, and native S3 and Azure backends. We also compare ReductStore against Rosbag and MongoDB so you can pick the right tool for each part of your stack.

Challenges in Robotic Data Management​

Robots operate in dynamic environments and continuously produce large volumes of data. The core challenges engineers run into are:

• High frequency, real time requirements: A drone navigating a city must process camera and sensor data in milliseconds. Storage solutions need to keep up with these streams and make data accessible fast enough for real time decision making.
• Limited on device storage: Most robots can't store everything they generate. Size, weight, and power constraints limit local capacity, so good data management strategies are essential.
• High data volume: An autonomous vehicle can produce up to 5 terabytes per hour across camera feeds, LiDAR, radar, GPS, and sensors. Most databases weren't designed around that kind of throughput.
• Cloud storage costs: Pushing everything to the cloud isn't practical. Providers charge per gigabyte stored and transferred, and with robots producing terabytes, the bill adds up fast. What you send to the cloud is a decision worth making deliberately.
• Data reduction complexity: Filtering out irrelevant data without losing important information is tricky. Without a clear strategy, you either waste storage or throw away data you later need.

General Strategies for Managing Robotic Data​

A few approaches that work well in practice:

• Use a time series object store: Systems like ReductStore are designed specifically for high frequency, timestamped binary data. They offer fast writes, efficient querying, and manageable costs at scale.
• Balance edge and cloud storage: Keep recent, critical data on the robot for fast access. Move older or lower priority data to the cloud for long term storage and analysis. Splitting storage this way cuts costs without adding much latency to what matters.
• Use volume based retention policies: Rather than deleting data based on age, use a FIFO (first in, first out) quota. Data is only removed when the storage limit is reached, which avoids unnecessary deletion during downtime or low activity periods.
• Compress where it makes sense: H.265 for video, JPEG for images. The right format can cut storage by an order of magnitude without losing anything you actually need.
• Prioritize what to keep: Not all sensor data has equal value. Define what's critical upfront and discard the rest early in the pipeline.

ReductStore: Built for Robotic Data​

ReductStore is a time series database built specifically for unstructured, binary data. It's designed to handle high frequency sensor streams from autonomous vehicles, drones, industrial robots, and IoT. It stores data with timestamps and labels, supports fast real time ingestion, and comes with batching, filtering, and edge to cloud replication built in.

Here's what's available today.

ROS Integration with reduct-bridge​

reduct-bridge connects live ROS 1 and ROS 2 systems directly to ReductStore. You configure it with a simple TOML file that defines your ROS topic inputs, pipelines, and the ReductStore destination. It subscribes to topics and stores each message as an individual record.

It also writes a $ros attachment to each entry with the message schema, topic name, and encoding — that metadata is what powers MCAP export and Grafana visualization, both covered below. If your stack is already on ROS, setup takes minutes.

Cloud Storage: Native S3 and Azure​

ReductStore supports native cloud storage backends for both Amazon S3 and Azure Blob Storage, no FUSE drivers needed. It uses a local cache for hot data and the cloud bucket for long term retention.

For records around 100KB (e.g. JPEG images), ReductStore can be >10x faster than traditional time series object stores at a fraction of the cost.

Visualize ROS Data in Grafana​

ReductStore has a Grafana integration through the ReductStore data source plugin. You can query ROS 2 messages directly in Grafana thanks to the ReductROS extension. The extension decodes binary CDR messages into JSON on the fly.

You can monitor sensor streams live, compare data across multiple robots, and set up alerts when metrics drift.

Export to MCAP for Foxglove​

You can export raw ROS messages stored in ReductStore directly to MCAP and open them in Foxglove. This is powered by the ReductROS extension as well, which reconstructs valid MCAP files on demand.

When records are ingested via reduct-bridge, the $ros attachment carries the schema and topic information. The extension uses this to reconstruct valid MCAP files on demand. You can cover a full time range across multiple topics in a single query, and split by duration or file size for long recordings.

Zenoh Native API​

ReductStore now includes a Zenoh native API alongside the existing HTTP API. Zenoh is a pub/sub protocol designed for robotics and distributed systems, and it's widely used in next generation ROS 2 deployments.

When enabled, ReductStore opens a Zenoh session with a subscriber for writes and a queryable for reads, both running in parallel with HTTP and sharing the same data. Zenoh keys become entry names, encodings map to content types, and attachments map to labels. Time range queries, conditional filters, and label lookups all work the same way they do over HTTP.

It's a natural fit if your robot stack already speaks Zenoh, or if you want to skip HTTP overhead on high frequency ingestion paths.

Query Language and Batching​

ReductStore uses a JSON based query language that supports filtering, aggregation, and time range operations. You can query data from a specific robot within a time window, compare sensor streams across robots in parallel, or filter by any label.

Retrieval is optimized through batching: multiple records are grouped into a single response based on a time range, which cuts down on request overhead and improves throughput. SDKs are available for Python, C++, JavaScript, Go, and Rust. More in the querying guide.

Replication and Edge to Cloud Replication​

ReductStore replicates at the bucket level based on conditions. That means you can choose to send only high priority sensor data to the cloud based on rules, labels, or events. Replication is incremental, so only new data is transferred. Labels stored alongside records let you define fine grained rules based on content, for example, only replicate records flagged as anomalies or sampled at 1 in 10 seconds.

Retention Strategies​

Volume based retention follows the FIFO principle: data is only deleted when storage is full, making room for new records. This is different from time based retention, which deletes data after a fixed age regardless of whether storage is full. After an outage, a time based policy might delete data the system never had the chance to process. Volume based retention avoids that.

Example Applications​

Comparing ReductStore with Rosbag and MongoDB​

Three tools come up most often when robotics teams think about data storage: Rosbag and MCAP, MongoDB, and ReductStore. They're built for different things.

Rosbag and MCAP are the standard for recording ROS sessions. They're great for capturing a snapshot during a test run. But they're file formats, not databases. Querying across many recordings requires custom scripts, there's no built in content indexing, and managing thousands of bag files quickly becomes its own problem. Use them for short recordings and local playback, not long term storage or fleet wide analysis.

MongoDB is flexible and works well for structured metadata, labels, and event logs. But it wasn't built for large binary payloads. For blob data, it relies on GridFS, which adds complexity and hurts performance. It also uses time based retention, so data can be deleted during idle periods even when storage isn't full.

ReductStore is designed specifically for the data robots actually produce: large, binary, timestamped records at high frequency. It connects to ROS via reduct-bridge, exports to MCAP for Foxglove, plugs into Grafana, and replicates selectively to S3 or Azure.

In practice: use Rosbag or MCAP for short test recordings, MongoDB for structured metadata and event logs, and ReductStore for raw sensor data that needs to be stored at scale, queried efficiently, and managed over time.

For a deeper look: MongoDB vs ReductStore: Choosing the Right Database for Robotics Applications.

Hands-On Example: Storing Robotic Data in ReductStore​

Let's walk through a practical example of storing and querying robotic data with ReductStore. We'll use trajectory data (coordinates, speed, orientation) to keep things simple, but the same approach works for any sensor stream.

You'll need Python 3.8+ installed.

Setting Up ReductStore​

Create a folder and add a docker-compose.yaml file:

version: "3.8"

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    volumes:
      - data:/data
    environment:
      - RS_API_TOKEN=my-token

volumes:
  data:
    driver: local

Start it with:

docker compose up -d

ReductStore will be available at http://127.0.0.1:8383. Check the container is running with docker ps, then install the Python libraries:

pip install reduct-py numpy

Store and Query Data​

Create a Bucket​

async def create_trajectory_bucket():
 async with Client("http://localhost:8383", api_token="my-token") as client:
  settings = BucketSettings(
	 quota_type=QuotaType.FIFO,
	 quota_size=1000_000_000,
	)
  await client.create_bucket("trajectory_data", settings, exist_ok=True)

The bucket uses a FIFO quota of 1 GB. Old data is only deleted when the limit is reached.

Generate Trajectory Data​

async def generate_trajectory_data(frequency: int = 10, duration: int = 1):
 interval = 1 / frequency
 start_time = datetime.now()

  for i in range(frequency * duration):
    time_step = i * interval
    x = np.sin(2 * np.pi * time_step) + 0.2 * np.random.randn()
    y = np.cos(2 * np.pi * time_step) + 0.2 * np.random.randn()
    yaw = np.degrees(np.arctan2(y, x)) + np.random.uniform(-5, 5)
    speed = abs(np.sin(2 * np.pi * time_step)) + 0.1 * np.random.randn()
    timestamp = start_time + timedelta(seconds=time_step)

    yield {
      "timestamp": timestamp.isoformat(),
      "position": {"x": round(x, 2), "y": round(y, 2)},
      "orientation": {"yaw": round(yaw, 2)},
      "speed": round(speed, 2),
    }
    await asyncio.sleep(interval)

This simulates a robot moving in 2D at 10 Hz for 1 second. X and y are position coordinates, yaw is orientation, and speed is derived from position changes.

Calculate Metrics​

def calculate_trajectory_metrics(trajectory: list) -> tuple:
  positions = np.array([[point["position"]["x"], point["position"]["y"]] for   point in trajectory])
  speeds = np.array([point["speed"] for point in trajectory])

  deltas = np.diff(positions, axis=0)
  distances = np.sqrt(np.sum(deltas**2, axis=1))
  total_distance = np.sum(distances)

  average_speed = np.mean(speeds)

  return total_distance, average_speed

Write to ReductStore​

async def store_trajectory_data():
  trajectory_data = []
  async for data_point in generate_trajectory_data(frequency=10, duration=1):
    trajectory_data.append(data_point)

  total_distance, average_speed = calculate_trajectory_metrics(trajectory_data)

  labels = {
    "total_distance": total_distance,
    "average_speed": average_speed
  }

  packed_data = pack_trajectory_data(trajectory_data)

  timestamp = datetime.now()

  async with Client("http://localhost:8383", api_token="my-token") as client:
    bucket = await client.get_bucket("my-bucket")
    await bucket.write("trajectory", packed_data, timestamp, labels=labels)


def pack_trajectory_data(trajectory: list) -> bytes:
  """Pack trajectory data json format"""
  return json.dumps(trajectory).encode("utf-8")

The labels (total distance and average speed) are stored alongside each record. You can later filter or replicate records based on these values.

Once data is written, you'll see the bucket populate in ReductStore:

Query by Label​

async def query_by_label(bucket_name, entry_name, label_key, label_value):
  async with Client("http://localhost:8383", api_token="my-token") as client:
    bucket = await client.get_bucket(bucket_name)

    async for record in bucket.query(
      entry_name,
      when={
        label_key: {"$gt": label_value}
      },
    ):
      # Do something with the record

Remove the 'when' condition to return all records with no filtering.

Main Function​

async def main():
  await create_trajectory_bucket()
    await store_trajectory_data()
    label_query_result = await query_by_label("my-bucket", "trajectory", "&total_distance", HIGH_DISTANCE)
    if label_query_result:
      print(f"Data queried by label: {label_query_result}")

asyncio.run(main())

Conclusion​

Robotics data pipelines don't have to be a mess of bag files, overpriced cloud storage, and custom scripts duct taped together. ReductStore covers ingestion, retention, replication, and querying in one place, with direct integrations into ROS, Foxglove, Grafana, S3, and Azure.

Start with what you need, and add the rest as your system grows. Check out reduct.store or read through the documentation to get going.

If you have any questions or comments, feel free to use the ReductStore Community Forum.

ReductStore 1.19.0 is now available. This release extends the storage model for robotics and telemetry workloads and introduces new integration points for ROS and Zenoh.

To download the latest release, visit the Download Page.

What's new in 1.19.0?​

The first major change in v1.19 is licensing. ReductStore Core is now open source under Apache 2.0, which makes the core database easier to evaluate, integrate, and extend in production systems.

The second major change is the data model. ReductStore now supports hierarchical entry names, similar to ROS topics, and adds entry attachments for schemas and metadata. This makes it possible to represent structured robotics data without flattening topic hierarchies or moving context into external systems.

The release also introduces a native Zenoh API for direct ingestion and querying over Zenoh, and ReductBridge for ROS1 and ROS2 integration.

Nested Data Model with Attachments​

The new hierarchical data model lets you organize data in a path-based structure, similar to ROS topics, Zenoh key expressions, or MQTT topics. Instead of relying on a flat namespace, ReductStore can now store data in a form that matches the structure used by upstream systems.

Each entry can also include attachments for schemas and metadata. These attachments preserve the context required by downstream tooling without changing the record payload itself. For example, the ROS Extension can use them to decode serialized ROS messages and export them to MCAP files.

Native Zenoh API​

Zenoh is increasingly used in robotics and edge environments as a low-overhead protocol for distributed data exchange. With the native Zenoh API, ReductStore can participate directly in Zenoh-based systems without requiring an additional bridge or adapter.

You can start ReductStore with the Zenoh API enabled using a minimal Docker configuration:

docker pull reduct/store:v1.19.0
docker run --env "RS_ZENOH_ENABLED=ON" \
  --env "RS_ZENOH_CONFIG={}" \
  --env "RS_ZENOH_SUB_KEYEXPRS=**" \
  -p 8383:8383 -p 36597:36597 -p 7446:7446 \
  reduct/store:v1.19.0

Once enabled, the API allows you to write data to ReductStore directly over Zenoh. If a sample includes a JSON attachment, ReductStore stores it as record labels:

import json
import zenoh

KEY = "factory/line1/camera"
PAYLOAD = b"<binary payload>"
LABELS = {"robot": "alpha", "status": "ok"}

with zenoh.open(zenoh.Config()) as session:
    session.put(
        KEY,
        PAYLOAD,
        attachment=json.dumps(LABELS).encode(),
    )

You can also query data through Zenoh. For conditional queries, pass a when expression in the query attachment. If you want all matching records returned individually, use zenoh.ConsolidationMode.NONE:

import json
import zenoh

KEY = "factory/line1/when-query"
CONSOLIDATION = zenoh.ConsolidationMode.NONE
attachment = json.dumps({"when": {"&status": {"$eq": "ok"}}}).encode()

with zenoh.open(zenoh.Config()) as session:
    replies = [
        reply
        for reply in session.get(
            KEY,
            timeout=5.0,
            attachment=attachment,
            consolidation=CONSOLIDATION,
        )
        if reply.ok
    ]

    for reply in replies:
        print(reply.ok.payload.to_bytes())

Data ingested through Zenoh is stored in ReductStore like data written through the HTTP API, so it remains available for querying, replication, and downstream tools and extensions.

ReductBridge for ROS Integration​

This release also introduces ReductBridge, a new project for integrating ReductStore with ROS1 and ROS2. ReductBridge automatically labels ROS messages and stores related schemas and metadata as attachments.

This makes raw ROS payloads usable after ingestion: they can be decoded later with the ROS Extension or exported to MCAP files. ROS support is the first target, but the same pattern can be extended to operating system metrics, logs, and other telemetry to build a unified storage layer with consistent labeling and metadata.

What’s Next​

The next area of work is data compression and storage efficiency, especially for robotics workloads where payload sizes and formats vary significantly. We plan to introduce backend compression optimized for ReductStore's storage model, where many records are packed into a single block.

We also plan to store metadata in Parquet format. This should improve query efficiency, make metadata accessible without scanning the underlying data blocks, and simplify integration with analytics and data lake tooling.

Compatibility and Migration​

Starting with v1.19, ReductStore Docker images no longer run as root for security reasons. If you deploy with Docker, make sure the mounted data directory is writable by UID/GID 10001:10001 before upgrading.

If you have questions or feedback, join the ReductStore Community forum.

Thanks for using ReductStore!

Grafana is a powerful tool for visualizing time-series data, and it is widely used for monitoring and analysis. However, it does not natively understand robotics data formats, such as ROS 2 messages, since they are usually stored in binary formats (e.g., CDR). ReductStore's flexible query engine and extension system can bridge this gap. With the ReductROS extension, you can extract ROS 2 messages as JSON directly in Grafana queries. This enables you to build rich dashboards and alerts on your robotics data without preprocessing it into a different format.

In this tutorial, we will create a Grafana dashboard supported by the ReductStore data source plugin. Using the ReductROS extension, the dashboard will convert ROS 2 messages to JSON upon query, enabling us to visualize and analyze the data directly in Grafana.

Prerequisites​

This tutorial assumes that you have the following set up:

• A ReductStore instance with ROS 2 data (either your own or from the demo instance). Some robotics data is available on our demo ReductStore instance.
• A Grafana 10+ instance (local or cloud) with the ReductStore data source plugin installed. You can use our Docker image, or install Grafana separately and add the plugin manually, following the documentation.

Set up the ReductStore data source in Grafana​

Let's run Grafana locally using Docker and connect it to ReductStore:

docker run -d -p 3000:3000 reduct/grafana:latest

Open Grafana at http://localhost:3000 and log in with the default credentials, admin/admin. Then, go to the "Data Sources" panel. There, you can find the default ReductStore data source if you are using our Docker image. Otherwise, you can add it manually on the "Add new connection" page.

Once the data source has been added, configure it using the following parameters:

• Name: reductstore-datasource
• Server URL: https://play.reduct.store/replica (or your ReductStore instance URL)
• API Token: reductstore for the demo instance (or your API token)

Click "Save & Test" to verify the connection. You should see a success message confirming that Grafana can connect to ReductStore.

Query and extract ROS 2 data to JSON​

Now we can create a new dashboard and add a panel to query ROS 2 data from ReductStore. Go to Dashboards -> New Dashboard -> Add new panel. In the query editor, select the ReductStore data source and provide the following query parameters to extract a ROS 2 topic as JSON:

• Bucket: orion
• Entry: any entry, you can use wildcards (e.g., *) to query multiple entries
• Scope: LabelAndContent , which allows us to view both labels and content of the messages if it's JSON

when the parameters are set, we can read data with the following query:

{
  "$each_t": "$__interval",
  "#ext": {
    "ros": {
      "extract": {}
    }
  }
}

The magic happens in the #ext section, where we activate the ros extension and specify that we want to extract data. The extension reads the serialized ROS content for the specified bucket/entry, decodes the ROS 2 messages, and returns them as JSON records with timestamps and labels. You can read more about the query format in the ReductROS documentation.

Scale and performance​

Since your data may be large and data extraction can be computationally intensive, here are some tips to ensure good performance:

• Use Grafana $__interval with $each_t for interval-based queries
• Add label filters to narrow down the data (e.g., filter by topic name or message type)
• Start with a narrow time range and expand as needed

Optional: Visualize binary topics (images)​

If you're interested in visualizing binary data, such as images or point clouds, you might be interested in the ReductROS extension. The ReductROS extension can encode binary data as base64 strings or in JPEG format. This data can then be displayed in Grafana's "Text" panel with HTML rendering. For instance, if your topic contains raw images, you can specify the encoding in the query.

{
  "$each_t": "$__interval",
  "#ext": {
    "ros": {
      "extract": {
        "encode": { "data": "jpeg" }
      }
    }
  }
}

Conclusion​

Using ReductStore and the ReductROS extension allows you to easily extract and visualize ROS 2 data in Grafana without preprocessing it into a different format. This enables you to build rich dashboards and alerts on your robotics data, allowing for better monitoring and analysis of your robotic systems.

Importantly, this approach preserves your data's original format, enabling flexible querying without the need for additional pipelines or transformations. You can also combine it with other Grafana features, such as alerting and annotations, to gain additional insights from your robotics data.

I hope this tutorial gives you a good starting point for visualizing your ROS 2 data in Grafana using ReductStore. If you have questions or want to discuss which edition fits your use case, please reach out on the ReductStore Community forum or via our contact page.

Starting with ReductStore v1.19, we are changing how we license and package the project. From this version onward, ReductStore is split into two editions: ReductStore Core and ReductStore Pro.

Previously, ReductStore shipped as a single edition under BUSL-1.1 (Business Source License). With v1.19, the core database is open source under Apache 2.0, while Pro continues under commercial terms.

What's changing​

With this release, ReductStore Core becomes the open-source foundation of the project and is now available under the Apache License 2.0. It includes the database server and the core functionality that most users rely on for edge deployments and everyday data workflows.

Alongside Core, we will continue offering ReductStore Pro, which is distributed under a commercial license. The Pro edition provides additional capabilities and support options for teams with more advanced requirements. See Pricing for a clear comparison of what is included in each edition.

Why we're doing this​

We want to better support open source communities and make it easier to contribute to, integrate with, and build on top of ReductStore Core. Just as importantly, we want users to be able to run ReductStore on the edge without licensing restrictions for the core database use cases.

At the same time, we want to keep a sustainable commercial model for companies that build more complex cloud setups or need advanced support and functionality around robotics and IIoT data formats. That is the role of ReductStore Pro.

What this means for you​

If you use the core database functionality, you can freely adopt, integrate, and distribute ReductStore Core under the Apache 2.0 license. Existing workflows and upgrades remain straightforward.

For teams that rely on advanced functionality or require commercial support, ReductStore Pro remains available as the supported commercial edition. We will clearly document which features belong to Core and which are part of Pro in the documentation and release notes.

If you have questions or want to discuss which edition fits your use case, please reach out on the ReductStore Community forum or via our contact page.

Drones in air-gapped environments produce a lot of data (camera images, telemetry, logs, model outputs). Storing this data reliably on each drone and syncing it to a ground station later can be hard. ReductStore makes this easier: it's a lightweight, time-series object store that works offline and replicate data when a connection is available.

This guide explains a simple setup where each drone stores data locally with labels, replicates records to a ground station based on what it detects, and keeps a clear audit trail of what was captured and replicated.

What we'll cover:

• Drone-to-Ground Architecture
• Setting Up the Drone Node
• Storing Drone Data with Labels
• Setting Up Selective Replication
• Querying for Audit Reports
• Why This Setup Works Well for Drones

Drone-to-Ground Architecture​

The architecture has three main components:

• Each drone runs a small ReductStore server to save images and telemetry locally on disk (this lets the drone operate fully offline).
• A ground station runs a ReductStore instance that receives replicated data for analysis and archiving.
• ReductStore replication tasks copy data from drone to ground based on labels and conditions (e.g., only records flagged as anomalies, plus context around them).

Each drone pushes its data to the ground whenever it is connected. If the network disconnects, replication continues when the drone reconnects. This approach provides offline capability, lets you decide which data to replicate, and keeps a clear record of what happened.

Setting Up the Drone Node​

Start by running ReductStore on the drone's companion computer. Here is a minimal docker-compose.yml:

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    environment:
      RS_API_TOKEN: <DRONE_TOKEN>
      RS_BUCKET_1_NAME: mission-data
      RS_BUCKET_1_QUOTA_TYPE: FIFO
      RS_BUCKET_1_QUOTA_SIZE: 10000000000 # 10 GB
    volumes:
      user: "0:0" # Run as root with host bind mount in this example
      - ./data:/data

Run it with:

docker compose up -d

This starts a ReductStore server with a mission-data bucket that uses FIFO retention. Old data is deleted only when the 10 GB limit is reached, so the drone always keeps as much history as possible.

If you prefer Snap instead of Docker:

sudo snap install reductstore

That starts a ReductStore server on port 8383 by default. You can then create the bucket using the Reduct CLI:

reduct-cli alias add drone -L http://localhost:8383 -t "<DRONE_TOKEN>"
reduct-cli bucket create drone/mission-data --quota-type FIFO --quota-size 10GB

Storing Drone Data with Labels​

Use labels to tag every record with mission context. This is what makes selective replication and auditing possible later. Here is an example using the Python SDK:

import asyncio
import time
import hashlib
from reduct import Client


async def main():
    async with Client("http://localhost:8383", api_token="<DRONE_TOKEN>") as client:
        bucket = await client.get_bucket("mission-data")

        # Read a camera frame
        payload = open("frame.jpg", "rb").read()
        checksum = hashlib.sha256(payload).hexdigest()
        timestamp = int(time.time() * 1_000_000)  # microseconds

        # Write with mission labels
        await bucket.write(
            "camera",
            payload,
            timestamp=timestamp,
            labels={
                "mission_id": "m-2026-02-24-01",
                "platform_id": "uav-07",
                "anomaly": "false",
                "confidence": "0.95",
                "checksum": checksum,
            },
            content_type="image/jpeg",
        )

        # Write telemetry as a CSV batch
        csv_data = "ts,lat,lon,alt,speed\n"
        csv_data += "1708771200000000,47.3769,8.5417,450.2,12.5\n"
        csv_data += "1708771201000000,47.3770,8.5418,451.0,12.8\n"

        await bucket.write(
            "telemetry",
            csv_data.encode(),
            timestamp=timestamp,
            labels={
                "mission_id": "m-2026-02-24-01",
                "platform_id": "uav-07",
                "anomaly": "false",
                "checksum": hashlib.sha256(csv_data.encode()).hexdigest(),
            },
            content_type="text/csv",
        )


asyncio.run(main())

The anomaly label is important: it lets the replication task decide what to sync based on what the drone actually sees. For example, if the drone detects something unusual (an object, a warning, a low confidence score), it sets anomaly=true. The replication task can then automatically sync that record — plus the context around it.

The checksum label gives you a simple way to verify data integrity during audits.

Setting Up Selective Replication​

Once the drone connects to a trusted network, replication sends only the relevant records to the ground station. The simplest approach is to replicate based on a label, for example only records where the drone detected an anomaly:

reduct-cli alias add drone -L http://localhost:8383 -t "<DRONE_TOKEN>"

reduct-cli replica create drone/mission-to-ground \
    mission-data \
    https://<GROUND_TOKEN>@<ground-address>/drone-data \
    --when '{"&anomaly": {"$eq": "true"}}'

This creates a replication task that copies only records where anomaly=true from the drone's mission-data bucket to the ground station.

Replicating with context (before and after)​

In many cases, you don't just want the anomaly record itself — you also want to see what happened before it. ReductStore supports this with the #ctx_before and #ctx_after directives. For example, to replicate each anomaly record plus 30 seconds of data before it and 10 seconds after:

{
  "&anomaly": { "$eq": "true" },
  "#ctx_before": "30s",
  "#ctx_after": "10s"
}

This is powerful for drone operations: imagine the drone's onboard model detects an unexpected object. ReductStore will replicate that record and the 30 seconds of camera frames leading up to the detection, so the ground team can review what happened.

You can provision this directly in Docker using environment variables:

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    environment:
      RS_API_TOKEN: <DRONE_TOKEN>
      RS_BUCKET_1_NAME: mission-data
      RS_BUCKET_1_QUOTA_TYPE: FIFO
      RS_BUCKET_1_QUOTA_SIZE: 10000000000
      RS_REPLICATION_1_NAME: mission-to-ground
      RS_REPLICATION_1_SRC_BUCKET: mission-data
      RS_REPLICATION_1_DST_BUCKET: drone-data
      RS_REPLICATION_1_DST_HOST: https://<ground-address>
      RS_REPLICATION_1_DST_TOKEN: <GROUND_TOKEN>
      RS_REPLICATION_1_WHEN: |
        {
          "&anomaly": { "$$eq": "true" },
          "#ctx_before": "30s",
          "#ctx_after": "10s"
        }
    volumes:
      user: "0:0" # Run as root with host bind mount in this example
      - ./data:/data

Querying for Audit Reports​

After a mission, you can query the ground station to check what was captured and replicated. Here is a simple example that lists all records from a specific mission:

import asyncio
from reduct import Client


async def main():
    async with Client("https://<ground-address>", api_token="<GROUND_TOKEN>") as client:
        bucket = await client.get_bucket("drone-data")

        # Query all camera records from a specific mission
        async for record in bucket.query(
            "camera",
            when={"&mission_id": {"$eq": "m-2026-02-24-01"}},
        ):
            print(
                f"ts={record.timestamp}, "
                f"anomaly={record.labels.get('anomaly')}, "
                f"checksum={record.labels.get('checksum')}, "
                f"size={record.size}"
            )


asyncio.run(main())

This gives you a clear log of every record in that mission: timestamp, anomaly flag, checksum, and size. You can use this to verify that all expected data arrived on the ground side.

To go further, compare the checksums on the drone with the ground side to confirm nothing was altered during transfer. You can also check the error logs of the replication task to see if any records failed to replicate.

Why This Setup Works Well for Drones​

Drones have specific constraints that general purpose databases don't handle well. Here is what makes this setup practical:

• Full offline operation. Drones store everything locally and don't need a network connection during the mission. Data is safe on disk until sync happens.
• Automatic sync when connected. When the drone lands or connects to a trusted network, replication picks up where it left off. No manual file transfers, no rsync scripts, no USB sticks.
• Smart replication with context. You don't have to sync everything. The replication task filters by labels and can include past records around each event using #ctx_before. The ground team gets exactly what they need to understand what happened.
• Disk never fills up unexpectedly. FIFO retention removes the oldest data only when the disk is full. The drone always keeps as much history as possible without running out of space mid mission.
• Easy auditing. Every record has a timestamp, labels, and a checksum. After a mission, you can query the ground station and verify exactly what was captured and what was synced.
• Store any file type. Camera frames, telemetry CSV, logs, MCAP files, model outputs. Everything goes into the same system with the same interface.

Next Steps​

If you want to go deeper, check out these articles:

• Distributed Storage in Mobile Robotics for a similar setup with mobile robots and S3 cloud backend
• How to Store and Manage Robotics Data for a broader look at ReductStore features for robotics
• Data Replication Guide for the full documentation on replication tasks, filters, and modes
• Conditional Query Reference for all available conditional query operators you can use in replication filters and queries

I hope you found this article helpful! If you have any questions or feedback, don't hesitate to reach out on our ReductStore Community forum.

We are pleased to announce the release of the latest minor version of ReductStore, 1.18.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.18.0?​

In this release, we have added support for resilient deployments to build a more robust, fault-tolerant, and highly available ReductStore cluster. Now, you can implement hot-standby configurations, automatic failover, and seamless recovery to ensure uninterrupted service even in the face of hardware failures or network issues. You can also elastically scale read-only nodes to handle increased read workloads without impacting the performance of the primary nodes.

Additionally, we have introduced a new Multi-entry API that allows you to efficiently manage and query multiple entries in a single request. This API is designed to optimize performance and reduce latency when working with large datasets, making it easier to retrieve and manipulate data in bulk.

Resilient Deployments​

In ReductStore v1.18.0, resilient deployments are now a first-class feature. Using the RS_INSTANCE_ROLE setting, you can build topologies that keep your ingestion endpoint available during node failures and scale reads independently from writes.

Hot standby (active-passive) for write availability​

Run two nodes against the same backend (a shared filesystem or the same remote backend). Only one node is active at a time: the active node holds a lock file and refreshes it, while the standby waits and takes over when the lock becomes stale.

• Set RS_INSTANCE_ROLE=PRIMARY for the active node and RS_INSTANCE_ROLE=SECONDARY for the standby.
• Put both nodes behind a single virtual endpoint (load balancer / reverse proxy).
• Route traffic only to the node that returns 200 OK on GET /api/v1/ready (the inactive node returns 503).
• Tune failover behavior with RS_LOCK_FILE_TTL (how long the standby waits) and RS_LOCK_FILE_TIMEOUT (how long a node waits to acquire the lock).

To avoid split-brain writes, don’t run both nodes in STANDALONE mode against the same dataset.

Read-only replicas for read scaling​

Add one or more REPLICA nodes to serve queries from the same dataset. Replicas never write and periodically refresh bucket metadata and indexes from the backend, so newly written data may appear with a small delay.

• Route writes to a dedicated ingestion node (or the active node in a hot-standby pair).
• Route reads to replicas to scale query workloads horizontally.
• Tune staleness with RS_ENGINE_REPLICA_UPDATE_INTERVAL.

For an end-to-end walkthrough (including S3-based standalone, active-passive, and replicas), see the S3 Backend tutorial. For architecture options and operational notes, see the Disaster Recovery guide.

Multi-entry API​

The new Multi-entry API makes it possible to work with multiple entries in a single request. In practice, this is most useful for querying: instead of running one query per sensor/stream and merging results on the client, you can request all the entries you need at once and process a single result stream (each returned record includes its entry name).

Here is a Python example using reduct-py to query multiple entries in one call:

import asyncio

from reduct import Client


async def main() -> None:
    async with Client("http://localhost:8383", api_token="my-token") as client:
        bucket = await client.get_bucket("my-bucket")

        # Query multiple entries in a single request (since ReductStore v1.18).
        # You can mix exact names and wildcards.
        entries = ["sensor-*", "camera"]
        async for record in bucket.query(
            entries,
            start="2026-02-05T10:00:00Z",
            stop="2026-02-05T10:05:00Z",
            when={"&score": {"$gte": 10}},
        ):
            payload = await record.read_all()
            print(record.entry, record.timestamp, len(payload))


if __name__ == "__main__":
    asyncio.run(main())

What’s Next​

We’re already working on the next improvements to make ReductStore easier to integrate into real-world data pipelines:

Native Zenoh API​

Zenoh is becoming a common choice for data exchange in distributed, edge-first systems (robotics, industrial IoT, and telemetry). In upcoming releases, we plan to add a native Zenoh API so ReductStore can join Zenoh networks seamlessly.

This will make it easier to ingest and serve data directly through Zenoh—without custom bridges—so your storage layer fits naturally into existing Zenoh-based deployments.

Entry attachments (metadata)​

Today, labels work well for filtering and replication, but many projects also need structured metadata tied to an entry itself: data format, schema version, units, encoding, calibration details, and other context.

We plan to introduce attachments for entries, allowing you to store and retrieve this kind of metadata alongside your data streams, making datasets more self-describing and easier to consume across teams and tools.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

Modern robots collect a lot of data from sensors, cameras, logs, and system outputs. Managing this data well is important for debugging, performance tracking, and training machine learning models.

Over the past few years, we've been building a storage system from scratch. As part of that work, we spoke with many robotics teams across different industries to understand their challenges with data management.

Here's what we heard often:

• Only a subset of what robots generate is actually useful
• Network connections are not always stable or fast
• On-device storage is limited (hard drive swaps is not practical)
• Teams rely on manual workflows with scripts and raw files
• It's hard to find and extract the right data later
• ROS bag files get large quickly and are difficult to manage

In this article, we compare four tools built to handle robotics data: ReductStore, Foxglove, Rerun, and Heex. We look at how they work, what they're good at, and which use cases they support.

If you're working with robots and need to organize, stream, or store data more effectively, this overview should help.

Key Criteria for Comparison​

When picking a data tool for robotics, focus on these areas:

• Data Types Robotics is a large field with many sensor types. The tool should support the data you work with, such as:
  • Telemetry: Lightweight (GPS, IMU, joints), ideal for monitoring.
  • Downsampled Data: Lower-rate images or lidar for incident review without high storage cost.
  • Full-Resolution: Raw sensor outputs for deep debugging or training. This is storage-intensive but essential for some applications.
• Integration The tool should work with what you already use, like ROS, Grafana, MQTT, cloud platforms (S3, Azure, Google Cloud), and your development environment to avoid extra glue code and simplify workflows.
• Performance and Scalability Data must move quickly (both locally and to the cloud). Large files or slow queries can block robots or delay analysis.
• Ease of Use and APIs A simple UI and solid API support make it easier to automate, scale, and adapt the tool to different use cases.

Tool Overviews​

ReductStore​

ReductStore is a storage and streaming system designed for robotics data. It works both on the robot and in central storage (on-premise/self-hosted or in the cloud) with the same interface and SDKs (in Python, C++, Go, Javascript/TypeScript or Rust). That means your code stays the same whether you're reading local, remote data (or creating a browser-based dashboard).

To move data to the cloud, ReductStore uses conditional replication. You can define rules to upload only certain records: by label, rules, or event. For example, replicate all incident data, or just 1 out of 10 entries for routine monitoring.

ReductStore handles storage limits on edge devices with FIFO retention. Old data is deleted only when the device is full. Each bucket can have different rules, so you can keep more images and less telemetry, for example.

With an S3 backend, ReductStore batches small records together before uploading. This cuts down the number of requests and lowers cloud storage costs. For observability, you can connect Grafana to ReductStore to create dashboards with system metrics and sensor data. For MCAP files, ReductStore supports shareable query links that open directly in Foxglove v1/v2.

It also lets you filter or merge records server-side. For example, you can pull all temperature readings above a threshold over a time range without downloading full datasets.

Want more technical detail? Check out The Missing Database for Robotics Is Out.

Foxglove and MCAP​

Foxglove is a browser-based visualization and observability tool for robotics. It supports ROS 1, ROS 2, and MCAP logs, and handles data types like telemetry, camera feeds, lidar, and depth maps.

It uses MCAP, an open-source log format built for robotics, to store high-resolution data efficiently. You can explore MCAP files interactively in Foxglove Studio or stream them programmatically.

Foxglove provides an agent that detects new MCAP files on the robot and uploads them to the cloud automatically. This requires robots to record short rosbag segments (typically a few minutes each) which are closed and rotated continuously.

It integrates natively with ROS topics, services, and actions, and offers WebSocket and REST APIs. It also connects to major cloud providers like AWS, Azure, and Google Cloud for scalable storage.

The interface is built for time-series and sensor data, with interactive 2D/3D views, plots, and drag-and-drop panels for quick setup and review.

Rerun​

Rerun is an open-source visualization solution for time-series and multimodal data. It supports data types like images, point clouds, lidar, depth maps, tensors, and other sensor streams.

Its main strength is combining flexible logging with a fast, built-in 3D viewer designed for robotics and extended reality (XR) applications. For large datasets, Rerun provides a column-oriented API to speed up ingestion and reduce memory usage. It also uses efficient internal structures to minimize allocations and optimize performance on edge devices.

Rerun doesn't offer native ROS integration yet, but it can be used in ROS projects by adding custom logging to nodes.

You can embed Rerun in Jupyter notebooks or web pages, and use loggers for Python, Rust, and C++ to stream data into the viewer.

The UI is built for real-time 3D exploration, with overlays and live tracking that make it easy to inspect different data types in the same visual space.

Heex​

Heex is a data capture and review platform for autonomous systems that focuses on collecting only key moments—like errors or specific events instead of logging everything. This reduces bandwidth and storage needs while keeping important context.

Robots using Heex record data continuously in short ROSbag segments. A small agent on the robot watches for triggers and uploads only selected segments to the cloud based on rules.

A core feature is RDA (Resource and Data Automation) for ROS 2, which automates what to record and when. Rules can be changed remotely without restarting the robot.

Data is stored in ROSbag and can be reviewed directly in the Heex dashboard, which includes built-in open-source version of Foxglove. This setup makes it easy to manage data across fleets and locations.

Heex supports both ROS 1 and ROS 2, and integrates with other systems through SDKs, APIs, and a CLI.

The interface includes customizable dashboards to monitor sensor data, errors, and system status. Timelines and streams are easy to navigate for quick analysis.

Comparative Analysis Table​

To help visualize the differences between the tools, here is a comparison table summarizing their main characteristics:

Conclusion​

Each tool addresses a different part of the robotics data workflow. ReductStore is ideal for distributed storage across many robots, with selective replication to the cloud and flexible integration with tools like Grafana and Foxglove. Foxglove excels at visualizing MCAP logs and ROS topics. Rerun offers flexible, real-time 3D inspection for custom applications. Heex focuses on capturing just the important moments for efficient fleet analysis.

Choosing the right tool depends on what kind of data you collect, how you process it, and where you need it to go. In many cases, combining tools can give you the best of all worlds.

Thanks for reading. I hope this article helps you decide on the right storage strategy for your vibration data. If you have questions or comments, feel free to visit the ReductStore Community Forum.

Mobile robots produce a lot of data (camera images, IMU readings, logs, etc). Storing this data reliably on each robot and syncing it to the cloud can be hard. ReductStore makes this easier: it's a lightweight, time-series object store built for robotics and industrial IoT. It stores binary blobs (images, logs, CSV sensor data, MCAP, JSON) with timestamps and labels so you can quickly find and query them later.

This introduction guide explains a simple setup where each robot stores data locally and automatically syncs it to a cloud ReductStore instance backed by Amazon S3.

Edge-to-Cloud Architecture​

The architecture has three main components:

• Each robot runs a small ReductStore server in order to save images and IMU data locally on disk (this let the robot operate offline).
• A cloud ReductStore instance runs on a server (e.g., EC2) and uses S3 for long-term storage.
• ReductStore replication tasks copies data from robot to cloud based on labels, events, or rules (e.g., 1 record every minute).

Each robot pushes its data to the cloud whenever it is connected to the network. This approach provides the robots with offline capability, allows you to decide which data to replicate, and easily scales to support many robots.

How Replication Works​

ReductStore uses an append-only replication model:

• The robot stores new data locally.
• ReductStore automatically detects new records.
• It sends them to the cloud in batches (or streams large files).
• If the network disconnects, replication continues when the robot reconnects.

You can replicate:

• everything
• or only specific sensors
• or only records with certain labels
• or based on rules (e.g., 1 record every S seconds or every N records)

This can be configured per robot using environment variables (provisioning), with the web console or via the CLI (as shown in this guide).

Cloud ReductStore With S3 Backend​

ReductStore supports storing all records directly in S3. It keeps a local cache for fast access and batches many small blobs into larger blocks to save on S3 costs.

Here is an example docker-compose.yml to run a ReductStore server that uses S3 as the remote backend and provisions buckets for robots:

services:
  reductstore:
    image: reduct/store:latest
    container_name: reductstore
    ports:
      - "8383:8383"
    environment:
      # AWS credentials and S3 bucket configuration
      RS_REMOTE_BACKEND_TYPE: s3
      RS_REMOTE_BUCKET: <YOUR_S3_BUCKET_NAME>
      RS_REMOTE_REGION: <YOUR_S3_REGION>
      RS_REMOTE_ACCESS_KEY: <YOUR_AWS_ACCESS_KEY_ID>
      RS_REMOTE_SECRET_KEY: <YOUR_AWS_SECRET_ACCESS_KEY>
      RS_REMOTE_CACHE_PATH: /data/cache
      # Bucket provisioning
      RS_BUCKET_ROBOT_1_NAME: robot1-data
      RS_BUCKET_ROBOT_2_NAME: robot2-data
      # .. additional buckets as needed
    volumes:
      user: "0:0" # Run as root with host bind mount in this example
      - ./cache:/data/cache

Run it with:

docker compose up -d

This starts a ReductStore server that writes to S3 automatically. There are many more configuration options available in the configuration documentation.

Setting Up Replication​

First spin up a local ReductStore on each robot. Here with Snap:

sudo snap install reductstore

That starts a ReductStore server on port 8383 by default. Then you can use the Reduct CLI to set up replication from the robot to the cloud instance:

# Point the CLI to the robot's local ReductStore
reduct-cli alias add local -L http://localhost:8383 -t "<ROBOT_API_TOKEN>"

# Create a bucket for that robot
reduct-cli bucket create local/robot1-data

# Create a replication task to the cloud
reduct-cli replica create local/robot1-to-cloud \
    robot1-data \
    https://<CLOUD_API_TOKEN>@<cloud-address>/robot1-data

This creates a replication task called robot1-to-cloud that copies all data from the robot's local robot1-data bucket to the cloud instance. You can customize replication further by adding filters or rules. See the replication guide for more details.

Storing Sensor Data​

There are many ways to store data. When it comes to high-frequency sensor data like IMU readings, a common approach is to store them in 1-second files. Images can be stored as binary blobs (e.g., JPEG or PNG files). Here is an example of storing IMU data as CSV files and images as binary blobs using the Python SDK (this stores 10,000 samples and one camera image for a given timestamp as an example):

import asyncio
import time
import random
from reduct import Client


async def main():
    async with Client("http://localhost:8383", api_token="<ROBOT_API_TOKEN>") as client:
        bucket = await client.get_bucket("robot1-data")

        # Current timestamp to index the data by time in ReductStore
        timestamp = int(time.time() * 1_000_000)  # microseconds

        # Generate 10'000 IMU samples
        rows = []
        for i in range(10_000):
            rows.append(
                {
                    "ts": timestamp + i * 100,  # microseconds
                    "linear_acceleration_x": round(random.uniform(-2, 2), 3),
                    "linear_acceleration_y": round(random.uniform(-2, 2), 3),
                    "linear_acceleration_z": round(random.uniform(8.0, 10.0), 3),
                }
            )

        # Convert to CSV (store 1 seconds of data per file)
        csv = (
            "ts,linear_acceleration_x,linear_acceleration_y,linear_acceleration_z\n"
            + "\n".join(
                f"{r['ts']},{r['linear_acceleration_x']},"
                f"{r['linear_acceleration_y']},{r['linear_acceleration_z']}"
                for r in rows
            )
        )

        # Write the IMU batch
        await bucket.write(
            entry_name="imu_logs",
            data=csv.encode(),
            timestamp=timestamp,
            labels={"sensor": "imu", "rows": "1000"},
            content_type="text/csv",  # MIME type
        )

        # Write one camera image
        with open("camera_image.png", "rb") as img:
            await bucket.write(
                entry_name="images",
                data=img.read(),
                timestamp=timestamp,
                labels={"sensor": "camera"},
                content_type="image/png",
            )


asyncio.run(main())

Querying Sensor Data Using ReductSelect​

If your IMU data is stored as CSV, the ReductSelect extension lets you:

• extract only certain columns
• filter rows based on conditions

Example: filter CSV rows where acc_x > 10:

{
    "#ext": {
        "select": {
            "csv": {"has_headers": True},
            "columns": [
                {"name": "ts", "as_label": "ts_ns"},
                {"name": "linear_acceleration_x", "as_label": "acc_x"},
                {"name": "linear_acceleration_y"},
                {"name": "linear_acceleration_z"},
            ],
        },
        "when": {"@acc_x": {"$gt": 1.9}},
    }
}

Python example:

import asyncio
from reduct import Client

when = { ... } # the JSON condition from above

async def main():
    async with Client("https://<cloud-address>", api_token="<TOKEN>") as client:
        bucket = await client.get_bucket("robot1-data")

        async for rec in bucket.query("imu_logs", when=when):
            data = await rec.read_all()
            print(data.decode())

asyncio.run(main())

This returns only the rows where linear_acceleration_x > 1.9, along with the timestamp.

Why This Setup Works Well for Robotics​

There are several advantages to using a specialized storage solution like ReductStore for mobile robotics:

• Robots can store data locally and operate offline without network connectivity.
• Automatic replication when connected to avoid manual uploads and simplify data management.
• Selective replication lets you control what data is sent to the cloud (i.e. decide on your reduction strategy) to save bandwidth and storage.
• Labels and timestamps make it easy to organize and query sensor data later.
• Store files of any type (images, CSV, logs, MCAP) in a single system without needing separate storage solutions for each data type.

Next Steps​