Skip to main content

Cloud Backend overview

The ESP RainMaker Neo cloud backend is the service that sits between devices and the people using them: it authenticates both users and devices, carries state in each direction, stores nodes and their groupings, and exposes all of this over REST or MQTT. It is open source esp-rainmaker-neo, built entirely from managed AWS services, and it runs in an AWS account you control.

This page is the map of the section. It covers two jobs that share the same repository — standing up and operating a deployment, and changing the backend itself.

There is also a hosted deployment

Espressif runs a public ESP RainMaker Neo deployment for evaluation and hobbyist use, and you do not need any of this section to use it — see Get Started. Everything below is about a deployment in your own account.

What it does for you​

  • Identity and access — Cognito user pools for end users and for administrators, exchanged for short-lived AWS credentials that sign REST calls and authorise MQTT.
  • Device plane — AWS IoT Core with X.509 mutual TLS, topic rules that route device messages to handlers, and connect/disconnect events that keep online state current.
  • State and storage — nodes, users, groups, associations and sharing requests in DynamoDB, with device state held in shadows.
  • APIs — an API Gateway REST API for end-users and admins, plus the MQTT topics devices and apps publish and subscribe to.
  • Admin Dashboard — the web console for the deployment used for configuration.
  • Voice assistant skills — Alexa and Google Voice Assistant skills for voice based configurations.

What you supply is the AWS account, the parameters that name your deployment, and — if you are extending it — handlers for whatever the platform does not already do.

How it is built​

There are no long-running servers. Every entry point is either an API Gateway request that invokes a Lambda, or an AWS IoT Core rule that fans out to one. State lives in DynamoDB, where partition and sort keys plus GSIs encode several access patterns per table so the table count stays small.

LayerBuilt from
AuthCognito user pools and an identity pool, fronted by an assume_role Lambda
ComputeOne Go Lambda binary per *_main.go entry point, grouped into feature packages
DataDynamoDB tables with overloaded indexes
Device planeAWS IoT Core with topic rules for device messages and for presence
DeliveryAWS CDK in Python, organised into stack groups that deploy in dependency order

Architecture overview has the full topology, including which rule invokes which handler.

Two ways to get a deployment​

You can install Espressif's published CloudFormation templates through the Cloud Installer — a small stack whose dashboard lets you pick a version and fill in parameters, with no local toolchain — or deploy from source with AWS CDK, which is the route to take if you intend to modify the backend or run it from your own CI. Both produce the same stacks. Deploying ESP RainMaker Neo compares them and describes what you take on once a deployment exists.

Working on the backend itself​

The repository is a Go workspace for the Lambda handlers plus a Python CDK app for the infrastructure, and the top-level Makefile is the entry point for every workflow: make go_build to build the binaries, make test for the Ginkgo unit tests against mocked AWS services, make itest for the Pytest suite that runs against a live deployment.

The mock-based tests carry the bulk of the coverage because they parallelise and run fast; the integration tests exist for the things only a real deployment shows you, such as a missing permission or an unattached IoT policy. Both have to pass, in that order.

Where to start​

You want toGo to
Understand the runtime topologyArchitecture overview
Get a deployment running todayDeploy With Installer
Add an endpoint, rule or handlerExtending the Cloud Backend
Verify a change against a deploymentIntegration tests
Call the service from a clientAPI Reference
Drive a real device end to endFlash a device
Run one day to dayOperations