Practical thoughts on AWS IoT Core & Rules

In the following piece I will iterate through a few interesting revelations I have made while extensively using the AWS IoT Core platform. As of January 2019 I have not found information regarding the following practices in the documentation, nor on AWS Forum or the internet. I hope that some of you experimenting with IoT Core will find this guide useful.

Using IoT Rules with Lambda integration

The use case was the following:

1. The client application publishes an Item to a certain topic, let’s say sensors/list via SDK.
2. The IoT Rule perceives the event and invokes a Lambda function.
3. The Lambda function queries a DynamoDb table and returns the result set to the client application.
4. The requirement is that the application publishes / subscribes on the same topic.

I had the preconception, that IoT Rule engine with Lambda integration out-of-the-box supports passing back Lambda response to the invoker (the client application in this case). To my great surprise it turned out not to be the case, furthermore, none of the other integrations support this functionality automatically.

I have searched for best practices but without success, nobody have publicly covered this scenario, or at least I was unable to find any.

My solution:

1. The client application publishes an Item to a certain topic, let’s say sensors/list via SDK.
2. The IoT Rule perceives the event and invokes a Lambda function.
3. The Lambda function queries a DynamoDb table and publishes the result set to the origin topic via SDK.
4. As the client application subscribes to the sensors/list topic, the callback function will receive the response by the Lambda function.

The problem with this approach:

As the Lambda function publishes to the same topic IoT Rule engine listens on, it invokes an other Lambda function and ultimately starts a chain, which results in an infinite loop of service calls.

The cleanest solution would be that the client application subscribes to an other topics, let’s say sensors/list/result, and the invoked lambda function publishes the database results into it.

As I have mentioned, in my case the explicit requirement was to publish / subscribe to the same topics, so I came up with a rather easy solution.

When you define IoT Rules, you are required to write a simple SQL like query, which queries your published payload and triggers services according based on the parameter values.

In my case, I put an “original_request”: true in the payload, and told the rule query statement to look for this parameter:

SELECT * FROM ‘sensors/list’ WHERE original_request=true

This way the rule engine knew that now that invoking the Lambda function is allowed.

One important constraint you will have to pay attention, according to AWS IoT documentation:

The payload for every publish request is limited to 128 KB. The AWS IoT service rejects publish and connect requests larger than this size.

It means, if you know that your message payload (published by the lambda function), will be greater than 128kb, you will have to implement a solution to split the message into chunks, and reconstruct them on the client side.

My best bet would be to split the message into smaller parts inside the Lambda function. Attach a unique identifier to the chunks, and send the pieces flagged by this unique ID and include the number of desired chunks in the payload. On the client side, store the chunks in a buffer and wait for the individual chunks to arrive.

In case you have any question, remark or practical advice regarding the above article please reach out or post a response.

Im am Peter, a Cloud Engineer and tech enthusiast formally working at IBM Budapest Lab as member of the Enterprise Content Delivery Network team. At the moment I am working as Consultant for the exclusive Eastern-European AWS Partner company called TC2. If you like this piece please 👏

👋 Find me on Github or Linkedin 👋