Subscribing a topic with QoS1 results in duplicate messages

[shekharhimanshu]

Hello,

I am not sure if this is by design or a bug, but subscribing a topic with QoS1 always results in duplicate messages. However, these messages don't have the DUP flag set to 1. So it is as if the AWS MQTT broker is re-sending the same message over and over again or there is some issue inside the SDK.

I tried to debug the issue and found few interesting observations:

• The duplicate messages are received roughly every 30 seconds or so and are received 4-6 times before dying out.
  • However, 30 second thing is not always true.
• Using QoS 0 when subscribing, stops these duplicate messages for some reason.
• Other MQTT client like mosquitto does not seem to have this issue when tested with the same AWS MQTT broker. (No duplicate messages with either QoS 0 or QoS 1)

For now, as a workaround I have refrained from using QoS 1 when subscribing to AWS MQTT topics but this behavior is something not desirable and might have unknown consequences when used in production.

Could you please look into this issue?

[vareddy-zz]

Hi @shekharhimanshu,
I tried reproducing the issue but I wasn't able to. I tested with all combinations of QoS for publish and subscribe with the AWS MQTT broker and they all behaved as expected.
What QoS are you using while publishing the messages? Also, what are you using to publish the messages, the SDK itself or some other MQTT client, like mosquitto?
Please let us know if you find any more issues.
Thanks,
Varun

[shekharhimanshu]

Hello @vareddy
Thats weird because I am able to reproduce the issue every single time.
I am publishing with QoS1 and using the SDK itself.

I will try to reproduce the issue with PubSub sample and let you know.
In the meantime, let me check if the new release behaves any differently.

[shekharhimanshu]

I tried with the latest SDK version as well and there was no change in the behavior.
Subscribing a topic with QoS1 still always results in duplicate messages.

[vareddy-zz]

Hi @shekharhimanshu,
I think I had configured my test program incorrectly as it wasn't waiting for as long as you stated (> 30 seconds) before quitting as I ran a long running test (~10 mins) and now see the repeated messages as you stated. My apologies!
Investigating this now.
Thanks for all the inputs!
Varun

[vareddy-zz]

Hi @shekharhimanshu,
The fix for this issue is now in v1.1.1 of the SDK.
Thanks!
Varun

[twebner]

Hey @vareddy,

Can you explain what you changed in the SDK to fix the duplicate issue? I was using the Embedded C SDK before and saw the duplicate issue there when publishing with QoS 1. I created an issue on their GitHub (aws/aws-iot-device-sdk-embedded-C#169) and the resolution was that since the AWS broker does not support the duplicate flag, we would have to add our own IDs in the payload and filter ourselves. However, it seems like the C++ SDK has solved the duplicate problem without forcing clients to implement custom logic?

[vareddy-zz]

Hi @twebner,
The puback packets were using an incorrect packet ID (which was auto-generated to be a unique packet ID). The puback packets need to have the same packet ID as the published packet. I added an extra member to the puback class and stored the published packet's ID, which was used for packet creation.
See Publish.hpp and Publish.cpp.
Thanks!
Varun

[twebner]

Thanks @vareddy! So it sounds like something similar could be done in the C SDK, right? Is there any way for you to contact that team and share your solution with them? Maybe you could reopen and comment on the issue I linked in my comment above?

[braju3112]

Hi @shekharhimanshu and @vareddy
I want to configure QOS = 1 for Subscribing a topic with QoS1 in AWS IOT. I am trying to do that but unable to find the correct solution. Could you explain the solution?

Thanks,
Mohan

[shekharhimanshu]

You can set QoS when creating a Subscription.

[braju3112]

Thank you @shekharhimanshu for your quick reply.
Could you explain with example? I am new to this AWS. If we want to set QOS =1, do we set through CLI or SDK for topic for subscribe the topic?

[shekharhimanshu]

See, PubSub sample for example.

If we want to set QOS =1, do we set through CLI or SDK for topic for subscribe the topic?

That depends upon what you are trying to do. If you are using this SDK, you do it with this SDK. If by CLI you mean the aws-cli, I haven't explored it much with regards to IoT pub sub. You can check the documentation to check for available commands.