Development chain does not produce any blocks

[ArseniiPetrovich]

We've been running a 701 runtime version of Moonbeam codebase to run a custom development chain. It was working quite fine for 3.5k blocks, and then suddenly it stop the block producing because of the following error:

[🌗] 🔮 Skipping candidate production because we are not eligible

Is this somehow related to the following PR => #828? I just don't want to update the network yet if this is not strictly necessary. Or am I doing smth wrong?
I think this error is described somewhere at the dependencies, but I'm having tough times trying to locate it.

[crystalin]

It might be related yes, it is not easy to say cause it depends of the client and the runtime and if you have done migrations or not.
So far you can use the flag --force-authoring to make sure your node produces blocks (They might not get into the canonical chain, the consensus will decide)

[ArseniiPetrovich]

@crystalin
After applying this flag I started to receive the following instead:

Oct 12 13:37:36 node-a [173644]: 2021-10-12 13:37:36 [🌗] Bad mandatory: Module { index: 18, error: 2, message: Some("CannotBeAuthor") }
Oct 12 13:37:36 node-a [173644]: 2021-10-12 13:37:36 [🌗] ❌️ Mandatory inherent extrinsic returned error. Block cannot be produced.

I've tried to query the chain state, but in the list of "candidates" for collation I receive the two collators that were producing the blocks before :(

[crystalin]

@ArseniiPetrovich So having the 2 candidates that were there before is good no ?
You need to ensure you have the key for it.
Can you increase the logs author=trace,cumulus=trace and paste a 5 minutes of log in the ticket please ?

[ArseniiPetrovich]

@crystalin
I realized that blocks actually started to be produced, despite the fact that error appears from time to time. Here are the logs: logs.txt, but I'm starting to think that this is exactly what was fixed in the #828, because the node sees that it's collator is not eligible to produce the block, but still seems to be trying and throwing an error. My other thought is that the error thrown is because of this force-authoring thing, but then why it didn't produce the blocks without it? 🤔
One way or another the blocks are producing, so I'm extremely happy about it :)

[crystalin]

@ArseniiPetrovich Yes, it is very likely related to this bug, but there is also a similar one that might happen if when you will migrate to 800. However nodes 0.13.1 have the fix for it included (without requiring the --force-authoring)

[ArseniiPetrovich]

Thanks for your help! :)