OTA Update Problem #74
Comments
|
For additional information, I was able to test the basic functionality using the Basic _AirUpdate example which downloads the "Hello Sming" program. I also had my program (garagectrl) download the same "Hello Sming" program from your server and it flashed and re-started the program correctly. So it would seem there is something about my garagctrl program that prevents it from flashing itself via OTA. My garagectrl program flashes via the serial port without any problems. Is there any program size limitations I should be aware of? I just used the simple "python -m SimpleHTTPServer 8000" invocation at the command-line from the firmware directory to serve the binary files. At a bit of a loss at the moment why this fails for my application only (it appears). |
|
I had a similar problem to you when using python simpleHTTPServer. I think it might be something to do with the headers which it sends which are different than those sent by the server hosting the update in the example. He is using nginx but I managed to get it to work using CherryPy, so you might want to try using a different webserver. |
|
I thought of the same thing and tried webfsd with the same results. Without additional feedback from the Sming developers, there seems to be a few worrisome issues with their approach. First, most implementations I've run across (including rboot) and Expressif's own solution seem to preserve the concept of maintaining separate boot-able images. You never flash into the same partition where your code is running and always maintain one "good" image. Of course you sacrifice precious flash to maintain 1 (or more) backup images. Likewise, the code needs to be built for the different partitions (e.g. start address varies). Another issue that I can't quite follow is although it downloads two segments, I'm not sure that it's flashing them both (see I actually like Sming and am using it as the basis for one of my projects. I hope I can end up leveraging the OTA method they provide because it sure would simplify programming my device since it will be installed in a hard-to-reach location. Thanks for the suggestion . . . I'll investigate some more . . . |
|
I tried it again with a simple CheeryPy file server. The results were identical. I think there is something more happening here than just an issue with downloading the segments correctly. Works fine with a serial download but OTA is another story altogether. Hope the Sming developers have some ideas to try ;-) |
|
I have had some success with this. Sometimes I'll get 2 -3 successful flashes before it goes south. Whenever it is successful, Item 1 is always at 0x0 0x4023C000. It always seems to be OK the first time after flashing across serial. A failed flash always has another address ie item: 0x0 0x402434C0 29888 bytes. It also seems that on a failed flash, I always get Download: http://10.10.254.109:80/fw/eagle.flash.bin I don't get any connection errors on a successful flash. doing an fsformat before flash "seems" to improve the odds of a successful write, but since the flash process does this anyway, I cant see why. BTW, Sming is awesome. Keep up the good work!!! |
|
I'm interested what happen if it fail during the update process. It directly write downloaded data to the flash memory? Without any CRC check? It is possible programm some minimal code which will check if new firmware exist and download it on the different place in the flash to keep update mechanism safe? |
It will write full updated flash segment from end to end (from start of first to the end of the last)
|
What's not entirely clear is where the downloaded image it stored prior to flashing it to the final destination in flash. Also, in Is it safe to assume your As I understand it, .irom0.text is run from memory-mapped SPI flash. What also concerns me is how that region is updated via your OTA mechanism without disturbing the code that is actively running there. You algorithm doesn't appear to maintain separate distinct update "slots" that are updated via OTA with a boot-loader configured to run the downloaded image on successful update. Likewise, I don't see any mention of separate linker files to build the application for one (or more) distinct .irom0.text regions. Basically, a more detailed explanation of how the OTA mechanism works and is implemented would be extremely helpful. For my situation, I don't believe my image is being corrupted during the download. I believe there is something more fundamentally broken with the current OTA strategy barring additional details on exactly how it works and any inherent limitations. I've been following the development of the rboot OTA framework and it appears to be a very reasonable approach to this problem. Additionally, the author has offered a good deal of information on how the mechanism works, limitations, and integration examples. Likewise, in this post he indicates that he has successfully integrated it into your Sming framework with minor changes. Perhaps it might be to your advantage to investigate his approach and perhaps offer it as a replacement (or alternative) mechanism for supporting OTA updates? It seems his solution already offers some of the features that yours currently lacks. Again, I appreciate the tremendous effort you've put into Sming and am generally pleased with the framework and it's on-going development. It just seems the current Sming OTA strategy, as it stands today, may have some fundamental limitations that preclude it's use as a general-purpose (and reliable) OTA solution. |
I'm very sorry, really I have a limitted time to write documentation for my code. Sorry. I understand what it isn't right, but it's to hard develop all and create help. If somebody can help with any parts of documentation it also will be great. Now it's saving downloaded image in filesystem space (and rewrite files). But if you have big flash it can be stored in the free space. After that code what make firmwaring located in IRAM and can work during IROM rewriting.
Why not, If somebody help with it, we can move to more powerful rboot solution. |
|
I understand how difficult it can be to write the code and then have spare cycles to document it. It can be the least enjoyable part of the process but unfortunately is one of the more critical parts for developing a community. At least you do provide sample projects which helps immensely. Perhaps Richard might share the changes he made to your linker script to get your examples to work with his OTA framework. I'll probably begin an investigation myself because I could see it as an extremely useful feature . . . one that has to be robust in order to really be useful. |
|
Yes, I'm always trying to provide examples for every developed class. Thank you very much, please keep me informed. Also I'm ready to answer any questions. |
|
You might be very interested in this: http://www.esp8266.com/viewtopic.php?p=18403&sid=2689c2c483e75ce37b6e88cd9fa7975c#p18403 Richard describes how he integrated rboot into your linker/makefiles for the simple Blink project. |
|
OTA core was updated, please check again. |
|
I was looking through the commits and it wasn't clear what changed to address this problem? Can you give me any details on the potential "fix"? Are you saying you've updated to the latest SDK and since SMING partially relies on the SDK OTA update mechanism this might have "fixed" the issue? I'm a bit confused. Also, I went ahead and implemented the rboot boot-loader (see above) based on Richard's changes to your Blink example project in my own project. It required a few more tweaks to his Makefile but I've successfully implemented it in my own Sming based project and it works like a charm. I have two "rom" slots that I can switch between and optionally update. It feels a little safer to only update the slot that is not currently being used. The only issue is I haven't figured out is how to deal with a spiffs image. My project doesn't use the filesystem so I safely omitted that option. I suppose the spiff images could be stored in an extra flash slot if the flash is large enough (often the case since the ESP8266 can only map 1MB of the flash for iromtext and an ESP-12 typically comes with 4 MB . . . so 512KB for each "rom/program" slot still leaves you with 3 MB untouched and potentially useful for your spiffs filesystem). Have you looked any further into the rboot approach? |
|
@gschmottlach It almost small fixes, most important - checking correct download state before flashing. I think what when you wrote:
You just miss reboot chip after flashing, because when code auto-started after firmwaring from serial it can't reboot automatically. But if you reboot it one time after flashing, it can run auto reboot when needed, and OTA can be finished automatically. So if you will test Sming OTA and reply here it will be very good. If you will share your experience about rboot, it will be great also. Thank you very much. |
|
About last changes: also WDT switching was added. It should solve unexpected restart problem. |
|
Back when I was investigating Sming's native OTA feature, I rebooted the chip (manually) after the OTA update. If I recall, the problem was it wouldn't boot at all and ended up with an exception on power-up. I tried this several times. I was able, however, to successfully build/execute your OTA example project and that worked correctly - just with my project it would not. Unfortunately, I ripped out the code in my original project and moved on to implement the 'rboot' OTA implementation which seems to work well for my use case. As I mentioned, it just seems "safer" to have a backup image available and not to update the image that is currently executing. Seems like it might be tricky to insure the OTA code doesn't try to access any functions in the memory-mapped irom.text section of the flash during the update. My experiences with 'rboot' have been positive so far. The alternative Makefile that Richard provided in the ESP8266.com posting (referenced above) was fairly complete and only needed a few tweaks to support building '*.c' files. Also, I had to add a few targets to build rboot.bin and utilize his updated linker files which produce the two rom0/rom1 binary images. Finally, his rboot-ota.c code had a small bug that prevented the binaries to be downloaded when using a simple Python server. I could certainly share the pertinent Makefile's and code changes as a zip/tgz archive if you're really interested in investigating it for Sming. Eventually I'll probably make my project public but it's still a a work in progress - not quite ready for prime time. If you have any specific questions, let me know. With work chewing up most my free time, it's been tough to focus on this little side project. Unfortunately I can't quickly repeat my earlier experiments with Sming OTA solution. |
|
You might be interested in this bugfix if you consider investigating rboot's OTA: |
|
@gschmottlach I'd be interested in seeing your changes. I am getting ready to implement OTA, and rboot does seem like a better approach. For my application, I can't risk something getting corrupted as my users won't really have the knowledge to re-flash via serial. Any info you can provide is appreciated. |
|
Hi Curtis - Sorry for not getting back sooner but I was out of town. I haven't updated my Makefile lately to match the changes they (continue) to make to the general Sming Makefile. It's highly likely I'll have to tweak a few things. I intend to (at some point) to update my project to the latest Sming release and merge any changes. In the interim, you might want to check out Richard's original integration into the Sming Blink example (see http://www.esp8266.com/viewtopic.php?p=18403&sid=2689c2c483e75ce37b6e88cd9fa7975c#p18403). I started with his instructions (and modified Makefile) and then tweaked it as necesary to fit my project's needs (e.g. library dependences etc...). Also, I added code to my application to do a DNS lookup of a MQTT broker that published a topic (JSON formatted) containing information of where to fetch the OTA binaries. From there you feed the URL to Richard's library and it fetches the binary, flashes it, and reboots. Not too tricky depending on your background. If it was in a public repo I'd share it with you but I still want to do some cleanup before making it generally available.. Let me know if you need more details . . . |
|
Closing: New OTA update using rBoot is failsafe and the recommended way for OTA. |
I'm trying to utilize the new OTA feature of Sming and I've wired up my application to listen for an MQTT topic (e.g. /home/garage/large/cmd) for a command to initiate the OTA. It receives the command which includes two files to update: eagle.flash.bin and eagle.irom0text.bin. The JSON formatted command looks as follows:
The offsets are in decimal but convert to 0x0 => eagle.flash.bin, 0x9000 => eagle.irom0text.bin. Anyway, the software contacts my webserver and successfully downloads the two files just fine. The thing I noticed, however, is that it appears to only flash one of the files/segments instead of both. I'm a little confused looking at
HttpFirmwareUpdate::applyUpdate(). It seems like it only flashes a single item. Anyway, after writing to flash it re-boots and just sits there - dead to the world. Perhaps by looking at the included trace you can provide me with some idea of what might be going on or perhaps how I can trouble-shoot this. Any insight into the expected theory of operation would be very helpful as well. Are there limitations which I should be aware of? It seems that it reflashes the same flash region where the code is currently executing out of which seems a little suspect - especially (as I understand it), the ESP8266 does execute in-place from flash by loading chunks of code into an in-memory (SRAM) cache.Here's the trace:
The text was updated successfully, but these errors were encountered: