Solution for upgrading ISM43362-M3G-L44 wifi module firmware on a ST B-L4S5I-IOT01A dev board

Why I'm trying to upgrade the wifi module's firmware.

I have been experiencing problems with network offload on Zephyr RTOS using a ST B-L4S5I-IOT01A dev board which has an onboard Inventek ISM43362-M3G-L44 wifi module.  Its not completely dead, I have it successfully connecting to my wifi, the offload processing works well enough that I can ping the module and telnet to the wifi coprocessor, but there's something not "connecting" between the processors in the sense that I can't get telnet to the dev board shell or anything else on Zephyr to connect in a conceptual sense to the wifi.  Offload "works" in that the C language callbacks fire and I can report my DHCP provided wifi IP address, which is how I can ping it, but its like the application TCP/UDP sockets are refusing to connect to the wifi interface, even though they kind of are doing that; most aggravating.

For a variety of reasons, I decided to upgrade the wifi firmware.  I've read elsewhere about how Azure connectivity with this board will NOT work unless the wifi firmware is upgraded, so that is very interesting although I'm not using Azure in this application.  Another reason is the usual troubleshooting procedure of I've done everything easier and more of an obvious solution already.  My final justification is I have no idea what version of the wifi coprocessor firmware was shipped on the board, so at least after an upgrade I'd know for certain what version I'm running, which probably provides at least some troubleshooting value.  

I did not decide to upgrade the wifi firmware for fun or because I was bored, LOL.

Description of the wifi module

The wifi module is similar to an ancient Hayes-compatible phone modem, serial attached with "AT" command set, that connects to wifi networks instead of phone lines.  Also instead of connecting to the device using a serial port like a traditional phone modem, this module has the option of SPI connection for higher speed, and on this ST dev board, it connects via that SPI interface.  So in one way of looking at it, it's exactly like a phone modem, except when looking at it in another way, everything is different from a phone modem (LOL?).

Finding references, instructions, tools for the upgrade process

If you google for the wifi module and firmware, which is a pretty good strategy most of the time, one of the top results is this URL:

https://www.inventeksys.com/es-wifi-firmware/

Sadly, the most recent firmware on that page is v3.5.2.3 from Oct 10 2016.  That's not going to help.  I don't think the CPU on this dev board had been released back in 2016.

Further research led me to:

https://www.inventeksys.com/ism4336-m3g-l44-e-embedded-serial-to-wifi-module/

From that device page, you can find a link to another, separate and newer, firmware upgrade page for the device:

https://www.inventeksys.com/iwin/firmware/

The highest version available on THIS download page is 3.5.2.7, that's cool.

Next we have to upload the firmware into the wifi module.  If you have a serial connection, with a switch selectable USB port or similar like the Arduino shields provide, this task is pretty easy.  If you have a PCB standard JTAG pinout and a JTAG programmer this task is also pretty easy.  The ST board has none of the above.  What the ST provided upgrade solution does have, is a very small binary program provided by ST, that enables SPI passthrough so STM32CubeProgrammer or anything else that speaks ST/Link can program what it thinks is the STM32 flash, but its actually programming the flash in the wifi module.  What could possibly go wrong? 

The software to upload the firmware is on the ST website, in the "docs" section not "tools":

https://www.st.com/en/evaluation-tools/b-l4s5i-iot01a.html#documentation

The best instructions for the firmware upgrade seem to be in the upload tool's README file, although Inventek provides a link to a .pdf which has some alternative options:

http://www.inventeksys.com/iwin/wp-content/uploads/Firmware-Upgrades.pdf

How the upgrade process was supposed to work

I followed the README upgrade instructions from ST.  First you download the firmware zip from inventeksys.com, using the correct, newer download page, not the Google-provided older download page.  Then, unzip and change the filename from *.bin.rename to *.bin, because of crazy windows anti-virus software, the firmware has to be double wrapped to get past.  Then copy the renamed firmware into the bin directory of the ST provided firmware upgrader.  The firmware upgrader batch file uploads a tiny bin file that puts the dev board into SPI passthrough, and then uses the CLI for STM32CubeProgrammer to burn the firmware into what it thinks is the STM32L4 chip but is actually the passed-through wifi module's flash chip.  It's a rather short and simple windows batch file.

Something undocumented, but cool, is when the wifi module is in passthrough mode the blue LED turns on.  At least ... I think that's what the LED implies.  The STM32L4 firmware is a binary blob and I didn't run the debugger on it so who really knows?

What actually happened during the upgrade process

There is a slight problem, if you run update_Wifi.bat on a Windows 11 PC with the newest version of STM32CubeProgrammer and newest ST/Link firmware on the dev board, the dev board bootloader pass thru is downloaded to the STM32L4 and it runs successfully, OK so far.  Then the command to bulk erase the Inventek module's flash runs, and eventually times out while in the process of erasing the flash (whoops).  Then the program stage runs, tries to re-issue the erase command before writing the firmware, however the flash chip is still erasing from the previous bulk erase command, so it crashes the write process and never writes the firmware to the wifi.  So now we have a wifi module with erased firmware flash.  That's not good.

It gets somewhat worse.  If you reinstall MCUBoot and a Zephyr image, the MCUBoot will jump to the Zephyr image and silently fail and lock up the STM32L4 during initial bootup of Zephyr before it even jumps to main(), so time can be wasted trying to troubleshoot your reinstall, well, obviously I forgot to ... confirm the Zephyr image or something.  So the board seems bricked and everyone is unhappy.

Unbricking and successful wifi module firmware upgrade process

The board, however, is not bricked.  I would theorize the problem is older versions of STM32CubeProgrammer and/or the ST/Link firmware would not return from the erase command until the erase was fully complete and/or the default delay to wait for bulk erase completion before timing out was longer.  Also I don't really see the point of bulk erasing the wifi flash if the next step in the batch file, the write step, begins by erasing the flash anyway.

So use vim or whatever editor you use to edit this line in update_Wifi.bat to skip the bulk erase step by commenting out, prefix the line with a "rem":

rem %PROGRAMMERPATH% -c port=%COMPORT% br=115200 p=EVEN db=8 sb=1 fc=OFF -e all

Then re-run the uploader script and it seems to work quite well, instead of timing out and screwing everything up past the failed erase command, it skips the bulk erase that doesn't work anyway, and runs the next command in the uploader script which successfully erases the flash, writes the firmware, and does a verify read pass in about two and a half minutes.

After this, unplug and plug the board back in, re-upload the MCUBoot firmware, re-upload the Zephyr RTOS image, and it works exactly the same as before.  Unfortunately this firmware upgrade had zero effect on my Zephyr networking offload problems.  But at least the dev board isn't bricked anymore, and I know it has the latest wifi module firmware version installed.

Summary

If you're trying to upgrade the firmware on a Inventek ISM43362-M3G-L44 board which is SPI attached on a ST B-L4S5I-IOT01A dev board, you'll run into two problems.  There are two firmware download pages at Inventek and Google will send you to the older download page with half a decade old firmware that doesn't work with the ST uploader.  The other problem is the ST batch file to automate firmware upgrades will brick your board, although with some extremely minor editing the script will work and un-brick your board and successfully upgrade your wifi module firmware.

This firmware upgrade process, overall, was somewhat more exciting than is necessary.

Zephyr RTOS with MCUboot "USAGE FAULT" after "Jumping to the first image slot", fixed.

Subjectively, what is it like to develop embedded software?

When things go wrong, its days of feeling like your hands are wrapped in mittens while wearing a blindfold, with your primary source of documentation being a Douglas Adams novel like "Hitchhikers Guide to the Galaxy".  Or even worse a Charles Stross "Laundry Series" novel.  Then, after days of struggle, the problem is fixed!  Development rapidly proceeds at a supersonic pace.  Until impact with the next block, stuck for a couple days of zero progress.  Such is life when doing embedded / IOT software development.

Take for example, this recent three day struggle:

Consider that for more than a year, you've been using older, lower RAM STM32 dev boards like the disco_l475_iot1, a truly fine board, other than only having about 128K of ram and Zephyr needing about 127.9K to run everything other than your application code, which will only require another 128K of ram LOL.  Or a classic F767 board like the nucleo_f767zi, a wonderful board and a joy to use, other than being wired ethernet instead of wifi.  The good news is the b_l4s5i_iot01a is a massive upgrade over the disco_l475_iot1 it has around six times the ram... awesome... its the board every "starved for memory" disco_l475_iot1 developer dreams of... this should be a simple port job ... Right?  Right?  Just recompile and upload?  Its never that easy in embedded development...

Not to put the cart before the horse, however, the problem I ran into is, in a technical sense only, documented at:

https://docs.zephyrproject.org/latest/services/device_mgmt/dfu.html

Where it explains you need to set the zephyr,code-partition in your DTS overlay file using a line looking something like:

zephyr,code-partition = &slot0_partition;

However, old dev boards automatically or automagically had that set, no developer effort required.  To make it crystal clear, although it disagrees with the docs above, code for old boards (admittedly, using older versions of Zephyr...) works perfectly without setting a code-partition variable.  I just verified using the latest version of Zephyr that a venerable 'F767 compiles and boots perfectly with MCUboot, no need to manually set a code-partition setting.

What happens if you do not configure a code-partition and compile for a nice new l4s5i board?  Certainly, not any error messages upon compilation, not the slightest indication of any future problems incoming.  Run your imgtool to sign the new image with firmware keys and mark as confirmed so MCUboot will be happy, upload to your dev board at address 0x08020000 as you do on that platform, reboot, and I kid you not, this is all you have to troubleshoot the problem:

(the usual MCUboot startup messages, it recognizes my primary image as good, etc)
I: Jumping to the first image slot
E: ***** USAGE FAULT *****
E: Attempt to execute undefined instruction
(boring register dump here)
E: Faulting instruction address (r15/pc): 0x0801bca8
and it crashes out and halts the system.

Notice that the PC address where it crashes out is NOT in the zephyr firmware slot ranges, its in the MCUboot range of addresses.  Interesting!  So its "obviously" a MCUboot bug, amirite?  Not so fast...

I think I have a good MCUboot compilation and source tree, in fact if I change the "west" command to force the same zephyr source tree to compile MCUboot for my old boards, they work fine.  So, a code bug in MCUboot?  I donno, there's not much space in "jump to the first slot image" to have a bug. 

Clearly (sarcasm) it can't be a problem with my code that is configured as per my old notes and runs fine on my old boards.  MCUboot is crashing at an MCUboot program counter address ONLY on the brand new dev board and fine on the other hardware.

Try one of several dev boards?  Sure, why not.  Turns out none of them work, all fail the same way.  OK then.  So its not hardware, its not Zephyr, its crashing at a MCUboot address, its not my build system, that works fine when told to compile for other boards...  I've "proven" it has to be a MCUboot problem.  

I spent a couple days learning about the innards of MCUboot.  Fun, but not productive and it didn't fix my problem.

After some days of messing around trying everything else under the sun, as typical for embedded development, I finally checked the Zephyr docs for device management (as opposed to using my documentation and source code comments and working source code) and sometime after Zephyr version 2.2.1 and before Zephyr 3.0.0 the online Zephyr docs were changed to reflect the need for a code-partition setting (apparently only for SOME boards, as most of my boards don't require this, and certainly never required it in the past).

Add that code-partition config option and recompile and imgtool sign the binary, upload with STM32CubeProgrammer, and we're off and running, where I should have been three days ago.

Two interesting summary points:  Good luck finding anything explaining this using Google, because I certainly did not, all I had to work with was the crash address 0x0801bca8 and that isn't finding much.  Also, yes, weird as it sounds, I have proven you can crash MCUboot while its running, by giving it a Zephyr firmware image that compiled perfectly yet was incorrectly configured.

By posting this on my blog maybe Google will index it, then people who experience crashes in MCUboot at address 0x0801bca8 on the newest L4 STM32 boards while trying to run old code that runs fine on older boards, will find this explanation and fix...

I could also tell the typical embedded development story of how my spyware blocking "lie protection" software thought the Ninja build tool, used by Zephyr, is spyware so it would crash the build for fun, not bother to log any errors or other explanation, just kind of randomly crash the suspected spyware program while in mid-build.  That was so much fun.

So... yeah.  That's what people mean when they talk about the Embedded IoT software development experience.  On the other hand, it's more fun than the above blog post implies, really, it is.  It's just that the struggles are real.