Various Sming fixes and features tasklist

[ADiea]

Hi everyone,

I started to port my work branch to main Sming repo. Because NOOS was not maintained for ~1year a lot of stuff remained on my local fork. I have reviewed my commits and I want to place all intended changes here as a task list for easy reference and will make multiple PRs in the future referencing this issue.
Of course some things in the list might not be applicable anylonger will compare and see for each one.

Feel free to comment, express your priorities etc, I hope in ~1-2 weeks it will all be done :)

Pending

• Possible readd connection->flush() in WebSocket::send() (removed due to instability, might be fixed) (improves latency)
• Improvements for Si4432 RF module library (bugfix)
• Memleak debug option by printing mem operations to console + use external tool (memleak detection)
• Introduce optional delegates for ChipSelection on SPI protocol (flexibility)
• Possible SDCard library fixes (will compare and see)
• Add SoftAP dhcp (will compare and see if applicable)
• Add lib for I2c IO extender PCF8574
• Add lib for Nextion displays

In progress

Finished

• Possible other SSL fixes, (will compare and see) -> in progress: PRs upstream (35 36 37 38 40)
• not needed Fix a leak in SSL axtls-8266/compat/lwipr_compat.c (bugfix) (was freeing vector->data first in ax_fd_init() but this is now already checked in calling function ax_init by checking data and capacity before initing)
• [LWIP] OPTS: reduce ram usage, improve tcp speed, cleanup #1001 LWIP review lwiopts.h (discuss better default options if applicable) (increase heap)
• Directly use os_random is webSocket #896 Set random seed with time at startup (better randomness) not needed, os_random is based on hardware rng and used throughout the project
• Modify image size and start address in ld script. (increased irom section) nothing to do see below comment
• Fix WebSocket key compare for some servers #897 WebSocketClient compare key in lowercase (bugfix)
• Possible some modifications on crash handler/postmortem (will compare and see) nothing to do original commit
• Make default debug uart speed configurable. #903 Modify UART speed from 115200 to 921600 (faster printing to console, bigger ammounts of logging possible like memleak)
• Implement m_vprintf for usage in axtls debugging and in the future already present in upstream, nothing to do
• Automatic store all debugf format strings in flash #927 Implement convenient way to move char* debug strings from global ram section to irom section (a lot of ram savings without sacrificing debug strings)
• Add SetLastKnownTime to systemclock.cpp to set a last known time before NTP time is available. nothing to do on closer look, application can use directly RTC to getRtcSeconds
  store and later setRtcSeconds
• Add software UART implementation: not ready, will not add due to availability of hardware UART1. A software UART will take up alot of interrupt time and possibly cause system instability if not used right so it's risky
• Guard CommandExecutor and CommandHandler #1014 Guard CommandExecutor to be able to exclude from build if not used/needed. (increase IROM/heap)

[slaff]

@ADiea It would be great to see all of those changes being part of the upstream Sming version. I can suggest the bug fixes to be the first PRs, followed by improvements related to memory (memleaks, heap increase, etc ) and finally the new features.

For the UART changes I am trying to rewrite Sming to "borrow" from Arduino's HardwareSerial (see https://github.com/slaff/Sming/tree/fix/uart1) so if you want wait with your UART changes a bit.

[ADiea]

Please go on with uart1 support because software_uart will kill the interrupts it's better to avoid it if we have alternative. Then we can switch debugging to uart1 and use uart0 for communicating to other devices (in my case a nextion display)

[slaff]

@ADiea I guess the time has come to cleanup the debug-in-flash branch and create a PR out of it so that we can start testing it. Also create another PR for "Modify image size and start address in ld script. (increased irom section)".

[ADiea]

I have a issue here: it's section conflict.

The feature works like this: from makefile I set a CUSTOM_BASE_FILE with filename and in header I declare LOG function as macro that defines a static global log_CUST_FILE_NAME_LINE_NUMBER like

static log_WebSocketClient33[] attribute section irom = "debug value is %d";

The problem is that for ex in Sming/SmingCore/Network/FTPServerConnection.cpp there are several classes. When debugf is replaced by static variable creation in irom section gcc will complain of section type conflict.
If I don't force the section of the variables created the compiler will put static global variables for each class in a different section for one translation file.

So one way is to make sure we don't have more than one class in a file and refactor the FTP file, or keep digging for a solution out of this I was thinking maybe a pragma to disable this error exists...

The benefit is huge it saved me a lot of RAM but I still need to find a fool-proof solution for this gcc error...

[slaff]

@ADiea are you ready to propose as PR those changes ?

• Fix a leak in SSL axtls-8266/compat/lwipr_compat.c (bugfix)
• Possible other SSL fixes, (will compare and see)
• Memleak debug ... + use external tool (memleak detection)

[ADiea]

I will give them priority. I think the SSL leak is already fixed in current version because there is a check now when making buffer realloc in lwipr_compat.c
Will start diffing!

[ADiea]

Regarding: Modify image size and start address in ld script. (increased irom section)

I was using these values for iram and irom sections, to allow more functions into the IRAM section

  iram1_0_seg :                         org = 0x40100000, len = 0xF000
  irom0_0_seg :                         org = 0x4020F000, len = 0x68000

Right now in standalone.rom.ls these are the values

  iram1_0_seg :                         org = 0x40100000, len = 0x8000
  irom0_0_seg :                         org = 0x4020a000, len = (1M - 0x0a000)

According to memory map range from 40108000 to 4011000 should not be used. However in my tests I have used if (that is the first image was bigger than 0x8000 and it loaded and executed correctly).

But because this is not documented properly. probably it should only be used for experimentation and not for production.

[slaff]

Haven't tested this but I guess you can disable the RAM cache by calling early enough Cache_Read_Enable(0, 0, 0) and then you could change the length from 0x8000 to 0x0C000.

[slaff]

@ADiea May be we can add to this release also that feature "Guard CommandExecutor to be able to exclude from build if not used/needed. (increase IROM/heap)" if you submit PR until Wednesday (8th March, 2017). If will align nicely with the other heap improvements that you have contributed.

[ADiea]

I have just received my projects's PCBs this weekend so was a little busy soldering :) Right now my priority is SPI modifications in order to have functionality on my board But I will also try to guard CommandExecutor because it's a simple PR.

[slaff]

@ADiea how about adding that change "Possible SDCard library fixes (will compare and see)" ?

[ADiea]

My current hardware is like this: I have a IO expander that controls Chip select lines for SD card and for another spi device - because I have too little io pins for all the stuff I need to do... Thus I have added a extension to spi base class, a chip select delegate possibility to control cs from application side.

Normally in Arduino world application is responsible for chip select spi class does nothing. But we have the SDCard code witch is an application on top of spi class, so the upper layer application only opens files on the sdcard it does not know when to chip select the sdcard...

I kept thinking what is the best approach to make a PR, I will start with the i2c io expander PR (PCF8574).
Plus sample

Then made SDCard modifications to allow flexibility to chip select using delegates from application code or to just define a CS pin like it is currently done (default)
Plus sample SD card using CS through I2Cexpander... this is exotic but maybe somebody else will find it useful.

[slaff]

Memleak debug option by printing mem operations to console + use external tool (memleak detection)

@ADiea How about adding the above described feature?

[ADiea]

Have not done ESP related dev in a while because my project/idea is on hold... for now I'll keep just an eye maybe help with reviews etc. That "feature" is not something very complicated it is something like this:

1. keep early program malloc / free details in a list until they can be printed
2. after the system is inited and debugf works print the buffer and then print every malloc normally
3. Print format is something like {m,123312, 12} meaning malloc 12 bytes at address 123312
4. Use javascript heap viewer to manually inspect mallocs - this is the tedious work...