heap and memory question ?

[Zaltora]

-official esp8266 RTOS sdk use heap_5.c memory management for freertos. esp-open-rtos too or another model ?

• "malloc" and "free" will allocate memory from the task heap (inside) ? or oustide of the task ?
• if a task "malloc" memory, other task can manipulate this memory ?
• i got a situation : one task "malloc" memory, send memory ptr with a queue to another task and this task after treatments "free" this memory. the memory is "free" truly ?

I got some memory problem.

[Zaltora]

if i do this:
#define "MY TEXT HERE"
and use it in my function, the text is on ROM memory or RAM ?
when i do this:
const char* text = "MY TEXT HERE"
The text is in ROM ?
I call multiple function with text and think the compiler will stock these text in ROM. its good ?
if not, can i force put them in ROM ?

[Zaltora]

#define configTOTAL_HEAP_SIZE ( ( size_t ) ( 32 * 1024 ) )

Max heap can be change ? i remenber, that an update of linker from @ourairquality save a lot of RAM. (Maybe not the same RAM)

[Zaltora]

Something interesting, the official RTOS SDK got ~70KB heap
SDK version:1.5.0-dev(a8c4880)
Heap :69984

[Zaltora]

I think that espressif use Instruction RAM/FLASH cache ram to get more heap.
esp8266 memory map
When we look last commit of them:

1. Change cache size from 32KB to 16KB, reverse this 16KB region as heap;

Someone know how to disable cache for this region: 0x40108000 ? if we can do this, it will allow 16KB heap more. (if we not change other things).

#define configTOTAL_HEAP_SIZE ( ( size_t ) ( 48 * 1024 ) )

[Zaltora]

After compare DPORT register for both SDK:
esp-open-rtos:
DPORT SPI_CACHE_RAM 3FF00024 0000001A
esp8266 rtos sdk:
DPORT SPI_CACHE_RAM 3FF00024 0000000A

Maybe we just need to setup this register to get more 16Kb more heap.

[ourairquality]

There are some clues in the include files:

/* Details for SPI_CACHE_RAM register */
#define DPORT_SPI_CACHE_RAM_BANK1  BIT(3)
#define DPORT_SPI_CACHE_RAM_BANK0  BIT(4)

So bit 4 has changed to be zero, and the include file suggests this disables bank 0 of the spi cache ram. Bank zero is probably 0x40108000 to 0x4010c000. They use the FreeRTOS heap5 allocator which can handle separate regions, and it seems to use a region that starts at the end of the used iram (and so also uses the remaining iram) and runs to 0x4010c000 (the end of bank 0).

The comments suggest some limitations using this cache ram, that it can not be a TX buffer, or task stack. The TX buffer limitation might be a hardware issue. Not sure why the task stack could not use this area.

Perhaps another option would be to use it as iram, to have the linker use 0x40100000 to 0x4010c000 for iram, to be able to link some code there if an app really needed that for speed etc.

Perhaps Cache_Read_Enable sets up this register, and we would need a custom version.

[Zaltora]

After do some test, i valid: Cache_Read_Enable setup the cache register. Unfortunately, this function must initialize or set other parts that I do not know.

They use heap5 and get more heap like 72Kb setting. They use more ram region to get this value.(where ?) . But if the RAM/CACHE is following the 32K ram region, We need really to change heap system ? or can we just increase heap size. (and linker update )

[Zaltora]

i found this
cache enable C version
infos:
function info

[Zaltora]

After replace Cache_Read_Enable function by the C version above and replace
Cache_Read_Enable(0, 0, 1); by Cache_Read_Enable(0, 0, 0); everywhere.
I got a succes !
DPORT SPI_CACHE_RAM 3FF00024 0000000A
The original version function didn't work when we set the last arg at 0.

[ourairquality]

Perhaps it is not necessary to replace Cache_Read_Enable rather as you did just replace the third argument with 0, and perhaps that could be done in core/spiflash-cache-enable.S by changing the a4 value passed to the ROM function Cache_Read_Enable to zero.

[Zaltora]

When i test the original function, put the third arg at 0 didn't change the register.

	/* map the first megabyte of flash */
	movi a2, 0
	movi a3, 0
	movi a4, 0

I clean and rebuild but no effect with core/spiflash-cache-enable.S too.

[ourairquality]

The attached patch appears adequate here. With this it was possible to use 0x40108000 to 0x4010c000 as scratch ram in a simple test.
patch.txt

[ourairquality]

However only aligned 32 bit memory read operations appear to work in this region. Even aligned 8 and 16 bit write operations fault. There does appear to be code in the exception handlers to deal with this, but it appears to only implement the read opcodes. So do we need to extend that exception handler to deal with store opcodes too, or is there some hardware workaround? If only 32 bit operators are efficient in this region then it might have an even bigger performance hit.

[vlad-ivanov-name]

Perhaps it's connected as instruction RAM (which only can be accessed by 32-bit store/loads)? And when it's used as cache, an external cache controller is used (otherwise it's hard to imagine how can it work both as SPI cache and RAM)

[Zaltora]

After test, Heap is on "dram0" size. dram : 81920 . After printf "sdk_system_print_meminfo" (it is buggy, i will fix it). i got this information
data : 0x3ffe8000 ~ 0x3ffe8964, len: 2404
rodata: 0x3ffe8968 ~ 0x3ffe9258, len: 2288
bss : 0x3ffe9258 ~ 0x3fff1500, len: 33448
heap : 0x3fff1500 ~ 0x3fffc000, len: 43776
//
i guess espressif use heap_5 to add iram1:
iram1 : 32768 =>Instruction RAM (40100000h - 40108000h)
?Seg0 : 16384 =>RAM/FLASH CACHE (40108000h - 4010C000h)

//Limitation of iram is do not allocate task and TX buffer. (So we can use it for queue,buffers,stream...)

[Zaltora]

RTOS ESPRESSIF SDK information:
data : 0x3ffe8000 ~ 0x3ffe8850, len: 2128
rodata: 0x3ffe8850 ~ 0x3ffe8d44, len: 1268
bss : 0x3ffe8d48 ~ 0x3ffef990, len: 27720
heap : 0x3ffef990 ~ 0x40000000, len: 67184

interesting map of heap. They use additionnal 4K from this segment
ETS system data RAM (0x3FFFC000 - 0x40000000)

i think they use only this part of iram as heap too:
?Seg0 : 16384 =>RAM/FLASH CACHE (40108000h - 4010C000h)

[Zaltora]

@ourairquality Ty for your patch, it is work well.
was looking to add last 4K Dram segment.
we need change ld script
dram0_0_seg : org = 0x3FFE8000, len = 0x14000
to
dram0_0_seg : org = 0x3FFE8000, len = 0x18000

I dont know what is the process to allocate memory. The change above don't change the size of heap. I guess it is "malloc" from libc that allocate memory and know the limit and we need recompile libc ?

pvPortMalloc = malloc;
vPortFree = free;

[ourairquality]

Much of that top 4k is still used, it is just not made available to the linker to use. The malloc area starts from the _heap_start defined by the linker and ending at the xPortSupervisorStackPointer defined at startup which is in that top 4k, so malloc can use what is not otherwise used.

Not sure what is above that. Tried writing over the area above the xPortSupervisorStackPointer and checking if it changes and it does not appear to be used.

Assuming it is the startup stack, then once scheduling is started it is probably no longer needed and perhaps the available heap could be increased to 0x40000000 rather that just the SP when scheduling was started?

Would anyone know what that top area of RAM is used for and if we could malloc over it?

[ourairquality]

Here's a quick patch to make all of the top area of RAM available for allocation once scheduling has started. This assumes that the area was all the start up stack and is no longer needed - an assumption not checked!
heap-end-patch.txt

[Zaltora]

From RTOS sdk:

 NEW VERSION: 1.3.0
17. Modify "ld" files, "dram0 len" should be 0x18000 in RTOS SDK.

Why they said "should be" ?
Your changes got effect. I do a program to malloc memory template until memory is full. and write a template into this memory. I got no crash. Maybe is a good news .
But total heap is only increase of 1000?
Before:

MAIN: Heap: 40588
heap : 0x3fff1520 ~ 0x3fffc000, len: 43744

Now:

MAIN: Heap: 41868
heap  : 0x3fff1520 ~ 0x40000000, len: 60128

Its maybe normal. I didnt know well how it is work.

[Zaltora]

the first heap is value from xPortGetFreeHeapSize()
the second heap value from sdk_system_print_meminfo() (I got a fixed one, the original is buggy, miss &)

[Zaltora]

I found three interesting article about memory and FreeRTOS / Newlib.
FreeRTOS heap 5
FreeRTOS newlib
newlib integration FreeRTOS

the esp-open-rtos project use the memory scheme of newlib ? f it is the case, why do not use heap_5 from Freertos ?

I use PR #519 and #523 in my project. I go not memory problem for now.

[Zaltora]

I recently update esp-open-rtos with lock implementation than use "configUSE_RECURSIVE_MUTEXES". This implementation use 1k heap.
lock functions are used by printf and others functions that i don't know.
Can we get a preprocessor option to disable recursive mutexes (swap with old system ?) without cause troubles and disable functions usage like printf.