Sming Host Emulator #1692

mikee47 · 2019-05-18T17:13:15Z

Initial PR providing support for building and running Sming applications directly within Host environment (Linux / Windows).

Implemented

Serial port driver emulation, based on gdbstub technique. Socket server support added to allow multiple terminal connection via telnet.
Flash memory emulated using backing file
Dummy gdbstub component (not required)
Task queues
Timers
Basic time functions
Network support for Linux and Windows

Dummy modules

Digital
Interrupts

Build

Use different build subdirectory for Linux/Windows (based on UNAME)
Add HostTests sample
- Runs some tests against ArduinoJson6 and filesystem, asserts on failure
- Remove related code from LiveDebug sample
Travis-CI
- Move script into separate build.sh file
- Host compilation requires more up to date GCC - conflict with std::isnan and std::isinf and stdc function declarations
- Build and run HostTests sample
Appveyor
- Add basic Host build

File layout changes

Move pwm.h into Components/drivers, consistent with ESP-IDF
Add esp_wifi component

Build conflicts

Replace itoa macro with function definition
Rename random() in WMath.cpp (name conflict)
#undef conflicting structure member names in gdb_syscall.h
Fix implementation of sprintf_P()

Fix IMPORT_FSTR for Linux/Windows host builds

.word only emits 2 bytes on Linux, so use .long which is fine for all 32-bit machines
Windows (COFF format) requires different implemention
Use .irom0.text, not .irom.text (fixes 'section changed' warning)

slaff · 2019-05-19T08:33:32Z

The Windows CI needs update as mentioned in #1333 (comment)

There are issues reported by Codacy that have to be addressed.

mikee47 · 2019-05-19T10:51:35Z

Appveyor build still isnt' quite right, will take a closer look later on.

I think it's simpler having a separate build.cmd but clearly build errors aren't getting reported back. There's also a strange error at line 2247:

c:/projects/sming/Sming/Arch/Esp8266/Compiler/lib\libmqttc.a: error adding symbols: Archive has no index; run ranlib to add one
collect2.exe: error: ld returned 1 exit status

We can now actively run (some) applications instead of just building, so with travis/appveyor those probably need to go in a separate section after the build. I'm fairly new to CI so perhaps best left to someone else?

mikee47 · 2019-05-20T07:34:15Z

All we need to do for C99 compliance is change asm() to __asm()__. Question is, which standard to apply - C99, C11, something else?

slaff · 2019-05-20T08:30:02Z

Question is, which standard to apply - C99, C11, something else?

I would suggest the following standards to be imposed on our own code: For C language - use C11, for C++ use C+11.
Also we should not impose our restrictions to third-party code.

slaff · 2019-05-20T08:40:18Z

@mikee47 Can you do the following change: extract only that change "Remove user_config.h file from all samples - identical to default" as a separate PR ? I would like to decrease the number of files that need to be reviewed in this PR. If you think that there is also another change that can be extracted as a separate PR, please, propose ti.

mikee47 · 2019-05-20T11:35:26Z

... extract only that change "Remove user_config.h file from all samples ...

Unfortunately this is a necessary change because it breaks the Host build - there's a lot of stuff in there which is platform-specific. If you look at the examples provided with the NON-OS SDK, the user_config.h is usually empty, so how it got quite so cluttered is strange.

slaff · 2019-05-20T11:37:45Z

Unfortunately this is a necessary change because it breaks the Host build

I was meaning to put only that change as a PR and then rebase this PR code on top of the the latest code(with merged includes-simplify PR) . But if that is too much of a hassle then leave it like that.

mikee47 · 2019-05-20T11:48:50Z

I see what you mean. That's no problem, I reckon the first 10 commits could fall into that category - everything before Initial host architecture implementation is just prep work and can be done as separate PRs. These are:

Fix Arduino Library #include paths
Fix framework #include paths
Move `Clock.cpp`, `HardwareSerial.cpp`, `System.cpp` and `WDT.cpp `from Esp8266 back into Core`
Move platform-dependent definitions out of `Core/RTC.h`
Standardise data types
Move `flashmem` into `esp_spi_flash` component
Move `pwm.h` into `Components/drivers`, consistent with ESP-IDF
Add `esp_wifi` component
Address issue of `weak` linkage
Use `os_timer*()` instead of `ets_timer*()`
Simplify and tidy Esp8266 system headers
Makefile updates

…sComplete`

…ing().c_str() Unsafe to return pointer to temporary

slaff · 2019-05-29T07:16:37Z

@mikee47 Please, merge feature/Arch-Host-DEV into this PR and rebase. The current version of feature/Arch-Host-DEV is pretty decent and I would like to allow other developers to test it, use it and improve it. Plus teach them how to use valgrind for memory leak detection, wireshark for network analysis and static code analysis for discovering security and other issues.

Implemented: * Serial port driver emulation, based on gdbstub technique. Socket server support added to allow multiple terminal connection via telnet. * Flash memory emulated using backing file * Dummy gdbstub component (not required) * Task queues * Timers * Basic time functions * Network support for Linux and Windows Dummy modules: * Digital * Interrupts Build * Use different build subdirectory for Linux/Windows (based on UNAME)

* Replace `itoa` macro with fucntion definition * Rename `random()` in WMath.cpp (name conflict) * `#undef` conflicting structure member names in `gdb_syscall.h` * Fix implementation of sprintf_P()

* `.word` only emits 2 bytes on Linux, so use `.long` which is fine for all 32-bit machines * Windows (COFF format) requires different implemention * Use `.irom0.text`, not `.irom.text` (fixes 'section changed' warning)

Add `HostTests` sample * Runs some tests against ArduinoJson6 and filesystem, asserts on failure * Remove related code from `LiveDebug` sample Travis * Move script into separate `build.sh` file * Use more recent linux distro (xenial vs. trusty) with less dated GCC - resolves conflict with `std::isnan` and `std::isinf` and stdc function declarations * Build and run `HostTests` sample Appveyor * Run mingw update/upgrade - default appveyor install is 5.3.0, current is 6.3.0 * Move build script into batch file * Add basic Host build and run `HostTests` sample

samples/HostTests/.cproject

samples/HostTests/app/test-files.cpp

Sming/Arch/Host/readme.md

.travis.yml

.travis/build.sh

slaff · 2019-05-30T09:05:59Z

running all our unit tests

Is there a decent Unit Test framework for C/C++ that is easy to install and use? Can you recommend something? Google Test and Catch2 seem quite promising but I have never used them.

mikee47 · 2019-05-30T09:58:28Z

running all our unit tests

Is there a decent Unit Test framework for C/C++ that is easy to install and use? Can you recommend something? Google Test and Catch2 seem quite promising but I have never used them.

Unfortunately my experience of formal testing frameworks is a little (lot!) out of date. I am aware that ArduinoJson uses 'catch' extensively (albeit an older version). From a casual inspection it looks simple enough, is lightweight and modern. I like the way Benoit freely adds test cases as bugs are introduced and fixed.

The ESP32-IDF uses Unity but that's more of a C framework and looks rather more complex.

Perhaps a way forward is to pick something that needs testing, devise a plan then see how well that fits a particular framework.

* This is required only for Esp8266 platform, the check is done in `Arch/Esp8266/app.mk` * Move `HttpServer_ConfigNetwork` sample `Makefile` modifications into `Makefile-user.mk`

* Move install section into `install.sh` * Run host tests in separate job, only continue to build once that has completed successfully as it should be a lot faster than an actual build * Cannot export variables from script so apply deploy condition explicitly * Define `SMING_ARCH` in `.travis.yml`, not in scripts * Move coding style checks into test stage * Use `TRAVIS_BUILD_STAGE_NAME`

* Move install into external script file

samples/Wifi_Sniffer/Makefile

Sming/Arch/Host/readme.md

slaff · 2019-05-30T13:30:41Z

I am planning to merge the PR later today. Do you think it is ready as it or you would like to add some small changes?

mikee47 · 2019-05-30T13:46:49Z

Good to go I think, unless you wanted any of the commits split out as separate PRs.

slaff · 2019-05-30T13:48:34Z

Good to go I think, unless you wanted any of the commits split out as separate PRs.

Not at the moment. Thanks a lot for your great work! I will merge the PR now and test it quickly.

separate PRs.

As a next step, as new PR, we need axTLS on Host.

slaff · 2019-05-30T14:32:09Z

.travis/build.sh

+make list-config
+
+# Build the framework
+make


We could try also parallel build splitting this command to the following two:

make submodules make -j

We could.... I gave it a brief try with appveyor but after one initial failure didn't bother any further as I figured what we really need is a clear and accurate log for when it goes wrong. I had a hell of a job with the initial CI updates on appveyor, had to remote terminal into the server to find out how to fix things so really rather not have to do that if poss.!

The --output-sync option might help but haven't tried it - I have make 3.81 under MinGW which doesn't have the option anyway. I use parallel building all the time under Windows, on an 8-core system it kind of speeds things up just a teeny bit :-)

I suspect on a VM parallel builds wouldn't help much anyway. My Linux install is running under VirtualBox and really doesn't like parallel builds at all, but that's my problem!

I suspect we'd also need lwip:

make submodules make lwip make -j

slaff · 2019-05-30T17:50:58Z

My Linux install is running under VirtualBox and really doesn't like parallel builds at all, but that's my problem!

Running make -j in $SMING_HOME is one of the few ways that make my Linux box unresponsive. So we leave the parallel build for now.

mikee47 · 2019-05-30T18:37:17Z

Running make -j in $SMING_HOME is one of the few ways that make my Linux box unresponsive. So we leave the parallel build for now.

make -j2 or -j3 seem OK...

frankdownunder · 2019-05-31T03:58:06Z

Hi guys, great work!
I thought I'd have a look at it, so I built the Basic_Serial sample but I got a buffer overrun:
Invoking 'all' for 'Host' architecture
C+ System/stringutil.cpp
C+ System/stringconversion.cpp

C+ Arch/Host/Platform/AccessPoint.cpp
C+ Arch/Host/Platform/WifiEvents.cpp
In file included from /usr/include/string.h:494,
from System/include/m_printf.h:12,
from Arch/Host/Components/esp_hal/include/esp_systemapi.h:42,
from Arch/Host/Components/esp_wifi/include/esp_wifi_types.h:8,
from Arch/Host/Components/esp_wifi/include/esp_wifi.h:3,
from Arch/Host/Platform/WifiEvents.cpp:11:
In function ‘void* memcpy(void*, const void*, size_t)’,
inlined from ‘void WifiEventsClass::WifiEventHandler(System_Event_t*)’ at Arch/Host/Platform/WifiEvents.cpp:72:3:
/usr/include/bits/string_fortified.h:34:33: error: ‘void* __builtin_memcpy(void*, const void*, unsigned int)’ forming offset [34, 36] is out of the bounds [0, 33] of object ‘log_string’ with type ‘const char [33]’ [-Werror=array-bounds]
return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from Arch/Host/Components/esp_hal/include/esp_systemapi.h:43,
from Arch/Host/Components/esp_wifi/include/esp_wifi_types.h:8,
from Arch/Host/Components/esp_wifi/include/esp_wifi.h:3,
from Arch/Host/Platform/WifiEvents.cpp:11:
Arch/Host/Platform/WifiEvents.cpp: In member function ‘void WifiEventsClass::WifiEventHandler(System_Event_t*)’:
System/include/debug_progmem.h:86:21: note: ‘log_string’ declared here
static const char log_string[] PROGMEM_DEBUG = "%u " fmt "\r\n";
^~~~~~~~~~
System/include/debug_progmem.h:107:19: note: in expansion of macro ‘debug_e’
#define debug_i debug_e
^~~~~~~
Arch/Host/Components/esp_hal/include/esp_systemapi.h:49:16: note: in expansion of macro ‘debug_i’
#define debugf debug_i
^~~~~~~
Arch/Host/Platform/WifiEvents.cpp:72:3: note: in expansion of macro ‘debugf’
debugf("station: %s leave, AID = %d", macToStr(evt->event_info.sta_disconnected.mac),
^~~~~~
In file included from /usr/include/string.h:494,
from System/include/m_printf.h:12,
from Arch/Host/Components/esp_hal/include/esp_systemapi.h:42,
from Arch/Host/Components/esp_wifi/include/esp_wifi_types.h:8,
from Arch/Host/Components/esp_wifi/include/esp_wifi.h:3,
from Arch/Host/Platform/WifiEvents.cpp:11:
In function ‘void* memcpy(void*, const void*, size_t)’,
inlined from ‘void WifiEventsClass::WifiEventHandler(System_Event_t*)’ at Arch/Host/Platform/WifiEvents.cpp:58:3:
/usr/include/bits/string_fortified.h:34:33: error: ‘void* __builtin_memcpy(void*, const void*, unsigned int)’ forming offset [31, 32] is out of the bounds [0, 30] of object ‘log_string’ with type ‘const char [30]’ [-Werror=array-bounds]
return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from Arch/Host/Components/esp_hal/include/esp_systemapi.h:43,
from Arch/Host/Components/esp_wifi/include/esp_wifi_types.h:8,
from Arch/Host/Components/esp_wifi/include/esp_wifi.h:3,
from Arch/Host/Platform/WifiEvents.cpp:11:
Arch/Host/Platform/WifiEvents.cpp: In member function ‘void WifiEventsClass::WifiEventHandler(System_Event_t*)’:
System/include/debug_progmem.h:86:21: note: ‘log_string’ declared here
static const char log_string[] PROGMEM_DEBUG = "%u " fmt "\r\n";
^~~~~~~~~~
System/include/debug_progmem.h:107:19: note: in expansion of macro ‘debug_e’
#define debug_i debug_e
^~~~~~~
Arch/Host/Components/esp_hal/include/esp_systemapi.h:49:16: note: in expansion of macro ‘debug_i’
#define debugf debug_i
^~~~~~~
Arch/Host/Platform/WifiEvents.cpp:58:3: note: in expansion of macro ‘debugf’
debugf("ip: %s, mask: %s, gw: %s", ip.toString().c_str(), mask.toString().c_str(), gw.toString().c_str());
^~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [/opt/Sming/Sming/modules.mk:7

mikee47 · 2019-05-31T06:03:47Z

Compiler's a lot fussier (a good thing!). Try this:

diff --git a/Sming/Arch/Host/Platform/WifiEvents.cpp b/Sming/Arch/Host/Platform/WifiEvents.cpp
index 95f525b3..f5b3f3be 100644
--- a/Sming/Arch/Host/Platform/WifiEvents.cpp
+++ b/Sming/Arch/Host/Platform/WifiEvents.cpp
@@ -23,5 +23,5 @@ WifiEventsClass::WifiEventsClass()
 }
 
-static String macToStr(const uint8_t mac[])
+static String macToStr(const uint8_t mac[6])
 {
 	return makeHexString(mac, 6, ':');

Could you do a gcc --version as well please?

frankdownunder · 2019-05-31T06:25:25Z

gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
I tried the fix, still complains

mikee47 · 2019-05-31T06:51:32Z

OK, try this:

diff --git a/Sming/Wiring/FakePgmSpace.h b/Sming/Wiring/FakePgmSpace.h
index 6d75fc63..cdabab52 100644
--- a/Sming/Wiring/FakePgmSpace.h
+++ b/Sming/Wiring/FakePgmSpace.h
@@ -237,5 +237,5 @@ extern "C"
 #define LOAD_PSTR(_name, _flash_str)                                                                                   \
 	char _name[ALIGNUP(sizeof(_flash_str))] __attribute__((aligned(4)));                                               \
-	memcpy_aligned(_name, _flash_str, sizeof(_name));
+	memcpy_aligned(_name, _flash_str, sizeof(_flash_str));
 
 #define _FLOAD(_pstr)                                                                                                  \

The issue arises because LOAD_PSTR uses the aligned length, which is longer than the string stored in flash. (This optimisation is to avoid copying byte-by-byte on the Esp which is inefficient.)

I'll need to think about a proper fix for this...

mikee47 · 2019-05-31T07:06:38Z

An alternative which won't break anything is adding -Wno-array-bounds -

diff --git a/Sming/build.mk b/Sming/build.mk
index f5e7c890..d98ed380 100644
--- a/Sming/build.mk
+++ b/Sming/build.mk
@@ -119,3 +119,3 @@ CFLAGS_COMMON	= -Wl,-EL -finline-functions -fdata-sections -ffunction-sections
 # compiler flags using during compilation of source files. Add '-pg' for debugging
-CFLAGS			= -Wall -Wundef -Wpointer-arith -Wno-comment $(CFLAGS_COMMON) \
+CFLAGS			= -Wall -Wundef -Wpointer-arith -Wno-comment -Wno-array-bounds $(CFLAGS_COMMON) \
          			-DARDUINO=106 -DENABLE_CMD_EXECUTOR=$(ENABLE_CMD_EXECUTOR) -DSMING_INCLUDED=1

mikee47 · 2019-05-31T07:08:33Z

From https://gcc.gnu.org/gcc-8/changes.html

The -Warray-bounds option has been improved to detect more instances of out-of-bounds array indices and pointer offsets. For example, negative or excessive indices into flexible array members and string literals are detected.

mikee47 · 2019-05-31T07:17:43Z

I'll open an issue for this

mikee47 changed the title ~~feature/Sming Host Emulator~~ Sming Host Emulator May 18, 2019

slaff added this to the 3.9.0 milestone May 19, 2019

mikee47 force-pushed the feature/Arch-Host branch from c82c9c1 to 513be2c Compare May 19, 2019 16:31

mikee47 added 5 commits May 24, 2019 18:40

Remove duplicate debug statement from `HttpServerConnection::onHeader…

89e8f8d

…sComplete`

(char*)getRemoteIp() output garbled, replace with getRemoteIp().toStr…

7d37668

…ing().c_str() Unsafe to return pointer to temporary

Move pwm.h into Components/drivers, consistent with ESP-IDF

c751372

Add esp_wifi component

d434850

WifiEvents - move trivial methods into header

17d1c40

mikee47 force-pushed the feature/Arch-Host branch 2 times, most recently from 79c880c to c01ebb2 Compare May 24, 2019 19:26

slaff mentioned this pull request May 25, 2019

Port Sming to esp32/Arduino #1333

Closed

3 tasks

slaff added the 3 - Review label May 29, 2019

mikee47 force-pushed the feature/Arch-Host branch from 01ad266 to 764e290 Compare May 29, 2019 09:29

mikee47 added 4 commits May 29, 2019 10:39

Fix build conflicts

b9c918f

* Replace `itoa` macro with fucntion definition * Rename `random()` in WMath.cpp (name conflict) * `#undef` conflicting structure member names in `gdb_syscall.h` * Fix implementation of sprintf_P()

Fix IMPORT_FSTR for Linux/Windows host builds

fb599d7

* `.word` only emits 2 bytes on Linux, so use `.long` which is fine for all 32-bit machines * Windows (COFF format) requires different implemention * Use `.irom0.text`, not `.irom.text` (fixes 'section changed' warning)

mikee47 force-pushed the feature/Arch-Host branch from 764e290 to ad14726 Compare May 29, 2019 09:39

slaff requested changes May 29, 2019

View reviewed changes

samples/HostTests/.cproject Outdated Show resolved Hide resolved

samples/HostTests/app/test-files.cpp Outdated Show resolved Hide resolved

Sming/Arch/Host/readme.md Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

.travis/build.sh Outdated Show resolved Hide resolved

mikee47 added 5 commits May 30, 2019 13:08

Remove ESP_HOME check from sample Makefile-user.mk

831bf2b

* This is required only for Esp8266 platform, the check is done in `Arch/Esp8266/app.mk` * Move `HttpServer_ConfigNetwork` sample `Makefile` modifications into `Makefile-user.mk`

Create tests directory and move HostTests sample

e39ab67

Revise Appveyor script

cf6d327

* Move install into external script file

Fix codacy advisories

17473c0

slaff reviewed May 30, 2019

View reviewed changes

samples/Wifi_Sniffer/Makefile Show resolved Hide resolved

slaff reviewed May 30, 2019

View reviewed changes

Sming/Arch/Host/readme.md Outdated Show resolved Hide resolved

mikee47 added 2 commits May 30, 2019 14:16

Update readme.md - interfaces aren't persistent

f3a72bc

Remove extraneous ESP_HOME check in Arch/Esp8266/app.mk

0602d7b

slaff approved these changes May 30, 2019

View reviewed changes

slaff merged commit a8ce067 into SmingHub:develop May 30, 2019

slaff removed the 3 - Review label May 30, 2019

slaff reviewed May 30, 2019

View reviewed changes

mikee47 deleted the feature/Arch-Host branch May 30, 2019 19:45

mikee47 mentioned this pull request May 31, 2019

Array bounds warning building Host with GCC 8 #1706

Closed

slaff mentioned this pull request Jul 1, 2019

Preparation for Release 4.0.0 #1674

Closed

4 tasks

mikee47 mentioned this pull request Nov 7, 2019

Suggestion for Code Quality and CI tools #1718

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sming Host Emulator #1692

Sming Host Emulator #1692

mikee47 commented May 18, 2019 •

edited

Loading

slaff commented May 19, 2019 •

edited

Loading

mikee47 commented May 19, 2019 •

edited

Loading

mikee47 commented May 20, 2019

slaff commented May 20, 2019

slaff commented May 20, 2019

mikee47 commented May 20, 2019

slaff commented May 20, 2019

mikee47 commented May 20, 2019

slaff commented May 29, 2019

slaff commented May 30, 2019 •

edited

Loading

mikee47 commented May 30, 2019

slaff commented May 30, 2019

mikee47 commented May 30, 2019

slaff commented May 30, 2019 •

edited

Loading

slaff May 30, 2019

mikee47 May 30, 2019 •

edited

Loading

slaff commented May 30, 2019

mikee47 commented May 30, 2019

frankdownunder commented May 31, 2019

mikee47 commented May 31, 2019 •

edited

Loading

frankdownunder commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 31, 2019

Sming Host Emulator #1692

Sming Host Emulator #1692

Conversation

mikee47 commented May 18, 2019 • edited Loading

slaff commented May 19, 2019 • edited Loading

mikee47 commented May 19, 2019 • edited Loading

mikee47 commented May 20, 2019

slaff commented May 20, 2019

slaff commented May 20, 2019

mikee47 commented May 20, 2019

slaff commented May 20, 2019

mikee47 commented May 20, 2019

slaff commented May 29, 2019

slaff commented May 30, 2019 • edited Loading

mikee47 commented May 30, 2019

slaff commented May 30, 2019

mikee47 commented May 30, 2019

slaff commented May 30, 2019 • edited Loading

slaff May 30, 2019

Choose a reason for hiding this comment

mikee47 May 30, 2019 • edited Loading

Choose a reason for hiding this comment

slaff commented May 30, 2019

mikee47 commented May 30, 2019

frankdownunder commented May 31, 2019

mikee47 commented May 31, 2019 • edited Loading

frankdownunder commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 31, 2019

mikee47 commented May 18, 2019 •

edited

Loading

slaff commented May 19, 2019 •

edited

Loading

mikee47 commented May 19, 2019 •

edited

Loading

slaff commented May 30, 2019 •

edited

Loading

slaff commented May 30, 2019 •

edited

Loading

mikee47 May 30, 2019 •

edited

Loading

mikee47 commented May 31, 2019 •

edited

Loading