Handle strings in the sketch - heap problem.

[robertgregor]

Hello, it seems, that all the strings used in the sketch are going to the heap. I have a sketch which is using ESP8266WebServer. In each handler, I am using the Strings to output. Now I've reached the situation, that if I put any more Strings in the sketch, the heap is full. It is very strange, thus based on the hint I am now doing this:
static const char XSD_SW_1[] = "blablba-bigstring";

Then in the handler method:
server.send(200, "text/xml", "");
WiFiClient client = server.client();
client.write(&XSD_SW_1[0], sizeof(XSD_SW_1)-1);

But looks like that that const is a part of the heap, because if I will do XSD_SW_1[] = ""; then everything is OK. If I leave big - aroung 1400 characters - maximum buffer of the Webserver write method, my sketch doesn't work at all - seems, it will even not start correctly.
Is there any way, how to tell to compiler, that the strings should stay on flash and should be read, when it is needed? In arduino, there is for this purpose the F() macro or PROGMEM directive. But I belive, this is avr specific.

[Makuna]

I did a little investigation on this. While it is possible to move strings off the heap, the attribute to do this is not compatible with the use like F() nor PSTR(). It requires that target be global in scope and a variable.

will work

PROGMEM const char titleString[] = "Title of my app";

setup() {
...
serial.print(titleString);
}

will not

setup() {
...
serial.print(F("Title of my app"));
}

[Makuna]

I now have a grasp on how to fix it. Its going to take a bit as a lot of original Arduino abstraction for it was removed from the code base and I have to add it all back to get it to work.

[igrr]

@Makuna could you please drop into our chat at https://gitter.im/esp8266/Arduino to discuss the changes required?

[robertgregor]

Hello,
for me it doesn't really work. Everything seems is going to the stack:
I have:
PROGMEM const char WSDL[] = "abcd\n";
void setup() {
Serial.write(&WSDL[0], sizeof(WSDL)-1);
}
Output is:
abcd
Free heap:32568

Then I will change:

PROGMEM const char WSDL[] = "/webservice/SimpleSwitch?xsd=1">/xsd:import/xsd:schema<message name="switchOnFor3Minutes"><part name="parameters" element="tns:switchOnFor3Minutes">.........................Long string";
void setup() {
Serial.write(&WSDL[0], sizeof(WSDL)-1);
}

Output:
/webservice/SimpleSwitch?xsd=1
Free heap:31704

So there should be some way, how to store the bytes to the flash and read it directly from flash. Especially for the Webserver.

[Makuna]

I have solution. It is in progress and a pull will be posted with the fixes. The current PROGMEM just mapped to ram.

It will work just like Arduino does today, storing strings into the instruction ".irom.text" section.

It will support F(), PSTR(), PROGMEM, Print extensions, String extensions, and str*_P methods.

[Makuna]

#236 pending with fixes

[Makuna]

this was merged in

[tontito]

Work:
client.write(F("
Password<input type="password" name="pass" placeholder="Example: ">"));

Not Work:
client.write(F("
Password<input type="password" name="pass" placeholder="Example: ">") + F("12345678"));

In file included from C:\Program Files (x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266/Arduino.h:212:0,
from C:\Program Files (x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src/WiFiClient.h:24,
from C:\Program Files (x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src/ESP8266WiFi.h:32,
from sketch_may14a.ino:1:
sketch_may14a.ino: In function 'void setup()':
C:\Program Files (x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266/WString.h:38:95: error: invalid operands of types 'const _FlashStringHelper' and 'const _FlashStringHelper' to binary 'operator+'
#define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))
^
sketch_may14a.ino:10:128: note: in expansion of macro 'F'
Derleme sırasında hata oluştu.

[Makuna]

This is invalid syntax and use of F(). It will not compile when targeting AVR either.

F is not a string class that does string manipulation, use String for that. It is simply a way to annotate string literals so they are stored in rom rather than ram.

The simple fix to your example is

client.write(F("Password12345678"));

But I suspect there is more to your case than your example? The following will work and is supported.

client.write( String(F("Password")) + F("12345678") );

[tontito]

Ok thanks. I fixed it

i have define to FLASH ROM
#define webcss F("body{font-family:Trebuchet MS;font-size:16px;-webkit-font-smoothing:antialiased}h1{text-align:center;color:#00f}")

error: invalid conversion from 'const __FlashStringHelper*' to 'uint8_t {aka unsigned char}' [-fpermissive]

How do I do that

[Makuna]

how are you using webcss? What are you passing it to? Provide the code for the call.

please read up on the differences between F() and PSTR() in the Arduino documentation.

[tontito]

webss is my css file send client.

ok now I'm looking at. thank you so much.

[robertgregor]

Hello, great work!!! Working well - PROGMEM and also F() macro. I have now a lot of stack free:))))
Thanks for fixing that!

[robertgregor]

Just a question: I know, that AVR is doing this: If I have two same strings in the sketch will it occupy only one space in the program flash? I.e.:
................
F("abcd");
............
............
F("abcd");

Will this occupy 5 bytes or 10 bytes? In case of AVR and arduino, it will take only 5 bytes (4+1 null terminator.)

[robertgregor]

Hello, seems, something is wrong. I have modified my sketch to use everywhere the F() macro, and when I tried to compile, the compiler freeze and never finish:

C:\RH\Arduino1.6.4/hardware/tools/esp8266/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -IC:\RH\Arduino1.6.4/hardware/tools/esp8266/sdk//include -c -Os -mlongcalls -mtext-section-literals -fno-exceptions -fno-rtti -std=c++11 -MMD -DF_CPU=80000000L -DARDUINO=10604 -DARDUINO_ESP8266_ESP01 -DARDUINO_ARCH_ESP8266 -DESP8266 -IC:\RH\Arduino1.6.4\hardware\esp8266com\esp8266\cores\esp8266 -IC:\RH\Arduino1.6.4\hardware\esp8266com\esp8266\variants\generic -IC:\RH\Arduino1.6.4\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src -IC:\RH\Arduino1.6.4\hardware\esp8266com\esp8266\libraries\ESP8266WebServer\src -IC:\RH\Arduino1.6.4\hardware\esp8266com\esp8266\libraries\EEPROM C:\Users\gregorro\AppData\Local\Temp\build3042939131809121947.tmp\SimpleSwitch.cpp -o C:\Users\gregorro\AppData\Local\Temp\build3042939131809121947.tmp\SimpleSwitch.cpp.o

What could be wrong???

[Makuna]

How much memory are you using for strings?

[robertgregor]

What do you mean? You mean how long the strings are in the sketch? The size of the ino file is around 64 kb, so the strings could have around 30 Kb.

[Makuna]

Unfortunately the tool chain does not seem to "pool" constant string instances. Every definition will create and consume flash memory. So it seems for now that you need to manually make sure you have no duplicates.

[Makuna]

@robertgregor could you copy a single line of code each of how you define a string and how you use it?

[Makuna]

I have attached a Ino that will cause the compiler to "puke". In the middle of all those prints, there is a comment that if you comment out a line, it works (clean load of the file). So that is the edge of memory size that causes problems I suspect.

the compile just endlessly repeats this with verbose compile option turned on.

at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4604)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4785)

[igrr]

This isn't the compiler, looks more like the Java preprocessor part has failed (the one which makes a .cpp out of .ino).
What happens if you add a .cpp file to your sketch and move your code there?

[Makuna]

@igrr You are correct. If I move the strings into a cpp, the issue that was demonstrated with the compiling seemly hanging is gone. Further, if I create enough strings, I will run into a nice error message about exceeding flash. So all is good.

...\AppData\Local\Temp\build9026207628654039897.tmp/ProgmemSizeTests.cpp.elf section `.irom0.text' will not fit in region `irom0_0_seg'
collect2.exe: error: ld returned 1 exit status
Error compiling.

[robertgregor]

HelIo, it seems that this construction hangs the compiler or preprocessor:
html += (i==1)?F("selected"):"";

[Makuna]

@robertgregor What type is html?

[robertgregor]

html is String object

[Makuna]

@robertgregor Please enter a unique issue for this problem; I can repro it.
The issue at hand is that the compiler never completes when it should be displaying an error. If you compile this same code for an AVR, you get...

StringTest.ino:61:46: error: conditional expression between distinct pointer types 'const __FlashStringHelper*' and 'const char*' lacks a cast [-fpermissive]
StringTest.ino:61:15: error: ambiguous overload for 'operator+=' (operand types are 'String' and 'const void*')

That line should read...

html += (i==1)?F("selected"):F("");

[sticilface]

I can get F() to work inside functions and my heap goes right down. I can't get it to work on a global variable though.

I've tried

static const String simplepage3 = "..."; which just goes on heap
any variations of PROGMEM just cause constant rebooting fail at start.

could someone give an example of how to store a large global string in flash?

appreciated.:)

[Makuna]

Please list the failure case example code.
Also, what build are you using.

[sticilface]

1.6.1-esp8266-1-1055-gcf89c32

ok so i can't reproduce the crashing bit not sure what i did there.
I can use
PROGMEM const char xyz[] = "......." to send global array to flash, then when i need to use it i String(xyz), is there no way to send global string to flash as a global and outside of a function? Sorry for so many noob questions here.. this is all so new to me!

[sticilface]

although i get a reset if i try String buf = Strin(xyz)

if i use Serial.write(&xyz[0], sizeof(xyz)-1); nothing happens

me-no-dev, suggested that i use const char * test = "<html>....</html>"; for my strings, but i did some testing by making loads of them, and they all consume heap. So how do i make a global string go to flash. I've got a function that subs in variables, and it works nicely on strings but not char arrays... as it uses indexOf() and +.. which doesn't work with char.. the F macro does work, but only in a function. should i define a function that returns the F() macro's string?

[Makuna]

Don't use the String class if you want avoid heap and use str*_P methods if you are using PSTR or PROGMEM or __FlashStringHelper if you are using F(). The String class copies everything into heap.

Make sure you understand the difference between a C/C++ string and a String class.

Please review the Arduino reference for progmem, PSTR, F.

Again, minimum but complete code examples help.

The Serial.write() method doesn't work with flash strings. Serial.print will if you define your flash string the way it wants.

static __FlashStringHelper xyz = F("..."); // define the string for use with methods that take F("")

Serial.print(xyz);  // print supports F(), so it will consume the above

[hallard]

Guys,
I tried to do some flash string optimization on my sketch, I know and checked that this works fine

• Serial.print(F("BlaBla"));

but as soon as I'm trying to do some printf() with flash, with F() it's not compiling and with PSTR() it compile but crash the sketch at runtime

Did I forgot something ?

Here complete example

 // This is working fine
  Serial1.print(F("valueRemoveFlagged(0x"));
  Serial1.printf("%04X", flags);
  Serial1.println(F(")"));

  // this is make ESP in reset at execution time;
  Serial1.printf(F("valueRemoveFlagged(%04X)"),flags);

[Makuna]

Did you check how printf was defined?
printf does not support being passed a flash based string.
Also note, printf is an extension on esp8266, it is not present on AVR or SAM builds.

This will work.

Serial1.printf(String(F("valueRemoveFlagged(%04X)")),flags);

or if you happen to have a lot of them in a row, a more efficient way

String formatStr;

formatStr = F("valueRemoveFlagged(%04X)");
Serial1.printf(formatStr , flags);

formatStr = F("different one %04X");
Serial1.printf(formatStr , flagsOther);

[hallard]

@Makuna
Sorry no, I didn't, it sounds you answered and confirmed what I thought, it's not possible (for now).
Thanks

[rogerclarkmelbourne]

Ivan.

Is this fixed ?

I've been looking at things like

const char *pointerToStaticString = "ABDCE";

and currently, this doesn't get compiled into the ROM it gets compile into the RAM

const char staticString[6]="ABCDE";

doesn't get compile into ROM either.

I presume this is either a gcc settings (compile or linker switch options) or a linker file issue

Is that what your fix is that is staged for release ?

Or an I looking at a different problem ?

[igrr]

const char arrays are not expected to go into flash by default, because special care is required to read byte-size data from flash (i.e. pgm_read_byte). However you may use progmem macros to move stuff into flash.

[rogerclarkmelbourne]

Umm

OK.
I'll need to look at your PROGMEM macro to see what it actually does, as I thought using those macros was a hack for the AVR, and not necessary on more modern architectures.

I thought that the ESP8266 was the same sort of architecture as on the STM32 where if we declare a either a char array or pointer to const string, these go into Flash

But perhaps because the flash is external to the ESP8266 this affects the way the compiler and linker can be configured.

[igrr]

Flash is memory-mapped into the address space in the ESP8266. However due to the way this is implemented in hardware, this memory has alignment restrictions — all reads have to be dword-aligned. So while you can modify the linker script to place all strings into flash, normal C library functions (like strcpy, strstr and others) will not be able to work with such strings. Reading individual bytes from strings placed into flash requires something similar to pgm_read_byte.

[rogerclarkmelbourne]

Thanks for the explanation.

I have some large strings which are SVG graphics, which I don't want to put into SPIFFS because I need to update them as part of the firmware via OTA.

So I'll need to look at the PROGMEM stuff, as currently they are getting put into RAM which is becoming a problem as I add more of them.

Anyway. Thanks again

Roger

[Makuna]

PROGMEM is an abstraction, albeit not the best way to do this; but it has rooted itself into Arduino and pretty much became the way to do this if you want to write standard Arduino code. Relying on const to do this for some chips is a mistake, especially if you write a library.

[ds380]

Thanks for sharing your knowledge, guys. I am new to Arduino and was puzzled when sprintf() refused to take F() constants. Rather it did, but the result was including some random text from RAM instead of my constant.