Serve LittleFS files with ethernet W5500 and esp8266webserver (ERR_CONTENT_LENGTH_MISMATCH)

[infrafast]

Basic Infos

• [X ] This issue complies with the issue POLICY doc.
• [ X] I have read the documentation at readthedocs and the issue is not addressed there.
• [X ] I have tested that the issue is present in current master branch (aka latest git).
• [X ] I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• [ X] I have filled out all fields below.

Platform

• Hardware: ESP8266
• Core Version: framework-arduinoespressif8266 @ 3.30101.0 (3.1.1)
• Development Env: PlatformIO
• Operating System: Windows

Settings in IDE

• Module: Espressif 8266 (4.1.0) > WeMos D1 R2 and mini
• Flash Mode:
• Flash Size: 4M
• lwip Variant: default
• Reset Method:
• Flash Frequency: [40Mhz]
• CPU Frequency: 80Mhz
• Upload Using: serial + OTA
• Upload Speed: 115200

Problem Description

Detailed problem description goes here.

I have a device that works both with WiFi and Ethernet using W5500 library and esp8266webserver.
Serving page works ok with WiFI but when using W5500 I have the similar problem as the one described in #2121
This happens randomly n some page, one time it can load, the other it will not and another page will fail.

I follow all threads about this issue both in github and other places and I made test with and without gziping the document, with and without merging my javascript files, with small (2/3k) and bigger file (30k).
I monitored the freeheap before and after pages are served and always have a reasonnable amount of memory available

The problem happens both when using "server.serveStatic" and "server.on"

I presume that the fix that was apply for the WiFi core has not been done for the W5500 part.

Hence I suspect the streamFile method template to be the cause.

I provide some extract of the code which are representative of the function calls (not the full one has it is too big).

MCVE Sketch

#include <Arduino.h>
#include <ESP8266WebServer.h>
#include <W5500lwIP.h>        

ESP8266WebServer server;
Wiznet5500lwIP ethDriver(D2);   

void streamJavascript(String path) {
  streamFile(path,"application/javascript");
}

void streamCSS(String path){
  streamFile(path,"text/css");
}

void handleIndexJS() {
  streamJavascript("/index.min.js.gz");
}

void handleStyleCSS() {
  streamCSS("/style.min.css.gz");
}

bool streamFile(String path, String dataType) {
  DEBUG_PRINT("\nFree heap() before call: %i",ESP.getFreeHeap());
  DEBUG_PRINT("\nRequested page -> %S",path.c_str());
  if (LittleFS.exists(path)){
      File dataFile = LittleFS.open(path, "r");
      if (!dataFile) {
          DEBUG_PRINT("\nNot found");
          return false;
      }
      uint16_t dataSentSize = server.streamFile(dataFile, dataType);
      if (dataSentSize != dataFile.size()) {
        DEBUG_PRINT("\nSent less data (%i) than expected! (%i)",dataSentSize,dataFile.size());

      }else{
          DEBUG_PRINT("\nPage served!");
      }
      DEBUG_PRINT("\nFree heap() after call: %i",ESP.getFreeHeap());
      dataFile.close();
  }else{
      DEBUG_PRINT("\nNot found");
      return false;
  }
  return true;
}

void setup() {
      ethDriver.begin(mac);         
      loopElapsedTime = millis();
      while (!ethDriver.connected() && (millis() - loopElapsedTime < ETHCONNECTTIMEOUT)) {
        //DEBUG_PRINT(".");          
      }
    server.on("/index.min.js", handleIndexJS);
    server.on("/style.min.css", handleStyleCSS);
}

void loop() {
   server.handleClient();
}

Debug Messages

Free heap() before call: 11856
Requested page -> /style.min.css.gz
Page served!
Free heap() after call: 10232
Free heap() before call: 12504
Requested page -> /index.min.js.gz
Sent less data (4166) than expected! (6808)
Free heap() after call: 9832

[infrafast]

Ok, I found the solution:
it's a timeout problem,

on ESP8266WebServer.h => contentLength = file.sendAll(_currentClient,5000);
instead of file.sendAll(_currentClient);

one may consider this to be fixed for next releases ?

[mcspr]

Do you mean that 5000ms timeout in the server code fixed the problem?

Arduino/libraries/ESP8266WebServer/src/ESP8266WebServer.h

Line 235 in d7cd4be

contentLength = file.sendAll(_currentClient);

Since neverExpires is implicitly added without argument, we end up with whatever set to the object on which we use sendAll()

Arduino/cores/esp8266/Stream.h

Lines 177 to 178 in d7cd4be

	size_t sendAll (Print* to, const oneShotMs::timeType timeoutMs = oneShotMs::neverExpires) { return sendGeneric(to, -1, -1, timeoutMs); }
	size_t sendAll (Print& to, const oneShotMs::timeType timeoutMs = oneShotMs::neverExpires) { return sendAll(&to, timeoutMs); }

Arduino/cores/esp8266/StreamSend.cpp

Lines 235 to 237 in d7cd4be

	esp8266::polledTimeout::oneShotFastMs timedOut(
	timeoutMs >= esp8266::polledTimeout::oneShotFastMs::neverExpires ? getTimeout()
	: timeoutMs);

Arduino/cores/esp8266/Stream.h

Line 69 in d7cd4be

unsigned long getTimeout () const { return _timeout; }

Arduino/cores/esp8266/Stream.h

Line 52 in d7cd4be

unsigned long _timeout = 1000; // number of milliseconds to wait for the next char before aborting timed read

[infrafast]

yes, I've put the delay in the sendall method called in the server
If default timeout is 1000, I'd recommend to increase it to 3000 at least (in my case with 5000 it definitely solved the issue and I could not reproduced it after heavy testing)

[d-a-v]

Considering the "streamSend" API, timeout is currently the input stream's timeout.
The output stream's timeout is not considered (Print* to in StreamSend.cpp). In your case, the output is network (webserver's client) which default is already 5000ms.
Using a timeout of max(input-timeout,output-timeout) would be more coherent and should solve your issue.
I can prepare a proposal for this.

[infrafast]

Great if you provide a fix, I'd be happy to retest my code with it and validate the use case

[d-a-v]

If you are using the git version of this core, can you try with #8874 ?

(or the unofficial "nightly" alpha release)

[infrafast]

I just had a look but I am using platformIO, not the Arduino IDE, do oyu know by chance the lines I should add in my platformio.ini file to add the platform in a new environement ?

[d-a-v]

@infrafast Have you had success trying with #8874 ?

[mcspr]

I just had a look but I am using platformIO, not the Arduino IDE, do oyu know by chance the lines I should add in my platformio.ini file to add the platform in a new environement ?

https://docs.platformio.org/en/stable/platforms/espressif8266.html#using-arduino-framework-with-staging-version

[infrafast]

perfect thanks for the link, I tried to setup my staging environment accordingly but I still don't know how I can point it to the right git address. I am not enought familiar with Git I am afraid I can't test

[infrafast]

@infrafast Have you had success trying with #8874 ?

since I don't know how to connect my platformIO to the git repo with this fix, could you rather send me the 4 files that you changd and I will update my local folder with them ? => [email protected] thx!

[mcspr]

@infrafast Have you had success trying with #8874 ?

since I don't know how to connect my platformIO to the git repo with this fix, could you rather send me the 4 files that you changd and I will update my local folder with them ? => [email protected] thx!

Replace package URI

platform_packages =
    platformio/framework-arduinoespressif8266 @ git+https://github.com/d-a-v/Arduino.git#StreamSendToStream

Or, use the existing Arduino repository that you have already cloned by following PIO documentation (path to the package may vary, see pio run -v banner for the full path; PR page has commandline instructions for git at the very bottom)

> pwd
/home/runner/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910

> git fetch https://github.com/d-a-v/Arduino.git StreamSendToStream
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 18 (delta 7), reused 16 (delta 7), pack-reused 0
Unpacking objects: 100% (18/18), 20.04 KiB | 301.00 KiB/s, done.
From https://github.com/d-a-v/Arduino
 * branch              StreamSendToStream -> FETCH_HEAD

> git switch --detach --recurse-submodules --discard-changes FETCH_HEAD
HEAD is now at c8c30c7e move template function into header file

# ... test stuff ...

# ... back to master ...

> git switch --recurse-submodules --discard-changes master

[infrafast]

Thanks, with your instruction I learned how to setup my env and manage to test :-) !

Here are the results:

• on 2 differents PC, tested with
  • chrome (macos and windows)
  • safari (ubuntu, macos, windows)
• using WiFi and Ethernet connectivity
• page load total ~12seconds
  
  I've performed in total about 20 loads and never had the issue.
  Without the fix, the problem manifest about 70% of the time.
  I can state that this works very well and it seems to even faster the page loading.
  Thank you very much for this fix and looking forward to have it integrated into the main branch.