Integrated handling for filesystem and gzipped binaries #8266
Conversation
The current platform.txt doesn't define the entries to copy signed binaries when performing a "Export compiled Binary". This pull request adds the nessesary enties.
ATENTION: This is tested on Linux only! See below for open questions/prerequisites. Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer) The steps are: 1. Compile the sketch 2. Export the generated sketch-binary 3. Generate the filesystem 4. Export the fs-binary 5. Generate gzipped versions of binaries 6. Generate signed versions of gzipped binaries Which of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?). There are some problems/inconveniences: 2. The signed sketch-binary is not exported when using "Export compiled Binary". (This point will be solved with my Pull-Request #8255) 3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools: “arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”. Both tools are designed to create & upload the fs, but not export them. So if preparing the files for an OTA when no board is connected, you have to: - ignore errors shown because upload is not possible - find & navigate to the “build.path” and copy the fs-binary manually 5. & 6. AFAIK, there is no “integrated” way to perform these steps. This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino. This is done by adding two new entries (“Filesystem” and “Binaries”) to the tools-menu. These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last. Menu-Entry “Filesystem” let You choose between: - Off - LitteFs: Create & Upload - LitteFs: Create only - SPIFFS: Create & Upload - SPIFFS: Create only Menu-Entry “Binaries” let You choose between: - Off - Export (.bin & .bin.signed) - Create & Export gzipped Binaries too Now You can choose with of the steps mentioned above will be automatically performed. After a (successful) compile, the files will be exported to a subdir “bin/{variant}” of Your sketch directory (“{variant}” replaced with the name of the board as shown behind “Tools -> Board:”). Signed variants of all Binaries are gererated similar to the “automatic signing” done for normal sketch binary. Open questions/prerequisites: As stated above I had tested this on Linux only. Even under this os there are some prerequisites: - gzip must be available (inside path) if it is not, a message is given and creating of gzipped binaries is skipped. - tkinter should be available if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”. So in this case a message is given and the fs will not be created. This “should” work under Windows/MacOs the same, but can't be tested by me (I had no PC with this OSes) An additional question for developers: As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)
|
Ok, bug found. It seams, that the python version used is a bit more strict (than mine) and imports only if there are no errors inside. There was an ident error inside utilities.py that must have raise the error. |
|
A try to get someone review this. |
|
edit Thanks for allowing to automate these steps !
Can you replace this dependancy by some python code ?
I'm not sure tkinter can be easily provided by the embedded python. Why not bailing out with an error when FS is empty, and/or add a another menu entry for empty FS ? |
|
Thanks for reviewing this.
Yes it can and I show how to install it in the message given if it's not already installed... see line 84 of utilities.py
That's what I already do. I give the message "can't ask because..." and then simply skip generation of the fs.
Yeah, that's a good point. |
* Integrated handling for filesystem and gzipped Binaries ATENTION: This is tested on Linux only! See below for open questions/prerequisites. Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer) The steps are: 1. Compile the sketch 2. Export the generated sketch-binary 3. Generate the filesystem 4. Export the fs-binary 5. Generate gzipped versions of binaries 6. Generate signed versions of gzipped binaries Which of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?). There are some problems/inconveniences: 2. The signed sketch-binary is not exported when using "Export compiled Binary". (This point will be solved with my Pull-Request #8255) 3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools: “arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”. Both tools are designed to create & upload the fs, but not export them. So if preparing the files for an OTA when no board is connected, you have to: - ignore errors shown because upload is not possible - find & navigate to the “build.path” and copy the fs-binary manually 5. & 6. AFAIK, there is no “integrated” way to perform these steps. This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino. This is done by adding two new entries (“Filesystem” and “Binaries”) to the tools-menu. These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last. Menu-Entry “Filesystem” let You choose between: - Off - LitteFs: Create & Upload - LitteFs: Create only - SPIFFS: Create & Upload - SPIFFS: Create only Menu-Entry “Binaries” let You choose between: - Off - Export (.bin & .bin.signed) - Create & Export gzipped Binaries too Now You can choose with of the steps mentioned above will be automatically performed. After a (successful) compile, the files will be exported to a subdir “bin/{variant}” of Your sketch directory (“{variant}” replaced with the name of the board as shown behind “Tools -> Board:”). Signed variants of all Binaries are gererated similar to the “automatic signing” done for normal sketch binary. Open questions/prerequisites: As stated above I had tested this on Linux only. Even under this os there are some prerequisites: - gzip must be available (inside path) if it is not, a message is given and creating of gzipped binaries is skipped. - tkinter should be available if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”. So in this case a message is given and the fs will not be created. This “should” work under Windows/MacOs the same, but can't be tested by me (I had no PC with this OSes) An additional question for developers: As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-) * Forgot to add new tools to git * Minor bugfix * Debug for failed check in Pull-Request * Ignore files generated by boards.txt.py & my extra files * More Debug for failed check in Pull-Request * Bugfix for check * And another bigfix :-( * Changed mod a+x (as all other .py) * Syntax error in utilities.py may be the reason for failed import. Correted * instead of external commd, use python module gzip
…pped-Binaries Conflicts: tools/postbuild.py tools/utillities.py
|
I made the info shown if tkinter was not installed a bit better, so I hope that's now all OK |
|
this is PR 8266 :-) |
|
On the way to #8285 (#8284 is the current last one) |
|
@d-a-v @earlephilhower: I thing that (part off) my original comment should be integrated into the documentation of the esp toolkit (when this PR was merged) and although mentioned in the tools “arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”. |
|
You can integrate documentation changes as part of your pull request. Everything is in |
Added description of changes to docu
|
@d-a-v I had done as suggested + make some improvements inspired by some of my fiends. |
…-and-gzipped-Binaries
|
@d-a-v: Is there any reason why this isn't merged till now ? |
…' of https://github.com/gneiss15/ArduinoEsp8266 into Integrated-handling-for-filesystem-and-gzipped-Binaries
doc/filesystem.rst
Outdated
| @@ -192,8 +192,55 @@ in the specific subdirectory. This mimics the POSIX behavior for | |||
| directory traversal most C programmers are used to. | |||
|
|
|||
|
|
|||
| Uploading files to file system | |||
| ------------------------------ | |||
| Uploading files to file system (new) | |||
There was a problem hiding this comment.
(new) => (combined firmware and filesystem)
There was a problem hiding this comment.
OK; good point (Thanks)
There was a problem hiding this comment.
Hm, I don't understand what happend here. You changed this, but inside the "resolved" view I still see my old version ???
I prefer You Version (with "combined firmware and filesystem" instead of "new"), but that's not shown here ???
There was a problem hiding this comment.
OK, because that seams to be true for all such comments here, I integrate theses changes into my local copy and commit theses changes later..
If this isn't the right way, please let me know.
| # Next, do the binary upload | ||
| # Combined in one rule because Arduino doesn't support upload.1.pattern/upload.3.pattern | ||
| tools.esptool.upload.pattern="{cmd}" -I "{runtime.platform.path}/tools/upload.py" --chip esp8266 --port "{serial.port}" --baud "{upload.speed}" "{upload.verbose}" {upload.erase_cmd} {upload.resetmethod} write_flash 0x0 "{build.path}/{build.project_name}.bin" | ||
| tools.esptool.upload.network_pattern="{network_cmd}" -I "{runtime.platform.path}/tools/espota.py" -i "{serial.port}" -p "{network.port}" "--auth={network.password}" -f "{build.path}/{build.project_name}.bin" |
There was a problem hiding this comment.
Is removing espota.py really part of this PR ?
This one is very sensitive and any change must be backward compatible.
It must be part of another dedicated PR.
There was a problem hiding this comment.
It's not removed... See You comment above (about old behaviour) there espota.py is still contained.
It's only modified to additionally upload the fs within one call to espota !
tools/netUploadGn.py
Outdated
| #!/usr/bin/env python3 | ||
|
|
||
| # Network-Upload-Wrapper | ||
| # 2021-08-08: G.N.: Written to support (optional) uploading of sketch & filesystem in one step |
There was a problem hiding this comment.
Should be part of another PR
There was a problem hiding this comment.
No, that is an essential part of this PR.
It does the combined upload when using netupload instead of espota, so it MUST be part of this PR.
tools/utillities.py
Outdated
| return path | ||
|
|
||
| def TkDialog( title, text, mbName ): | ||
| try: |
There was a problem hiding this comment.
No, it is the base of (for ex.) ConfirmDialog & WarningDialog which are used inside postbuild.py
There was a problem hiding this comment.
Anything with python must be cross-compatible with all archs.
A portable version of python is shipped by the installer and I doubt tkinter will work in that case.
|
@d-a-v wrotes: There are some unrelated changes which should not be part of it : some breaking or unexplained changes and also a softwareserial commit which should be removed. Because I am although a bit confused what "resolve conversation" above means/does.. I had done:
Some of the changes that are shown inside my branch are although shown inside the main repro (as already committed), but when comparing my branch against the master these changes are shown a additional changes ??? confusing.. master...gneiss15:Integrated-handling-for-filesystem-and-gzipped-Binaries |
|
OK, I hope I had found the correct way of removing unwanted changes. As far as I could see, the only "open" difference is the removal of "SoftwareSerial" here I don't know how to revert that.. |
|
Reviewing is made against current core's source (master branch). |
All this trouble comes from the merge mentioned above (05.09) with tag 8c06e83 which now links to a "page not found" ??? I still have no idea how to resolve the mentioned problems... Should I remove my whole repro and redo all changes ? |
|
If you are not familiar with git, best is probably to reapply your changes to a fresh master. |
|
OK, I will do so (reapply your changes to a fresh master). Unfortunately that means I have to make a new fork... I will open a new pull request when I had done this..
No, tkinter is not shipped with this python, it must be installed separate (at least under linux) |
That's the reason why such change can't be accepted by maintainers. |
|
As I already mentioned the presense of tkinter is not mandatory. If it isn't installed I simply can't ask the user what is his choice in the situation when the FS is/will be empty. I then simply give a message and don't create the FS. BTW.: It will take some time to create the new repro (I am away for some days).. |
|
I close this now, to open a new one with a new fresh fork |
ATENTION: This is tested on Linux only! See below for open questions/prerequisites.
Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer)
The steps are:
Which of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?).
There are some problems/inconveniences:
2. The signed sketch-binary is not exported when using "Export compiled Binary".
(This point will be solved with my Pull-Request #8255)
3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools:
“arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”.
Both tools are designed to create & upload the fs, but not export them.
So if preparing the files for an OTA when no board is connected, you have to:
This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino.
This is done by adding two new entries (“Filesystem” and “Binaries”) to the tools-menu.
These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last.
Menu-Entry “Filesystem” let You choose between:
Menu-Entry “Binaries” let You choose between:
Now You can choose with of the steps mentioned above will be automatically performed.
After a (successful) compile, the files will be exported to a subdir “bin/{variant}” of Your sketch directory (“{variant}” replaced with the name of the board as shown behind “Tools -> Board:”).
Signed variants of all Binaries are gererated similar to the “automatic signing” done for normal sketch binary.
Open questions/prerequisites:
As stated above I had tested this on Linux only.
Even under this os there are some prerequisites:
if it is not, a message is given and creating of gzipped binaries is skipped.
if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.
So in this case a message is given and the fs will not be created.
This “should” work under Windows/MacOs the same, but can't be tested by me (I had no PC with this OSes)
An additional question for developers:
As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)