Skip to content

Conversation

@mcspr
Copy link

@mcspr mcspr commented Jun 28, 2025

Hiding these behind a merge-commit does not seem sensible, and I don't want to lose history info. Hope messages below are pretty self-descriptive.
CI is updated to reference up-to-date toolchain and produces a .zip with filesystem after make install
edit: https://github.com/mcspr/newlib-xtensa/actions/runs/15948100054/job/44984645502

Functional changes are limited to splitting *_pgmspace.c & str{,n}cpy into separate .c to provide .o per function and avoiding messing with the newlib sources too much
Rest mostly deals with crusted over automake/conf sources, port file overrides, header chain order tweaks, removing some unused files coming from original patch and correcting files that were changed after upstream merges over the years.

ps. would worth a toolchain bump, also including the 'always-inline' commit for sys/pgmspace.h
could also push for 4.5.0, since I don't really plan on changing that much as I thought originally

ps2. newlib-4.0.0 used ancient autoconf-2.64 and automake-1.11 (which btw ubuntu ships w/ a broken debian patch even to this day)

mcspr added 14 commits June 27, 2025 20:51
this was not done for newlib-xtensa sources since newlib-1.19
amends c0f3596
using dummy implementation already used in esp8266/Arduino
remove #if 0 from newlib string.h implementations, explicitly list files that need overrides
amends 42f6e16
upstream _default_types.h already provides a way to override stdint.h types with a macro
similarly, use machine/types.h to provide lwip2 declaraction helper

remove direct newlib sys/types.h changes

amends 08a150b
amends 0343751
Change newlib header to allow redefinitions instead of implicitly requiring sys/pgmspace.h
Keeps newlib sys/stdio.h, avoid c/p its code
Remove unused sys/xtensa/include/config.h
Update sys/config.h instead
Redundant and unlike AVR this is a plain type w/o any extra attributes
don't interpret pointer value as float or double
sync with sys/xtensa/sys/pgmspace.h & don't swallow original strings
Don't link the whole .o when only one function is used
-H "Authentication: ${{ secrets.GITHUB_TOKEN }}" \
-H "Accept: application/vnd.github.VERSION.raw" \
-o "$flocal" \
https://api.github.com/repos/esp8266/Arduino/contents/$fremote
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need any authentication here. This is only ever getting the master version of the 2 files, no?

Copy link
Author

@mcspr mcspr Jun 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not technically required, it is helpful for cases when CI breaks and has to be re-run multiple times. Otherwise, user has to wait for an hour to continue after performing 60 requests.

I also c/p the request from my other incorrectly written workflow, it is actually authorization: bearer $TOKEN not authenticate: $TOKEN

@earlephilhower
Copy link
Owner

Nice work, cleans things up a lot. Did you decide to just punt on printf-nano and go full-fat for 4.5.0?

Same as `gh-api`, allow 5000 vs. 60 api requests per hour
ref. https://docs.github.com/en/rest/rate-limit/rate-limit
@mcspr
Copy link
Author

mcspr commented Jun 30, 2025

Nice work, cleans things up a lot. Did you decide to just punt on printf-nano and go full-fat for 4.5.0?

Reverting 3b97a5e plus this patch to move ssputws_r and sswprint_r out of nano-formatted-io if-else seems ok
https://inbox.sourceware.org/newlib/[email protected]/#Z31newlib:libc:stdio:Makefile.inc
https://inbox.sourceware.org/newlib/[email protected]/#Z31newlib:libc:stdio:Makefile.inc

diff --git a/newlib/libc/stdio/Makefile.inc b/newlib/libc/stdio/Makefile.inc
index e25680212..09a508928 100644
--- a/newlib/libc/stdio/Makefile.inc
+++ b/newlib/libc/stdio/Makefile.inc
@@ -39,9 +39,7 @@ libc_a_SOURCES += \
 	%D%/sprint_r.c \
 	%D%/swprint_r.c \
 	%D%/ssputs_r.c \
-	%D%/ssputws_r.c \
-	%D%/ssprint_r.c \
-	%D%/sswprint_r.c
+	%D%/ssprint_r.c
 endif
 
 libc_a_SOURCES += \
@@ -99,6 +97,8 @@ libc_a_SOURCES += \
 	%D%/snprintf.c \
 	%D%/sprintf.c \
 	%D%/sscanf.c \
+	%D%/ssputws_r.c \
+	%D%/sswprint_r.c \
 	%D%/stdio.c \
 	%D%/svfiwprintf.c \
 	%D%/svfiwscanf.c \
-- 
2.43.0

After patch build files list matches 4.0.0, so it should build exactly the same result. Fingers crossed, nothing else breaks, given that it still builds & links std::regex_search(L"...", std::wregex(L"...")) correctly

No idea how this is used in the wild w/ anyone else's esp8266 builds, btw. wchar_t is necessary for "C99 support" label, but not sure how much of it is actually used. libstdc++-v3 linkage is a nasty side-effect, as library tries to implicitly create various objects using wchar_t. e.g. <iostream> or <locale> internals, where whenever wchar_t support is declared there is now an implicit wstring, numeric, time, money formatting / scanning, etc.

There is also --disable-wchar_t, but gcc-10.x may have some missing overloads (otherwise, just a small patch of std include decls). For example, removing wide char like that saves ~110KiB when using std::cout

There is also https://github.com/picolibc/picolibc which is based on newlib, but it also includes tinystdio port that fully supports wide chars. tbh that might be even better, just have to rebase it yet again

@mcspr mcspr force-pushed the xtensa-4_0_0-lock-arduino branch 2 times, most recently from 49ac679 to 07c6da0 Compare July 2, 2025 16:48
@earlephilhower
Copy link
Owner

It's a holiday week here and so can't dig too much into this, but looks good at a glance, still.

I found some time formatting issues also WRT nano-printf. strftime or something similar, would need to see if I have the logs. There was one example that formatted a date using it that also failed to link in CI w/nano-printf. AdvancedWebServer, maybe? It was a WebServer example for sure...

mcspr added 4 commits July 3, 2025 03:26
sync w/ picolibc va_list typing (which is often confusing for clang, too)
sys/_stdio.h & others, editor should see stdarg.h dependency
Same as memmove_P check, but for libc func

Note that esp8266/Arduino currently re-maps ROM functions
> PROVIDE ( strcpy = 0x4000bec8 );
> PROVIDE ( strncpy = 0x4000c0a0 );

But, both are still available via `ets_...`
Another c/p of existing source
4.0.0 build system does not prepend library name to outputs in libc/sys/
4.5.0 does (but, libc/machine is preferred)

move atexit.c & previously missing time.c

amends e9fa704
@mcspr mcspr force-pushed the xtensa-4_0_0-lock-arduino branch from 07c6da0 to 961364f Compare July 3, 2025 01:13
@mcspr
Copy link
Author

mcspr commented Jul 3, 2025

btw sorry for force-pushes above. I intended to work on a test branch, but it has gone horribly wrong...

From public arduino-pico builds I've seen at least one erroneous PR run.
earlephilhower/arduino-pico#2969

Both arm & riscv fail with UploadHudeFile.ino
libraries/WebServer/examples/UploadHugeFile/UploadHugeFile.ino breaks on wcsftime, flooding output with more undefined references to `swprintf' follow

And it is also related to #include <regex>, coming from #include <uri/UriRegex.h>
https://github.com/earlephilhower/arduino-pico/blob/master/libraries/WebServer/src/uri/UriRegex.h

Using the same idea, I've repackaged 230224 with lib{c,g,m}.a and rsync --delete --exclude=c++ed includes. esp8266/Arduino#9259 builds every available example. edit: Latest run is 4.5.x merge, current 4.0.0 is around 48f9336 after I've noticed the issue with time()

Since WebServer still uses regex.h and not C++ header, instead I rigged tests/sanity_check.sh to build something linking with either regex or iostream.

Happy holidays :)

@mcspr
Copy link
Author

mcspr commented Aug 9, 2025

If you checked the sources here, pls also take a look at merge & release branches for 4.5.0
https://github.com/mcspr/newlib-xtensa/tree/xtensa-4_0_0-to-4_5_0-merge (just merging up to version)
https://github.com/mcspr/newlib-xtensa/tree/xtensa-4_5_0-lock-arduino (above + patches mentioned)

I could put up it as PR, but these probably could be just fetched manually.
iirc from above, it is safe to replace xtensa-lx106-elf/include & .a packaged in the current toolchain, so it would not be necessary to rebuild everything in esp-quick-toolchain. I've done that manually last time, but I guess it can be a CI job / script here / script in esp8266 repo?

nb. for some weird reason, merging one version tag at a time is much easier than attempting to jump from 4.0.0 to 4.5.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants