[core] refactor: extract and centralize HTTP fetcher#4016
[core] refactor: extract and centralize HTTP fetcher#4016khassel merged 3 commits intoMagicMirrorOrg:developfrom
Conversation
- Extract HTTP logic from Calendar module into HTTPFetcher class - Migrate calendar module to use centralized HTTPFetcher - Add modern improvements: * AbortController with configurable timeout (default 30s) * undici Agent for self-signed certificates * Improved Retry-After header handling (respects server hints) - Implement intelligent retry strategies: * 401/403: 5x interval, min 30 minutes * 429: Retry-After header parsing, fallback 15 minutes * 5xx: Exponential backoff (max 3 retries) * 4xx: 2x interval, min 15 minutes - Provide structured error objects with translation keys - Comprehensive unit tests with msw (Mock Service Worker) - Tests cover all error scenarios and authentication methods
|
My idea was to make it easier for the translators, but you're right, that was not very clean. Instead of taking the easy way out and simply removing them, I used automatic translation (which has delivered very good results in the past) to translate the strings. In doing so, I even noticed duplicate strings in the pt translation, which I removed right away. |
|
This broke my calendar: [2026-01-22 23:27:21.769] [ERROR] [calendar] https://calendar.google.com/calendar/ical/xxx.calendar.google.com/public/basic.ics - iCal parsing failed: UNTIL rule part MUST have the same value type as DTSTART
[2026-01-22 23:27:21.770] [ERROR] [calendar] Calendar Error. Could not fetch calendar: https://calendar.google.com/calendar/ical/xxx.calendar.google.com/public/basic.ics iCal parsing failed: UNTIL rule part MUST have the same value type as DTSTART |
|
Found the breaking entry: DTSTART;VALUE=DATE:20160313
DTEND;VALUE=DATE:20160314
RRULE:FREQ=YEARLY;UNTIL=20190312;BYMONTHDAY=13;BYMONTH=3
DTSTAMP:20260122T223427Z
UID:a21aa3d4-c1c2-4d74-b929-535e168a566f
CREATED:20170127T230833Z
LAST-MODIFIED:20190330T190642Z
SEQUENCE:1
STATUS:CONFIRMED
SUMMARY:xxx
TRANSP:OPAQUE
X-MOZ-FAKED-MASTER:1
X-MOZ-GENERATION:2
END:VEVENTI was able to solve the problem by deleting and recreating the entire calendar series in Google Calendar, but other users might encounter similar issues... |
|
Oh. I'll take a look at it by the weekend at the latest. |
|
don't know if it is a real problem, that was the only entry with a |
|
I think it is confused between full day and not for some reason, rrule until doesn’t use DATE: as it’s not required there |
When parsing recurring events with VALUE=DATE (all-day events), preserve the VALUE=DATE parameter in the DTSTART string passed to rrule-temporal. This ensures that rrule-temporal can correctly validate that UNTIL values have the same type as DTSTART. Previously, date-only information was lost when building the RRULE string, causing rrule-temporal to incorrectly assume DTSTART was DATE-TIME and reject valid DATE-only UNTIL values with the error: "UNTIL rule part MUST have the same value type as DTSTART" The fix uses local date components (getFullYear, getMonth, getDate) which is consistent with how dateOnly dates are created in the dateParameter function using new Date(year, month, day). Fixes issue reported in MagicMirrorOrg/MagicMirror#4016 where Google Calendar birthday events with yearly RRULE and DATE-only UNTIL in the past failed to parse. Test coverage: - Yearly recurring DATE events with UNTIL - Monthly recurring DATE events with UNTIL - DATE events with COUNT instead of UNTIL - Edge case: VALUE=DATE with TZID (correctly ignores TZID)
|
I think I found the root cause 🙂 The error comes from The problem: node-ical never told rrule-temporal that DTSTART was VALUE=DATE, so the validation failed. The stricter validation is good - it just exposed a bug in node-ical that's been there all along. Recreating the entry helped because Google Calendar probably removed the UNTIL or changed the format. But the underlying issue remains for other users with similar entries. I've opened a fix: jens-maus/node-ical#432 This should handle Google Calendar birthdays and similar recurring all-day events correctly. |
When parsing recurring events with VALUE=DATE (all-day events), preserve the VALUE=DATE parameter in the DTSTART string passed to rrule-temporal. This ensures that rrule-temporal can correctly validate that UNTIL values have the same type as DTSTART. Previously, date-only information was lost when building the RRULE string, causing rrule-temporal to incorrectly assume DTSTART was DATE-TIME and reject valid DATE-only UNTIL values with the error: "UNTIL rule part MUST have the same value type as DTSTART" The fix uses local date components (getFullYear, getMonth, getDate) which is consistent with how dateOnly dates are created in the dateParameter function using new Date(year, month, day). Fixes issue reported in MagicMirrorOrg/MagicMirror#4016 where Google Calendar birthday events with yearly RRULE and DATE-only UNTIL in the past failed to parse. Test coverage: - Yearly recurring DATE events with UNTIL - Monthly recurring DATE events with UNTIL - DATE events with COUNT instead of UNTIL - Edge case: VALUE=DATE with TZID (correctly ignores TZID)
When parsing recurring events with VALUE=DATE (all-day events), preserve the VALUE=DATE parameter in the DTSTART string passed to rrule-temporal. This ensures that rrule-temporal can correctly validate that UNTIL values have the same type as DTSTART. Previously, date-only information was lost when building the RRULE string, causing rrule-temporal to incorrectly assume DTSTART was DATE-TIME and reject valid DATE-only UNTIL values with the error: "UNTIL rule part MUST have the same value type as DTSTART" The fix uses local date components (getFullYear, getMonth, getDate) which is consistent with how dateOnly dates are created in the dateParameter function using new Date(year, month, day). Fixes issue reported in MagicMirrorOrg/MagicMirror#4016 where Google Calendar birthday events with yearly RRULE and DATE-only UNTIL in the past failed to parse. Test coverage: - Yearly recurring DATE events with UNTIL - Monthly recurring DATE events with UNTIL - DATE events with COUNT instead of UNTIL - Edge case: VALUE=DATE with TZID (correctly ignores TZID)
This includes a new version of `node-ical` which should resolve a calendar issue that was reported [here](#4016 (comment)) and [here](#4010 (comment)).
Migrates the Newsfeed module to use the centralized HTTPFetcher class introduced in MagicMirrorOrg#4016, following the same pattern as the Calendar module. Changes: - Refactored NewsfeedFetcher from function constructor to ES6 class - Replaced manual fetch() + timer handling with HTTPFetcher composition - Uses structured error objects with translation keys - Inherits smart retry strategies (401/403, 429, 5xx backoff) - Inherits timeout handling (30s) and AbortController - Removed js/module_functions.js (scheduleTimer no longer needed) - Removed #module_functions import from package.json
This migrates the Newsfeed module to use the centralized HTTPFetcher class (introduced in #4016), following the same pattern as the Calendar module. This continues the refactoring effort to centralize HTTP error handling across all modules. ## Changes **NewsfeedFetcher:** - Refactored from function constructor to ES6 class (like the calendar module in #3959) - Replaced manual fetch() + timer handling with HTTPFetcher composition - Uses structured error objects with translation keys - Inherits smart retry strategies (401/403, 429, 5xx backoff) - Inherits timeout handling (30s) and AbortController **node_helper.js:** - Updated error handler to use `errorInfo.translationKey` - Simplified property access (`fetcher.url`, `fetcher.items`) **Cleanup:** - Removed `js/module_functions.js` (`scheduleTimer` no longer needed) - Removed `#module_functions` import from package.json ## Related Part of the HTTPFetcher migration effort started in #4016. Next candidate: Weather module (client-side → server-side migration).
Summary
PR #3976 introduced smart HTTP error handling for the Calendar module. This PR extracts that HTTP logic into a central
HTTPFetcherclass.Calendar is the first module to use it. Follow-up PRs would migrate Newsfeed and maybe even Weather.
Before this change:
fetch()callsWhat this PR adds:
Architecture
Before - Decentralized HTTP handling:
After - Centralized with HTTPFetcher:
Complexity Considerations
Does HTTPFetcher add complexity?
Even if it may look more complex, it actually reduces overall complexity:
calendarfetcherutils.js#3976) - we're extracting, not addingFuture Benefits
Weather migration (future PR):
Moving Weather from client-side to server-side will enable:
Moving the weather modules from client-side to server-side will be a big undertaking, but I think it's a good strategy. Even if we only move the calendar and newsfeed to the new HTTP fetcher and leave the weather as it is, this PR still makes sense, I think.
Breaking Changes
None
I am eager to hear your opinion on this 🙂