Add retry on 503 error

[omkar-334]

Fixes #1225.

Do we need to add this for wikipedia as well?

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

Added HTTP 503 retry handling to Common Crawl WARC downloads using wget flags --retry-on-http-error=503, --waitretry=5, and --tries=5. This allows the downloader to automatically retry failed downloads when Common Crawl servers return 503 Service Unavailable errors.

Key changes:

• Modified wget command to include retry flags for 503 errors with 5-second wait between retries and maximum 5 attempts
• Updated all test assertions to reflect the new command structure

Note: The Wikipedia downloader (nemo_curator/stages/text/download/wikipedia/download.py:56) also uses wget for downloads and could benefit from similar retry logic if Wikipedia dumps experience similar transient failures.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The changes are straightforward and well-tested: adding standard wget retry flags to handle 503 errors. All tests have been properly updated to match the new command structure. The implementation follows wget best practices for retry logic.
• No files require special attention

Important Files Changed

Filename	Overview
nemo_curator/stages/text/download/common_crawl/download.py	Added wget retry flags (--retry-on-http-error=503, --waitretry=5, --tries=5) to handle 503 errors gracefully
tests/stages/text/download/common_crawl/test_download.py	Updated test assertions to expect the new wget retry flags in command invocations

_{Last reviewed commit: 15e7e4a}

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

--retry-on-http-error requires wget 1.21+ (released Jan 2021). Consider documenting this requirement or adding a fallback for older versions

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[omkar-334]

cc @praateekmahajan

[greptile-apps]

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[sarahyurick]

Based on https://status.commoncrawl.org/ retrying on 503 makes sense. Not sure if we should add others like 429, etc.

Adding -c could be good too.

WDYT @omkar-334 @praateekmahajan @ayushdg ?

[omkar-334]

hi @sarahyurick
i went through the wget docs.
They say that

A combination with ‘-O’/‘--output-document’ is only accepted if the given output file does not exist.

So the comment is true, i don't think we can/should add -c, since the output file would already exist.
what do you think?

[sarahyurick]

Fair enough, I am fine with skipping it. Thanks!

[greptile-apps]

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[sarahyurick]

/ok to test 15e7e4a

[sarahyurick]

Thanks @omkar-334 , LGTM!