Misleading HTTP 403 responses when `bulk.limits.max.targets-per-flat-request` is exceeded

[nicolapace]

We would like to report what appears to be misleading HTTP status codes returned by dCache when the bulk request size exceeds the configured limit (bulk.limits.max.targets-per-flat-request).

Description

When the number of targets in a bulk request exceeds the configured limit, dCache replies with HTTP 403 – Forbidden. This is extremely confusing for clients, as it strongly suggests an authorization or permission problem, while the real cause is a request size / server-side limit.

We recently spent a significant amount of time debugging permissions, credentials, and authorizations before discovering that the real issue was simply that the request exceeded the configured bulk limit.

Concrete example (IN2P3 tape staging)

During a CMS mini-data challenge, FTS was staging files from tape at CCIN2P3 using the dCache Tape REST API. We observed many failures in FTS with errors like:

STAGING [1] [Tape REST API] Stage call failed: HTTP 403 : Permission refused

Key things:

• Errors did not appear in dCache logs visible to the site operators.
• The site initially could not find any trace of the failing requests.
• The error message strongly suggested an authorization issue, which turned out to be incorrect.

The root cause was later identified as follows:

• The site was using the default value of
  bulk.limits.max.targets-per-flat-request = 500
• The FTS instance is configured to submit up to 2000 in a single bulk request.
• When the limit was exceeded, dCache returned HTTP 403.
• After increasing the limit to 2000, the errors immediately disappeared.

The problem

Returning 403 Forbidden in this case is misleading because:

• The client is authorized.
• The request is syntactically valid.
• The failure is due to request size / server configuration limits, not permissions.

Suggested improvement

When the bulk request exceeds bulk.limits.max.targets-per-flat-request, dCache should return a more appropriate HTTP status code, for example:

• 413 Content Too Large, or
• Another 4xx code that clearly indicates a request size or limit violation.

Additionally, returning a clearer error message in the response body (explicitly stating that the bulk request exceeds the configured limit) would greatly improve debuggability.
This issue has occurred before at other sites as well, which suggests this is not site-specific but a general usability issue in dCache’s API behavior.

[DmitryLitvintsev]

500 is default so not surprising sites are hitting these

bulk service log file should have these errors in it:

... 6B97-353D-4943-8599-1BA289F47DDA.root, /WAX/11/store/mc/RunIISummer20UL17NanoAODv9/ttH_ttSemiLep_HTo2LongLivedTo2G2Q_MH-125_MFF-55_ctau-20mm_TuneCP5_13TeV_powheg-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/70000/53C1296A-FC2F-904B-8988-A55E54DFB06F.root], activity: STAGE: org.dcache.services.bulk.BulkPermissionDeniedException: The number of targets 558 exceeds maximum 500 for directory expansion NONE.

Original author of the code treated this condition as "Forbidden to create requests larger than LIMIT" hence 403. But I agree this is misleading, especially if no cause is provided in reponse body.

I will fix it. Meanwhile it would be nice to have this broadly announced at, say, WLCG BDT meeting that sites need to set

bulk.limits.max.targets-per-flat-request 

in excess of 2000. Since 2000 is what FTS is using.

[vokac]

Limits were discussed in WLCG BDT meeting few years ago and as a result dCache updated default values #7116. It is not clear to me what lead CERN FTS Ops to set value different than default 200.

[maarten-litmaath]

Might 2000 instead of 200 have been a typo?

[vokac]

According CERN FTS Ops the FTS instance used by CMS rely on non-default StagingBulkSize (configured probably on CMS request on Feb 12 2025). So, I would keep current bulk.limits.max.targets-per-flat-request value which is compatible (bigger) when it comes to FTS default configuration (200).

Improving error / HTTP code is always welcome.

[nicolapace]

Thanks for the clarifications.

Just to be clear, from an FTS perspective this issue is only about the misleading HTTP 403 error, not about the actual limit values or configurations.

Receiving a 403 strongly suggests an authorization problem, which led to significant unnecessary failures and consecutive debugging, while the real cause was simply a bulk-size limit. A more appropriate HTTP status code and a clearer error message would greatly improve debuggability and also could be handled automatically by new versions of FTS (it could automatically change limits if it sees failures for those kind of errors, but if it sees 403 it won't do that).