-
Notifications
You must be signed in to change notification settings - Fork 21
feat: bound incoming request and add postgres service #76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: bound incoming request and add postgres service #76
Conversation
|
👋 Thanks for assigning @tankyleo as a reviewer! |
4bfb403 to
a163ae7
Compare
rust/server/src/vss_service.rs
Outdated
| use std::pin::Pin; | ||
| use std::sync::Arc; | ||
|
|
||
| const MAXIMUM_REQUEST_BODY_SIZE: u16 = 65_535; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems very conservative, given that monitors could get quite large. Given that the VSS service is actually a storage service, it also might make sense to make this configurable (in contrast to lightningdevkit/ldk-server#80, but even there we set the limit to 10MB).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned elsewhere: I guess a static upper bound is a good first step, but if we're really concerned about DoS we might need some dynamic rate limiting on a per-IP basis. Although then the question becomes how much of that should be considered the concern of the VSS service itself, and how much we'd just expect users to slap a load balancer/Cloudflare in front of the service to handle that for them
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've made the configuration changes suggested here, capping at 20 MB for the maximum size.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, as mentioned elsewhere, individual values might be quite a bit larger than 10 or 20MB. I think something along the lines of 100MB would be a more reasonable maximum, though per request it raises the question of how much a DoS protection this actually is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updated the max to 1GB per Matt's recommendation. We get basic protection with this but I could spent some time on this and propose a coherent strategy.
5d391cc to
fbdd957
Compare
|
🔔 1st Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 4th Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 5th Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 6th Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 7th Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 8th Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 9th Reminder Hey @tankyleo! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we also add some test coverage to ensure the database / backends actually support the configured maximum values, i.e., that we're not otherwise limited somehow?
rust/server/src/vss_service.rs
Outdated
| use std::pin::Pin; | ||
| use std::sync::Arc; | ||
|
|
||
| const MAXIMUM_REQUEST_BODY_SIZE: u16 = 65_535; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, as mentioned elsewhere, individual values might be quite a bit larger than 10 or 20MB. I think something along the lines of 100MB would be a more reasonable maximum, though per request it raises the question of how much a DoS protection this actually is.
|
🔔 10th Reminder Hey @tankyleo! This PR has been waiting for your review. |
|
🔔 11th Reminder Hey @tankyleo! This PR has been waiting for your review. |
97b6b25 to
93faf38
Compare
|
Thanks for the review @tnull and @TheBlueMatt. I have updated the maximum request body size and added a test to check that our backend(s) support the configured value. It's ready for another look when you have the time. |
|
🔔 12th Reminder Hey @tankyleo! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally looks good, mod some comments.
As noted above already, I do wonder how much we're gaining at all we we're limiting to 1GB per request. In any case, probably better than nothing. So generally concept ACK from my side, but I'll leave it to @tankyleo to decide.
|
I mean at least it avoids someone being able to OOM us by just sending us trash 🤷♂️ Of course VSS exposed directly would probably still be OOM'able with a few parallel requests, but hopefully a decent frontend proxy can limit the impact by buffering the requests first. |
15417ec to
7121dcc
Compare
|
🔔 13th Reminder Hey @tankyleo! This PR has been waiting for your review. |
tankyleo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank your for the PR sorry for the delay on my part, a few bits, looking good !
7121dcc to
f061dc1
Compare
f061dc1 to
8a49df9
Compare
|
Hi @tankyleo |
564f34a to
5d0d841
Compare
tankyleo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a few more nits below, feel free to squash the fixups
5d0d841 to
04a9213
Compare
|
Hi @tankyleo, |
tankyleo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM two nits, good to consolidate everything into two commits, one for the docker file, and one for the max_request_body_size feature.
As with the other PR, please remove the prefixes, and capitalize the first word of the commit message.
rust/impls/src/postgres_store.rs
Outdated
| } | ||
|
|
||
| #[cfg(test)] | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should have caught this, we don't need an additional line here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has been removed.
rust/server/vss-server-config.toml
Outdated
|
|
||
| # Maximum request body size in bytes. Can be set here or be overridden by env var 'VSS_MAX_REQUEST_BODY_SIZE' | ||
| # Defaults to the maximum possible value of 1 GB if unset. | ||
| # max_request_body_size = 1073741824 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm sounds like the trailing whitespace is still there ? feel free to use something like git diff --check to check
Add VssServiceConfig to make the maximum request body size configurable through the server configuration file, with a hard limit of 1GB. Additionally, add test coverage for 1GB maximum supported value size and verifies that storage backends can handle the configured maximum value size.
04a9213 to
e0ae2f9
Compare
enigbe
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've addressed the last nits. When you approve I'll squash the remaining fixup commit.
What this PR does
TODOmaximum_request_body_sizein [server_config] or via environment variable