Socket Leak Denial of Service

[thetayloredman]

Hi. I'm one of the owners of a small Minecraft network using Waterfall and SubServers 2. After many "too many file descriptors" errors for the last few months, me and the other owner (cc @ilikestohack) have isolated the issue. There appears to be some sort of socket leak in Waterfall where non-Minecraft packets are accepted (of course) and never properly closed, leaving them in the CLOSE_WAIT state.

We use a tool called Uptime Kuma for status monitoring, which sends very simple TCP pings to every monitor we have set up. One of those monitors is :25565 (Waterfall) and we then discover the proxy's Java process is over the allocated process maximum file descriptors. We assumed this was a SubServers bug, but later discovered it was actually a bunch of open SOCKETS all originating from my IP address. I had been un-knowingly DoSing my own Minecraft server.

What currently occurs:

• Connection comes in and is established (syn, syn-ack, ack)
• Ping packet of some sort is sent
• Kuma attempts to disconnect (fin packet)
• Connection is stuck in CLOSE_WAIT waiting for Waterfall to send the FIN-ACK packet.

What should occur:

• Waterfall sends the FIN-ACK packet.

Unusual conditions:

• TCP pinging service

The resolution for this is quick, but this results in almost a complete crash of the proxy. No new connections are accepted and the log file is filled with errors. Over 524285 file descriptors were opened by Waterfall around once every two weeks during the leak.

[Janmm14]

Can you do a quick check to see whether bungeecord has the same behaviour?

[thetayloredman]

Can you do a quick check to see whether bungeecord has the same behaviour?

@Janmm14 This issue only occurs for me rarely. If I can force a repro with a netcat or something I will attempt that.

[Janmm14]

So usually waterfall closes the connections of your tcp pinging without problem?

My idea was since you have it set up and such, that you let the tcp pinging happen like 100 or 1000 times and then check java process after 2 mnutes or so for open tcp connections?

[thetayloredman]

So usually waterfall closes the connections of your tcp pinging without problem?

My idea was since you have it set up and such, that you let the tcp pinging happen like 100 or 1000 times and then check java process after 2 mnutes or so for open tcp connections?

I will attempt this, but it's not something can do quickly. Kuma has a maximum interval of 1 ping every 30 seconds (iirc), or 2 pings a minute/120 an hour/2,880 a day/174 days for 500000 descriptors (almost what we obtained), but including the two kumas running, 87 days (which is accurate to how often this happens)

I will also be charting n of open files, and keep track of that information from here-on, and will update when I get a chance.

cc @ilikestohack (the other owner)

[Janmm14]

I think with those rates, if the number of open files of a test bungeecord after a day should already indicate whether the problem exists in bungeecord or was introduced in waterfall.

[thetayloredman]

I think with those rates, if the number of open files of a test bungeecord after a day should already indicate whether the problem exists in bungeecord or was introduced in waterfall.

I think just running a Bungee instance and mocking one of those connections (SYN, SYN-ACK, ACK, FIN) will tell me the same information (if I receive FIN-ACK, ACK I know we are all clear in Bungee)

[thetayloredman]

@ilikestohack Repro?