-
Notifications
You must be signed in to change notification settings - Fork 59
Remove finalize() methods to eliminate Finalizer lock contention
#533
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Problem
-------
In highly parallel environments, creating DuckDBResultSet and
DuckDBPreparedStatement objects causes severe mutex contention on
the global java.lang.ref.Finalizer lock. Profiling with Pyroscope
showed that ~48% of mutex contentions originated from:
DuckDBResultSet.<init>
→ Object.<init>
→ Finalizer.register
→ Finalizer.<init> [GLOBAL LOCK]
When a class overrides finalize(), the JVM registers every new
instance with the Finalizer system using a global lock. In
applications executing many concurrent queries (e.g., using ZIO,
Akka, or thread pools), this single lock becomes a severe
bottleneck, significantly degrading throughput.
Background
----------
The finalize() methods were added in April 2020 (commit 934af9f)
as a safety net to release native JNI resources if users forgot
to call close(). However, finalize() was deprecated in Java 9
(2017) due to:
- Unpredictable execution timing (GC-dependent)
- Performance overhead (extra GC cycles for weak reachability)
- Global lock contention in the Finalizer registration
- Single-threaded Finalizer thread becoming a bottleneck
The modern replacement (java.lang.ref.Cleaner) was introduced in
Java 9, but this driver targets Java 8 compatibility.
Solution
--------
Remove finalize() from all four classes that had it:
- DuckDBConnection
- DuckDBPreparedStatement
- DuckDBResultSet
- DuckDBSingleValueAppender
All these classes already implement AutoCloseable with proper
close() methods. Users should use try-with-resources:
try (Connection conn = DriverManager.getConnection("jdbc:duckdb:");
PreparedStatement stmt = conn.prepareStatement(sql);
ResultSet rs = stmt.executeQuery()) {
// ...
}
This is standard JDBC best practice and ensures deterministic
resource cleanup without relying on GC finalization.
Impact
------
- Eliminates Finalizer lock contention entirely
- Improves throughput in high-concurrency scenarios
- No behavior change for users who properly close resources
- Users who relied on finalize() for cleanup will now leak
resources if they don't call close() (but finalize() was
never guaranteed to run anyway)
finalize() methods to eliminate Finalizer lock contention
|
Hi, thanks for the PR! I agree that |
This is a backport of the PR duckdb#533 to `v1.5-variegata` stable branch. Problem ------- In highly parallel environments, creating DuckDBResultSet and DuckDBPreparedStatement objects causes severe mutex contention on the global java.lang.ref.Finalizer lock. Profiling with Pyroscope showed that ~48% of mutex contentions originated from: DuckDBResultSet.<init> → Object.<init> → Finalizer.register → Finalizer.<init> [GLOBAL LOCK] When a class overrides finalize(), the JVM registers every new instance with the Finalizer system using a global lock. In applications executing many concurrent queries (e.g., using ZIO, Akka, or thread pools), this single lock becomes a severe bottleneck, significantly degrading throughput. Background ---------- The finalize() methods were added in April 2020 (commit 934af9f) as a safety net to release native JNI resources if users forgot to call close(). However, finalize() was deprecated in Java 9 (2017) due to: - Unpredictable execution timing (GC-dependent) - Performance overhead (extra GC cycles for weak reachability) - Global lock contention in the Finalizer registration - Single-threaded Finalizer thread becoming a bottleneck The modern replacement (java.lang.ref.Cleaner) was introduced in Java 9, but this driver targets Java 8 compatibility. Solution -------- Remove finalize() from all four classes that had it: - DuckDBConnection - DuckDBPreparedStatement - DuckDBResultSet - DuckDBSingleValueAppender All these classes already implement AutoCloseable with proper close() methods. Users should use try-with-resources: try (Connection conn = DriverManager.getConnection("jdbc:duckdb:"); PreparedStatement stmt = conn.prepareStatement(sql); ResultSet rs = stmt.executeQuery()) { // ... } This is standard JDBC best practice and ensures deterministic resource cleanup without relying on GC finalization. Impact ------ - Eliminates Finalizer lock contention entirely - Improves throughput in high-concurrency scenarios - No behavior change for users who properly close resources - Users who relied on finalize() for cleanup will now leak resources if they don't call close() (but finalize() was never guaranteed to run anyway)
This is a backport of the PR #533 to `v1.5-variegata` stable branch. Problem ------- In highly parallel environments, creating DuckDBResultSet and DuckDBPreparedStatement objects causes severe mutex contention on the global java.lang.ref.Finalizer lock. Profiling with Pyroscope showed that ~48% of mutex contentions originated from: DuckDBResultSet.<init> → Object.<init> → Finalizer.register → Finalizer.<init> [GLOBAL LOCK] When a class overrides finalize(), the JVM registers every new instance with the Finalizer system using a global lock. In applications executing many concurrent queries (e.g., using ZIO, Akka, or thread pools), this single lock becomes a severe bottleneck, significantly degrading throughput. Background ---------- The finalize() methods were added in April 2020 (commit 934af9f) as a safety net to release native JNI resources if users forgot to call close(). However, finalize() was deprecated in Java 9 (2017) due to: - Unpredictable execution timing (GC-dependent) - Performance overhead (extra GC cycles for weak reachability) - Global lock contention in the Finalizer registration - Single-threaded Finalizer thread becoming a bottleneck The modern replacement (java.lang.ref.Cleaner) was introduced in Java 9, but this driver targets Java 8 compatibility. Solution -------- Remove finalize() from all four classes that had it: - DuckDBConnection - DuckDBPreparedStatement - DuckDBResultSet - DuckDBSingleValueAppender All these classes already implement AutoCloseable with proper close() methods. Users should use try-with-resources: try (Connection conn = DriverManager.getConnection("jdbc:duckdb:"); PreparedStatement stmt = conn.prepareStatement(sql); ResultSet rs = stmt.executeQuery()) { // ... } This is standard JDBC best practice and ensures deterministic resource cleanup without relying on GC finalization. Impact ------ - Eliminates Finalizer lock contention entirely - Improves throughput in high-concurrency scenarios - No behavior change for users who properly close resources - Users who relied on finalize() for cleanup will now leak resources if they don't call close() (but finalize() was never guaranteed to run anyway)
|
@staticlibs Would it be possible to release a 1.4.4.0 with this fix, please? |
|
No, sorry, only bugfixes go to 1.4 branch now and this is actual behaviour change. I would think there is some (broken) code in the wild that relies on these finalizers. 1.5.0.0 is now planned for Feb 16 and this change goes there. |
|
@staticlibs Do you guys publish SNAPSHOT versions? This is impacting my application. I need to update ASAP. Can't wait for Feb 16 |
|
Unfortunately SNAPSHOTS got disabled after the update on Maven Central side and there is no easy way to restore them there, see #338 for details. We are going to restore snapshots in a custom Maven repo under If you can use custom repos, I suggest to create one manually (like in a github repo) and put there the JDBC driver JAR from CI downloads (for example, from the bottom of this run page) under correct name and dir structure. |
I'd argue that this PR fixes a bug. I understand the behaviour change PoV but I don't think we should stop ourselves from releasing a bug fix because some people have written some broken code using a bug. Maybe we can release the |
Problem
In highly parallel environments, creating
DuckDBResultSetandDuckDBPreparedStatementobjects causes severe mutex contention on the globaljava.lang.ref.Finalizerlock. Profiling with Pyroscope showed that ~48% of mutex contentions originated from:When a class overrides
finalize(), the JVM registers every new instance with theFinalizersystem using a global lock. In applications executing many concurrent queries (e.g., using ZIO, Akka, or thread pools), this single lock becomes a severe bottleneck, significantly degrading throughput.Background
The
finalize()methods were added in April 2020 (commit 934af9f) as a safety net to release native JNI resources if users forgot to callclose(). However,finalize()was deprecated in Java 9 (2017) due to:FinalizerregistrationFinalizerthread becoming a bottleneckThe modern replacement (
java.lang.ref.Cleaner) was introduced in Java 9, but this driver targets Java 8 compatibility.Solution
Remove
finalize()from all four classes that had it:DuckDBConnectionDuckDBPreparedStatementDuckDBResultSetDuckDBSingleValueAppenderAll these classes already implement
AutoCloseablewith properclose()methods. Users should use try-with-resources:This is standard JDBC best practice and ensures deterministic resource cleanup without relying on GC finalization.
Impact
Finalizerlock contention entirelyfinalize()for cleanup will now leak resources if they don't callclose()(butfinalize()was never guaranteed to run anyway)