Composite State Store Part 2: EVM database implementation#2755
Conversation
Kbhat1
changed the title
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2755 +/- ##
==========================================
+ Coverage 46.89% 56.60% +9.70%
==========================================
Files 1965 2033 +68
Lines 160600 166253 +5653
==========================================
+ Hits 75312 94105 +18793
+ Misses 78760 63877 -14883
- Partials 6528 8271 +1743
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| for keyStr, value := range keyValues { | ||
| if value != nil { // Skip tombstones | ||
| keys = append(keys, []byte(keyStr)) | ||
| values = append(values, value) | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
Kbhat1
force-pushed
the
feature/evm-ss-router-config
branch
from
ac83d2f to
0935d03
Compare
Kbhat1
force-pushed
the
feature/evm-ss-database
branch
from
7271005 to
7ef1b00
Compare
Kbhat1
force-pushed
the
feature/evm-ss-database
branch
from
7ef1b00 to
906d2ed
Compare
| for storeType, pairs := range changes { | ||
| db := s.databases[storeType] | ||
| if db == nil { | ||
| continue | ||
| } | ||
| if err := db.ApplyBatch(pairs, version); err != nil { | ||
| return fmt.Errorf("failed to apply batch for %s: %w", StoreTypeName(storeType), err) | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| for storeType, pairs := range changes { | ||
| db := s.databases[storeType] | ||
| if db == nil { | ||
| continue | ||
| } | ||
|
|
||
| wg.Add(1) | ||
| go func(db *EVMDatabase, pairs []*iavl.KVPair) { | ||
| defer wg.Done() | ||
| if err := db.ApplyBatch(pairs, version); err != nil { | ||
| errCh <- err | ||
| } | ||
| }(db, pairs) | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| for _, db := range s.databases { | ||
| if v := db.GetLatestVersion(); v > maxVersion { | ||
| maxVersion = v | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| for _, db := range s.databases { | ||
| if err := db.SetLatestVersion(version); err != nil { | ||
| return err | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| for _, db := range s.databases { | ||
| v := db.GetEarliestVersion() | ||
| if minVersion < 0 || v < minVersion { | ||
| minVersion = v | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| for _, db := range s.databases { | ||
| if err := db.SetEarliestVersion(version); err != nil { | ||
| return err | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| for _, db := range s.databases { | ||
| if err := db.Close(); err != nil { | ||
| lastErr = err | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| go func(db *EVMDatabase, pairs []*iavl.KVPair) { | ||
| defer wg.Done() | ||
| if err := db.ApplyBatch(pairs, version); err != nil { | ||
| errCh <- err | ||
| } | ||
| }(db, pairs) |
Check notice
Code scanning / CodeQL
Spawning a Go routine Note
Kbhat1
force-pushed
the
feature/evm-ss-database
branch
from
906d2ed to
851b64c
Compare
| for _, db := range s.databases { | ||
| wg.Add(1) | ||
| go func(db *EVMDatabase) { | ||
| defer wg.Done() | ||
| if err := db.Prune(version); err != nil { | ||
| errCh <- err | ||
| } | ||
| }(db) | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| go func(db *EVMDatabase) { | ||
| defer wg.Done() | ||
| if err := db.Prune(version); err != nil { | ||
| errCh <- err | ||
| } | ||
| }(db) |
Check notice
Code scanning / CodeQL
Spawning a Go routine Note
yzang2019
left a comment
yzang2019
left a comment
There was a problem hiding this comment.
Please also address all lint issues
Kbhat1
force-pushed
the
feature/evm-ss-database
branch
from
851b64c to
3801df9
Compare
| "encoding/binary" | ||
| "fmt" | ||
| "path/filepath" | ||
| "runtime" |
Check notice
Code scanning / CodeQL
Sensitive package import Note
Adds configuration and type definitions for EVM state stores: - EVMStateStoreConfig in config/config.go for enabling EVM SS - EVMStoreType alias to EVMKeyKind from common/evm - AllEVMStoreTypes() for iteration over store types - StoreTypeName() for DB directory naming Also renames common/evm/memiavl_keys.go to keys.go and ParseMemIAVLEVMKey to ParseEVMKey for broader applicability.
Adds the core EVM state store components: db.go - EVMDatabase (single PebbleDB for a specific EVM data type): - Uses DefaultComparer (lexicographic byte ordering) - Versioned key encoding with big-endian version suffix - ApplyBatch for efficient batch writes - EVMIterator for version-aware iteration (for state dumps/full scans) store.go - EVMStateStore (manages multiple EVMDatabase instances): - Manages databases for nonce, code, codehash, codesize, storage - ApplyChangesetParallel for concurrent writes to multiple DBs - Uses commonevm.ParseEVMKey directly for key routing
- Use runtime.NumCPU() for MaxConcurrentCompactions - Remove sync writes for version metadata (improves performance) - Parallelize EVMStateStore.Prune across all databases
- Fix errcheck: properly handle Close() error returns with _ = or defer func - Fix gosec G115: add nolint comments for safe int64/uint64 conversions - Fix staticcheck S1009: remove redundant nil checks before len()
- Use atomic.Int64 for latestVersion/earliestVersion to fix race condition - Add github.com/cockroachdb/pebble v1.1.2 as direct dependency (was indirect) - All lint fixes from previous commit included
Kbhat1
force-pushed
the
feature/evm-ss-database
branch
from
40f2a73 to
7c8c4b4
Compare
Metadata keys (latest_version, earliest_version) are stored without a version suffix, unlike data keys. The iterator was incorrectly decoding all keys first, which stripped 8 bytes from metadata key names, causing the skip check to fail. Now checks raw iterator key against metadata key names before calling decodeKey, matching the pattern used in Prune().
3 participants
Describe your changes and provide context
Testing performed to validate your change