LogRecord and SpanByte changes #1186
Open
+31,088
−26,663
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Move CountdownWrapper to its own file - Rename Lockable* to Transactional* - Rename Transient* to Ephemeral*
Other work to get CreateNewRecordRMW clean
…ns first pass done)
vazois
reviewed
Nov 20, 2025
| firstRead = true; | ||
| } | ||
|
|
||
| // Note: We may be sending to multiple replicas, so cannot serialize LogRecords directly to the network buffer |
Collaborator
There was a problem hiding this comment.
Every connection has a distinct network buffer, so we can technically serialize it directly to the network buffer.
Is there another reason why we cannot write the key,value pair directly to the network buffer as previously?
TalZaccai
approved these changes
Nov 21, 2025
| /// </summary> | ||
| public PinnedSpanByte GetAvailableNetworkBufferSpan() => PinnedSpanByte.FromPinnedPointer(curr, (int)(end - curr)); | ||
|
|
||
| public void IncrementRecordDirect(int size) |
Contributor
There was a problem hiding this comment.
Looks like this is not used anywhere? Do we need it?
| db.MainStore.Reset(); | ||
| if (db.ObjectStore?.Log.TailAddress > 64) | ||
| db.ObjectStore?.Reset(); | ||
| if (db.Store.Log.TailAddress > 64) |
Contributor
There was a problem hiding this comment.
Should this be PageHeader.Size?
libs/server/Storage/Session/Common/ArrayKeyIterationFunctions.cs
Outdated
Show resolved
Hide resolved
| var valueSB = scratchBufferBuilder.FormatScratch(sizeof(long), value).SpanByte; | ||
| valueSB.ExtraMetadata = DateTimeOffset.UtcNow.Ticks + expiry.Ticks; | ||
| return SET(ref _key, ref valueSB, ref context); | ||
| var input = new RawStringInput(RespCommand.APPEND, ref parseState, arg1: DateTimeOffset.UtcNow.Ticks + expiry.Ticks); |
…threshold configuration option
…object-log segments); PendingContext is no longer itself a TSourceLogRecord;
…an dispose the log devices. Remove KVSettings from CachedSizeTracker as it was unused Fix some Tsavorite LogRecord-lifetime issues on scan and conditional insert/push
… into tedhar/storage-v2
Fix CacheSerializedData to not overwrite if CopyUpdater already set the object Fix foldover checkpoint storage of object log tail
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR converts Garnet to use a new ObjectAllocator with a revised in-memory and on-disk format that will use only a single log file, change ISessionFunctions to be (Disk)LogRecord-based and propagate this to operations such as Compaction and Migration, remove TKey and TValue from TsavoriteKV, and many related refactors.
Highlights:
(ReadOnly)Span<byte>extensions.ReadOnlySpan<byte>at the StorageSession and below, including Tsavorite. There are no longer any byte[] keys. The TKey type argument is gone from Tsavorite and Garnet.(ReadOnly)Span<byte>or objects implementing IHeapObject. TValue has been removed from TsavoriteKV.Major remaining tasks to be addressed in follow-on PRs:
(ReadOnly)Span<byte>path by replacing byte* usage (such as was done for Utility.HashBytes).