sharedSchema: add more colum info checks for using shared schema (#3875) #3877
Conversation
Signed-off-by: ti-chi-bot <[email protected]>
ti-chi-bot
added
do-not-merge/hold
|
@hongyunyan This PR has conflicts, I have hold it. |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
Summary of ChangesHello @ti-chi-bot, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the accuracy and robustness of the shared schema mechanism by extending the column information checks to include more detailed attributes. This ensures that schema changes, especially those involving default values, generated columns, or hidden properties, are correctly identified. To validate these enhancements, a new, comprehensive integration test suite has been added and integrated into the CI pipeline, simulating schema evolution and verifying data consistency across different replication targets. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the shared schema mechanism by including more column attributes in the schema comparison and hashing logic. This improves the accuracy of detecting schema differences. A new comprehensive integration test has been added to validate these changes.
My review found a critical issue: there are merge conflict markers in tests/integration_tests/run_light_it_in_ci.sh that must be resolved. I also have a suggestion to further improve the hashing logic in pkg/common/table_info_helper.go for better performance by including GeneratedExprString in the hash calculation. Overall, the changes are good and the new test is thorough.
| <<<<<<< HEAD | ||
| 'row_format tiflash multi_rocks fail_over_ddl_M' | ||
| ======= | ||
| # ds_memory_control | ||
| 'row_format tiflash multi_rocks fail_over_ddl_M correctness_for_shared_column_schema' | ||
| >>>>>>> e9cc243ed (sharedSchema: add more colum info checks for using shared schema (#3875)) |
There was a problem hiding this comment.
This file contains merge conflict markers (<<<<<<<, =======, >>>>>>>). Please resolve the conflicts before merging.
| binary.BigEndian.PutUint64(buf, uint64(boolToInt(col.Hidden))) | ||
| sha256Hasher.Write(buf) | ||
| binary.BigEndian.PutUint64(buf, col.Version) | ||
| sha256Hasher.Write(buf) |
There was a problem hiding this comment.
It's great that you've added more column attributes to the hash calculation for better accuracy. I noticed that GeneratedExprString is checked in sameColumnsAndIndices but not included in the hash here. This will lead to hash collisions if two schemas differ only by GeneratedExprString, which can degrade performance as it requires a linear scan to find the correct schema. Since GeneratedExprString is a string, it should be straightforward to include it in the hash. Consider adding it to the hash calculation. You would also need to update the corresponding test case in TestGetOrSetColumnSchema_SameColumnsAndIndices_ChecksAdditionalColumnAttrs to expect expectDigestSame: false for the 'generated expr string' case.
The same applies to OriginDefaultValue and DefaultValue, though hashing them is more complex as they are interface{}. You could consider JSON marshalling them before hashing if performance with many schemas differing only in default values becomes a concern.
sha256Hasher.Write(buf)
sha256Hasher.Write([]byte(col.GeneratedExprString))
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hongyunyan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
hongyunyan
merged commit 83c7c3d
into
pingcap:8.5.5-release.3-hotfix-0101
2 participants
This is an automated cherry-pick of #3875
What problem does this PR solve?
Issue Number: close #3876
What is changed and how it works?
This pull request significantly improves the accuracy and robustness of the shared schema mechanism by extending the column information checks to include more detailed attributes. This ensures that schema changes, especially those involving default values, generated columns, or hidden properties, are correctly identified. To validate these enhancements, a new, comprehensive integration test suite has been added and integrated into the CI pipeline, simulating schema evolution and verifying data consistency across different replication targets.
Highlights
sameColumnsAndIndicesfunction inpkg/common/table_info_helper.gonow performs more thorough checks, comparing additional column properties likeOriginDefaultValue,DefaultIsExpr,GeneratedStored,Hidden,GeneratedExprString, andVersionto ensure accurate schema identification.correctness_for_shared_column_schema, has been introduced. This suite validates the shared schema handling, particularly focusing on schema evolution scenarios and data consistency across various sink types.Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note