-
Notifications
You must be signed in to change notification settings - Fork 13.8k
[FLINK-38819] Add debug logs for serialisation errors #27370
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
6490362 to
9a44dec
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the improvement.
Left a few of comments.
PTAL~
...link-json/src/main/java/org/apache/flink/formats/json/AbstractJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...link-json/src/main/java/org/apache/flink/formats/json/AbstractJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...k-json/src/main/java/org/apache/flink/formats/json/canal/CanalJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...k-json/src/main/java/org/apache/flink/formats/json/canal/CanalJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/flink/formats/json/debezium/DebeziumJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/flink/formats/json/debezium/DebeziumJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...on/src/main/java/org/apache/flink/formats/json/maxwell/MaxwellJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...on/src/main/java/org/apache/flink/formats/json/maxwell/MaxwellJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...flink-json/src/main/java/org/apache/flink/formats/json/ogg/OggJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
...flink-json/src/main/java/org/apache/flink/formats/json/ogg/OggJsonDeserializationSchema.java
Outdated
Show resolved
Hide resolved
|
Thanks @RocMarshal for the review. I have addressed the comments. PTAL! |
...link-json/src/main/java/org/apache/flink/formats/json/AbstractJsonDeserializationSchema.java
Show resolved
Hide resolved
ed9d6fd to
e51e5b3
Compare
|
@flinkbot run azure |
| if (!ignoreParseErrors) { | ||
| throw new IOException( | ||
| format( | ||
| "Unknown \"type\" value \"%s\". The Canal JSON message is '%s'", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the IOException is the same as the debug content - can we use the same variable?
| } | ||
|
|
||
| @Test | ||
| void testIgnoreParseErrorsLogsDebugMessage() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you add a test case for fail on missing as well please. This also comes out of JsonParserToRowDataConverters as a JsonParseException.
What is the purpose of the change
The pull request logs deserialisation errors as debug logs when
ignoreParseErrorsis set to true.Brief change log
(for example:)
Verifying this change
Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
@Public(Evolving): NoDocumentation