Skip to content

[PLUGIN-1847] Add SqlServerErrorDetailsProvider#539

Merged
psainics merged 1 commit intodata-integrations:developfrom
cloudsufi:fem/sqlserver
Jan 29, 2025
Merged

[PLUGIN-1847] Add SqlServerErrorDetailsProvider#539
psainics merged 1 commit intodata-integrations:developfrom
cloudsufi:fem/sqlserver

Conversation

@psainics
Copy link
Contributor

@psainics psainics commented Jan 21, 2025

ErrorDetailsProvider - SqlServer [Source|Sink] plugin

Jira : PLUGIN-1847

Description

Implement Program Failure Exception Handling in SqlServer Source/Sink plugin to catch known errors

Code change

  • Added SqlServerErrorDetailsProvider.java
  • Modified DBUtils.java
  • Modified SqlServerSink.java
  • Modified SqlServerSource.java

Tests

  • Test Case (Violate check constrain)
Raw Logs

2025-01-29 09:20:32,606 - ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):o.a.s.u.Utils@98] - Aborting task
io.cdap.cdap.api.exception.WrappedStageException: Stage 'SQL Server4' encountered : io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '547', errorMessage: The INSERT statement conflicted with the CHECK constraint "CHK_Gender". The conflict occurred in database "master", table "dbo.users", column 'gender'.
  at io.cdap.cdap.etl.common.ErrorDetails.handleException(ErrorDetails.java:77)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:67)
  at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.closeWriter(SparkHadoopWriter.scala:373)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$1(SparkHadoopWriter.scala:145)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
  at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:135)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
Caused by: io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '547', errorMessage: The INSERT statement conflicted with the CHECK constraint "CHK_Gender". The conflict occurred in database "master", table "dbo.users", column 'gender'.
  at io.cdap.cdap.api.exception.ProgramFailureException$Builder.build(ProgramFailureException.java:229)
  at io.cdap.cdap.api.exception.ErrorUtils.getProgramFailureException(ErrorUtils.java:198)
  at io.cdap.plugin.common.db.DBErrorDetailsProvider.getProgramFailureException(DBErrorDetailsProvider.java:196)
  at io.cdap.plugin.common.db.DBErrorDetailsProvider.getExceptionDetails(DBErrorDetailsProvider.java:170)
  at io.cdap.cdap.etl.common.ErrorDetails.handleException(ErrorDetails.java:75)
  ... 14 common frames omitted
Caused by: java.sql.BatchUpdateException: The INSERT statement conflicted with the CHECK constraint "CHK_Gender". The conflict occurred in database "master", table "dbo.users", column 'gender'.
  at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2101)
  at io.cdap.plugin.db.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:99)
  at io.cdap.cdap.etl.spark.io.TrackingRecordWriter.close(TrackingRecordWriter.java:46)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:65)
  ... 13 common frames omitted
2025-01-29 09:20:32,609 - WARN  [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):i.c.p.d.s.ETLDBOutputFormat@106] - com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
  at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:237)
  at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:1369)
  at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:3724)
  at io.cdap.plugin.db.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:104)
  at io.cdap.cdap.etl.spark.io.TrackingRecordWriter.close(TrackingRecordWriter.java:46)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:65)
  at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.closeWriter(SparkHadoopWriter.scala:373)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$2(SparkHadoopWriter.scala:150)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1549)
  at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:135)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
  
POST v3/namespaces/{namespace-id}/apps/{app-id}/workflows/DataPipelineWorkflow/runs/{run-id}/classify
[
{
  "stageName": "SQL Server4",
  "errorCategory": "Plugin-'DB Integrity Constraint Violation'-'SQL Server4'",
  "errorReason": "The INSERT statement conflicted with the CHECK constraint \"CHK_Gender\". The conflict occurred in database \"master\", table \"dbo.users\", column 'gender'. For more details, see https://docs.microsoft.com/en-us/sql/relational-databases/errors-events/database-engine-events-and-errors",
  "errorMessage": "Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '547', errorMessage: The INSERT statement conflicted with the CHECK constraint \"CHK_Gender\". The conflict occurred in database \"master\", table \"dbo.users\", column 'gender'.",
  "errorType": "USER",
  "dependency": "true",
  "errorCodeType": "SQLSTATE",
  "errorCode": "23000",
  "supportedDocumentationUrl": "https://docs.microsoft.com/en-us/sql/relational-databases/errors-events/database-engine-events-and-errors"
}
]

image

@psainics psainics added the build label Jan 21, 2025
@psainics psainics self-assigned this Jan 21, 2025
@psainics psainics marked this pull request as ready for review January 23, 2025 07:16
Comment on lines 34 to 42
protected ErrorType getErrorTypeFromErrorCode(int errorCode, String sqlState) {
if ((errorCode >= 50000 && errorCode <= 59999) || (errorCode >= 0 && errorCode <= 4999)) {
// User-defined or low-level errors
return ErrorType.USER;
} else if (errorCode >= 60000 && errorCode <= 99999) {
// System-level errors
return ErrorType.SYSTEM;
} else {
// Errors outside the defined range
return ErrorType.UNKNOWN;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how did we came up with this categorization?

Copy link
Contributor Author

@psainics psainics Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI [ChatGPT]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we verify the categorization?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the range 5000-6000

@Override
protected ErrorType getErrorTypeFromErrorCode(int errorCode, String sqlState) {
if ((errorCode >= 50000 && errorCode <= 59999) || (errorCode >= 0 && errorCode <= 4999)
|| (errorCode >= 5000 && errorCode <= 6000)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we combine it from 0-6000?

@psainics psainics merged commit 10c7548 into data-integrations:develop Jan 29, 2025
8 of 10 checks passed
@psainics psainics deleted the fem/sqlserver branch January 29, 2025 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants