PLUGIN-1936: Read data in a streaming manner by sgarg-CS · Pull Request #131 · data-integrations/servicenow-plugins

sgarg-CS · 2025-11-10T13:54:24Z

PLUGIN-1936

Issue : OutOfMemory (OOM) Errors seen in the ServiceNow Connector frequently when transfering large number of records

RCA: The entire response is converted into string and the JSON is parsed in memory leading to OOM issues.

Proposed Fix: Read the records incrementally i.e. one by one in a streaming manner.

Changes :

Refactored the existing method: prepareResponse :in the RESTAPIResponse class to return a reusable InputStream in the RESTAPIResponse object.
Added openNextPage to initialise the Gson's JsonReader and read the result array in the response and closeCurrentPage to close the JsonReader
nextKeyValue() : Refactored the existing logic to stream one record at a time.
Changed the return type of the following methods from Map<String, String> to RESTAPIResponse
- fetchData()
- fetchTableRecordsRetryableMode()
- fetchTableRecords()

Testing

Sunish-Dahiya · 2025-12-30T06:35:06Z

src/main/java/io/cdap/plugin/servicenow/ServiceNowBaseConfig.java

  }

+  /**
+   * Check whether the result is empty or not.


Also explain how are we checking is result is empty

It specifically looks for a top-level key named result. Once found, it opens the associated array and checks for the presence of a first element. If it does contain at least one element, then it return false, otherwise true.

Updated the method comment. ba5eb53

Sunish-Dahiya · 2025-12-30T06:55:19Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

    return GSON.fromJson(ja, type);
  }

+  public List<Map<String, String>> parseResponseToResultListOfMap(InputStream in) throws ServiceNowAPIException {


Why are we creating a List of map? Instead we can directly read it as a JsonObject, something like:

gson.fromJson(reader, ServiceNowRecordObject.class);

Refactored the code to parse the the RestAPIResponse into result. ba5eb53

APIResponse apiResponse = GSON.fromJson(new JsonReader(new InputStreamReader(in, StandardCharsets.UTF_8)), APIResponse.class); return apiResponse.getResult();

The calling method : serviceNowTableAPIClient.parseResponseToResultListOfMap in ServiceNowConnector.getTableData expects a result of type: List<Map<String, String>>

Can you check the APIResponse class once? When you say ServiceNowRecordObject class, do you expect that class to hold the result array returned by the ServiceNow Table REST API ? Please clarify.

Sunish-Dahiya · 2025-12-30T06:55:58Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

                                                                  String startDate, String endDate, int offset,
                                                                  int limit) throws ServiceNowAPIException {
-    final List<Map<String, String>> results = new ArrayList<>();
+    //final List<Map<String, String>> results = new ArrayList<>();


please remove

removed ba5eb53

Sunish-Dahiya · 2025-12-30T06:56:06Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

+    final RestAPIResponse[] restAPIResponse = new RestAPIResponse[1];
    Callable<Boolean> fetchRecords = () -> {
-      results.addAll(fetchTableRecords(tableName, valueType, startDate, endDate, offset, limit));
+      // results.addAll(fetchTableRecords(tableName, valueType, startDate, endDate, offset, limit));


please remove

removed ba5eb53

Sunish-Dahiya · 2025-12-30T06:57:46Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

    Callable<Boolean> fetchRecords = () -> {
-      results.addAll(fetchTableRecords(tableName, valueType, startDate, endDate, offset, limit));
+      // results.addAll(fetchTableRecords(tableName, valueType, startDate, endDate, offset, limit));
+      restAPIResponse[0] = fetchTableRecords(tableName, valueType, startDate, endDate, offset, limit);


can you please explain what are we trying to do here? why restAPIResponse is an array

In Java, when we use a variable from an outer scope inside a lambda expression or a Callable, that variable must be effectively final. This means its reference cannot change after it is initialised. so, I was creating an array earlier for RestAPIResponse and returning only the first element. I've now replaced it with AtomicReference. ba5eb53

Sunish-Dahiya · 2025-12-30T07:14:38Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowMultiRecordReader.java

  private final ServiceNowMultiSourceConfig multiSourcePluginConf;
  private ServiceNowTableAPIClientImpl restApi;
+  private final Gson gson = new Gson();
+  private final Type mapType = new TypeToken<Map<String, String>>() { }.getType();


should use an object instead of map

Do we need to replace TypeToken<Map<String, String>>() with TypeToken<Object>() here? Please confirm.

Sunish-Dahiya · 2025-12-30T07:17:02Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowMultiRecordReader.java

+      return false;
+    }
+    this.jsonReader = new JsonReader(new InputStreamReader(in, StandardCharsets.UTF_8));
+    this.jsonReader.setLenient(true);


why we need this?

In general, the ServiceNow Table REST API provides standard-compliant JSON that does not strictly require jsonreader.setlenient(true) for basic parsing. However, there are specific scenarios where enabling leniency becomes necessary or is a recommended safeguard.

If the API call fails (e.g., 401 Unauthorised, 404 Not Found, or a ServiceNow "maintenance" page), the server might return an HTML body instead of JSON. A strict parser will fail immediately but a lenient one may provide a more readable error or allow you to handle the string manually.

Sunish-Dahiya · 2025-12-30T07:17:47Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowMultiRecordReader.java

+      JsonToken top;
+      try {
+        top = jsonReader.peek();
+        LOG.info("Peeking JSON token for table {}: {}", tableName, top);


please remove, it should be a debug log

Yes, Changed it to DEBUG ba5eb53

Sunish-Dahiya · 2025-12-30T07:18:46Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowMultiRecordReader.java

+  }
+
+  public void closeCurrentPage() {
+    LOG.info("Closing current page for table {}", tableName);


not needed?

Yes, had added this logger during dev testing to troubleshoot an issue. Removed it now. ba5eb53

Sunish-Dahiya · 2025-12-30T07:19:56Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowRecordReader.java

+    return restAPIResponse;
+  }
+
+  private boolean openNextPage() throws IOException, ServiceNowAPIException {


looks redundant, as same function is present in src/main/java/io/cdap/plugin/servicenow/source/ServiceNowMultiRecordReader.java

can we abstract these out in a common file. Same comment for nextKeyValue()

Added the implementation of nextKey(), openNextPage() and closeCurrentPage() to the ServiceNowBaseRecordReader class ba5eb53

Sunish-Dahiya · 2026-01-14T08:57:14Z

src/main/java/io/cdap/plugin/servicenow/ServiceNowBaseConfig.java

  }

+  /**
+   * Determines if the "result" array in a ServiceNow REST API response is empty. It specifically looks for a top-level


can you please format it as per java doc.
/** One line summary
*

Additional details

params
*@return
*/

Formatted 6115173

Sunish-Dahiya · 2026-01-14T08:59:58Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

                                                                  int limit) throws ServiceNowAPIException {
-    final List<Map<String, String>> results = new ArrayList<>();
+    // Using AtomicReference to capture a value inside a lambda that needs to be accessed outside.
+    AtomicReference<RestAPIResponse> responseRef = new AtomicReference<>();


this seems like a overkill, it's better to abstract the logic in a function instead of lambda to simplify the code here

Refactored the code by separating out the logic in a private method. PTAL. 6115173

Sunish-Dahiya · 2026-01-14T09:01:30Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

-                                          String tableName) {
-    List<Map<String, String>> result = parseResponseToResultListOfMap(restAPIResponse.getResponseBody());
+                                          String tableName) throws ServiceNowAPIException {
+    List<Map<String, String>> result = parseResponseToResultListOfMap(restAPIResponse.getBodyAsStream());


Instead of a Map, we should use well defined Java Objects.
Since we are already refactoring the code we can always change the caller functions, I don't see that as a blocker

Sunish-Dahiya · 2026-01-14T09:01:43Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

-    List<Map<String, String>> result = parseResponseToResultListOfMap(restAPIResponse.getResponseBody());
+                                          String tableName) throws ServiceNowAPIException {
+    List<Map<String, String>> result = parseResponseToResultListOfMap(restAPIResponse.getBodyAsStream());
+    // List<Map<String, String>> result = parseResponseToResultListOfMap(restAPIResponse.getResponseBody());


I can still see the comment

Sunish-Dahiya · 2026-01-14T09:02:21Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

    return GSON.fromJson(ja, type);
  }

+  public List<Map<String, String>> parseResponseToResultListOfMap(InputStream in) {


Should use well defined Objects here instead of Map

Sorry, it's not clear to me what's the ask here. Can you please elaborate more what you mean by well defined objects ? Are you looking for a POJO class here?

If I understand it correctly this Map is used to store the response of the API, if yes then response can be represented by a POJO class instead of a generic map

Yes, it is used to store the ServiceNow API Response. Can we use this POJO class ?

Sunish-Dahiya · 2026-01-14T09:07:30Z

src/main/java/io/cdap/plugin/servicenow/restapi/RestAPIResponse.java

+        inputStream, MAX_PAGE_BYTES + 1); // +1 to detect overflow
+      responseBody = IOUtils.toByteArray(boundedInputStream);
+      LOG.info("RAW JSON: {}", new String(responseBody, StandardCharsets.UTF_8));
+      if (responseBody.length > MAX_PAGE_BYTES) {


Loading the complete page defeats the whole purpose of this change altogether. We are making this change to avoid OOM issues when we load the entire page into memory and loading it into a bounded buffer is not solving that problem.

Regarding multiple access of the input stream: We can always refactor out code(validation etc.) such that all the processing happens record by record instead of the whole page.
Basically when the connector request for a record we can do all the processing/validation is required, instead of doing it beforehand. This would solve the problem of multiple access of InputStream.

Sunish-Dahiya · 2026-01-14T09:08:39Z

src/main/java/io/cdap/plugin/servicenow/sink/service/ServiceNowSinkAPIRequestImpl.java


-      JsonObject responseJSON = jsonParser.parse(apiResponse.getResponseBody()).getAsJsonObject();
+      JsonObject responseJSON = jsonParser.parse(
+        new InputStreamReader(apiResponse.getBodyAsStream(), StandardCharsets.UTF_8)).getAsJsonObject();


Are we switching to stream for POST methods also? Can't we restrict it to read/GET operations only?

Sunish-Dahiya · 2026-01-14T09:10:50Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowBaseRecordReader.java

  }
+
+  /**
+   * The refactored nextKeyValue() — uses Gson JsonReader to stream one record at a time.


Please rephrase the comment here, refactored doesn't provide any context to the reader, the purpose of comment is not to provide the history of the change but explain the logic

Rephrased. PTAL. 6115173

Sunish-Dahiya · 2026-01-14T09:11:31Z

src/main/java/io/cdap/plugin/servicenow/source/ServiceNowBaseRecordReader.java

+    closeCurrentPage();
+    LOG.debug("Fetching data for table {} at offset {}", tableName, split.getOffset());
+    RestAPIResponse resp = fetchData();
+    LOG.debug("Fetched data for table {} at offset {}", tableName, split.getOffset());


do we really need these many debug logs?

Not really, these were more for dev-testing. I can remove them. 4b89ca5

itsankit-google

Please fix the compilation failure in cdap-e2e-tests:

Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project servicenow-plugins: Compilation failure
[ERROR] /tmp/servicenow-plugins/servicenow-plugins/plugin/src/e2e-test/java/io/cdap/plugin/servicenowsink/actions/ServiceNowSinkPropertiesPageActions.java:[66,78] incompatible types: com.google.gson.JsonObject cannot be converted to java.util.Map<java.lang.String,java.lang.String>

itsankit-google

Still failures with JsonSyntaxExcsption : https://storage.googleapis.com/e2e-tests-cucumber-reports/servicenow-plugins/refs/pull/131/merge/cucumber-reports/advanced-reports/cucumber-html-reports/report-feature_4_2095121949.html

itsankit-google · 2026-02-16T16:36:14Z

src/main/java/io/cdap/plugin/servicenow/apiclient/ServiceNowTableAPIClientImpl.java

    return GSON.fromJson(ja, type);
  }

+  public List<JsonObject> parseResponseToResultListOfMap(InputStream in) {


this method is returning List<JsonObject> but the name of method is parseResponseToResultListOfMap, please fix.

Renamed the method accordingly. 016cdfb

itsankit-google · 2026-02-16T16:38:58Z

pom.xml

+    <dependency>
+      <groupId>commons-io</groupId>
+      <artifactId>commons-io</artifactId>
+      <version>2.5</version>


why using 2.5 when newer versions are available?

This is unused now. Removed it. 016cdfb

itsankit-google · 2026-02-16T16:39:05Z

pom.xml

    </dependency>
+    <dependency>
+      <groupId>commons-io</groupId>
+      <artifactId>commons-io</artifactId>


Also please move the version to properties section.

Not needed now.

sgarg-CS marked this pull request as ready for review November 25, 2025 04:20

Sunish-Dahiya reviewed Dec 30, 2025

View reviewed changes

Sunish-Dahiya requested a review from albertshau December 30, 2025 07:20

itsankit-google added the build label Dec 30, 2025

sgarg-CS requested a review from Sunish-Dahiya January 13, 2026 12:44

Sunish-Dahiya reviewed Jan 14, 2026

View reviewed changes

sgarg-CS force-pushed the feature/PLUGIN-1936 branch 5 times, most recently from 5a062f7 to 47c753b Compare February 16, 2026 14:31

PLUGIN-1936: Streaming Implementation

c24f96b

sgarg-CS force-pushed the feature/PLUGIN-1936 branch from 26c0bb1 to c24f96b Compare February 16, 2026 14:41

itsankit-google requested changes Feb 16, 2026

View reviewed changes

PLUGIN-1936: Fix e2e-tests

ab1a341

itsankit-google self-requested a review February 16, 2026 16:07

itsankit-google requested changes Feb 16, 2026

View reviewed changes

PLUGIN-1936: Refactor executePost method to use a shared httpClient

14ed9f2

itsankit-google reviewed Feb 16, 2026

View reviewed changes

sgarg-CS added 2 commits February 16, 2026 23:56

PLUGIN-1936: Rename method and remove unused imports and dependency

016cdfb

PLUGIN-1936: Fix failing UT

edc544a

Conversation

sgarg-CS commented Nov 10, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itsankit-google left a comment

Choose a reason for hiding this comment

Uh oh!

itsankit-google left a comment

sgarg-CS commented Nov 10, 2025 •

edited by atlassian bot

Loading