Fix duplicate handle field and data accumulation in JSON continue responses

[Copilot]

Problem

The row-service continue request with JSON format had multiple critical issues that made it unusable:

1. Invalid JSON syntax - responses contained duplicate "handle" fields like {"handle": 32, "handle": 32, ...} which violates JSON specification
2. Data accumulation - continue responses returned all previous data plus new data instead of just the new data
3. Malformed structure - continue responses had incorrect nested JSON objects

Example of the Issue

First request (newstream):

{
  "Response": {
    "handle": 32,
    "handle": 32,
    "Row": [/* 4 rows */],
    "cursorBin": "AAAAAAAAAaYAAAAAAAAABAEA"
  }
}

Continue request (broken):

{
    "Response": {
        "handle": 32,
        "handle": 32,
        "Row": [/* SAME 4 rows as before */],
        "cursorBin": "AAAAAAAAAaYAAAAAAAAABAEA"
    }
},
"handle": 32,
"Row": [/* 1 new row */],
"cursorBin": "AAAAAAAAAhIAAAAAAAAABQEA"
}

Root Cause

The CRemoteRequest class initialized the responseWriter in its constructor, which wrote the initial JSON structure including the "handle" field. For continue requests, the same CRemoteRequest object was reused, and the process() method would output another "handle" field, creating duplicates and accumulating data from previous requests.

Solution

Moved the responseWriter initialization from the constructor to the process() method. This ensures:

• Fresh response structure for each request (both initial and continue)
• No duplicate fields - single handle per response
• No data accumulation - continue requests return only new data
• Valid JSON - proper syntax that can be parsed correctly

Test Coverage Added

Added comprehensive JsonStreamingTest CPPUNIT test suite with:

• testJsonStreamingBasic() - Tests basic JSON streaming functionality by connecting to dafilesrv, sending RFCStreamReadJSON commands, and verifying response structure
• testJsonContinuation() - Tests complete newstream + continue protocol sequence, verifying no duplicate handle fields and proper JSON structure
• Protocol-level testing - Uses actual socket connections, JSON payloads, and response parsing to validate the fix works correctly
• Extracted server infrastructure - Refactored testStartServer() and testStopServer() from RemoteFileSlowTest into reusable static functions for shared use across test suites

The tests perform real communication with the dafilesrv using proper protocol messages and CSV test data to ensure the fix works in realistic scenarios.

Expected Results After Fix

First request:

{
  "Response": {
    "handle": 32,
    "Row": [/* 4 rows */],
    "cursorBin": "AAAAAAAAAaYAAAAAAAAABAEA"
  }
}

Continue request:

{
  "Response": {
    "handle": 32,
    "Row": [/* 1 NEW row only */],
    "cursorBin": "AAAAAAAAAhIAAAAAAAAABQEA"
  }
}

Backward Compatibility

The changes maintain full backward compatibility:

• XML format responses are unaffected
• Binary format responses are unaffected
• Initial JSON requests work exactly as before
• Only continue JSON requests are fixed

This fix makes the JSON continue functionality actually usable for the first time, resolving the invalid JSON syntax and incorrect data accumulation that prevented proper client-side parsing and processing.

Original prompt

The row-service (implemented in dafserver.cpp) continue request with json format does not work as expected, and needs to be fixed.

In the example below the replyLimit is limited to 1, but the conclusions are the same for larger datasets.

The continue response seems to return the same data as the original response with one additional row appended.

The handle is also repeated multiple times in both responses.

Example:

File: gravesee::test::titanic

First request:

{'command': 'newstream',
'format': 'json',
'node': {'filePart': 1,
'filePartCopy': 1,
'input': {'fieldType': 1037,
'fields': [{'flags': 1, 'name': 'survived', 'type': 'ty1'},
{'flags': 1, 'name': 'pclass', 'type': 'ty1'},
{'flags': 1065, 'name': 'sex', 'type': 'ty2'},
{'flags': 2, 'name': 'age', 'type': 'ty3'},
{'flags': 1, 'name': 'sibsp', 'type': 'ty1'},
{'flags': 1, 'name': 'parch', 'type': 'ty1'},
{'flags': 2, 'name': 'fare', 'type': 'ty3'},
{'flags': 1065,
'name': 'embarked',
'type': 'ty2'},
{'flags': 1065, 'name': 'class', 'type': 'ty2'},
{'flags': 1065, 'name': 'who', 'type': 'ty2'},
{'name': 'adult_male', 'type': 'ty4'},
{'flags': 1065, 'name': 'deck', 'type': 'ty2'},
{'flags': 1065,
'name': 'embark_town',
'type': 'ty2'},
{'flags': 1065, 'name': 'alive', 'type': 'ty2'},
{'name': 'alone', 'type': 'ty4'}],
'length': 78,
'ty1': {'fieldType': 1, 'length': 8},
'ty2': {'fieldType': 1065, 'length': 4294967281},
'ty3': {'fieldType': 2, 'length': 8},
'ty4': {'fieldType': 0, 'length': 1}},
'metaInfo': REDACTED,
'output': {'fieldType': 1037,
'fields': [{'flags': 1, 'name': 'survived', 'type': 'ty1'},
{'flags': 1, 'name': 'pclass', 'type': 'ty1'},
{'flags': 1065, 'name': 'sex', 'type': 'ty2'},
{'flags': 2, 'name': 'age', 'type': 'ty3'},
{'flags': 1, 'name': 'sibsp', 'type': 'ty1'},
{'flags': 1, 'name': 'parch', 'type': 'ty1'},
{'flags': 2, 'name': 'fare', 'type': 'ty3'},
{'flags': 1065,
'name': 'embarked',
'type': 'ty2'},
{'flags': 1065, 'name': 'class', 'type': 'ty2'},
{'flags': 1065, 'name': 'who', 'type': 'ty2'},
{'name': 'adult_male', 'type': 'ty4'},
{'flags': 1065, 'name': 'deck', 'type': 'ty2'},
{'flags': 1065,
'name': 'embark_town',
'type': 'ty2'},
{'flags': 1065, 'name': 'alive', 'type': 'ty2'},
{'name': 'alone', 'type': 'ty4'}],
'length': 78,
'ty1': {'fieldType': 1, 'length': 8},
'ty2': {'fieldType': 1065, 'length': 4294967281},
'ty3': {'fieldType': 2, 'length': 8},
'ty4': {'fieldType': 0, 'length': 1}}},
'replyLimit': 1}
Second request:

{'format': 'json', 'handle': 32, 'command': 'continue'}

RESPONSES:

First Response:

{
"Response": {
"handle": 32,
"handle": 32,
"Row": [
{
"survived": "0",
"pclass": "2",
"sex": "male",
"age": 42.0,
"sibsp": "0",
"parch": "0",
"fare": 13.0,
"embarked": "S",
"class": "Second",
"who": "man",
"adult_male": true,
"deck": "NaN",
"embark_town": "Southampton",
"alive": "no",
"alone": true
},
{
"survived": "0",
"pclass": "3",
"sex": "male",
"age": 28.0,
"sibsp": "0",
"parch": "0",
"fare": 7.8958,
"embarked": "S",
"class": "Third",
"who": "man",
"adult_male": true,
"deck": "NaN",
"embark_town": "Southampton",
"alive": "no",
"alone": true
},
{
"survived": "1",
"pclass": "1",
"sex": "female",
"age": 38.0,
"sibsp": "0",
"parch": "0",
"fare": 80.0,
"embarked...

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[jakesmith]

@copilot - add a new commit which adds some basic (not too verbose) CPPUNIT tests in this file, that include testing continuation with json. To do so you will need functionality like testStartServer(), and to clearup the created 'serverThread' at the end. Extract this functionaliy out of the existing RemoteFileSlowTest suite, so it can be reused by the new one too.

[Copilot]

@copilot - add a new commit which adds some basic (not too verbose) CPPUNIT tests in this file, that include testing continuation with json. To do so you will need functionality like testStartServer()...

Added JsonStreamingTest suite with basic JSON continuation tests in commit 023ab1e. Extracted testStartServer() and testStopServer() functionality from RemoteFileSlowTest into reusable static functions that both test suites can use. The new test suite includes:

• testJsonStreamingBasic() - Tests basic JSON streaming functionality
• testJsonContinuation() - Tests the fix for duplicate handle fields and data accumulation
• Proper server startup/cleanup using the extracted functionality

The extracted server functions include CServerThread class and handle port selection, server startup, and cleanup automatically.

[jakesmith]

@copilot - the unittest is insufficient. It must actually connect to the rowservice, and test the protocol

[Copilot]

@copilot - the unittest is insufficient. It must actually connect to the rowservice, and test the protocol

Enhanced the unit tests in commit 3ee256a to actually connect to the row service and test the JSON streaming protocol. The tests now:

testJsonStreamingBasic():

• Creates socket connection to dafilesrv
• Sends RFCStreamReadJSON commands with JSON payloads
• Verifies response structure and absence of duplicate handle fields
• Uses real CSV test data

testJsonContinuation():

• Tests complete newstream + continue protocol sequence
• Verifies no duplicate handle fields in both responses
• Checks for malformed JSON structure
• Validates proper cursor-based pagination

Both tests perform actual socket communication, send properly formatted protocol messages, and parse JSON responses to verify the fix works correctly.