issue-29: Add a consumer to process requests in chunks [fix]

[wiitt]

Solves #31
This PR adds the way to handle files grater in size than 1Gb. It adds and utilizes a consumer that processes responses in chunks.

How to test
Download files of various types and sizes.

The code has been already tested with the following request parameters:

params = struct(...
    'deviceCode', 'ASLCORAS300KHZMAIN',...
    'dateFrom', '2025-01-08T00:00:00.000Z',...
    'dateTo', '2025-01-09T00:00:00.000Z',...
    'extension','dat'...
    ); 
params = struct(...
    'deviceCode', 'CMAPCYCLOPS01',...
    'dateFrom', '2008-10-07T18:50:00.000Z',...
    'dateTo', '2008-10-07T18:52:00.000Z',...
    'extension','mpg'...
    );
params = struct(...
    'deviceCode', 'VWPiez62473',...
    'dateFrom', '2012-11-11T00:00:00.000Z',...
    'dateTo', '2012-11-12T00:00:00.000Z',...
    'extension','dat'...
    );
params = struct(...
    'deviceCode', 'SF_2100_0075',...
    'dateFrom', '2009-07-02T00:00:00.000Z',...
    'dateTo', '2009-07-03T00:00:00.000Z',...
    'extension','dat'...
    );

[wiitt]

I wonder if the tests are supposed to be passed

[spencerwplovie]

I wonder if the tests are supposed to be passed

I had a similar issue with the tests not passing initially in #26, but it looks like they ended up passing sometime before Ben merged the branch. If I remember correctly, I think he mentioned that the test suite needs to be refactored in some places. It looks like Isla, the previous scientific programmer co-op, was working on this in #10, but the PR was closed.

[spencerwplovie]

It might be handy to have a check when the user passes in a value for chunkSize to make sure it's less than the 2^30 bytes limit

[spencerwplovie]

Also, I'm not quite sure how to pass in a value for chunkSize. It's not a parameter, so would the user need to run chunkSize = int; on the command line before they run a data product order?

[wiitt]

Good point. Checking the chunkSize value would be helpful. I'll add it.

As for how to set this parameter up, it is done when the class is initialized. At the moment when an instance of this class is created, a user can pass input argument to the class like

chunkSize = 2^29;
consumer = onc.ChunkedResponseConsumer(chunkSize);
response = request.send(uri, options, consumer);

I might need to add empty brackets to line 28 onc/+util/do_request.m where the consumer is initialized. Or, to spare people from guessing, I can make passing of the parameter explicit by doing it in the way it is done in the code snippet above.

[spencerwplovie]

Should probably say function buffsize = start(obj)

[spencerwplovie]

chunks (minor typo but it's worth fixing)

[wiitt]

Sorry that it took so long to get back to this PR because of dealing with other tasks

[wiitt]

Hmm, now the updated code doesn't pass checks even though the updates are minor
I'm checking it

[KristenMeyerONC]

Hmm, now the updated code doesn't pass checks even though the updates are minor I'm checking it

There was a deployment today that may have interfered with the checks.

[spencerwplovie]

Variable should be called and passed in as chunkSize to be consistent