Tdl 12607 add page size as a configurable property

[prijendev]

Description of change

• Added is a configurable pagination parameter page_size.
• Added pagination test case.

QA steps

• Give the page_size parameter in the config and verify that tap calls the API with the given pagination parameter.
• Verify that if page_size parameter is not provided then tap consider default page_size 1000.

Risks

Rollback steps

• revert this branch

[namrata270998]

Suggested change

	# If page_size found in config then used it else use default page size.
	# If page_size is found in config then use it, else use the default page size.

[kspeer825]

This needs test coverage around the new configurable property:

• Can the tap handle types that are not int?
• What happens if I pass in a page size of zero?
• Does the tap actually fall back to the default and perform functionally?

[prijendev]

This needs test coverage around the new configurable property:

• Can the tap handle types that are not int?
• What happens if I pass in a page size of zero?
• Does the tap actually fall back to the default and perform functionally?

We have updated the code to handle different values of the page_size parameter.
The tap passes the default value of page_size if page_size is 0, empty string, invalid string, negative int, negative float.
Tap converts valid string and float values to int.

[kspeer825]

Could you update this validation to ensure all pages are unique rather than the first 2, see example here https://github.com/singer-io/tap-mixpanel/blob/2cf3a86231da042f76f4e3ef9fa8c23600b6f593/tests/tap_tester/test_all_fields_pagination.py#L134-L153

[prijendev]

Updated pagination test case.

[kspeer825]

Why is the page size changed here? And could you update this test to cover a larger page size than 1 so that we are validating multiple records per page?

[prijendev]

We have updated the page size to 2 for a pagination test case. Smaller page size takes a long time to complete the sync. So, here we are using a smaller page size just for the pagination test case, and for the rest of the test cases, we are reverting back it to 1000.

[shantanu73]

This seems to be an old link (old JIRA environment).
Is there a new JIRA associated with it, if not can we remove this ?

[prijendev]

@cosimon @kspeer825 Should we remove such a Jira link from the tap-tester? We are not sure whether it is an old Jira link or not because it is not accessible to us.