Add support for continuation tokens

[johnbatty]

Operations that return many items use continuation tokens to allow the items to be returned in batches.

REST responses that do not contain all items include an HTTP header x-ms-continuationtoken that provides a continuation token value. This value must be provided on a subsequent call as a continuationToken parameter to query the remaining items (repeated as necessary until the server does not include the x-ms-continuationtoken header).

The Python SDK has a "response" object that includes the continuation token, with the actual response as a value field.

The Rust SDK already modifies the spec to handle responses that return lists of items, as the server sends them in a wrapper:

{
   count: Option<i32>,
   value: Vec<type>
} <type>List

We could change this to include the continuation token as per the Python SDK (and perhaps change our wrapper type to be <...>Response rather than <...>List.

{
   count: Option<i32>,
   value: Vec<type>
   continuation_token: Option<String>
} <type>Response

Deserialization of this struct from the protocol would still work because the continuation_token field is declared as an Option, so would always be set to None as the field is in the headers rather than the body. However, we could modify the code generator to fill in this value from the response headers before the value is returned to the application.

Useful links:

• https://jessehouwing.net/azure-devops-accessing-apis-with-large-volumes-of-data/#:~:text=x-%20ms-continuation-token%20is%20returned%20by%20Azure%20DevOps%20in,the%20%26continuationToken%3D%24%20%7Bx-ms-continuation-token%7D%20query%20parameter%20to%20the%20call.
• Return Continuation Token in response azure-devops-python-api#152

[johnbatty]

Worth noting that the Azure SDK has done some work in this area, notably:

• azure_core::Pageable
• azure_core::Continuable
• initial work to support x-ms-pageable for generated specs
• Sketch for returning headers with responses

Need to decide whether we leverage this or just implement something similar to the Azure DevOps Python package.

[johnbatty]

A recent enhancement to the autorust enables access to headers on responses by using send() on requests rather than into_future(). This allows continuation tokens to be obtained by apps.

I'll update the docs and add an example to show how to do this.

[johnbatty]

Added an example that demonstrates how to do large queries with continuation tokens by using the new send() and raw_response() functions: build_list_continuation_token

[kyle-rader-msft]

@johnbatty I'm curious on your thoughts about also handling pagination for APIs that don't use continuation tokens like the Git Commit Get Changes API, which only uses top and skip params, but doesn't have any indication if there are more results to fetch. You have to just attempt to fetch them yourself.