-
Notifications
You must be signed in to change notification settings - Fork 4
TREE-417: Reworked assertion documentation #129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
210da3e
d7bc1e4
2e633f2
bd98343
7486103
c7aaead
8838fe1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,57 +1,46 @@ | ||
| --- | ||
| title: 'Writing assertions for API checks' | ||
| description: 'Writing assertions for API checks' | ||
| title: 'Defining Assertions' | ||
| description: 'Add assertions to validate your check results' | ||
| sidebarTitle: 'Assertions' | ||
| --- | ||
|
|
||
|
|
||
|
|
||
| The response of an API request can be checked for correctness and timeliness by using assertions on the response data. Assertions are flexible statements that combine preset modifiers with custom values to meet the needs of a broad set of use cases. | ||
| Uptime monitors and API checks allow you to define assertions to validate a response and check its data for correctness. | ||
|
|
||
| ## How assertions work | ||
|
|
||
| Assertions are statements you create that check one aspect of the HTTP response. You can create multiple assertions for one check that assert various aspects of a response, for example: | ||
| Assertions let you validate specific parts of a check's response. For example: | ||
|
|
||
| - HTTP response status equals 200. | ||
| - HTTP response body equals the text "success". | ||
| - HTTP response time is lower than 2000 milliseconds. | ||
| - HTTP response header "X-Custom-Header" equals "SomeValue". | ||
| - HTTP response JSON object has a key called "accountBalance" with a value greater than 9999 | ||
| - **URL monitor**: HTTP response status equals 200. | ||
| - **TCP monitor**: Response contains expected string (e.g. OK). | ||
| - **DNS monitor**: Resolved IP equals 93.184.216.34. | ||
| - **API check**: HTTP response header "X-Custom-Header" equals "SomeValue". | ||
|
|
||
| ### Sources | ||
|
|
||
| In each assertion, a **source** is connected to a **comparison** and a **target**. | ||
|
|
||
|  | ||
|
|
||
| In some cases a **property** is added, for example when asserting headers or JSON response bodies. | ||
| In some cases a [property](#property) is added, for example when asserting API check headers or JSON response bodies. | ||
|
|
||
|  | ||
|
|
||
| Assertions are executed from top to bottom. If one assertion fails, the full check is considered as failed. | ||
|
|
||
| ## Source | ||
|
|
||
| On each API check assertion, the following sources are available: | ||
| Supported assertion sources vary by monitor type. Refer to the documentation of the specific monitor (API, DNS, TCP, etc.) to see which sources are available. | ||
|
|
||
| - **Status code:** The HTTP response status, parsed as an integer. | ||
| - **JSON body:** The response body parsed as a JSON object. | ||
| - **Text body:** The response body as plain text. | ||
| - **Headers:** The response headers as an array of key/value pairs. | ||
| - **Response time:** The response time of the full API request in milliseconds, parsed as an integer value. | ||
| ### Property | ||
|
|
||
| ## Property | ||
| The property field is a free-form text input that lets you point to a specific part of the data you want to validate. It’s available for the following types of assertion [sources](#sources): | ||
|
|
||
| The property field is a free form text field used to identify the name of a header, the part of a JSON object using JSON path or part of a text body. | ||
| - **JSON response bodies**: Use a JSON path expression in the form of dot-separated strings to | ||
| target nested properties in an object, i.e. `$.product.size` or an item in an array, i.e. `$.[1].key`. [Learn more](#json-responses-with-json-path). | ||
|
|
||
| - With **JSON response bodies**, the property field accepts **JSON path** expressions in the form of dot-separated strings to | ||
| target nested properties in an object, i.e. `$.product.size` or an item in an array, i.e. `$.[1].key` | ||
| - **Text response bodies**: Provide a regular expression with a capture group to pick out parts, | ||
| i.e. `<!doctype (.{4})` would grab the word `html` from a body return `<doctype html>`. [Learn more](#text-body-assertions-with-regular-expressions). | ||
|
|
||
| - With **text response bodies**, the property field accepts a **regular expression** with a capture group to pick out parts, | ||
| i.e. `<!doctype (.{4})` would grab the word `html` from a body return `<doctype html>` | ||
|
|
||
| - With **headers**, you provide the header you want to assert in the property field, i.e. `Content-Type`. You can even add a | ||
| regular expression after that to tease out a specific part of the header. | ||
|
|
||
| Read more about asserting JSON response bodies below. | ||
| - **API check headers**: Enter the header name you want to assert on i.e. `Content-Type`. You can even add a | ||
| regular expression after that to tease out a specific part of the header. [Learn more](#using-regular-expressions). | ||
|
|
||
| ## Comparison | ||
|
|
||
|
|
@@ -75,19 +64,14 @@ Response time is empty? JSON Object is less than? We block out the comparisons w | |
|
|
||
| The target field is a free form text field that determines the desired outcome of your assertion. | ||
|
|
||
|
|
||
| ## JSON responses with JSON path | ||
|
|
||
| You can use **JSON path** to specify which field of a JSON response body should be asserted. JSON path is a query language | ||
| For monitors that support JSON body assertions, you can use **JSON path** to specify which field of a JSON response body should be asserted. JSON path is a query language | ||
| similar to Xpath for XML, but in general a lot more intuitive and simpler to use. | ||
|
|
||
| > For Checkly to be able to parse the JSON body, the `Content-Type` header of the response should be set to `application/json`. | ||
|
|
||
| ### JSON path primer | ||
|
|
||
| JSON path only uses a handful of operators in its queries. Not all of them are useful in the context of assertions. | ||
| Here is an adapted set of examples based on [Stefan Goessner's 2007 introduction post](http://goessner.net/articles/JsonPath/). | ||
|
|
||
| The following JSONPath operators are available: | ||
|
|
||
| JSONPath | Description | ||
| -----------------|------------ | ||
|
|
@@ -153,7 +137,6 @@ JSONPath | Description | |
| `$.store..price` | The price of everything in the store | ||
| `$.store.book.length` | The length of the book array | ||
| `$..book[2]` | The third book | ||
| `$..book[(@.length-1)]` | The last book via script subscript | ||
| `$..book[-1:]` | The last book via slice | ||
| `$..book[0,1]` | The first two books via subscript union | ||
| `$..book[:2]` | The first two books via subscript array slice. | ||
|
|
@@ -163,8 +146,7 @@ JSONPath | Description | |
| `$..book[?(@.price<30 && @.category=="fiction")]` | Filter all fiction books cheaper than 30 | ||
| `$..*` | All members of JSON structure | ||
|
|
||
| Use this [online editor to play around](https://jsonpath.com/), or look at the examples below. We use this [jsonpath NPM | ||
| module](https://github.com/dchester/jsonpath) under the hood. | ||
| Use this [online editor](https://jsonpath.com/) to try out your own JSONPath expressions. For a full description of the syntax and semantics, see [RFC 9535](https://datatracker.ietf.org/doc/rfc9535/). | ||
|
|
||
| ### Asserting basic types | ||
|
|
||
|
|
@@ -218,23 +200,24 @@ Regular expressions give you the power to extract specific parts of text from a | |
| You can use regular expressions with two assertions sources: | ||
|
|
||
| 1. **Text body:** Use the property field to add your regex. | ||
| 2. **Headers:** First select the header you are interested in the property field, then click "add regex". | ||
| 2. **API check headers:** First select the header you are interested in the property field, then click "add regex". | ||
|
|
||
| Here is an example: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This example shows the JS .match() function. We should probably remove this too, if we're removing the reference above to .match() (lines 223-224).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Totally missed this, thanks. I've updated the example, could use another sanity check since I'm no regex expert. |
||
|
|
||
| Under the hood, we use the stock Javascript regular expressions implementation. Specifically, we use the [.match()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) | ||
| method. We *do not use the `/g` modifier* and return the first matched group the expression finds. | ||
| ``` | ||
| The quick brown fox jumps over the lazy dog. It barked. | ||
| ``` | ||
|
|
||
| Sounds more complicated than it is. Here is an example: | ||
| And the following regular expression: | ||
|
|
||
| ```javascript | ||
| const paragraph = 'The quick brown fox jumps over the lazy dog. It barked.' | ||
| const regex = /quick (.*) fox/ | ||
| const found = paragraph.match(regex) | ||
| console.log(found) | ||
|
|
||
| // [ 'quick brown fox', | ||
| // 'brown', | ||
| // index: 4, | ||
| // input: 'The quick brown fox jumps over the lazy dog. It barked.' ] | ||
| ``` | ||
| quick (.*) fox | ||
| ```` | ||
|
|
||
| The assertion extracts: | ||
|
|
||
| ``` | ||
| brown | ||
| ``` | ||
|
|
||
| In the example above we return the string `brown` because it is the first capture group, the `(.*)` bit. | ||
|
|
@@ -244,25 +227,18 @@ The first item `quick brown fox` is the full match, which we do not return. | |
|
|
||
| ### Text body assertions with regular expressions | ||
|
|
||
| If our API returns HTML, there might be a `lang="en"` attribute in the `<html>` element. We can capture the two character | ||
| country/language code value of that attribute with the following expression. | ||
| When a check returns a text-based response, you can use regular expressions to extract and assert on specific parts of the response body. | ||
|
|
||
| For example, an HTML document might include a `lang="en"` attribute on the `<html>` element. You can capture the two-character language code using the following regular expression: | ||
|
|
||
|  | ||
|
|
||
| The expression `lang="(.{2})"` means 'grab any of the first two characters between `lang="` and the next `"`'. If we were | ||
| sure there are only non-capital characters, we could tighten it up a bit with `lang="([a-z]*)"`. | ||
|
|
||
| ### Header assertions with regular expressions | ||
|
|
||
| We can use regular expressions with headers too. In this example, we check if the `max-age` property of a `Strict-Transport-Security` | ||
| response header is above a `100000` | ||
|
|
||
|  | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
| ### API check header assertions with regular expressions | ||
|
|
||
| We can use regular expressions with [API check headers](/detect/synthetic-monitoring/api-checks/configuration#headers) too. In this example, we check if the `max-age` property of a `Strict-Transport-Security` | ||
| response header is above a `100000`. | ||
|
|
||
|  | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we are not following the rfc and only support a subset of features. let's try to set right expectations:
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sbezludny Is a link to the RFC necessary at all then? Or do you think its worth linking it while mentioning the exceptions you highlighted?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The underlying library we're using says it follows RFC9535. 🤔
I guess it is out-dated let's then adjust it accordingly.