Observed words cause a 400 response.

[kinguardo]

When running a scan with these 3 options enabled:

learn observed words
request
response

if an observed word contains a square bracket then it will be included in the header name. This causes the front-end to throw a 400 response very often because brackets aren't allowed in header names

[kinguardo]

same applies to semi colon ;

[kinguardo]

any thoughts on this?

[AkikoOrenji]

Related to this. Certain characters are going to cause some servers to error e.g. 400 etc. I've noticed param miner doesn't treat 400 responses as a trigger to start narrowing the current header set to find the header causing the problem.
If so it would eventually filter the problem header out and log it as 'interesting' but still analyze the other headers in the batch.
At the moment the default behaviour is if a 400 code is seen in the batch it just continue on with next batch potentially missing any valuable responses from other headers in that batch that would not cause a 400 error.

Anyone know if its configurable anywhere to set certain server response codes to be 'interesting' ?

[AkikoOrenji]

@kinguardo you can kind of work around this by configuring the Reshaper plugin to remove characters in requests which you know are going to cause problems. Its a fiddle and requires trial and error is required but is a partial workaround.

You can use the following as a template in Reshaper. You'll need to adjust the When's 'Matches Text' for a unique identifier in your applications URLs.

{"rules":[{"whens":[{"@class":".WhenEventDirection","negate":false,"useOrCondition":false,"dataDirection":"Request"},{"@class":".WhenMatchesText","negate":false,"useOrCondition":false,"identifier":{"text":"","variables":[]},"identifierPlacement":"Last","sourceText":{"text":"","variables":[]},"matchText":{"text":"appidentifier","variables":[]},"messageValue":"HttpRequestUriPath","messageValueType":"Text","messageValuePath":{"text":"","variables":[]},"matchType":"Contains","useMessageValue":true}],"thens":[{"@class":".ThenSetVariable","useMessageValue":true,"sourceMessageValue":"HttpRequestHeaders","sourceIdentifier":{"text":"","variables":[]},"sourceIdentifierPlacement":"Last","sourceMessageValueType":"Text","sourceMessageValuePath":{"text":"","variables":[]},"useReplace":true,"regexPattern":{"text":"[\\[\\]@]+","variables":[]},"text":{"text":"","variables":[]},"replacementText":{"text":"","variables":[]},"destinationMessageValueType":"Text","destinationMessageValuePath":{"text":"","variables":[]},"targetSource":"Global","variableName":{"text":"asdf","variables":[]}},{"@class":".ThenSetVariable","useMessageValue":false,"sourceMessageValue":"HttpRequestBody","sourceIdentifier":{"text":"","variables":[]},"sourceIdentifierPlacement":"Last","sourceMessageValueType":"Text","sourceMessageValuePath":{"text":"","variables":[]},"useReplace":true,"regexPattern":{"text":"amp;","variables":[]},"text":{"text":"%s","variables":[{"variableSource":"Global","name":"asdf","tag":"{{g:asdf}}"}]},"replacementText":{"text":"","variables":[]},"destinationMessageValueType":"Text","destinationMessageValuePath":{"text":"","variables":[]},"targetSource":"Global","variableName":{"text":"asdf","variables":[]}},{"@class":".ThenSetValue","useMessageValue":false,"sourceMessageValue":"HttpRequestBody","sourceIdentifier":{"text":"","variables":[]},"sourceIdentifierPlacement":"Last","sourceMessageValueType":"Text","sourceMessageValuePath":{"text":"","variables":[]},"useReplace":false,"regexPattern":{"text":"[\\[\\]]+","variables":[]},"text":{"text":"%s","variables":[{"variableSource":"Global","name":"asdf","tag":"{{g:asdf}}"}]},"replacementText":{"text":"","variables":[]},"destinationMessageValueType":"Text","destinationMessageValuePath":{"text":"","variables":[]},"destinationMessageValue":"HttpRequestHeaders","destinationIdentifier":{"text":"","variables":[]},"destinationIdentifierPlacement":"Only"}],"enabled":true,"autoRun":true,"name":"RemoveSpecials"}],"variables":[]}