Support custom HTTP headers

[vidok]

Problem Description

Sometimes, you need to use custom HTTP headers to bypass the authorization. One example is JWT tokens that are transferred via "Cookie" header.

Proposed Solution

You should be able to add custom HTTP headers to the crawl configuration.

[seanstory]

Can we narrow this to specific headers? Like I think a custom Authorization and Cookie make reasonable sense. But I don't want the user to be able to fully override user-agent or Accept or something.

[vidok]

Can we narrow this to specific headers? Like I think a custom Authorization and Cookie make reasonable sense. But I don't want the user to be able to fully override user-agent or Accept or something.

Defining a list of allowed HTTP headers that customers can change is a good idea. Dealing with Authorization might be a little bit tricky because there is http_header_service.rb.

[navarone-feekery]

@vidok and I discussed whether this should be a per-domain configuration, and I think in the long-run that we do want this to be configurable per-domain. However, that will require a decent amount of refactoring because the HTTP client is shared currently.

I think for now having the custom headers be shared for all domains is acceptable and we can tackle having separate HTTP clients/headers for each domain in the future.

For that reason, maybe the better approach is to extend http_header_service.rb to support non-auth headers, and these headers will apply to all domains. If we build a separate header configuration we'll need to figure out how to merge them together in the future.

What do you all think of this approach?

[navarone-feekery]

For that reason, maybe the better approach is to extend http_header_service.rb to support non-auth headers, and these headers will apply to all domains.

I take this back; it seems very auth-specific and will also require a lot of refactoring.

I think the approach taken in #162 is good, just missing the list of allowed HTTP headers that customers can change (excluding authorization).