Language-independent paywall detection

[mircealungu]

Current state: paywall detection is done based on hardcoded strings that have to be maintained for various languages.

Suggested approach:

• using BERT embeddings

Moreover, besides paywall detection another related task is removing the non-content bits that still get extracted by Readability. News articles very often have this sections entitled "Related articles" or "Get a subscription". They should be cleaned out and probably embeddings at the sentence level could help with this.