WebCrawler

A basic prototype of Web Search Engine where one can search for all the articles related to a particular topic on Wikipedia. An attempt to build an end-to-end system.

##What lies beneath?

To build such a system we have used the following technologies:

1. Nodejs

2. Kafka

3. Storm / Trident

4. ElasticSeach

5. Distributed Remote Procedure Calls (DRPC)

6. Probabilistic Data Structures - CountMin-Sketch and BloomFilters