Modular Crawlers Based

[korczis]

Currently Microcrawler expects / supports only HTTP(S) Crawlers.

It will be nice if the Architecture will be modular (or extension friendly) and allow extension of supported protocols / schemas.

• HTTP(S)
  • http(s)://hostname
  • http(s)://username:password@hostname
• SQL - MySQL, PostgreSQL
  • mysql://username:password@hostname/database?query="..."
  • postgreql://username:password@hostname/database?query="..."
• NoSQL - Mongo
  • mongodb://username:password@hostname/database?query="..."
• Twitter
  • twitter://query="..."