Scraping to find new events

[bahorn]

As mentioned in the readme, I'm interested in maintaining more of an effort to scrape sites and such to discover new events.

One approach that can use the existing data we are collecting is the tech society Github list.

As github provides atom feeds, we can track changes across all their repositories to see when their sites change by just appending a ".atom" to the url.

Also sneaky and we can catch things before publicly announced :)

[bahorn]

So, I setup a channel called events in the gitter community, which is where notifications from the bot should be posted.

[bahorn]

Ok, so I attempted to get the approach where I append .atom to organization URLs to get updates.

That doesn't work because apparently they do it from when it was last generated? You can see my attempt in the current revision 8f6f0a4

Best approach now would be to just use an API token to list all organization repos, check if anything new turned up (like a 2021 event repo, etc) or changes to known website repositories.

[bahorn]

https://api.github.com/users/{org}/events

Will list all recent events that organization, which I just pushed code to use. Just needs scheduling now. Will probably aim to run on a wednesday at 4am. Should capture most new stuff then.

[bahorn]

Now scheduled, and it seems to work!

I'll keep this issue open for progress on the other methods of scraping.