Skip to content

Add importers for various services and refactor base importer class#62

Draft
aquarion wants to merge 2 commits intofeature/refactor_steamfrom
feature/restructure
Draft

Add importers for various services and refactor base importer class#62
aquarion wants to merge 2 commits intofeature/refactor_steamfrom
feature/restructure

Conversation

@aquarion
Copy link
Copy Markdown
Owner

@aquarion aquarion commented Mar 4, 2026

Still to do:

Importer Conversions

Convert remaining import scripts from imports/ to class-based importers in src/lifestream/importers/:

Simple/Medium Complexity

  • atproto_posts.py - Bluesky/AT Protocol posts
  • historic.py - Historic data import
  • oyster.py - Oyster card data
  • oyster_csv.py - Oyster CSV import
  • mailbox-stats.py - Mailbox statistics
  • steambadges.py - Steam badges (uses Selenium)
  • planetside.py - Planetside 2 achievements

OAuth-based (More Complex)

  • foursquare.py - Foursquare checkins
  • tumblr.py - Tumblr posts
  • fitbit_day.py - Fitbit daily data
  • instagram.py - Instagram photos

Game APIs (Complex)

  • gw2.py - Guild Wars 2 achievements
  • ffxiv.py - Final Fantasy XIV data
  • destiny2.py - Destiny 2 achievements
  • wow.py - World of Warcraft achievements

Facebook (May be deprecated)

  • facebook_page.py - Facebook page posts
  • facebook_posts.py - Facebook personal posts

Twitter (May be deprecated)

  • tweets.py - Twitter/X posts

Cleanup Tasks

  • Remove imports/lifestream/ directory after all scripts converted
  • Update bin/run_import.sh to use new CLI or remove
  • Update any documentation referencing old structure

Optional Improvements

  • Add type hints throughout codebase
  • Add more comprehensive tests for importers
  • Create importer discovery system (auto-register from package)
  • Add progress bars for long-running imports
  • Refactor C901 complexity issues (tracked in Reduce function complexity (C901 violations) #60)

Things we've done

  • Introduced BaseImporter class to standardize importer functionality.
  • Added FlickrImporter for importing photos from Flickr.
  • Implemented GithubCommitsImporter to fetch commits from GitHub repositories.
  • Created LastfmImporter to import recent scrobbles from Last.fm.
  • Developed MastodonImporter for importing toots from Mastodon instances.
  • Added SteamImporter to import achievements from Steam.
  • Implemented SwitchbotImporter for temperature and humidity data from SwitchBot sensors.
  • Created WordpressImporter to import posts from WordPress sites.
  • Refactored tests to accommodate changes in module structure and improve clarity.

aquarion added 2 commits March 4, 2026 12:41
- Introduced BaseImporter class to standardize importer functionality.
- Added FlickrImporter for importing photos from Flickr.
- Implemented GithubCommitsImporter to fetch commits from GitHub repositories.
- Created LastfmImporter to import recent scrobbles from Last.fm.
- Developed MastodonImporter for importing toots from Mastodon instances.
- Added SteamImporter to import achievements from Steam.
- Implemented SwitchbotImporter for temperature and humidity data from SwitchBot sensors.
- Created WordpressImporter to import posts from WordPress sites.
- Refactored tests to accommodate changes in module structure and improve clarity.
- Introduced BaseImporter class to standardize importer functionality.
- Added FlickrImporter for importing photos from Flickr.
- Implemented GithubCommitsImporter to fetch commits from GitHub repositories.
- Created LastfmImporter to import recent scrobbles from Last.fm.
- Developed MastodonImporter for importing toots from Mastodon instances.
- Added SteamImporter to import achievements from Steam.
- Implemented SwitchbotImporter for temperature and humidity data from SwitchBot sensors.
- Created WordpressImporter to import posts from WordPress sites.
- Refactored tests to accommodate changes in module structure and improve clarity.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant