Full sync every time?

[hutcheon]

Just set this up, and am loving it, but sync takes a long time (about 2.5 hours). Every time it syncs, it pulls everything from audiobookshelf (that only takes a couple minutes), and the rest of the time is searching for matching hardcover books.

Maybe I'm missing something, or there's a problem with my setup, but after a first full sync, shelfbridge has a database of everything. I was expecting additional syncs to be quick... only acting on changed progress or new books on audiobookshelf side, not throwing out all of that progress and searching for a match for every book from scratch again on every sync.

This is two syncs, less than 10 hours apart (docker container didn't stop, but I did fix the cause of that error):

✅ Sync complete for user: tyler in 8839.1s
══════════════════════════════════════════════════
📚 SYNC COMPLETE (8839.1s)
══════════════════════════════════════════════════
📚 Library Status                  🌐 Hardcover Updates
├─ 1 of 2 libraries processed      ├─ 205 API calls made
├─ 1 libraries excluded by filter  ├─ 205 successful
├─ 942 total books                 ├─ 1 failed
├─ 2 currently reading             └─ 78 skipped (no changes)
├─ 282 completed
└─ 658 never started               ✅ Sync Status
                                   ├─ 1 error occurred
📊 Processing Results              ├─ Cache updated
├─ 205 marked complete             └─ Ready for next sync
└─ 78 skipped (no change
══════════════════════════════════════════════════


✅ Sync complete for user: tyler in 8801.9s
══════════════════════════════════════════════════
📚 SYNC COMPLETE (8801.9s)
══════════════════════════════════════════════════
📚 Library Status                  🌐 Hardcover Updates
├─ 1 of 2 libraries processed      ├─ 207 API calls made
├─ 1 libraries excluded by filter  ├─ 207 successful
├─ 942 total books                 ├─ 0 failed
├─ 2 currently reading             └─ 77 skipped (no changes)
├─ 282 completed                   
└─ 658 never started               ✅ Sync Status
                                   ├─ All updates successful
📊 Processing Results              ├─ No errors occurred
├─ 2 progress updated              ├─ Cache updated
├─ 205 marked complete             └─ Ready for next sync
└─ 77 skipped (no change)          
══════════════════════════════════════════════════

Config:

users:
  - id: tyler
    abs_url: https://snip
    abs_token: secret
    hardcover_token: secret
    libraries:
      exclude: ["Podcasts"]    # Skip these libraries

global:
   timezone: "America/Toronto"
   sync_schedule: "27 3 * * *"
   auto_add_books: false
   audiobookshelf_semaphore: 5
   audiobookshelf_rate_limit: 1200

[rohit-purandare]

Hey! What you described is definitely what should happen, it should only find changed books and sync. Everything else should leverage cache. I'll look into this, if you're able to pull the full logs from /logs even better!

[hutcheon]

log.txt.gz

[hutcheon]

It is creating a ".book_cache.db" file in the data directory (with leading period). It is a valid sqlite3 db with reasonable looking contents

sqlite> .tables
books          library_stats  sync_tracking
sqlite> select count(*) from books;
485
sqlite> select * from library_stats ;
┌─────────┬─────────────┬─────────────────────┬───────────────────┬─────────────────┬─────────────────────┬──────────────────────────┐
│ user_id │ total_books │ books_with_progress │ in_progress_books │ completed_books │ never_started_books │       last_updated       │
├─────────┼─────────────┼─────────────────────┼───────────────────┼─────────────────┼─────────────────────┼──────────────────────────┤
│ tyler   │ 942         │ 284                 │ 2                 │ 282             │ 658                 │ 2025-09-06T07:29:00.045Z │
└─────────┴─────────────┴─────────────────────┴───────────────────┴─────────────────┴─────────────────────┴──────────────────────────┘
sqlite> select * from sync_tracking ;
┌─────────┬────────────┬─────────────┬─────────────────────┬──────────────────────────┐
│ user_id │ sync_count │ total_syncs │     created_at      │        updated_at        │
├─────────┼────────────┼─────────────┼─────────────────────┼──────────────────────────┤
│ tyler   │ 10         │ 10          │ 2025-09-05 15:06:47 │ 2025-09-06T07:27:00.303Z │
└─────────┴────────────┴─────────────┴─────────────────────┴──────────────────────────┘

[rohit-purandare]

The logs are super helpful, I see what is going on. Working on a bug fix now!

[hutcheon]

Updated to the v1.22.1 image today, looks like there's still a problem. The errors are gone, but still appears to still be doing a full sync every time. Upon starting, it ran a full sync (2.5 hours), then I restarted the container and it ran another 2.5 hours. Oddly, I finished a book today, and neither of the syncs updated the status to completed. I'll let it run overnight and pull the logs tomorrow.

[rohit-purandare]

dang ok, yes please provide logs. this should not happen

[hutcheon]

log2.txt.gz

[hutcheon]

Not to detract from this, but an interesting note is that I see ShelfBridge touching things from my Podcasts library (which is excluded). I had a Linux Matters podcast episode in progress when this sync ran, and I see it as the first thing it worked on:

22:47:13 [info]: Title/author search initiated for "Linux Matters" service="shelfbridge" version="1.22.1" searchTitle="Linux Matters" normalizedSearchTitle="linux matters" searchAuthor="Linux Matters" searchNarrator="N/A" userFormat="ebook" maxResults=5 confidenceThreshold="70.0%" sourceBookMetadata={"series":"N/A","seriesPosition":"N/A","publicationYear":2024,"bookId":"31f229bd-27c5-4729-8de3-e1f7b15d040a"}
22:47:22 [info]: Title/author search results for "Linux Matters" service="shelfbridge" version="1.22.1" resultCount=2 searchMetadata={"searchTitle":"Linux Matters","searchAuthor":"Linux Matters","searchNarrator":"N/A","searchTimestamp":1757285242820,"searchStrategy":"title_and_author"} results=[{"title":"Building Embedded Linux Systems","author":"Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, Philippe Gerum","bookId":"1722590","hasEditions":false,"editionCount":0,"series":"N/A","seriesPosition":"N/A","publicationYear":"N/A","identifiers":{"isbn_10":"N/A","isbn_13":"N/A","asin":"N/A"},"formats":[],"usersCount":"N/A","ratingsCount":"N/A","averageRating":"N/A"},{"title":"Professional Linux Programming","author":"Jon Masters, Richard Blum","bookId":"341787","hasEditions":false,"editionCount":0,"series":"N/A","seriesPosition":"N/A","publicationYear":"N/A","identifiers":{"isbn_10":"N/A","isbn_13":"N/A","asin":"N/A"},"formats":[],"usersCount":1,"ratingsCount":"N/A","averageRating":"N/A"}]
22:47:22 [info]: Title/author scoring results for "Linux Matters" service="shelfbridge" version="1.22.1" targetTitle="Linux Matters" targetAuthor="Linux Matters" threshold="70.0%" candidates=[{"rank":1,"title":"Professional Linux Programming","author":"Jon Masters, Richard Blum","score":"38.6%","confidence":"none","isBookMatch":false,"passesThreshold":false,"breakdown":{"title":{"score":"35.4%","weight":"35.0%","comparison":"\"Linux Matters\" vs \"Professional Linux Programming\""},"author":{"score":"52.9%","weight":"25.0%","comparison":"\"Linux Matters\" vs \"Jon Masters, Richard Blum\""},"series":{"score":"60.0%","weight":"15.0%","reason":"No series data for comparison"},"activity":{"score":"30.0%","weight":"10.0%","value":"N/A"},"year":{"score":"20.0%","weight":"5.0%","comparison":"2024 vs 2007"},"penalties":[]}},{"rank":2,"title":"Building Embedded Linux Systems","author":"Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, Philippe Gerum","score":"36.9%","confidence":"none","isBookMatch":false,"passesThreshold":false,"breakdown":{"title":{"score":"38.2%","weight":"35.0%","comparison":"\"Linux Matters\" vs \"Building Embedded Linux Systems\""},"author":{"score":"42.2%","weight":"25.0%","comparison":"\"Linux Matters\" vs \"Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, Philippe Gerum\""},"series":{"score":"60.0%","weight":"15.0%","reason":"No series data for comparison"},"activity":{"score":"30.0%","weight":"10.0%","value":"N/A"},"year":{"score":"20.0%","weight":"5.0%","comparison":"2024 vs 2003"},"penalties":[]}}]
22:47:22 [info]: No suitable match found for "Linux Matters" service="shelfbridge" version="1.22.1" searchedTitle="Linux Matters" searchedAuthor="Linux Matters" threshold="70.0%" candidatesEvaluated=2 bestScore="38.6%" reason="Best candidate failed book identification criteria" outcome="MATCH_REJECTED"
22:47:22 [info]: Skipping Linux Matters: No ISBN, ASIN, or title/author match found service="shelfbridge" version="1.22.1"

I can open this as a separate issue if this isn't an understanding problem.

[rohit-purandare]

1. I need to do more investigation into your logs. My initial thought is if a title / author match is not found it will submit it for search every time. When a match meeting criteria is found it will cache, but until then it will search. Since Hardcover is pretty rate limiting I would bet this is what slows it down. However, this search should only happen for in-progress or completed items.
2. I think I fixed Podcasts being ignored in 1.22.2Please take a look and as always logs are much appreciated!

[hutcheon]

Good news is that Podcasts are no longer showing up in the logs, but it's still doing a full sync, and none of the progress is getting sync'd to Hardcover.

When I ran shelfbridge for the first time, I already had a bunch of stuff in Hardcover, but it updated my "Currently Listening" correctly. After that, I think it updated the progress at least once, but since then, it hasn't sync'd any changes.

Here's another batch of logs... this is on 1.22.2, and I deleted my database before starting. Like the others, it's two syncs, both taking a long time. There was one sync failure, which I fixed between the runs.

log3.txt.gz

[rohit-purandare]

Thank you, I have a test build up that I have confidence will resolve this. Let me do some more digging on the logs and get back to you