Separation of concerns

[randompersona1]

Modules should practice separation of concerns. Attached is the import tree of the main entrypoint. All of that is executed even when only usdb_syncer --version is run. This also makes it harder to avoid circular imports.

[RumovZ]

I think the main issue is that our single entry point is in usdb_syncer.gui. Apart from that, most modules have very clear functions, and circular imports would be detected by static code analysis (at least they were with pylint).

[randompersona1]

Circular imports aren't detected statically right now. They are detected by virtue of the tests importing modules, which may or may not recursively import the entire tree (see the issue?).

I agree that a major reactor of the entrypoint is step 1. But I also have a problem with:

• the txt parser using other modules liberally
• song_routines basically doing everything
• remote_config.py vs discord.py
• postprocessing importing everything
• song_loader importing everything
• SongId and SyncMetaId being under the main module

All of this currently works obviously. But it is more difficult to extend than it could be.

[RumovZ]

I wouldn't mind improvement to any of these parts, but some context:

song_routines basically doing everything

song_routines was created for synchronisation between the DB, the file system, caches and USDB. So by design it uses a lot of other modules, which I don't see as a bad thing per se. If you want to modularise it, keep in mind that performance matters here.

remote_config.py vs discord.py

Think of remote_config like settings. The "discord" function there is a type-safe accessor to the remote config field. It just looks weird because that's currently the only field.

song_loader importing everything

Another integration module. Would be interested to hear how you would change it.

By the way, the graph seems to include type checking imports, which makes it seem like there are a bunch of circular imports.

[randompersona1]

I don't have concrete solutions right now, and refactoring would take major effort. I wanted a tracking issue to bring up the problem.

By the way, the graph seems to include type checking imports, which makes it seem like there are a bunch of circular imports.

True. We know, of course, there aren't any, since the code works. But to me, having so much circular code, regardless of whether it is a problem to the interpreter is still a symptom of the underlying issue. That issue being that a large part of the codebase essentially exists in a single namespace, data and logic flowing freely forwards and back, with module separation being mostly ignored.

For reference, below is the graph of fastapi. There's a very distinct top-to-bottom flow in contrast to us.

[RumovZ]

For reference, below is the graph of fastapi.

To be honest, I don't see much of a difference. I'm not sure you can gain any insights from an arbitrary visual rendering. You'd have to analyse edge ratios and cycles. Plus, apps and libraries are apples and oranges.

That issue being that a large part of the codebase essentially exists in a single namespace, data and logic flowing freely forwards and back, with module separation being mostly ignored.

I don't get either of these points.

1. What single namespace? Every single module constitutes a namespace.
2. Logic flowing forwards and back would be a cycle, which, as established, we don't have.
3. How is module separation ignored?

That being said, I'm aware there are issues. For example:

1. The cycles if considering type imports you've already mentioned.
2. Type modules like usdb_song and sync_meta using db.
3. GUI classes relying on access to MainWindow attributes.
4. ...

But these are due to various restrictions. Python has no access modifiers. OOP is notoriously bad at separation of concerns, unless you go the Java route and add endless boilerplate in the form of artificial wrapper classes. Qt requires us to stuff everything into MainWindow.

In my opinion, the current architecture is extremely functional, so this is more of academic interest to me. But I'm genuinely interested in ideas on how to improve the design.