Allow overriding a component which is included via dependency (PACMAN-1198)

[projectgus]

Related area

Components

Is your feature request related to a problem?

Recently in MicroPython we had the situation:

• We include espressif/esp_tinyusb component in our project via idf_component.yml.
• espressif/esp_tinyusb component includes the espressif/tinyusb component as a dependency.
• We needed to apply a "hot fix" by overriding the version of the espressif/tinyusb component (specifically, redirecting it to a branch in a github repo).

Our idf_component.yml looks something like:

dependencies:
  espressif/esp_tinyusb:
    version: "~1.7.6"

I tried explicitly changing it to something like:

dependencies:
  espressif/esp_tinyusb:
    version: "~1.7.6"
  espressif/tinyusb:
    git: https://github.com/micropython/tinyusb-espressif.git
    version: cherrypick/dwc2_zlp_fix

Component manager sees both components, but ultimately the version added as a dependency of esp_tinyusb replaces the first one (without any error or warning printed):

NOTICE: Processing 5 dependencies:
NOTICE: [1/5] espressif/esp_tinyusb (1.7.6~1)
NOTICE: [2/5] espressif/mdns (1.1.0)
NOTICE: [3/5] espressif/tinyusb (e4c0ec3caab3d9c25374de7047653b9ced8f14ff)
NOTICE: [4/5] espressif/tinyusb (0.18.0~6)
NOTICE: [5/5] idf (5.5.1)

Describe the solution you'd like

It'd be very helpful if the logic of override_path could be generalised to allow any dependency in the project to be overridden by the top level manifest. i.e. something like:

  espressif/esp_tinyusb:
    version: "~1.7.6"
  espressif/tinyusb:
    override: true   # New flag, means that any `espressif/tinyusb` component in the project will be replaced with this one
    git: https://github.com/micropython/tinyusb-espressif.git
    version: cherrypick/dwc2_zlp_fix

Describe alternatives you've considered

The only way forward that I can see would be to also make a repo with a fork of esp_tinyusb component, change it's tinyusb dependency, and then have my project manifest point to that.

I didn't try this as it was too fiddly.

Additional context

No response

I have checked existing list of Feature requests and I have provided enough information.

• I confirm I have checked existing list of Feature requests and I have provided enough information.

[kumekay]

Hi @projectgus Angus! It's nice to see you here (and through all your micropython contributions)

Thank you for bringing this is, it's true that there is no direct way to overwrite. We have already discussed this issue internally before, but didn't come up with the best solution.

As a workaround in this situation, we usually recommend adding "git override" hotfix as a submodule inside project's components directory (or to EXTRA_COMPONENTS_DIR) - these have higher priority than managed components. I'm not sure whether it can be used in micropython project.

As for proper solution, we considered giving direct project dependencies higher weight, but it comes with new set of issues, like git components may not have defined version. So, override flag sounds feasible. It will require non-trivial modifications so it will take some time to deliver.

[projectgus]

Hi @kumekay! Nice to hear from you as well, and thanks for the extra context.

I hadn't realised an in-tree submodule will override anything in the manifest, thanks for pointing that out. We could possibly do this (although it's a little awkward as we already have a submodule for upstream tinyusb, and in this case we only want to maintain a MicroPython fork of the Espressif fork of upstream TinyUSB for as long as our hotfix is needed, and then we'll go back to using Espressif's TinyUSB fork.)

The override idea is not urgent from my perspective, but I did want to raise it as sooner or later it'll probably come up again for us.

[lukecyca]

This remains an awkward limitation in the component dependency system. A very common scenario:

• Application depends on a dependency like espressif/onewire_bus
• Application developer finds a bug in the library, fixes it, and opens a pull request
• Application developer wishes to temporarily override the dependency to use their fork, while they wait to the upstream maintainers to review, merge, and release the fix

All current workarounds require too much reconfiguration, especially since one expects the override to be quite temporary.

[kohait00]

This remains an awkward limitation in the component dependency system. A very common scenario:

• Application depends on a dependency like espressif/onewire_bus
• Application developer finds a bug in the library, fixes it, and opens a pull request
• Application developer wishes to temporarily override the dependency to use their fork, while they wait to the upstream maintainers to review, merge, and release the fix

All current workarounds require too much reconfiguration, especially since one expects the override to be quite temporary.

Just wanted to underscore the comment from @lukecyca w.r.t the importance of this for the day-to-day user/dev.
As for me, it was the reason I turned away from ESP projects for quite a while now, since I stumbled upon the related #45 . And trying to find the location of this error in order to fix it myself proved impossible. I hoped it to be fixed sometime later and was astonished to see it still being in discussion.

Understanding that this is a rather peculiar feature for the pro developpers, its this base that drives the platform forward. Please dont forget.

[kumekay]

Hi @lukecyca @kohait00 Thank you for bringing our attention to this issue, we have scheduled it and will address very soon.