Proposal for improving SDK selection with the host and admin installs

[marcpopMSFT]

Proposal for improving the SDK selection for the host and have it work better with admin installs (especially visual Studio).

Include a proposal for fast SDK version switching. Not-included is api work to share the SDK location to callers of the host/dotnetup.

[jkotas]

If I understand this correctly, this environment variable is meant to allow de-coupling location of dotnet.exe/dotnet and the actual SDK bits.

Current state: The SDK has to be in the known directory structure under that dotnet used to launch it. For example, if dotnet is located at /foo/dotnet, the SDK has to be located at /foo (/foo/sdk/10.0.123/,..) too.

Proposal change: Optionally override the SDK location using env variable. For example, if dotnet is located at /foo/dotnet, the SDK can be located at /bar (e.g. /bar/sdk/10.0.123/,..).

Is this correct understanding?

[baronfel]

That's a huge part of it, yeah. The thought being that if we can't always control PATH on Windows due to the choices we've made in our MSIs in the past, we can at least make the dotnet binaries that are invoked use the SDKs and Runtimes that the user intended.

Then as a value-add on top of that, if the muxer also allowed for specific SDK version selection to be driven externally then you can put together a really good UX around experimenting with different SDK versions in the same workspace without having to make any permanent changes. This is something that folks struggle with today, as you have to create/update global.jsons to get the same experience. This isn't high friction, but it's also not no friction, and there's substantial prior art in other ecosystems for making it easy to switch tool chains on a temporary basis.

[nagilson]

The debugger etc still rely on the PATH so a lot of features won't work in this scenario. We explored the alternative of updating the muxer at the system hive but decided against that for concerns, such as an MSRC that could happen in a pre-release muxer (very unlikely.)

[marcpopMSFT]

So dotnet use will work for your window but your debugger (and potentially other scenarios not going through the CLI) will be in a bad place until net11?

[nagilson]

Right - I'd have to open my IDE/workspace in a way that gets the effects of running dotnetup (aka launch from the terminal). But users might not notice that nuance.

[jkotas]

we can at least make the dotnet binaries that are invoked use the SDKs and Runtimes that the user intended.

Won't you also need this to control the runtime selection for dotnet myapp.dll to make it work well? The proposal talks about SDK selection only.

[jjonescz]

I feel like usually env vars have precedence over config files (since env vars can be set easily on the command line for one off scenarios). Is there a reason to have the precedence inverted here?

[marcpopMSFT]

We're thinking of the most common use case being dotnetup and the global.json path is a fallback hence having the precedence flipped. @baronfel thoughts here? I think jjonescz is thinking about this from a more engineering focused position where envs usually are the overrides.

[jjonescz]

It's not obvious to me what's the difference between DOTNET_SDK_ROOT and DOTNET_ROOT; could the doc perhaps explain that difference?

[marcpopMSFT]

One controls the SDK location and one controls the runtime location. DOTNET_ROOT is generally for when launching apps (with an app host presumably) and doesn't today control the SDK location. We could have it affect the SDK as well but that would be breaking hence proposing a new value. I'd have to think through whether the SDK_ROOT should affect the runtime ROOT and if there are customer scenarios who would want those different. Would a customer who's installed a user local hive want to run an app using the admin hive?

[jkotas]

We could have it affect the SDK as well but that would be breaking

Do we know how breaking it would be? Having a single env. variable that sets the location for both runtime and SDK would be much easier to explain.

[marcpopMSFT]

That's a good question. We'd need to understand what scenarios customers have where they are setting DOTNET_ROOT today. In particular, cases where they are setting it to a different location than their SDK and on a box where they are using their SDK.

Do we have any idea how common that is? I assume we don't have a lot of info for this so we'd be a bit blind (which is what makes me nervous about doing that). I'd be ok with one value though fwiw.

[marcpopMSFT]

@richlander @baronfel for thoughts on making DOTNET_ROOT apply to the SDK as well and if that's too risky.

[marcpopMSFT]

we can at least make the dotnet binaries that are invoked use the SDKs and Runtimes that the user intended.

Won't you also need this to control the runtime selection for dotnet myapp.dll to make it work well? The proposal talks about SDK selection only.

For the dotnetup scenarios, we are going to be setting DOTNET_ROOT in addition to DOTNET_SDK_ROOT. For the muxer, we are proposing that the environment variable function like the global.json paths setting (with the addition of env expansion). I previously suggested having the muxer override DOTNET_ROOT for the global.json path scenario but there was pushback hence keeping two separate values.

What's the behavior today if someone uses global.json paths and calls dotnet myapp.dll?

[jkotas]

What's the behavior today if someone uses global.json paths and calls dotnet myapp.dll?

I believe that dotnet myapp.dll does not consult global.json nor DOTNET_ROOT. dotnet myapp.dll is going to run on the runtime that is in the subdirectory of where dotnet is located.

[marcpopMSFT]

What's the behavior today if someone uses global.json paths and calls dotnet myapp.dll?

I believe that dotnet myapp.dll does not consult global.json nor DOTNET_ROOT. dotnet myapp.dll is going to run on the runtime that is in the subdirectory of where dotnet is located.

So in the case of someone using dotnetup where we've set both DOTNET_SDK_ROOT and DOTNET_ROOT, they'll get the runtime they expect. In the case where a customer sets only the SDK one themselves without going through dotnetup, they might be confused by the behavior but I think the result is ok (for the same reason that we're ok with DOTNET_ROOT not affecting the chosen sdk today).

[jkotas]

So in the case of someone using dotnetup where we've set both DOTNET_SDK_ROOT and DOTNET_ROOT, they'll get the runtime they expect.

They will get the global runtime, they won't get the runtime that DOTNET_SDK_ROOT and DOTNET_ROOT point to. It is not the behavior that I would expect.

[marcpopMSFT]

Let me try to clarify as there are a few scenarios. Kind of sounds like you're saying the current behavior for scenarios 2 and 3 as they exist today are not expected.

1. No variables set -- uses the runtime next to the muxer on the PATH
2. Only DOTNET_ROOT -- uses the runtime from dotnet muxer on the path ignoring DOTNET_ROOT. Unexpected?
3. Only DOTNET_SDK_ROOT -- Ignores global.json and SDK_ROOT. Uses muxer on the path. Unexpected?
4. Both are set to the same value -- ???
5. Both are set to different values -- ???

[jkotas]

No variables set -- uses the runtime next to the muxer on the PATH

myapp.exe uses runtime from the default location. dotnet myapp.dll uses the runtime next to the muxer on the PATH that does not have to be the one in the default location.

Only DOTNET_ROOT -- uses the runtime from dotnet muxer on the path ignoring DOTNET_ROOT. Unexpected?

The effect of DOTNET_ROOT is very limited and makes sense in the shipping bits today.

It stops making sense to me in the motivating scenario for this proposal. (Paraphrasing) The motivating scenario for this proposal is to allow one to completely relocated the .NET runtime and SDK, without touching PATH (assuming that the muxer on the PATH is recent enough). If DOTNET_ROOT is ignored for dotnet myapp.dll, .NET runtime is not fully relocated.

[agocke]

What would happen to a loose zip file in this system if you set the environment variable at the system or user level?

[MichaelSimons]

This link 404s.

[richlander]

It would be very useful to see an in-depth appendix that demonstrates how these tools work, on both Windows and some unix (to provide sufficient comparse/contrast). Otherwise, it seems like we're designing in a bit of a vaccum.

[marcpopMSFT]

What would happen to a loose zip file in this system if you set the environment variable at the system or user level?

Per offline discussion, this is the big sticking point with this proposal as the moment as we want to ensure that a customer downloading a zip independently of dotnetup can navigate to it (or add it to their path) and use it without getting super confused. The current proposal will lead to unexpected behavior in that scenario. We're exploring whether it's possible to only redirect for the centrally installed muxer (or just the program files\dotnet one on windows as windows is the most challenging to resolve).

[jkotas]

Isn't that a confusing behavior?

[marcpopMSFT]

This was a compromise with Andy as he didn't want to change the runtime load behavior. We're still talking with Elinor with how this might work but per my comment below, I think we're going to focus on finding a way to get feedback that is fewer long-term implications. If we get strong signal during previews, we'll revisit.

[jkotas]

I dislike CLI tools that change the behavior based on the path they are launched from. It means e.g. copying/linking the tool to some other place or launching it via an alternative path produces different results...

[jkotas]

It is simplest solution, it has the least amount of magic, and it does not require any changes in the admin installed .NET. I have strong preference for this solution, at least as a starting point.

[marcpopMSFT]

Ultimately, the team is trying to find a point in time solution that allows us to get feedback on dotnetup on windows machines with admin installs. I think given the concerns raised from runtime folks about making permanent changes to unblock feedback on an experiment, we'll probably start with options that are more temporary in nature (like dotnetup elevating and modifying your system path or having a "dotnetup dotnet" command or one of the targeted changes for specific terminals/profiles).

[richlander]

The .NET ecosystem has long needed better SDK selection capabilities similar to tools like nvm or rustup.

I did a search on the doc and seem that nvm and rustup show up just once each in the doc, in that first sentence. I don't think we should merge this design until an in-depth analysis is addded that compare/contrasts the proposal with the "long needed" capabilities that are "similar to". In fact, the entire premise of the proposal is that we should do something similar, yet, there is no commentary provided on how well the design achieves that. If similarity isn't a goal, then maybe the comparison with other ecosystems should be removed.

The RID simplification spec I wrote is most similar to this one and could offer a guide on how to provide industry context. Searches on "rust", "python" and "manylinux" within that doc will demonstrate the research the team did to generate the design and how the final design differs from the existing SOTA in only nuanced and practical ways. The final doc documented the design in the context of that foundational and inspirational research and helped readers understand the need for the change and why we were adopting (subtly) different approaches. As the primary writer of that doc, the compare/contrast with existing solutions helped me build up confidence on the strength of our plan.

At least some of the comments in the doc likely wouldn't be present in their current form if industry analysis was present.

For fairness, most of our self-litigated analysis is not present in our spec, just the final derived insight. We also looked at golang and it isn't mentioned at all. If an analysis can be shared compactly like I did, that likely works well.

[marcpopMSFT]

This proposal was strictly focused on trying to enable us to get feedback from previews on machines that had Windows Admin installs and the muxer is unique enough that it may require a unique solution. I went ahead and added some compact ecosystem context to the primary document that outlines the cli-acquisition proposal here.

I'm going to leave this open as we continue to have conversations but per my comment above, we have a few options that are potentially more temporary in nature to unblock feedback and we'll focus on those. We'll gather feedback on usage of dotnetup previews and use that to drive how we solve the admin problem long-term.