Add an optional experimental feature that allows the VS extension to let the the discovered bindings written out to a file.

[gasparnagy]

⚡️ What's your motivation?

Provide an alternative way for the Visual Studio extension to discover the bindings without the need to load the test assembly and run code from it, that is potentially causing assembly loading errors, like reqnroll/Reqnroll.VisualStudio#97

🤔 What's changed?

Added BindingProviderService.OnBindingRegistryBuildingCompleted(): A handler that can write the discovered bindings to a JSON file for the Visual Studio extension if instructed though an environment variable.

Currently the VS extension loads the test assembly to a console app (e.g. <extension>\Connectors\Reqnroll-Generic-net8.0\reqnroll-vs.exe, tries to resolve its dependencies (magic++) and then invoke the BindingProviderService.DiscoverBindings method from the loaded assembly that returns a JSON string with the binding information. The connector does a bit of further processing on it (gets step definition source line numbers from PDB), and then returns it to the VS extension, so that it can properly display the binding info.

This is performed after every successful build.

This sometimes causes assembly loading errors and requires a compilation of the console app for each .NET version.

The new concept would be the following:

• This new concept is only an alternative way. The other would still work and the VS extension could decide which one to use.
• If the new concept would be used, then
  • instead of loading the test assembly to a console app, we would invoke dotnet test --no-build, by setting the environment variables REQNROLL_DRY_RUN=true and REQNROLL_BINDING_OUTPUT=true before.
  • ideally the execution should be filtered for a single feature file
  • this will simulate running the tests (without actually invoking any user code), but OnBindingRegistryBuildingCompleted would split out the same JSON string that was produced by BindingProviderService.DiscoverBindings to a file (reqnroll_bindings.json)
  • the VS extension would load back the JSON from the file
  • it would resolve the source file references from the PDB file (for that you don't need to fully load the assembly, so less error-prone)
  • would return it to the VS extension

Open questions:

• How to make sure that the other formatters are not activated during this. Maybe we could extend the idea of Set individual formatters with key-value based environment variables #769 with something, like REQNROLL_FORMATTERS_ALL=false that would disable all "other" formatters.
• How to filter for a feature file? This is an optimization, because for a big project even the dry-run execution takes longer time (the big project with 5000 tests requires 9s on my machine). But the VS extension knows about the feature files, so it can compose a filter condition probably.
• How to get the bindings for a project that does not yet have feature files? Probably we don't need the bindings for this anyway, but we can fall back to the old method in this case if nothing else helps.

Plan:

• My plan is to include this formatter as an internal hidden formatter in v3.0, so that we can experiment with it. If it is not going to be useful, we can remove it with the next release.
• The only problem that we need to solve in the Reqnroll codebase is how to disable all other formatters.

What do you think?

🏷️ What kind of change is this?

• ⚡ New feature (non-breaking change which adds new behaviour)

♻️ Anything particular you want feedback on?

📋 Checklist:

• I've changed the behaviour of the code

---

This text was originally taken from the template of the Cucumber project, then edited by hand. You can modify the template here.

[clrudolphi]

Both the legacy connectors and this approach both run something out of process, right? I'm confused as to how this approach avoids assembly load conflicts while the existing approach has those problems. What is different that helps?

[clrudolphi]

Rather than dry-run a feature's worth of tests why not generate a pseudo-test (without test framework attributes) in each test assembly that can be invoked to trigger binding discovery?

[gasparnagy]

Both the legacy connectors and this approach both run something out of process, right? I'm confused as to how this approach avoids assembly load conflicts while the existing approach has those problems. What is different that helps?

Because with this, the test runner (dotnet test) does the assembly loading, not our connector.

Rather than dry-run a feature's worth of tests why not generate a pseudo-test (without test framework attributes) in each test assembly that can be invoked to trigger binding discovery?

Yeah, we could do that, but that would be ugly as the pseudo-test would be also visible for the user.

[clrudolphi]

we could do that, but that would be ugly as the pseudo-test would be also visible for the user.

I was thinking in terms of an additional class added to the produced assembly not an additional method on each generated feature class. Something like the assembly using class. It's present and visible, yes, but not likely to bother anyone.

[gasparnagy]

[...] but not likely to bother anyone.

It would, I know. ☹️ I have a few clients working in regulated domains and they need to track every test. That's unfortunate, but we cannot change this.

[gasparnagy]

I have reworked the PR and not use the formatter API for this. It is much simpler this way and easier to remove if not useful.

[Copilot]

Pull Request Overview

This PR adds an experimental feature that enables the Visual Studio extension to discover bindings through file output instead of loading test assemblies directly. This addresses assembly loading errors that occur when the VS extension tries to load and execute test assemblies.

Key changes:

• Introduces a new IBindingProviderService interface and implementation that can write discovered bindings to a JSON file
• Adds environment variable support (REQNROLL_BINDING_OUTPUT) to trigger binding file output during test runs
• Integrates the binding provider service into the dependency injection container and binding registry building process

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
BindingProviderService.cs	Adds new interface and implements binding output functionality with environment variable detection
RuntimeBindingRegistryBuilder.cs	Integrates binding provider service and calls it when building completes
DefaultDependencyProvider.cs	Registers the new binding provider service in the DI container
BindingJsonSourceGenerator.cs	Updates JSON serialization options to ignore null properties
RuntimeBindingRegistryBuilderTests.cs	Updates test constructor to include mock binding provider service

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[meliyesil]

I have reqnroll vs2022 extension 2025.1.256 and dotnet 9.0.304
Getting the following exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.BadImageFormatException: Could not load file or assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Reference assemblies cannot be loaded for execution. (0x80131058)
File name: 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' ---> System.BadImageFormatException: Cannot load a reference assembly for execution.
at ReqnrollConnector.ReflectionExecutor.Execute(String optionsJson, Assembly testAssembly, AssemblyLoadContext assemblyLoadContext, IDictionary2 analyticsProperties) at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span1 copyOfArgs, BindingFlags invokeAttr)
--- End of inner exception stack trace ---
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span1 copyOfArgs, BindingFlags invokeAttr) at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at ReqnrollConnector.ReflectionExtensions.ReflectionCallMethod[T](Object obj, String methodName, Type[] parameterTypes, Object[] args) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\NetExtensions\ReflectionExtensions.cs:line 26 at ReqnrollConnector.ReflectionExtensions.ReflectionCallMethod[T](Object obj, String methodName, Object[] args) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\NetExtensions\ReflectionExtensions.cs:line 7 at ReqnrollConnector.ReflectionExecutor.<>c__DisplayClass0_0.<Execute>b__2(Object instance) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\ReflectionExecutor.cs:line 23 at Some1.Map[TResult](Func2 map) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\NetExtensions\Optional\Some.cs:line 26 at ReqnrollConnector.ReflectionExecutor.Execute(DiscoveryOptions options, Func3 testAssemblyFactory, ILogger _log, IAnalyticsContainer analytics) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\ReflectionExecutor.cs:line 21
at ReqnrollConnector.Runner.ExecuteDiscovery(DiscoveryOptions options, Func3 testAssemblyFactory) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\Runner.cs:line 50 at ReqnrollConnector.Runner.<>c__DisplayClass4_0.<Run>b__0(ConnectorOptions options) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\Runner.cs:line 30 at FunctionalExtensions.Map[TSource,TResult](TSource this, Func2 fn) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\NetExtensions\FunctionalExtensions.cs:line 5
at ReqnrollConnector.Runner.Run(String[] args, Func3 testAssemblyFactory) in D:\a\Reqnroll.VisualStudio\Reqnroll.VisualStudio\Connectors\Reqnroll.VisualStudio.ReqnrollConnector.Generic\Runner.cs:line 26 Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: D. Path '', line 0, position 0. at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at Reqnroll.VisualStudio.Connectors.OutProcReqnrollConnector.Deserialize(RunProcessResult result, Func2 formatErrorMessage)

[gasparnagy]

@meliyesil Please report it to https://github.com/reqnroll/Reqnroll.VisualStudio/issues