Tentacle reports that it could not find a part of the path `Output.log`

[sburmanoctopus]

Team

• I've assigned a team label to this issue

What happened?

Tentacle sometimes reports that it could not find part of the path to Output.log

Reproduction

This was noticed while monitoring for RPC retry errors. It would only happen rarely, but it was noticed.

Error and Stacktrace

Server exception: 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/etc/octopus/Tentacle/Work/20231022132150-45254-2/Output.log'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Octopus.Tentacle.Util.OctopusPhysicalFileSystem.OpenFile(String path, FileMode mode, FileAccess access, FileShare share) in /opt/buildagent/work/639265b01610d682/source/Octopus.Tentacle/Util/OctopusPhysicalFileSystem.cs:line 135
   at Octopus.Tentacle.Util.OctopusPhysicalFileSystem.OpenFile(String path, FileAccess access, FileShare share) in /opt/buildagent/work/639265b01610d682/source/Octopus.Tentacle/Util/OctopusPhysicalFileSystem.cs:line 129
   at Octopus.Tentacle.Services.Scripts.ScriptLog.GetOutput(Int64 afterSequenceNumber, Int64& nextSequenceNumber) in /opt/buildagent/work/639265b01610d682/source/Octopus.Tentacle/Services/Scripts/ScriptLog.cs:line 38
   at Octopus.Tentacle.Services.Scripts.ScriptService.GetResponse(ScriptTicket ticket, RunningScript script, Int64 lastLogSequence) in /opt/buildagent/work/639265b01610d682/source/Octopus.Tentacle/Services/Scripts/ScriptService.cs:line 87
   at Octopus.Tentacle.Services.Scripts.ScriptService.CompleteScript(CompleteScriptCommand command) in /opt/buildagent/work/639265b01610d682/source/Octopus.Tentacle/Services/Scripts/ScriptService.cs:line 69
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage)
   at Halibut.HalibutRuntime.HandleIncomingRequest(RequestMessage request)
   at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor)
   at Halibut.ServiceModel.HalibutProxyWithAsync.ThrowExceptionFromReceivedError(ServerError error, ILog logger)
   at Halibut.ServiceModel.HalibutProxyWithAsync.MakeRpcCall(MethodInfo asyncMethod, Object[] args)
   at Halibut.ServiceModel.HalibutProxyWithAsync.InvokeAsyncT[T](MethodInfo asyncMethod, Object[] args)
   at System.Reflection.AsyncDispatchProxyGenerator.InvokeAsync[T](Object[] args)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.<>c__DisplayClass20_2.<<Finish>b__5>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Utils.AsyncHalibutFeatureExtensionMethods.WhenEnabled[T](AsyncHalibutFeatureWithResult`1 asyncHalibutFeatureWithResult, Func`1 action)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.<>c__DisplayClass20_0.<<Finish>b__3>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.<>c__DisplayClass8_1`1.<<ExecuteWithNoRetries>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.<>c__DisplayClass8_0`1.<<ExecuteWithNoRetries>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Retries.RpcCallNoRetriesHandler.ExecuteWithNoRetries[T](Func`2 action, Boolean abandonActionOnCancellation, TimeSpan abandonAfter, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.ExecuteWithNoRetries[T](RpcCall rpcCall, Func`2 action, ILog logger, Boolean abandonActionOnCancellation, ClientOperationMetricsBuilder clientOperationMetricsBuilder, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.Finish(ScriptServiceVersion scriptServiceVersionToUse, ScriptStatusResponseV2 lastStatusResponse, CancellationToken scriptExecutionCancellationToken)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.ObserveUntilCompleteThenFinish(ScriptServiceVersion scriptServiceVersionToUse, ScriptStatusResponseV2 scriptStatusResponse, CancellationToken scriptExecutionCancellationToken)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.ExecuteScript(CancellationToken scriptExecutionCancellationToken)
   at Octopus.Tentacle.Client.TentacleClient.ExecuteScript(StartScriptCommandV2 startScriptCommand, Action`1 onScriptStatusResponseReceived, Func`2 onScriptCompleted, ILog logger, CancellationToken scriptExecutionCancellationToken)
   at Octopus.Server.Orchestration.Targets.Tentacles.TentacleRemoteEndpointFacadeV2.ExecuteCommand(StartScriptCommand startScriptCommand, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/TentacleRemoteEndpointFacadeV2.cs:line 71
   at Octopus.Server.Orchestration.Targets.Common.RemoteEndpointFacadeCancellationTokenDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/RemoteEndpointFacadeCancellationTokenDecorator.cs:line 36
   at Octopus.Server.Orchestration.Targets.Tentacles.Observability.ErrorLoggingRemoteEndpointFacadeDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/Observability/ErrorLoggingRemoteEndpointFacadeDecorator.cs:line 75
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.TentacleExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 files, Nullable`1 forceIsolationLevel, Boolean raw, ITaskLog taskLog, String isolationMutexName, CancellationToken cancellationToken, Nullable`1 isolationMutexTimeout) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ExecutionTargets/TentacleExecutionTarget.cs:line 70
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteRawScript(ScriptCollection scripts, IReadOnlyList`1 files, Boolean isRaw, ITaskLog taskLog, CancellationToken cancellationToken, Nullable`1 isolationMutexTimeout, ExecutionIsolation isolationLevel) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 118
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.ScriptCommandBuilder.Execute(ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/CommandBuilders/ScriptCommandBuilder.cs:line 67
   at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.TentacleMachineHealthController.<>c__DisplayClass2_0.<<CheckMachineHealth>b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/TentacleMachineHealthController.cs:line 40
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.InvokeActionHandler(Machine target, ActionHandlerInvocation actionHandler, ActionAndTargetScopedVariables actionAndTargetScopedVariables, IExecutor executor, TargetManifest targetManifest, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 238
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.ExecuteOnWorker(TargetManifest targetManifest, Worker worker, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 136
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 80
   at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.TentacleMachineHealthController.CheckMachineHealth(Machine machine, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/TentacleMachineHealthController.cs:line 61
   at Octopus.Server.Orchestration.ServerTasks.HealthCheck.Controllers.MachineHealthController.CheckHealth(Machine machine, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/Controllers/MachineHealthController.cs:line 106
   at Octopus.Server.Orchestration.ServerTasks.HealthCheck.HealthCheckService.PerformHealthCheck(Machine machine, IHealthResultCollator healthResultCollator, CancellationToken cancellationToken, ExceptionHandling exceptionHandling, Func`3 customAction) in ./source/Octopus.Server/Orchestration/ServerTasks/HealthCheck/HealthCheckService.cs:line 102

More Information

No response

Workaround

No response