Log errors relating to invalid input

[sgryt]

So callers have a chance of diagnosing problems in their data. Or rule out that error, and continue looking elsewhere.

[ploeh]

Logging should, IMO, be based on IObservable<T>/IObserver<T>. The concrete logging sink should probably, at the beginning, be an Azure BLOB.

[moodmosaic]

How about doing something like this? (Using Azure Blob storage instead of an in-memory list.)

diff --git a/Src/Qaiain.CommandLine/Program.fs b/Src/Qaiain.CommandLine/Program.fs
index d8a6ac1..4699917 100644
--- a/Src/Qaiain.CommandLine/Program.fs
+++ b/Src/Qaiain.CommandLine/Program.fs
@@ -195,7 +195,13 @@ let ERROR_INVALID_DATA = 13

 // End of constants -------

+type Failure = {
+    XmlInput : string
+    DateTime : DateTime
+    ErrorMessage : string }
 open FSharpx.Validation
+open System.Reactive.Subjects

 [<EntryPoint>]
 let main argv = 
@@ -213,6 +219,12 @@ let main argv =
         | Mail.ErrorMessage.InvalidMessageReference -> "Invalid message reference, possibly due to a non-existing block blob."

     let handle msg = Mail.handle getMessage deleteMessage send msg
+    let failuresInMemory = System.Collections.Generic.List<Failure>()
+    let failureSubject = new Subject<Failure>()
+    failureSubject
+    |> Observable.subscribe failuresInMemory.Add
+    |> ignore

     match queue |> AzureQ.dequeue with
     | Some(msg) ->
@@ -221,6 +233,11 @@ let main argv =
             queue.DeleteMessage msg
             ERROR_SUCCESS
         | Failure f ->
-            f |> toString |> Console.Error.WriteLine
+            failureSubject.OnNext
+                {
+                    XmlInput = msg.AsString
+                    DateTime = DateTime.UtcNow
+                    ErrorMessage = f |> toString
+                }
             ERROR_INVALID_DATA

[ploeh]

As far as I can tell, the current behaviour of Qaiain is that if an error occurs, the message stays in the queue.

This may be problematic. However, it depends on the reason for the error.

In the case of the UnknownMessageType error type, we immediately know that Qaiain can never handle that message, in which case it ought to be moved to a Poison Queue.

In the case of the InvalidMessageReference error type, this may actually be an intermittent error, so it makes sense to let the message stay in the queue so that it can be retried a number of times. However, it's important to prevent such a message from being replayed for ever, so after appropriate retries, it should also be moved to a Poison Queue.

Instead of logging, this should take care of the troubleshooting aspect. Before doing any work on this issue, however, @sgryt should tell us if he still needs this, and if the above sketch of a solution is the appropriate direction to take.