Decoding frame, resizing, pointer forwarding, encodeing frame example needed.

[HEBOS]

Hi @jlennox,

Thank you for clarifying this issue.
It is awesome, how much effort you have put into this project.
There is so much code inside.

My use case is simple:

• grab frame by frame from the input video
• convert it to RGBA
• get the pointer of the frame which should still reside in device memory.
• forward the frame to the Darknet (YOLOV wrapper) Detect method (the overload which accepts IntPtr as function argument).
• resize image
• write frame using NVIDIA encoder

All steps need to be performed in GPU, so no device-host copying should take place.

Is that possible with your project?
If yes, can you please provide some hints, or an example?

P.S.
It would be also also brilliant if GPU frame from your project can be converted to OpenCV GpuMat. The C# wrapper which can be used for that is EmguCV, so all other GPU operations can be done using EmguCV (i.e. OpenCV). The number of possible operations, which can be done on GpuMat by OpenCV is enormous.

Any help is appreciated.

Thanks

[jlennox]

I never did get image resizing working for purely GPU methods. It has been awhile since I worked on this but I was experimenting with a DirectX 11 method of resizing but never got it working. There might be something inside the nVidia samples but I think I recall them being entirely done in CPU there. It looks like I am doing resizing but that's just how the surface is rendered to the window.

If you find something or have an idea on resizing, let me know and I can help experiment/incorporate it.

I think this method might convert from the directx surface2d to something OpenCV can use? https://docs.opencv.org/3.4/de/dc1/group__core__directx.html#gaaea63f9b492b4a84e1e9ef6d4eba377c
EmuguCV does not appear to directly support convertFromD3D11Texture2D so you'd need to add your own DllImport.

I am not sure, however, if OutputArray is GPU based or how to use that with EmuguCV (presumably it's an IInputArray which GpuMat accepts?)

[HEBOS]

Can you please provide an example, how to get object, which is can be sent to convertFromD3D11Texture2D function, as I'm not sure if I understand the code correctly.

What I tried is that I modified DisplayThreadCore to get GPU pointer which might be used for convertFromD3D11Texture2D.
The code doesn't raise any exception, but the SaveAsBitmap method (modified version, without bitmap.SaveTo line) gives the same peformance, as SaveAsGpuPointer, so I'm not sure if the implementation is right.
Strange enough is that context.Push says that "Pushes a context to the current CPU thread..."???
Does that mean that something is being done using CPU memory?

`private void DisplayThreadCore()
{
var frame = _framesChannel.Reader
.ReadAsync(_cts.Token)
.AsTask().Result;

        if (frame.IsFinalFrame)
        {
            _renderingCompleted.Set();
            return;
        }

        using var _lease = _nv12BufferPool.Lease(frame.Buffer);
        using var _ = _context.Push();
        {
            //_window.FrameArrived(frame);

            //if (_args.WriteBitmap)
            //{
            //    SaveAsBitmap(
            //        frame,
            //        Path.Combine(_args.BitmapPath, $"output-{_displayedFrames:D8}.bmp"));
            //}

            SaveAsGpuPointer(frame);
        }


        _globalStopWatch.Stop();
        Console.WriteLine($"Frame {_displayedFrames} - elapsed: {_globalStopWatch.ElapsedMilliseconds}");

        

        ++_displayedFrames;
    }`

Thanks

[jlennox]

Hmmm. The Yolo wrapper does not appear to accept GPU pointers. It's always expecting to make a CPU->GPU copy. In googling if Yolo itself supports this, I did not find any results. I suspect that it unfortunately may not. You might need to update Yolo to be able to support receiving a GPU pointer directly.

That wrapper even has an entirely unneeded cpu->cpu buffer copy! Which is really unfortunate for something which requires performance. https://github.com/AlturosDestinations/Alturos.Yolo/blob/master/src/Alturos.Yolo/YoloWrapper.cs#L223

This code is in general a mystery to me. var size = Marshal.SizeOf(imageData[0]) * imageData.Length; should be the same as var size = imageData.Length; but there's an extra memory access and Marshal.SizeOf is also not fast.

To use this you'd need to have a GPU->CPU copy unfortunately, and if you do that, I'd highly recommend you remove that extra buffer copy from the wrapper and instead use fixed on the passed in array. AlturosDestinations/Alturos.Yolo#149

[HEBOS]

Wow. Great that you spotted that. I didn't dig into that code at all.
Can you please answer if the code below (paired with code in my previous comment) would work?

private static void SaveAsGpuPointer(FrameInformation frame) { var rgbSize = frame.GetRgba32Size(); using (var destPtr = CuDeviceMemory.Allocate(rgbSize)) { frame.DecodeToDeviceRgba32(destPtr, new Size(512, 288)); } }

Also, I spotted that your code takes around 150 frames, and then pauses for 3-4 seconds. It seems like it does some buffering.
I'm using Ge Force 970M.

Thanks

[jlennox]

@HEBOS Just an FYI, the original Alturos.Yolo was unmaintained and there's now a maintained fork at https://github.com/zgabi/Yolo.Net