problem extracting frames from 480p Youtube videos using DirectX Samplegrabber

Hi.

I'm a computer programmer and I made a small converter (video files to animations).
It uses DirectX and SampleGrabber to extract the frames from the video files.
I recommend K-Lite to all the users of my application.
However I have problems with some video files, for example with 480p Youtube videos.
The samplegrabber is not automatically connected to the video decoder. I also tried to manually insert
it between the video decoder and the video renderer, it doesn't "want" to.
The video decoder is "LAV decoder". I tried with "ffdshow decoder" but
it crashes my application with all the video files I tried.

Here is an example of such file: https://drive.google.com/open?id=0ByKxAD_t9uvLVXI1alI0WGxEVXc
And a test app: https://drive.google.com/open?id=0ByKxAD_t9uvLUm9NSTNlcDJYOVE
The Snapshot button and the Callback event checkbox don't do anything on this video file.

Windows version: 8.1 Pro x64
K-Lite version: 12.2.7

I tried reinstalling K-Lite, no change.

Any thoughts on what can I try next...?

Thank you in advance.

Regards,
David

Sample Grabber has very limited capabilities. You can see its limitations at the bottom of this page:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd377544(v=vs.85).aspx

Most likely cause of failure in your case is due to VIDEOINFOHEADER2.

One solution would be to use for example VMR9 as renderer and grab samples directly from there. See its documentation.

Thank you for the suggestion.

But before deciding to use the Samplegrabber I did some research on Google and found other programmers having problems with VMR9 too. Not 100% similar with my problems but enough to not choose it.

And another thing: taking snapshots with VMR9 is so slow. On my 4 x 2 GHz processor it takes 0.55 sec to get a 1080p snapshot (and you can clearly see the playback stopping during this), while with samplegrabber I can set the playback rate to 4x and still get snapshots in realtime (without affecting the playback).

A solution would be to try first with the samplegrabber and, if it doesn't work, to set the video window to VMR9 and load the video file again.
But it's a lot of work...

Set the sample grabber to request RGB24 if you haven't already. Then try with NullRenderer.

Admin wrote:Set the sample grabber to request RGB24 if you haven't already.

Already did.

Admin wrote:Then try with NullRenderer.

The problem is not connecting the sample grabber output pin to null renderer, but to connect the sample grabber input pin to the decoder output pin.
Unfortunately still no change.

GraphStudioNext is an excellent to for experimenting with filter graphs.

I am unable to access your sample file, so I just download a random 480p youtube video.

I am able to (manually) construct these graphs:
LAV Splitter Source > LAV Video Decoder > Sample Grabber (RGB24) > Null renderer
LAV Splitter Source > LAV Video Decoder > Sample Grabber (RGB32) > Color Space Converter > VMR
LAV Splitter Source > LAV Video Decoder > Sample Grabber (YUY2) > AVI Decompressor (YUY2) > Color Space Converter > VMR

You can click on a pin for a lot of details about the pin and the pin connection.

Yes, it's a good tool, I used it many times.

Found out why couldn't connect the output pin of the decoder to the input pin of the sample grabber: it wanted a "Color Space Converter" filter inserted before the sample grabber. Although the output of the decoder and the input of the sample grabber looks identical.



Now it gets the snapshots but it look like with the wrong size and/or bit depth:

Now:


It should be:



This is the video: https://www.youtube.com/watch?v=tIJQkR-ofFo

And I'm using this https://addons.mozilla.org/en-US/firefox/addon/youtube-video-and-audio-dow/?src=userprofile to get the 480p with no sound. I tried with other addons too, same problem.



Late edit:

Seems that the TSampleGrabber component from DSPack is set to detect VideoInfoHeader2:

Code:

function TSampleGrabber.GetBitmap(Bitmap: TBitmap; Buffer: Pointer; BufferLen: Integer): Boolean;
var
   hr: HRESULT;
   BIHeaderPtr: PBitmapInfoHeader;
   MediaType: TAMMediaType;
   BitmapHandle: HBitmap;
   DIBPtr: Pointer;
   DIBSize: LongInt;
begin
   Result := False;
   if not Assigned(Bitmap) then
      Exit;
   if Assigned(Buffer) and (BufferLen = 0) then
      Exit;
   hr := SampleGrabber.GetConnectedMediaType(MediaType);
   if hr <> S_OK then
      Exit;
   try
      if IsEqualGUID(MediaType.majortype, MEDIATYPE_Video) then
      begin
         BIHeaderPtr := nil;
         if IsEqualGUID(MediaType.formattype, FORMAT_VideoInfo) then
         begin
            if MediaType.cbFormat = SizeOf(TVideoInfoHeader) then // check size
               BIHeaderPtr := @(PVideoInfoHeader(MediaType.pbFormat)^.bmiHeader);
         end
         else if IsEqualGUID(MediaType.formattype, FORMAT_VideoInfo2) then
         begin
            if MediaType.cbFormat = SizeOf(TVideoInfoHeader2) then // check size
               BIHeaderPtr := @(PVideoInfoHeader2(MediaType.pbFormat)^.bmiHeader);
         end;
         // check, whether format is supported by TSampleGrabber
         if not Assigned(BIHeaderPtr) then
            Exit;
         BitmapHandle := CreateDIBSection(0, PBitmapInfo(BIHeaderPtr)^,
            DIB_RGB_COLORS, DIBPtr, 0, 0);
         if BitmapHandle <> 0 then
         begin
            try
               if DIBPtr = nil then
                  Exit;
               // get DIB size
               DIBSize := BIHeaderPtr^.biSizeImage;
               if DIBSize = 0 then
               begin
                  with BIHeaderPtr^ do
                     DIBSize := GetDIBLineSize(biBitCount, biWidth) * biHeight * biPlanes;
               end;
               // copy DIB
               if not Assigned(Buffer) then
               begin
                  // get buffer size
                  BufferLen := 0;
                  hr := SampleGrabber.GetCurrentBuffer(BufferLen, nil);
                  if (hr <> S_OK) or (BufferLen <= 0) then
                     Exit;
                  // copy buffer to DIB
                  if BufferLen > DIBSize then // copy Min(BufferLen, DIBSize)
                     BufferLen := DIBSize;
                  hr := SampleGrabber.GetCurrentBuffer(BufferLen, DIBPtr);
                  if hr <> S_OK then
                     Exit;
               end
               else
               begin
                  if BufferLen > DIBSize then // copy Min(BufferLen, DIBSize)
                     BufferLen := DIBSize;
                  Move(Buffer^, DIBPtr^, BufferLen);
               end;
               Bitmap.Handle := BitmapHandle;
               Result := True;
            finally
               if Bitmap.Handle <> BitmapHandle then // preserve for any changes in Graphics.pas
                  DeleteObject(BitmapHandle);
            end;
         end;
      end;
   finally
      FreeMediaType(@MediaType);
   end;
end;

But for that video file MediaType.formattype is set to FORMAT_VideoInfo.

Found a way to solve the problem for this situation: I set RGB32 instead of RGB24.
This way it doesn't require the Color space converter, it doesn't distort the snapshot and to convert from RGB32 to RGB24 is easy.

My question is: is this solution "universal" or do I have to detect the RGB type for each video file, set it and convert later?

Another thing:  here http://stackoverflow.com/questions/6436722/regarding-the-scope-of-sample-grabber-in-directshow/6440568#6440568 is mentioned a way to force RGB24.
Is this better? Unfortunately I couldn't test it. I downloaded DirectX Sdk but couldn't find "RGBfilters" (searched Google afterwards but still nothing)...

LAV Video decoder supports outputting in a large number of colorspaces (see its options). It does conversion internally when needed. Outputting RGB32 should work with any input file.

OK
Thank you for helping me with this problem.

An offtopic question: from an application how to test if K-Lite is installed (maybe even if it is the latest version)? I ask because I want to implement in my application if it can't render a video file should verify if K-Lite is installed and advice the user to install it (or update it if it's possible to implement).

You can check for this registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\KLiteCodecPack_is1
on 64-bit system:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\KLiteCodecPack_is1

Or check for presence of LAV:
HKEY_CLASSES_ROOT\CLSID\{083863F1-70DE-11D0-BD40-00A0C911CE86}\Instance\{171252A0-8820-4AFE-9DF8-5C92B2D66B04}
HKEY_CLASSES_ROOT\CLSID\{083863F1-70DE-11D0-BD40-00A0C911CE86}\Instance\{EE30215D-164F-4A92-A4EB-9D4C13390F9F}

Thank you again

Hi,

I'm back with a nearby problem: my tool can't properly extract frames from mpeg-dash containers (for example mp4 video-only files).
I am not sure where is the problem: in DSPack, in video codec, my code or a combination.

Steps to reproduce the problem:

In Mozilla FireFox install this addon: https://addons.mozilla.org/en-US/firefox/addon/youtube-video-and-audio-dow/?src=api
Go to this video: https://www.youtube.com/watch?v=XN9wbIE04lQ&ytbChannel=Venita%20Cavalieri
Using the addon, download a video-only file (any resolution/bitrate).
Download and install my tool: https://drive.google.com/open?id=0ByKxAD_t9uvLOVJsUVc4aXlTT0U
Use the Import Video command. Select only the footage (frame 230 time 7.624 to frame 1056 time 35.219) and click on OK.
Wait a little bit until it finishes.
You'll notice it skipped some frames and added others at the end (from after the selection).

This happens only with this type of video files.

Could you help me with this problem too?

Thank you.

Sorry, I can't help you with this.

Ok

Thanks anyway.

Something very interesting: I just tried with X-Codeck Pack 2.7.4 and the problem disappeared...

Late edit:

Here is the graph screenshot for the video file when using X-Codec pack:



And here is the graph screenshot when using K-Lite:

You can change that with:
Codec Tweak Tool > Preferred Splitters > MP4 > Merit

But there should be no difference in behavior between "LAV Splitter Source" and "LAV Splitter". It is basically the same filter, except that the first one also performs the file reading.

The difference in behavior is probably because you are comparing two different versions. That other pack most likely has an older version. Playback of your sample file works normally here with K-Lite 13.0.0.

I tested with the latest 13.0.0 and it's the same problem.
You said "playback it's the same". I agree, it's the same. But in the steps for reproducing the problem I said "You'll notice it skipped some frames and added others at the end (from after the selection)." That means the problem is in capturing frames, not in playback. So you should compare the captured frames with the frames from the video file.

Anyway, I opened an issue on Lav filters git here: https://github.com/Nevcairiel/LAVFilters/issues/129
The author said "mpeg-dash" are not supported by LAV...
Although the Intelligent Connect is using LAV filters to open "mpeg-dash" video files...

So far I haven't found a good solution...

Dash is partially supported by FFmpeg (which is what LAV uses). At least the files that are created by that downloading tool. Dash streams are not supported.

I am not using your tool. That is pointless, since I don't know what it does internally. Based on your described symptoms, it appears the samplegrabber captures a different frame for timestamp X than you get during playback. You need to figure out why it is shifted. Start by logging all timestamps. Maybe that reveals something.

The samplegrabber uses an event (onBuffer) to signal the buffer with the bitmap data was filled.
Then other code moves the data from buffer into an array of bitmaps and signals the samplegrabber to continue.

At some point in the mpeg-dash files the buffer is not filled in the proper moment in playback.
And the code tries to retrieve frames from beyond the finish marker to achieve the required number of frames.

Using that old version (september 2015) of LAV filters solves the problem. This shows the problem may not be in my coding...

Here are additional versions of LAV Filters that you can test:
https://files.1f0.de/lavf/nightly/

SampleGrabber also has a mode without callback function:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd407288(v=vs.85).aspx#run_the_graph
I don't know if that makes any difference.

Here is an alternative frame grabber:
http://blog.monogram.sk/janos/2007/12/16/monogram-frame-grabber-v1001/
(programming interface can be found in the comments)

Thank you

Admin wrote:Here are additional versions of LAV Filters that you can test:
https://files.1f0.de/lavf/nightly/

I tested them but unfortunately no difference

Admin wrote:
SampleGrabber also has a mode without callback function:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd407288(v=vs.85).aspx#run_the_graph
I don't know if that makes any difference.

It says "In buffering mode, the Sample Grabber stores a copy of every sample". This situation has to be avoided, because it requires a LOT of RAM.
My tool uses the callback mode (the event OnBuffer is triggered by the callback function). It stores the frames in a buffer of 8..40 bitmaps. When the buffer is full, stops playback and compresses the bitmaps into an internal format then resumes playback.
Even with buffering mode there are no guarantees it will store every sample on mpeg-dash files.

Oh, on the same page found something I was searching for some time: how to deactivate playback clock so the frames could be captured at maximum speed

Admin wrote:
Here is an alternative frame grabber:
http://blog.monogram.sk/janos/2007/12/16/monogram-frame-grabber-v1001/
(programming interface can be found in the comments)

That seems promising, but unfortunately:
- colorspace.cpp and grab_types.h from comments are not sources;
- I don't see a way to set it programatically + to redirect the output from HDD to RAM (in that 8..40 bitmaps buffer)...
I'll try to talk to the guy, although it doesn't seem to be active from 2009.

A version from 2015 worked OK didn't it? If you can find the last working version, then that might give a clue as to what is the cause of the problem.

You don't need any source for that filter. grab_types.h define an interface to access the filter settings. Enumerate the filters in the graph, then check if it supports that interface (with QueryInterface), and if it does, you can use those functions.

Admin wrote:A version from 2015 worked OK didn't it? If you can find the last working version, then that might give a clue as to what is the cause of the problem.

You don't need any source for that filter. grab_types.h define an interface to access the filter settings. Enumerate the filters in the graph, then check if it supports that interface (with QueryInterface), and if it does, you can use those functions.

I will try, thank you for the suggestions.
But still have to find a way to switch the output to RAM instead of files + to pause the samplegrabber when the buffer is full.

Meantime I implemented another fix. It's not that professional but seems to work:
At some point I noticed if the video file is closed, the window with the video components freed and recreated and the video file reopened, the capturing problems are gone.
A sort of "hard reset" for the video file. I think this way the previous data for seeking (the user playing the video file and moving back/forth to find the area for import) is flushed...
So, if the code is finding "ftypdash" in the file's header, it's reloading the video file and recreates the window. This adds a few seconds to the processing but for the moment it's acceptable.

So seeking directly to the desired spot works, but samplegrabber gets confused after multiple seeks? Would be interesting to see if there is any pattern that triggers the problem. Maybe seeking backwards?