Extracating raw values of slices (I/P/B Slices)

[makisukurisu]

Hi! Is it possible to extract values of slices (not values/pixels of the video stream itself) using this library?
To avoid confusion, when I'm talking about slices here, I'm talking about I/P/B (and other) as defined by H.264

For example, I can see that there's a pict_type attribute on a frame, so this got me thinking that I can extract the values of slices (they have to extracted somewhere to reconstruct the image, right?)

I've tried using to_ndarray method, and it's counterpart - from_ndarray to alter the videostream, but even if it returns some kind of representation of the slices, I can't use it to get back the stream.
to_ndarray returns 2-dim array, but from_ndarray requires 3-dims. Also the to_ndarray method returns (in my case) different shape, then my video. For example, 19201080 returns 16201920.

Perhaps I misunderstood how these methods should be used (it looks like I can input per-channel pixel values with from_ndarray?)

For some context, I'm designing a steganography algorithm that will embed data to (alter value of) I/P/B slices (blocks) of compressed H.264 video. I guess I can implement parsing on my own, of course, but if there's a ready-made solution for decoding/encoding video files, I'd like to use that instead.

Thanks, and sorry for my ramble.

[kenwaytis]

I can explain the image format. Your original video is likely in yuv420p, where the memory layout typically places the complete Y plane first, followed by the U and V planes. Therefore, the array height becomes 1080 (Y) + 540 (U/V) = 1620, while the width remains the same as the Y plane's width, 1920.

If you want to obtain RGB or BGR, you can directly specify the frame's format, such as frame.to_ndarray(format='rgb24').

Regarding fetching I/P/B frames, it seems PyAV cannot specify decoding. If the goal is modification rather than precise retrieval, maybe consider decoding and then re-encoding?

[makisukurisu]

Thanks for the response, I've decided to go a different route, and modified ffmpeg's macroblock decoder/encoder to achieve my goal (with some technical difficulties, because it's not as simple as I thought it would be).

So basically - what you suggested at the end there, more or less.

I'll close this discussion, as it's quite clear that PyAV is not designed to do what I wanted it to do.

Once again, thanks!