Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add VP6[A] video decoding support #5369

Merged
merged 1 commit into from
Oct 5, 2021

Conversation

torokati44
Copy link
Member

This is an alternative to #3004.

In contrast to that:

  • This is a pure Rust solution, with no reliance on clang, no iffy FFI, no funky business with adding a quasi-libc on web, and no CI troubles.
  • It is much smaller in the WASM build: only an increase of about 90k, instead of 700k (or 240k even without libswscale).
    This is more in line with what the H.263 decoder added, which was about 80k, although that was including the colorspace conversion code.
  • Uses h263-rs-yuv for the colorspace conversion.
    Most likely as a result of this, it is measurably slower, at least on web. See for example: https://z0r.de/2289
    Merging yuv: Remove chroma interpolation, use iterators h263-rs#13 (then bumping the crate git ref) and web: Enable the bulk-memory WebAssembly extension #5225 should alleviate this somewhat.
  • It relies on a gutted-out NihAV instead of grabbing parts of FFmpeg or libav.
    Some more size reduction can be achieved by dropping a few more parts that are not strictly necessary, but I think this is already acceptable.
  • The decoder itself is under MIT license instead of LGPL, which matches Ruffle well.
    Thanks again to the author: https://codecs.multimedia.cx/nihav-relicensed-code/
  • Adds no unsafe blocks, aside from a few really small ones, in a single file, in one of the the decoder crates.

Similar to that:

  • It adds the vp6 feature to ruffle-core, enabled by default.
  • Includes VP6WithAlpha decoding and FrameDependency (keyframe) reporting.
  • Crops the unwanted encoded pixels on the right/bottom manually, to match bounds.

I plan to move ownership of https://github.com/torokati44/nihav-vp6 to the Ruffle organization after review, but before merging.

Copy link
Member

@kmeisthax kmeisthax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This more or less looks good to me. I'll merge after the repo transfer (so you have a chance to update the Cargo ref).

I merged the h263-rs PR you mentioned, so you might want to bump that at some point. You should also ensure the documentation on licensing makes it to LICENSE.md's third-party software table (which, itself, probably needs updating).

@torokati44
Copy link
Member Author

I have transferred the repo, updated the Cargo.toml ref, and added a line to LICENSE.md about this.

Thank you for merging the h263-rs PR. I will submit a separate dependency bump for that soon - I didn't want to bunch that up into these changes.

@torokati44
Copy link
Member Author

I have just now realized that we might be waiting for each other, because I was entirely unclear about the h263-rs git bump.
I want to do that after this is merged, everywhere at once, so there isn't any inconsistency/mismatch across the multiple refs in the main repo at any point in time.

@torokati44 torokati44 force-pushed the video-vp6-nihav branch 2 times, most recently from 01ef816 to c711676 Compare October 2, 2021 22:57
@Herschel
Copy link
Member

Herschel commented Oct 3, 2021

Getting a panic on the following file
vp6.zip

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidData', core\src\backend\video\software.rs:273:75
stack backtrace:
Loaded 'C:\Windows\System32\dbghelp.dll'.
   0: std::panicking::begin_panic_handler
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\/library\std\src\panicking.rs:515
   1: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\/library\core\src\panicking.rs:92
   2: core::result::unwrap_failed
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\/library\core\src\result.rs:1599
   3: enum$<core::result::Result<nihav_duck::codecs::vp56::VP56Header,enum$<nihav_core::codecs::DecoderError> > >::unwrap<nihav_duck::codecs::vp56::VP56Header,enum$<nihav_core::codecs::DecoderError> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\core\src\result.rs:1281
   4: ruffle_core::backend::video::software::vp6::impl$1::decode_frame
             at D:\projects\ruffle\core\src\backend\video\software.rs:273
   5: ruffle_core::backend::video::software::impl$2::decode_video_stream_frame
             at D:\projects\ruffle\core\src\backend\video\software.rs:77
   6: ruffle_core::display_object::video::Video::seek_internal
             at D:\projects\ruffle\core\src\display_object\video.rs:250
   7: ruffle_core::display_object::video::Video::seek
             at D:\projects\ruffle\core\src\display_object\video.rs:219
   8: ruffle_core::display_object::video::impl$1::post_instantiation
             at D:\projects\ruffle\core\src\display_object\video.rs:376
   9: ruffle_core::display_object::impl$7::post_instantiation
             at D:\projects\ruffle\core\src\display_object.rs:479
  10: ruffle_core::display_object::movie_clip::MovieClip::instantiate_child
             at D:\projects\ruffle\core\src\display_object\movie_clip.rs:1162
  11: ruffle_core::display_object::movie_clip::MovieClip::place_object
             at D:\projects\ruffle\core\src\display_object\movie_clip.rs:3187
  12: ruffle_core::display_object::movie_clip::impl$0::run_frame_internal::closure$0
             at D:\projects\ruffle\core\src\display_object\movie_clip.rs:1097
  13: ruffle_core::tag_utils::decode_tags<ruffle_core::display_object::movie_clip::impl$0::run_frame_internal::closure$0>
             at D:\projects\ruffle\core\src\tag_utils.rs:333
  14: ruffle_core::display_object::movie_clip::MovieClip::run_frame_internal
             at D:\projects\ruffle\core\src\display_object\movie_clip.rs:1115
  15: ruffle_core::display_object::movie_clip::impl$1::run_frame
             at D:\projects\ruffle\core\src\display_object\movie_clip.rs:1772
  16: ruffle_core::display_object::impl$7::run_frame
             at D:\projects\ruffle\core\src\display_object.rs:479
  17: ruffle_core::player::impl$1::run_frame::closure$0
             at D:\projects\ruffle\core\src\player.rs:1224
  18: ruffle_core::player::impl$1::update::closure$0<ruffle_core::player::impl$1::run_frame::closure$0,tuple$<> >
             at D:\projects\ruffle\core\src\player.rs:1611
  19: ruffle_core::player::impl$1::mutate_with_update_context::closure$0<ruffle_core::player::impl$1::update::closure$0,tuple$<> >
             at D:\projects\ruffle\core\src\player.rs:1552
  20: ruffle_core::player::GcArena::mutate<ruffle_core::player::impl$1::mutate_with_update_context::closure$0,tuple$<> >
             at C:\Users\Michael Welsh\.cargo\git\checkouts\gc-arena-fea9b4bbd970d646\4931b3b\src\gc-arena\src\arena.rs:161
  21: ruffle_core::player::Player::mutate_with_update_context<ruffle_core::player::impl$1::update::closure$0,tuple$<> >
             at D:\projects\ruffle\core\src\player.rs:1488
  22: ruffle_core::player::Player::update<ruffle_core::player::impl$1::run_frame::closure$0,tuple$<> >
             at D:\projects\ruffle\core\src\player.rs:1610
  23: ruffle_core::player::Player::run_frame
             at D:\projects\ruffle\core\src\player.rs:1212
  24: ruffle_core::player::Player::tick
             at D:\projects\ruffle\core\src\player.rs:518
  25: ruffle_desktop::impl$0::run::closure$0
             at D:\projects\ruffle\desktop\src\main.rs:397
  26: winit::platform_impl::platform::event_loop::impl$2::run_return::closure$0<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll>,ruffle_desktop::impl$0::run::closure$0>
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop.rs:206
  27: alloc::boxed::impl$45::call_mut<tuple$<enum$<winit::event::Event<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> > >,ref_mut$<enum$<winit::event_loop::ControlFlow> > >,dyn$<core::ops::function::FnMut<tuple$<enum$<winit::event::Event<enum$<ruffl
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\alloc\src\boxed.rs:1579
  28: winit::platform_impl::platform::event_loop::runner::impl$3::call_event_handler::closure$0<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop\runner.rs:245
  29: std::panic::impl$30::call_once<tuple$<>,winit::platform_impl::platform::event_loop::runner::impl$3::call_event_handler::closure$0>
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panic.rs:347
  30: std::panicking::try::do_call<std::panic::AssertUnwindSafe<winit::platform_impl::platform::event_loop::runner::impl$3::call_event_handler::closure$0>,tuple$<> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panicking.rs:401
  31: std::panicking::try<tuple$<>,std::panic::AssertUnwindSafe<winit::platform_impl::platform::event_loop::runner::impl$3::call_event_handler::closure$0> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panicking.rs:365
  32: std::panic::catch_unwind<std::panic::AssertUnwindSafe<winit::platform_impl::platform::event_loop::runner::impl$3::call_event_handler::closure$0>,tuple$<> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panic.rs:434
  33: winit::platform_impl::platform::event_loop::runner::EventLoopRunner<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::catch_unwind<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll>,tuple$<>,winit::platform_impl::platform::event_loop::
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop\runner.rs:152
  34: winit::platform_impl::platform::event_loop::runner::EventLoopRunner<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::call_event_handler<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop\runner.rs:239
  35: winit::platform_impl::platform::event_loop::runner::EventLoopRunner<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::move_state_to<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop\runner.rs:341
  36: winit::platform_impl::platform::event_loop::runner::EventLoopRunner<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::main_events_cleared<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop\runner.rs:227
  37: winit::platform_impl::platform::event_loop::flush_paint_messages<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop.rs:703
  38: winit::platform_impl::platform::event_loop::thread_event_target_callback::closure$0<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop.rs:2036
  39: core::ops::function::FnOnce::call_once<winit::platform_impl::platform::event_loop::thread_event_target_callback::closure$0,tuple$<> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\core\src\ops\function.rs:227
  40: std::panic::impl$30::call_once<isize,winit::platform_impl::platform::event_loop::thread_event_target_callback::closure$0>
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panic.rs:347
  41: std::panicking::try::do_call<std::panic::AssertUnwindSafe<winit::platform_impl::platform::event_loop::thread_event_target_callback::closure$0>,isize>
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panicking.rs:401
  42: std::panicking::try<isize,std::panic::AssertUnwindSafe<winit::platform_impl::platform::event_loop::thread_event_target_callback::closure$0> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panicking.rs:365
  43: std::panic::catch_unwind<std::panic::AssertUnwindSafe<winit::platform_impl::platform::event_loop::thread_event_target_callback::closure$0>,isize>
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\std\src\panic.rs:434
  44: winit::platform_impl::platform::event_loop::runner::EventLoopRunner<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::catch_unwind<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll>,isize,winit::platform_impl::platform::event_loop::thr
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop\runner.rs:152
  45: winit::platform_impl::platform::event_loop::thread_event_target_callback<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop.rs:2221
  46: DefSubclassProc
  47: DefSubclassProc
  48: CallWindowProcW
  49: DispatchMessageW
  50: SendMessageTimeoutW
  51: KiUserCallbackDispatcher
  52: NtUserDispatchMessage
  53: DispatchMessageW
  54: winit::platform_impl::platform::event_loop::EventLoop<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::run_return<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll>,ruffle_desktop::impl$0::run::closure$0>
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop.rs:221
  55: winit::platform_impl::platform::event_loop::EventLoop<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::run<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll>,ruffle_desktop::impl$0::run::closure$0>
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\platform_impl\windows\event_loop.rs:191
  56: winit::event_loop::EventLoop<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll> >::run<enum$<ruffle_desktop::custom_event::RuffleEvent, TaskPoll>,ruffle_desktop::impl$0::run::closure$0>
             at C:\Users\Michael Welsh\.cargo\registry\src\github.com-1ecc6299db9ec823\winit-0.25.0\src\event_loop.rs:154
  57: ruffle_desktop::App::run
             at D:\projects\ruffle\desktop\src\main.rs:322
  58: ruffle_desktop::main::closure$0
             at D:\projects\ruffle\desktop\src\main.rs:602
  59: enum$<core::result::Result<ruffle_desktop::App,alloc::boxed::Box<dyn$<std::error::Error>,alloc::alloc::Global> > >::map<ruffle_desktop::App,alloc::boxed::Box<dyn$<std::error::Error>,alloc::alloc::Global>,tuple$<>,ruffle_desktop::main::closure$0>
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\core\src\result.rs:748
  60: ruffle_desktop::main
             at D:\projects\ruffle\desktop\src\main.rs:602
  61: core::ops::function::FnOnce::call_once<enum$<core::result::Result<tuple$<>,alloc::boxed::Box<dyn$<std::error::Error>,alloc::alloc::Global> >, 1, 18446744073709551615, Err> (*)(),tuple$<> >
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
The program '[21316] ruffle_desktop.exe' has exited with code 0 (0x0).

Gated behind the "vp6" feature, enabled by default.
Utilizing a heavily stripped-down version of the NihAV project,
retaining only the VP6 decoder, relicensed under MIT.
Including VP6WithAlpha decoding, proper FrameDependency reporting,
and cropping the unwanted encoded pixels on the right/bottom manually.
@torokati44
Copy link
Member Author

torokati44 commented Oct 4, 2021

Thanks a lot for checking, I found the cause of the panic and fixed it.
I simply must have forgotten to test with VP6A videos (it panicked on z0r.de/3926 as well) after I switched from hard-coding test video sizes to parsing the header to get the actual size.

I didn't skip the first 24 bits encoding the offset to the alpha frame, which precede the proper VP6 header. The decoder itself does this on its own, which is why it worked with fixed sizes.

Copy link
Member

@Herschel Herschel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@QualityLow
Copy link

@torokati44 Fantastic work, thank you so much! Being able to view embedded videos in flash loops does a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants