Project PIRI Initial Proposal
Warning
Potential Project of USC CSCI 576 Multimedia Systems Design
Proof of Concept for a High-Performance, Fully-Fledged, and Universal Web Player using WebAssembly
Background
The web keeps eating software, and Atwood's Law has never felt more true. Google Workspace swapped desktop apps like MS Word, PowerPoint, and Excel for Google Docs, Slides, and Sheets. Figma went further and replaced heavy design apps like Adobe XD and Sketch. These web apps perform about as well as the desktop versions, and nobody has to install or update anything. You open a URL and get the latest, full-featured app.
Video players and editors mostly stayed behind, because video codecs and encodings are hard. WebAssembly (WASM) looked like the way out. It runs at near-native speed on the web and can run lower-level executables online. In 2019, FFMpeg, the all-in-one cross-platform tool for recording, converting, and streaming audio and video, was ported to the web. But WASM took a hit after the Spectre vulnerability, which got shared memory and high-resolution timers disabled at the beginning of 2018. By 2020 a new secure approach was standardized, shared memory came back, and fast web-based video apps were possible again.
So we propose PIRI, a "Google Docs" of video players. A high-performance, fully-fledged, universal web player.
PIRI has three parts.
- A WASM-based video player (for consumers)
- A WASM-based video editor (for prosumers)
- A WASM-based video streaming SDK (for companies)
We plan to grow PIRI into all three, but we want to start with a simple proof-of-concept web player.
Connections to CSCI 576: Multimedia Systems Design
PIRI handles and represents many media types (images, video, audio, graphics, and so on) to make multimedia content. That maps straight onto the course's focus on capturing media and assembling it into multimedia content, here applied in a modern web environment.
Compression and Storage
Building PIRI means understanding generic and specific media compression algorithms, including ITU/ISO standards such as JPEG, MPEG1/2/4, H.261/263/264, HEVC, mp3, AAC, and Dolby Atmos. The course covers compression and storage, and PIRI is a hands-on chance to use these standards to make video play and store efficiently on the web.
Distribution
PIRI is about the distribution of multimedia content over networks to many kinds of clients. That covers wired and wireless network distribution, Quality of Service (QoS), priority queuing, and streaming protocols such as MPEG-DASH, Apple's HLS, and Adobe HDS. The course looks at the same distribution mechanisms and at how hard it is to deliver high-quality multimedia over uneven networks.
Digital Rights Management
PIRI uses WebAssembly's secure execution environment for digital rights management (DRM) through watermarking and encryption. The course discusses protecting and managing distributed multimedia content, and this is that problem in practice.
End-to-End Multimedia Systems Design
PIRI is a distributed end-to-end multimedia system, with content creation, compression, storage, and distribution all in one design. The course sets out to explain the architecture of modern multimedia apps, and PIRI is a practical case study that touches both the theory and the engineering.
Recent Protocols and Technologies
PIRI uses recent protocols like MPEG-DASH, Apple's HLS, and Adobe HDS, and could explore newer things like stereoscopic and holographic displays. That gives students a way to work with the latest in multimedia systems, which the course also covers through examples and guest lectures.
Multimodal Media Analysis
Since PIRI could let people edit and analyze video right in the browser, it touches on multimodal media analysis, another course topic. It is a place to try out multimedia data classification and natural language queries over media.
Commercial Values
Universal Video Playback
Today's players have limits. VLC Media Player needs an install, and IINA only runs on macOS. A universal web player would play video the same way on every platform and codec, so nobody needs a separate player per platform, and it would handle a wide range of file formats, MKV included.
Accessibility and Engagement
If people can watch and edit video right in the browser, more people can use it and they use it more. No special installs means anyone can get to the content, which could raise engagement and satisfaction, especially on video-heavy social media and content distribution platforms.
Video Streaming Infrastructure
PIRI also proposes a different take on streaming infrastructure, an SDK that makes encoding and streaming video files simple. It removes the codec preprocessing step that can degrade video quality, a real pain for the big video services. Streaming and playing video without that preprocessing lets smaller teams compete with the big players, without the path dependency and vendor lock-in of products like AWS Elemental MediaLive. So an SDK for encoding and streaming could save organizations a lot of money. Hooking it into Content Delivery Networks (CDNs) could also make streaming much more efficient, with lower latency and better quality for live streams and HD content.
Security and Sustainability
WebAssembly's secure execution environment helps protect sensitive and copyrighted content. And by cutting down server-side video processing, PIRI could use less energy and leave a smaller carbon footprint.
Delivery Plans
By the end of the semester, we aim to deliver the following.
A Proof of Concept WASM Video Player
It shows FFMpeg playing video on the web, as a first look at a high-performance, fully-fledged web player.
A Proof of Concept WASM Video Trimmer
It shows FFMpeg editing video, to prove that web-based video editing tools can run at near-native speed with WASM.
Where this goes
PIRI wants to fill a real gap with a universal web player and editor built on WebAssembly and FFMpeg. If it fixes what today's playback and editing tools get wrong, people could watch and edit video on any platform without installing anything, and get a good experience doing it.