The WebCodecs API Explained: Frame-Level Video Access, and Why It's a Different Job Than MSE or ffmpeg.wasm

Two other posts on this blog cover browser APIs that make streaming video possible: Media Source Extensions (MSE), which hands finished, container-wrapped segments to a <video> element, and ffmpeg.wasm, which repackages downloaded segments into an MP4 without decoding a single frame. There's a third API that sits below both of them, and it gets confused with each often enough to be worth untangling on its own: WebCodecs. It doesn't play video and it doesn't touch containers — it does exactly one thing, which is turn compressed video into raw frames and back, using the browser's own hardware codecs, directly from JavaScript.

What WebCodecs actually exposes

WebCodecs is a W3C API, first shipped in Chromium browsers in 2021 and later added to Safari and Firefox, that gives JavaScript direct access to the same encoder and decoder implementations the browser already uses internally for <video> playback and WebRTC calls. Its core interfaces work in encoded/decoded pairs:

• VideoDecoder — takes an EncodedVideoChunk (a single compressed frame's worth of bytes, tagged as a keyframe or delta frame) and produces a VideoFrame — a raw, decoded image the page can draw to a <canvas>, upload to WebGL, or pass to another API.
• VideoEncoder — the reverse: takes a raw VideoFrame and produces an EncodedVideoChunk, using a codec and bitrate the caller configures.
• AudioDecoder / AudioEncoder — the same pair for audio, working with AudioData instead of VideoFrame.

Every call is asynchronous and callback-driven — frames come back through an output callback as the underlying hardware or software codec finishes each one, not in one blocking batch. A page can check whether a given codec configuration is even usable before committing to it, via VideoDecoder.isConfigSupported(), which is the WebCodecs equivalent of the MediaSource.isTypeSupported() check described in the MSE post.

The part that trips people up: no containers, no networking

WebCodecs operates entirely below the container format. It has no concept of .m3u8, .ts, .mp4, or fMP4/CMAF — it only understands individual encoded chunks and raw frames, and it has no idea how those chunks got assembled into a file or where the file came from. That means a page working with real video has to do two jobs WebCodecs itself doesn't offer:

• Fetching the segments or file over the network — the same bridge-first fetchBytes()pattern this site's extension bridge uses for everything else.
• Demuxing — parsing the container to pull out individual encoded chunks and their timestamps, since a .ts or .mp4 file is a container format wrapping compressed frames, not a raw sequence of them. WebCodecs expects the caller to have already done this, typically with a separate library such as mp4box.js for fMP4 or a hand-rolled MPEG-TS parser.

This is the opposite division of labor from MSE. A SourceBuffer accepts a whole container-wrapped chunk and does the demuxing and decoding internally, as a black box, and hands the result straight to the video element — the page never sees an individual frame. WebCodecs demands the frames be extracted first, but then gives the page the raw decoded image data to do anything with, not just paint it to a video element. hls.js and dash.js are built on MSE for exactly this reason: they need finished playback, not frame access. Tools built on WebCodecs — browser-based video editors, cloud-gaming clients, real-time filters over WebRTC video — need the opposite trade.

Where WebCodecs shows up in practice

The common thread across real WebCodecs use cases is that something needs to touch the pixels or re-encode them, not just play them back:

• WebRTC Insertable Streams pipe incoming call video through a VideoFrame so a page can apply a background blur or filter mid-call before the frame reaches the screen.
• In-browser video editors and exporters decode a source clip frame by frame, apply cuts or overlays, and re-encode the result — work that a container-copy tool like this site's converter is deliberately not built to do.
• Cloud gaming and low-latency remote rendering use VideoDecoderto turn a server's encoded frame stream into pixels with as few intermediate copies and as little added latency as possible.

Why this matters for a planned /trim feature

The ffmpeg.wasm post already flagged the relevant limit: /convert's FFmpeg command runs in stream-copy mode (-c copy), repackaging the existing compressed bitstream into a new container without decoding a single frame. That's fast and lossless, but it means any cut has to land on a keyframe boundary — FFmpeg's -ss/-to flags in copy mode can only start or end where a full, independently decodable frame already exists, not at an arbitrary timestamp in the middle of a GOP.

A trimmer built on WebCodecs instead could decode frames on either side of the requested cut points, re-encode only the handful of frames needed to produce a clean start and end, and stream-copy everything in between — a hybrid that gets closer to frame-accurate trimming than a pure copy-only tool without paying the cost of re-encoding an entire clip. That's meaningfully more implementation work than one FFmpeg command: it means writing or importing a demuxer for whatever container the segments arrive in, driving VideoDecoder/VideoEncoder directly, and re-muxing the result by hand, since WebCodecs still won't do any of that packaging for you. It's why a first version of /trim is more likely to reuse the same -ss/-to -c copy approach described in the ffmpeg.wasm post — snapping to the nearest keyframe — before a frame-accurate WebCodecs path would be worth the added complexity.

Browser support and its practical limits

WebCodecs is available in every current Chromium-based browser and in Safari, and Firefox has been rolling out support as well, but parity isn't total: which specific codecs a VideoDecoder or VideoEncoder can actually open still depends on what the underlying platform ships, exactly like MediaSource.isTypeSupported() for MSE. Hardware H.264 decode is close to universal at this point, but hardware AV1 or HEVC encode support varies a lot more by device and OS, which is why any real WebCodecs-based tool has to call isConfigSupported() before committing to a codec rather than assuming one is available.

FAQ

Does WebCodecs replace MSE? No. MSE is for assembling a container-wrapped stream into something a <video> element can play; WebCodecs is for working with individual encoded chunks and decoded frames directly. A player just needs playback and should keep using MSE (via hls.js or dash.js); a tool that needs to inspect, filter, or re-encode individual frames needs WebCodecs instead.

Does WebCodecs replace ffmpeg.wasm?Not for this site's current use case. /convertonly needs to repackage segments into a new container without touching the compressed data, which is exactly what FFmpeg's stream-copy mode does in one call. WebCodecs would only help for jobs that need actual frame-level access — like a frame-accurate trim — not a pure container swap.

Can WebCodecs read an .m3u8 playlist or an .mp4 file directly? No — it has no built-in demuxer. Something else has to parse the container first (the way this site's own m3u8 parser already parses playlists) and hand WebCodecs individual encoded chunks; WebCodecs only takes over once the container has already been unwrapped.