<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" xml:lang="en-US">
  <title>BigBinary Blog: Web APIs</title>
  <subtitle>Deep technical writing on Ruby on Rails, performance, scaling, monitoring and engineering.</subtitle>
  <link rel="self" type="application/atom+xml" href="https://www.bigbinary.com/blog/categories/web-apis/feed.xml"/>
  <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/categories/web-apis"/>
  <id>https://www.bigbinary.com/blog/categories/web-apis</id>
  <updated>2024-03-17T00:00:00.000Z</updated>
  <author>
    <name>BigBinary</name>
    <uri>https://www.bigbinary.com/</uri>
  </author>
  <icon>https://www.bigbinary.com/favicon.ico</icon>
  <logo>https://www.bigbinary.com/images/bigbinary_square_big_logo.png</logo>
  <rights>Copyright 2024 Big Binary LLC</rights>
  <generator uri="https://nextjs.org">Next.js</generator>
  <entry>
      <title type="text"><![CDATA[Universal video playback with MP4 and Range headers]]></title>
      <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/mp4_transmuxing_and_streaming_support-loom-alternative-part-3"/>
      <link rel="enclosure" type="image/jpeg" href="https://www.bigbinary.com/blog/images/social_preview_images/universal_playback_and_streaming_support.jpg"/>
      <id>https://www.bigbinary.com/blog/mp4_transmuxing_and_streaming_support-loom-alternative-part-3</id>
      <published>2024-03-17T00:00:00.000Z</published>
      <updated>2024-03-17T00:00:00.000Z</updated>
      <author>
        <name>Unnikrishnan KP</name>
        <uri>https://www.bigbinary.com/blog/authors/unni</uri>
      </author>
      <category term="NeetoRecord"/>
      <category term="web-apis"/>
      <media:thumbnail url="https://www.bigbinary.com/blog/images/social_preview_images/universal_playback_and_streaming_support.jpg" width="1200" height="627"/>
      <summary type="text"><![CDATA[How to convert WEBM video generated by the browser efficiently to MP4 and provide streaming support]]></summary>
      <content type="html" xml:base="https://www.bigbinary.com/blog/mp4_transmuxing_and_streaming_support-loom-alternative-part-3"><![CDATA[<p>This is part 3 of our blog on how we are building
<a href="https://www.neeto.com/neetorecord">NeetoRecord</a>, a Loom alternative. Here are
<a href="https://www.bigbinary.com/blog/build-web-based-screen-recorder-loom-alternative-part-1">part 1</a>
and
<a href="https://www.bigbinary.com/blog/persistant-storage-for-recordings-in-s3-loom-alternative-part-2">part 2</a>.</p>
<p>In Part 1 of our blog, we uploaded the recording from the browser to S3 in
small parts and stitched them together to get the final WEBM video file. We
could use this WEBM file to share our recording with our audience, but it has a
few drawbacks:</p>
<ol>
<li>
<p>WEBM is not universally supported. Though most modern browsers support WEBM,
a few browsers, especially devices in the Apple ecosystem, do not play WEBM
reliably.</p>
</li>
<li>
<p>Metadata for timestamps and duration are not present in WEBM videos. So,
these videos are not &quot;seekable.&quot; It means these videos do not show the video
length, and we cannot move back and forth using the seek bar. The video
starts playing back from the beginning when the user tries to push the seek
bar.</p>
</li>
</ol>
<p>Hence, we needed to convert the WEBM videos to a universally supported format to
solve the above problems. We chose MP4.</p>
<h2>MP4</h2>
<p>MP4 is a widely used multimedia file storage format for video storage and
streaming. It is an international standard that works with a vast range of
devices. MP4 refers to the digital container file that acts as a wrapper around
the video, not the video itself. The video content within MP4 files is encoded
with MPEG-4, a common encoding standard.</p>
<p>We chose MP4 because:</p>
<ol>
<li>MP4 works with
<a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video">HTML5 video player</a>.</li>
<li>Supports multiple streaming protocols.</li>
<li>Comprehensive support for user devices and browsers.</li>
</ol>
<h2>WEBM to MP4 conversion</h2>
<h3>AWS MediaConvert service</h3>
<p>Since our WEBM files were in an S3 bucket, our first idea was to use an AWS
service to do the WEBM to MP4 conversion. We configured
<a href="https://aws.amazon.com/mediaconvert/">AWS Elemental MediaConvert service</a> and
connected it to our WEBM bucket. When a user uploads a WEBM file to the bucket,
MediaConvert picks it up, converts it to MP4, and uploads it to a new bucket.</p>
<p>MediaConvert worked as expected, but we had to find another solution because:</p>
<ol>
<li>Cost - we found it too expensive for our use case.</li>
<li>Performance - It took a long time to do the conversion. While the smaller
recordings took about 20-30s, large ones took minutes. The time taken grew
linearly with the size of the WEBM file.</li>
</ol>
<h3>Manual transcoding using FFMPEG using AWS Lambda</h3>
<p>Converting WEBM to MP4 involves transcoding. Transcoding is the process of
changing the audio/video codecs in a container file. Codecs are algorithms used
to encode and decode digital media data. Converting to MP4 would mean using
codecs that are part of the MPEG-4 family. Eg: H.264 for video and AAC for
audio. <a href="https://ffmpeg.org">FFMPEG</a> is a popular open source tool that can be
used for transcoding WEBM to MP4.</p>
<pre><code>ffmpeg -i input.webm -c:v libx264 -c:a aac  output.mp4
</code></pre>
<ul>
<li><code>-c:v libx264</code> sets the video codec to libx264, which is a widely supported
H.264 codec.</li>
<li><code>-c:a aac</code> sets the audio codec to AAC, which is a commonly used audio codec.</li>
</ul>
<p>We could run FFMPEG on our web server and run the transcoding process. But that
will not be easy to scale. So, we decided to use a serverless solution that
would automatically scale. Since our input files were on AWS S3, AWS Lambda was
the obvious choice.</p>
<p>We installed FFMPEG on <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> using a
<a href="https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html">Layer</a> as
described in this
<a href="https://aws.amazon.com/blogs/media/processing-user-generated-content-using-aws-lambda-and-ffmpeg">post</a>.</p>
<p>We configured our input S3 bucket (one to which WEBM was uploaded) to trigger
Lambda whenever a new file was uploaded. FFMPEG would then transcode WEBM to MP4
and store the output in another S3 bucket.</p>
<p>This worked as expected. But performance was still a problem. Time taken was
proportional to the input file size and took longer than we found acceptable.</p>
<h3>Transmuxing instead of transcoding</h3>
<p>Transmuxing or stream copy is a fast process that doesn't involve re-encoding
but instead directly copies the existing audio and video streams into a new
container format. This approach works well when the codecs used in the input
file (WebM) are compatible with the output container format (MP4).</p>
<p>Popular browsers like Chrome, Brave, Safari etc. use the H264 codec for video
encoding. This is compatible with MP4. So transmuxing works flawlessly. But
Firefox uses the VP8 or VP9 codec, which is incompatible with MP4. Since we were
planning to build a Chrome extension for NeetoRecor,d we only needed to worry
about Chrome and we could ignore Firefox users for now.</p>
<pre><code>ffmpeg -i input.webm -c:v copy -c:a copy output.mp4
</code></pre>
<p>We modified the ffmpeg command as shown above. It now uses the <code>-c:v copy</code> and
<code>-c:a copy</code> options, which copies the video and audio from the input file to the
output file without re-encoding. MP4 conversion now become extremely fast, and
the time taken did not increase significantly with the size of the input file.</p>
<h2>Streaming</h2>
<p>Now that we successfully generated MP4 files, it was time to think of delivering
the file efficiently to the client (browser) for playback. We had two problems
to solve:</p>
<ol>
<li>
<p>S3 is a storage service. It is not suitable for content delivery.</p>
<ul>
<li>Relatively high data transfer costs.</li>
<li>Storage is in one geographical region, resulting in slower delivery over
the network.</li>
</ul>
</li>
<li>
<p>Video files are large in size. Downloading the entire file and then playing
it back is not efficient in terms of speed and data transfer. We needed to
find a way to allow streaming of the files. ie. deliver chunks of data as and
when it was needed by the client.</p>
</li>
</ol>
<h3>Cloudfront as CDN</h3>
<p>CloudFront is a content delivery network (CDN) service provided by AWS. It can
be used as a CDN for S3, and this combination is a common architecture for
distributing content globally with low latency and high transfer speeds.</p>
<p>We created a CloudFront distribution, which is connected to our MP4 bucket. Once
the distribution is deployed, we can access the MP4 files using the CloudFront
domain name. When users request content through CloudFront, CloudFront checks
its cache for the requested content. If the content is in the cache and is still
valid (based on cache-control headers), CloudFront serves the content directly
from its edge locations, reducing latency. If the content is not in the cache or
is expired, CloudFront retrieves the content from the S3 bucket, caches it, and
serves it to the user. This helps reduce the load on our S3 bucket and improves
the performance of content delivery.</p>
<h3>The HTTP Range request header</h3>
<p>HTTP Range requests allow clients to request specific portions of a file from a
server. This feature enables users to stream or download only parts of the file
they need, reducing bandwidth usage and improving user experience. At first, the
client could request the range for the beginning of the video file, and then, as the
playback proceeded, request subsequent parts. If the user moves back and forth
the video using the seek bar, corresponding ranges can be requested.</p>
<pre><code>GET /example.mp4 HTTP/1.1
Host: example.com
Range: bytes=5000-9999
</code></pre>
<p><code>Range: bytes=5000-9999</code> is the Range header indicating the specific bytes the
client wants to retrieve. In this case, the client requests bytes 5000 to 9999
of the MP4 file. The numbering starts from zero, so byte 5000 means the 5001st
byte in the file.</p>
<p>Server responds with a <code>206</code> response (Partial content) with the request
sequence of bytes in the body. If the server does not support range requests,
then it responds with a <code>200</code> along with the full content.</p>
<h4>Checking if the server supports Range requests</h4>
<p>We can perform a check by issuing a HEAD request to the server to see if the
server supports Range requests.</p>
<pre><code>curl -I http://abc.com/1.mp4
</code></pre>
<p>If range requests are supported, then the server responds with a
<code>Accept-Ranges: bytes</code> header.</p>
<pre><code>HTTP/1.1 200 OK
…
Accept-Ranges: bytes
Content-Length: 146515
</code></pre>
<p>We did the test on our S3 bucket directly first, and then through Cloudfront.
Both
<a href="https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html">S3</a>
and
<a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html">Cloudfront</a>
supports Range request headers.</p>
<p>MP4, as mentioned above, supports streaming. It has metadata to help the server
deliver it in chunks as requested. The HTML5 video player supports progressive
download automatically by making use of HTTP Range headers.</p>
<p>So now we have our video in a file format that supports streaming (MP4), web
server that supports Range headers (S3 and Cloudfront) and a client that uses
Range headers for progressive download - all the ingredients needed to support
streaming.</p>
]]></content>
    </entry>
  <entry>
      <title type="text"><![CDATA[Building a web-based screen recorder]]></title>
      <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/build-web-based-screen-recorder-loom-alternative-part-1"/>
      <link rel="enclosure" type="image/jpeg" href="https://www.bigbinary.com/blog/images/social_preview_images/building_web_based_screen_recorder.jpg"/>
      <id>https://www.bigbinary.com/blog/build-web-based-screen-recorder-loom-alternative-part-1</id>
      <published>2024-03-15T00:00:00.000Z</published>
      <updated>2024-03-15T00:00:00.000Z</updated>
      <author>
        <name>Unnikrishnan KP</name>
        <uri>https://www.bigbinary.com/blog/authors/unni</uri>
      </author>
      <category term="NeetoRecord"/>
      <category term="web-apis"/>
      <media:thumbnail url="https://www.bigbinary.com/blog/images/social_preview_images/building_web_based_screen_recorder.jpg" width="1200" height="627"/>
      <summary type="text"><![CDATA[Building a web based screen recorder using browser's MediaStream Recording API]]></summary>
      <content type="html" xml:base="https://www.bigbinary.com/blog/build-web-based-screen-recorder-loom-alternative-part-1"><![CDATA[<p>This is part 1 of our blog on how we are building
<a href="https://www.neeto.com/neetorecord">NeetoRecord</a>, a Loom alternative. Here are
<a href="https://www.bigbinary.com/blog/persistant-storage-for-recordings-in-s3-loom-alternative-part-2">part 2</a>
and
<a href="https://www.bigbinary.com/blog/mp4_transmuxing_and_streaming_support-loom-alternative-part-3">part 3</a>.</p>
<p>At <a href="https://neeto.com">neeto</a>, the product team, developers, and the UI team
often communicate using short videos and screen recordings. We relied on popular
solutions like Loom and Bubbles. But they allowed only a small number of
recordings in their free versions, and soon, they presented us with the upgraded
screens - upgrades were quite expensive for our team due to our team size and
the number of recordings we made daily.</p>
<p>So, we decided to build our own solution. We found the browser's
MediaStream Recording API.</p>
<h2>MediaStream Recording API</h2>
<p>The MediaStream Recording API, sometimes called the MediaRecorder API, is
closely affiliated with the
<a href="https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API">Media Capture and Streams API</a>
and the
<a href="https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API">WebRTC API</a>. The
MediaStream Recording API enables capturing the data generated by a MediaStream
or HTMLMediaElement. Captured video data is in WebM format. We can play it back
later using the
<a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement">HTMLVideoElement</a>
on any video player that supports WebM playback.</p>
<p>We will build a basic recorder that records the screen, audio from the
microphone, and then plays it back. We will first look at different fragments of
code for recording the screen, recording audio, playing back in the browser and
then downloading the video file. At the end, we will combine them into a
fully working web-based screen recorder program.</p>
<h3>Record the screen</h3>
<pre><code class="language-javascript">let mediaRecorder;
let recordedChunks = [];

const stream = await navigator.mediaDevices.getDisplayMedia({
  video: true,
});

mediaRecorder = new MediaRecorder(stream);

mediaRecorder.ondataavailable = event =&gt; {
  recordedChunks.push(event.data);
};
</code></pre>
<p><code>getDisplayMedia()</code> is provided by the
<a href="https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API">WebRTC</a> (Web
Real-Time Communication) API. It captures the contents of the user's screen or
specific application windows. <code>getDisplayMedia()</code> method prompts the user to
select and grant permission to capture the contents of a display or portion
thereof (such as a window) as a MediaStream.</p>
<p>There is a similar method called <code>getUserMedia()</code>. It is typically used for
applications like video conferencing and live streaming. When you call
<code>getUserMedia()</code>, the browser prompts the user for permission to access their
camera and microphone.</p>
<p>When there is recorded data available, <code>ondataavailable()</code> callback is
triggered. We could process the data in this callback. In our case, we collect
the data by appending it to an array named <code>recordedChunks</code>.</p>
<h3>Record audio</h3>
<pre><code class="language-javascript">let audioStream = await window.navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true },
});
</code></pre>
<p>To capture audio, we use <code>getUserMedia()</code>. It is typically used for applications
like video conferencing and live streaming. When you call <code>getUserMedia()</code>, the
browser prompts the user for permission to access their camera and microphone.
But we want to capture only the audio, so we pass the <code>audio</code> parameter.</p>
<p>The <code>audio</code> key accepts a set of parameters that would let us control the
quality and properties of the captured audio stream. In our example, we have
enabled <code>echoCancellation</code> and <code>noiseSuppression</code> - two good features that would
enhance the quality of our screen recordings. The complete list of audio options
is available
<a href="https://www.bigbinary.com/blog/%5Bhttps://developer.mozilla.org/en-US/docs/Web/API/MediaTrackSettings#instance_properties_of_audio_tracks">here</a>.</p>
<p>The audio stream could be composed of multiple audio tracks - the microphone,
system sounds, etc. We will add these tracks to the video stream we had
previously set up using <code>getDisplayMedia()</code>.</p>
<pre><code class="language-javascript">audioStream.getAudioTracks().forEach(audioTrack =&gt; stream.addTrack(audioTrack));
</code></pre>
<h3>Playback the recording</h3>
<p>We now have an array named <code>recordedChunks</code>, which contains sequential chunks of
the recorded data. Video players need video data as a
<a href="https://developer.mozilla.org/en-US/docs/Web/API/Blob">Blob</a>. A blob is a
file-like object of immutable, raw binary/text data. We must convert our
<code>recordedChunks</code> array into a <code>Blob</code> to be played back or written into a file.</p>
<p>To construct a Blob from other non-blob objects and data, we can use the
<code>Blob()</code> constructor.</p>
<pre><code class="language-javascript">const blob = new Blob(recordedChunks, {
  type: &quot;video/webm&quot;,
});
</code></pre>
<p>Suppose we have an
<a href="https://www.w3schools.com/tags/tag_video.asp">HTML video tag</a> in our page.</p>
<pre><code class="language-html">&lt;video id=&quot;recordedVideo&quot; controls&gt;&lt;/video&gt;
</code></pre>
<p>When the video recording is stopped, we could create an Object URL for our
recording <code>blob</code> and attach it to the video player.</p>
<pre><code class="language-javascript">mediaRecorder.onstop = () =&gt; {
  let recordedVideo = document.getElementById(&quot;recordedVideo&quot;);
  recordedVideo.src = URL.createObjectURL(blob);
};
</code></pre>
<p>We can now play the recording on the HTML video player.</p>
<p>Similarly, we can create a download link using the Object URL for the recording
<code>blob</code>.</p>
<pre><code class="language-javascript">let a = document.createElement(&quot;a&quot;);
let url = URL.createObjectURL(blob);
a.href = url;
</code></pre>
<p>We can now download and play the recording locally on any video player
supporting WebM playback.</p>
<h2>Putting it all together</h2>
<p>We have glued together the code fragments discussed above and created a
<a href="https://www.bigbinary.com/blog/neeto_record/basic_screen_recorder.html">demo</a> for a basic web-based
screen recorder.</p>
<p>You may view the source code
<a href="https://gist.github.com/unnitallman/6a054300f8bba645d42fd04008ea6ff1">here</a>.</p>
<h2>Next steps</h2>
<p>Now that we have a basic screen recorder in place, we have to consider the
following:</p>
<ol>
<li>Persistent storage.</li>
<li>Chunked uploading.</li>
<li>CDN support.</li>
<li>Playback with support for streaming.</li>
</ol>
<p>We will cover these topics in the next set of blogs. Stay tuned.</p>
]]></content>
    </entry>
</feed>