In Part 1 of our blog, we uploaded the recording from the browser to S3 insmall parts and stitched them together to get the final WEBM video file. Wecould use this WEBM file to share our recording with our audience, but it has afew drawbacks:

1. WEBM is not universally supported. Though most modern browsers support WEBM,a few browsers, especially devices in the Apple ecosystem, do not play WEBMreliably.

2. Metadata for timestamps and duration are not present in WEBM videos. So,these videos are not "seekable." It means these videos do not show the videolength, and we cannot move back and forth using the seek bar. The videostarts playing back from the beginning when the user tries to push the seekbar.

Hence, we needed to convert the WEBM videos to a universally supported format tosolve the above problems. We chose MP4.

MP4

MP4 is a widely used multimedia file storage format for video storage andstreaming. It is an international standard that works with a vast range ofdevices. MP4 refers to the digital container file that acts as a wrapper aroundthe video, not the video itself. The video content within MP4 files is encodedwith MPEG-4, a common encoding standard.

We chose MP4 because:

1. MP4 works withHTML5 video player.
2. Supports multiple streaming protocols.
3. Comprehensive support for user devices and browsers.

WEBM to MP4 conversion

AWS MediaConvert service

Since our WEBM files were in an S3 bucket, our first idea was to use an AWSservice to do the WEBM to MP4 conversion. We configuredAWS Elemental MediaConvert service andconnected it to our WEBM bucket. When a user uploads a WEBM file to the bucket,MediaConvert picks it up, converts it to MP4, and uploads it to a new bucket.

MediaConvert worked as expected, but we had to find another solution because:

1. Cost - we found it too expensive for our use case.
2. Performance - It took a long time to do the conversion. While the smallerrecordings took about 20-30s, large ones took minutes. The time taken grewlinearly with the size of the WEBM file.

Manual transcoding using FFMPEG using AWS Lambda

Converting WEBM to MP4 involves transcoding. Transcoding is the process ofchanging the audio/video codecs in a container file. Codecs are algorithms usedto encode and decode digital media data. Converting to MP4 would mean usingcodecs that are part of the MPEG-4 family. Eg: H.264 for video and AAC foraudio. FFMPEG is a popular open source tool that can beused for transcoding WEBM to MP4.

ffmpeg -i input.webm -c:v libx264 -c:a aac  output.mp4

• -c:v libx264 sets the video codec to libx264, which is a widely supportedH.264 codec.
• -c:a aac sets the audio codec to AAC, which is a commonly used audio codec.

We could run FFMPEG on our web server and run the transcoding process. But thatwill not be easy to scale. So, we decided to use a serverless solution thatwould automatically scale. Since our input files were on AWS S3, AWS Lambda wasthe obvious choice.

We installed FFMPEG on AWS Lambda using aLayer asdescribed in thispost.

We configured our input S3 bucket (one to which WEBM was uploaded) to triggerLambda whenever a new file was uploaded. FFMPEG would then transcode WEBM to MP4and store the output in another S3 bucket.

This worked as expected. But performance was still a problem. Time taken wasproportional to the input file size and took longer than we found acceptable.

Transmuxing instead of transcoding

Transmuxing or stream copy is a fast process that doesn't involve re-encodingbut instead directly copies the existing audio and video streams into a newcontainer format. This approach works well when the codecs used in the inputfile (WebM) are compatible with the output container format (MP4).

Popular browsers like Chrome, Brave, Safari etc. use the H264 codec for videoencoding. This is compatible with MP4. So transmuxing works flawlessly. ButFirefox uses the VP8 or VP9 codec, which is incompatible with MP4. Since we wereplanning to build a Chrome extension for NeetoRecor,d we only needed to worryabout Chrome and we could ignore Firefox users for now.

ffmpeg -i input.webm -c:v copy -c:a copy output.mp4

We modified the ffmpeg command as shown above. It now uses the -c:v copy and-c:a copy options, which copies the video and audio from the input file to theoutput file without re-encoding. MP4 conversion now become extremely fast, andthe time taken did not increase significantly with the size of the input file.

Streaming

Now that we successfully generated MP4 files, it was time to think of deliveringthe file efficiently to the client (browser) for playback. We had two problemsto solve:

1. S3 is a storage service. It is not suitable for content delivery.

   • Relatively high data transfer costs.
   • Storage is in one geographical region, resulting in slower delivery overthe network.

2. Video files are large in size. Downloading the entire file and then playingit back is not efficient in terms of speed and data transfer. We needed tofind a way to allow streaming of the files. ie. deliver chunks of data as andwhen it was needed by the client.

Cloudfront as CDN

CloudFront is a content delivery network (CDN) service provided by AWS. It canbe used as a CDN for S3, and this combination is a common architecture fordistributing content globally with low latency and high transfer speeds.

We created a CloudFront distribution, which is connected to our MP4 bucket. Oncethe distribution is deployed, we can access the MP4 files using the CloudFrontdomain name. When users request content through CloudFront, CloudFront checksits cache for the requested content. If the content is in the cache and is stillvalid (based on cache-control headers), CloudFront serves the content directlyfrom its edge locations, reducing latency. If the content is not in the cache oris expired, CloudFront retrieves the content from the S3 bucket, caches it, andserves it to the user. This helps reduce the load on our S3 bucket and improvesthe performance of content delivery.

The HTTP Range request header

HTTP Range requests allow clients to request specific portions of a file from aserver. This feature enables users to stream or download only parts of the filethey need, reducing bandwidth usage and improving user experience. At first, theclient could request the range for the beginning of the video file, and then, as theplayback proceeded, request subsequent parts. If the user moves back and forththe video using the seek bar, corresponding ranges can be requested.

GET /example.mp4 HTTP/1.1Host: example.comRange: bytes=5000-9999

Range: bytes=5000-9999 is the Range header indicating the specific bytes theclient wants to retrieve. In this case, the client requests bytes 5000 to 9999of the MP4 file. The numbering starts from zero, so byte 5000 means the 5001stbyte in the file.

Server responds with a 206 response (Partial content) with the requestsequence of bytes in the body. If the server does not support range requests,then it responds with a 200 along with the full content.

Checking if the server supports Range requests

We can perform a check by issuing a HEAD request to the server to see if theserver supports Range requests.

curl -I http://abc.com/1.mp4

If range requests are supported, then the server responds with aAccept-Ranges: bytes header.

HTTP/1.1 200 OKAccept-Ranges: bytesContent-Length: 146515

We did the test on our S3 bucket directly first, and then through Cloudfront.BothS3andCloudfrontsupports Range request headers.

MP4, as mentioned above, supports streaming. It has metadata to help the serverdeliver it in chunks as requested. The HTML5 video player supports progressivedownload automatically by making use of HTTP Range headers.

So now we have our video in a file format that supports streaming (MP4), webserver that supports Range headers (S3 and Cloudfront) and a client that usesRange headers for progressive download - all the ingredients needed to supportstreaming.

In the previous blog, we learned how to use the Browser APIs to record the screenand generate a WEBM file. We now need to upload this file to persistent storageto have a URL to share our recording with our audience.

Uploading a large file all at once is time-consuming and prone to failure due tonetwork errors. The recording is generated in parts, each part pushed to anarray and joined together. So it would be ideal if we could upload these smallerparts as and when they are generated, and then join them together in the backend oncethe recording is completed. AWS'sSimple Storage Service (S3) made a perfect fit asit provides cheap persistent storage, along withMultipart Uploadsfeature.

S3 Multipart Uploads allow us to upload large objects in parts. Rather thanuploading the entire object in a single operation, multipart uploads break itdown into smaller parts, each ranging from 5 MB to 5 GB. Once uploaded, theseparts are aggregated to form the complete object.

Initialization

The process begins with an initiation request to S3, where a unique upload ID isgenerated. This upload ID is used to identify and manage the individual parts ofthe upload.

s3 = Aws::S3::Client.newresp = s3.create_multipart_upload({  bucket: bucket_name,  key: object_key})upload_id = resp.upload_id

Upload Parts

Once the upload is initiated, we can upload the parts to S3 independently. Eachpart is associated with a sequence number and an ETag (Entity Tag), a checksumof the part's data.

Note that the minimum content size for a part is 5MB (There is no minimum size limiton the last part of your multipart upload). So we store the recording chunks inlocal storage until they are bigger than 5MB. Once we have a part greater than5MB, we upload it to S3.

part_number = 1content = recordedChunksresp = s3.upload_part({  body: content,  bucket: bucket_name,  key: object_key,  upload_id: upload_id,  part_number: part_number})puts "ETag for Part #{part_number}: #{resp.etag}"

Completion

Once all parts are uploaded, a complete multipart upload request is sent to S3,specifying the upload ID and the list of uploaded parts along with their ETagsand sequence numbers. S3 then assembles the parts into a single object andfinalizes the upload.

completed_parts = [  { part_number: 1, etag: 'etag_of_part_1' },  { part_number: 2, etag: 'etag_of_part_2' },  ...  { part_number: N, etag: 'etag_of_part_N' },]resp = s3.complete_multipart_upload({  bucket: bucket_name,  key: object_key,  upload_id: upload_id,  multipart_upload: {    parts: completed_parts  }})

Aborting and Cancelling

At any point during the multipart upload process, you can abort or cancel theupload, which deletes any uploaded parts associated with the upload ID.

s3.abort_multipart_upload({  bucket: bucket_name,  key: object_key,  upload_id: upload_id})

The uploaded file will finally be available at s3://bucket_name/object_id

S3 Multipart Uploads offers us several advantages:

Fault tolerance

We can resume uploads from where they left off in case of network failures orinterruptions. Also, uploading large objects in smaller parts reduces thelikelihood of timeouts and connection failures, especially in high-latency orunreliable network environments.

Upload speed optimization

With multipart uploads, you can parallelize the process by uploading multipleparts concurrently, optimizing transfer speeds and reducing overall upload time.

At neeto, the product team, developers, and the UI teamoften communicate using short videos and screen recordings. We relied on popularsolutions like Loom and Bubbles. But they allowed only a small number ofrecordings in their free versions, and soon, they presented us with the upgradedscreens - upgrades were quite expensive for our team due to our team size andthe number of recordings we made daily.

So, we decided to build our own solution. We found the browser'sMediaStream Recording API.

MediaStream Recording API

The MediaStream Recording API, sometimes called the MediaRecorder API, isclosely affiliated with theMedia Capture and Streams APIand theWebRTC API. TheMediaStream Recording API enables capturing the data generated by a MediaStreamor HTMLMediaElement. Captured video data is in WebM format. We can play it backlater using theHTMLVideoElementon any video player that supports WebM playback.

We will build a basic recorder that records the screen, audio from themicrophone, and then plays it back. We will first look at different fragments ofcode for recording the screen, recording audio, playing back in the browser andthen downloading the video file. At the end, we will combine them into afully working web-based screen recorder program.

Record the screen

let mediaRecorder;let recordedChunks = [];const stream = await navigator.mediaDevices.getDisplayMedia({  video: true,});mediaRecorder = new MediaRecorder(stream);mediaRecorder.ondataavailable = event => {  recordedChunks.push(event.data);};

getDisplayMedia() is provided by theWebRTC (WebReal-Time Communication) API. It captures the contents of the user's screen orspecific application windows. getDisplayMedia() method prompts the user toselect and grant permission to capture the contents of a display or portionthereof (such as a window) as a MediaStream.

There is a similar method called getUserMedia(). It is typically used forapplications like video conferencing and live streaming. When you callgetUserMedia(), the browser prompts the user for permission to access theircamera and microphone.

When there is recorded data available, ondataavailable() callback istriggered. We could process the data in this callback. In our case, we collectthe data by appending it to an array named recordedChunks.

Record audio

let audioStream = await window.navigator.mediaDevices.getUserMedia({  audio: { echoCancellation: true, noiseSuppression: true },});

To capture audio, we use getUserMedia(). It is typically used for applicationslike video conferencing and live streaming. When you call getUserMedia(), thebrowser prompts the user for permission to access their camera and microphone.But we want to capture only the audio, so we pass the audio parameter.

The audio key accepts a set of parameters that would let us control thequality and properties of the captured audio stream. In our example, we haveenabled echoCancellation and noiseSuppression - two good features that wouldenhance the quality of our screen recordings. The complete list of audio optionsis availablehere.

The audio stream could be composed of multiple audio tracks - the microphone,system sounds, etc. We will add these tracks to the video stream we hadpreviously set up using getDisplayMedia().

audioStream.getAudioTracks().forEach(audioTrack => stream.addTrack(audioTrack));

Playback the recording

We now have an array named recordedChunks, which contains sequential chunks ofthe recorded data. Video players need video data as aBlob. A blob is afile-like object of immutable, raw binary/text data. We must convert ourrecordedChunks array into a Blob to be played back or written into a file.

To construct a Blob from other non-blob objects and data, we can use theBlob() constructor.

const blob = new Blob(recordedChunks, {  type: "video/webm",});

Suppose we have anHTML video tag in our page.

<video id="recordedVideo" controls></video>

When the video recording is stopped, we could create an Object URL for ourrecording blob and attach it to the video player.

mediaRecorder.onstop = () => {  let recordedVideo = document.getElementById("recordedVideo");  recordedVideo.src = URL.createObjectURL(blob);};

We can now play the recording on the HTML video player.

Similarly, we can create a download link using the Object URL for the recordingblob.

let a = document.createElement("a");let url = URL.createObjectURL(blob);a.href = url;

We can now download and play the recording locally on any video playersupporting WebM playback.

Putting it all together

We have glued together the code fragments discussed above and created ademo for a basic web-basedscreen recorder.

You may view the source codehere.

Next steps

Now that we have a basic screen recorder in place, we have to consider thefollowing:

1. Persistent storage.
2. Chunked uploading.
3. CDN support.
4. Playback with support for streaming.

We will cover these topics in the next set of blogs. Stay tuned.