In the previous blog, we learned how to use the Browser APIs to record the screenand generate a WEBM file. We now need to upload this file to persistent storageto have a URL to share our recording with our audience.

Uploading a large file all at once is time-consuming and prone to failure due tonetwork errors. The recording is generated in parts, each part pushed to anarray and joined together. So it would be ideal if we could upload these smallerparts as and when they are generated, and then join them together in the backend oncethe recording is completed. AWS'sSimple Storage Service (S3) made a perfect fit asit provides cheap persistent storage, along withMultipart Uploadsfeature.

S3 Multipart Uploads allow us to upload large objects in parts. Rather thanuploading the entire object in a single operation, multipart uploads break itdown into smaller parts, each ranging from 5 MB to 5 GB. Once uploaded, theseparts are aggregated to form the complete object.

Initialization

The process begins with an initiation request to S3, where a unique upload ID isgenerated. This upload ID is used to identify and manage the individual parts ofthe upload.

s3 = Aws::S3::Client.newresp = s3.create_multipart_upload({  bucket: bucket_name,  key: object_key})upload_id = resp.upload_id

Upload Parts

Once the upload is initiated, we can upload the parts to S3 independently. Eachpart is associated with a sequence number and an ETag (Entity Tag), a checksumof the part's data.

Note that the minimum content size for a part is 5MB (There is no minimum size limiton the last part of your multipart upload). So we store the recording chunks inlocal storage until they are bigger than 5MB. Once we have a part greater than5MB, we upload it to S3.

part_number = 1content = recordedChunksresp = s3.upload_part({  body: content,  bucket: bucket_name,  key: object_key,  upload_id: upload_id,  part_number: part_number})puts "ETag for Part #{part_number}: #{resp.etag}"

Completion

Once all parts are uploaded, a complete multipart upload request is sent to S3,specifying the upload ID and the list of uploaded parts along with their ETagsand sequence numbers. S3 then assembles the parts into a single object andfinalizes the upload.

completed_parts = [  { part_number: 1, etag: 'etag_of_part_1' },  { part_number: 2, etag: 'etag_of_part_2' },  ...  { part_number: N, etag: 'etag_of_part_N' },]resp = s3.complete_multipart_upload({  bucket: bucket_name,  key: object_key,  upload_id: upload_id,  multipart_upload: {    parts: completed_parts  }})

Aborting and Cancelling

At any point during the multipart upload process, you can abort or cancel theupload, which deletes any uploaded parts associated with the upload ID.

s3.abort_multipart_upload({  bucket: bucket_name,  key: object_key,  upload_id: upload_id})

The uploaded file will finally be available at s3://bucket_name/object_id

S3 Multipart Uploads offers us several advantages:

Fault tolerance

We can resume uploads from where they left off in case of network failures orinterruptions. Also, uploading large objects in smaller parts reduces thelikelihood of timeouts and connection failures, especially in high-latency orunreliable network environments.

Upload speed optimization

With multipart uploads, you can parallelize the process by uploading multipleparts concurrently, optimizing transfer speeds and reducing overall upload time.