That's where Crunchy Data comes in. They provide managed Postgres service. Checkout this tweet from DHH about Crunchy Data.

In our internal discussion, one of the BigBinary engineers brought up the issue of "latency". Since the PostgreSQL server will not be in the same data center, what would be the latency? How much impact will it have on performance?

We didn't know the answer so we thought we would do some benchmarking.

Benchmarking

To do the comparison we needed another hosting provider where we can run PostgreSQL on the same data center. We chose to work with Digital Ocean.

To compare the two services, we wrote a benchmark script in Ruby similar to the one Ben Johnson wrote in Go for his GopherCon talk in 2021.

In this benchmark, we're using Ruby's Benchmark module to measure the performance of a series of database operations. Here's what the code does:

1. It establishes a connection to the database only once, at the beginning of the script. This is done outside the benchmarked operations because establishing connections can be a slow operation because of TLS negotiation, and we don't want to account for that time in our measurements.

2. It then performs the following operations 10,000 times, measuring each one:

   • Drops a table named 't' if it exists.
   • Creates a new table 't' with two columns: 'id' (an auto-incrementing primary key) and 'name' (a text field).
   • Inserts a single row into the table with the name 'jane'.
   • Selects the 'name' from the table where the 'id' is 1 (which should be 'jane').

3. After all 10,000 iterations, it calculates and prints the average time for each operation in microseconds.

require "pg"
require "benchmark"

class PostgresBenchmark
  def initialize(connection_string)
    @conn = PG.connect(connection_string)
  end

  def run(iterations = 10_000)
    total_times = Hash.new { |h, k| h[k] = 0 }

    iterations.times do |i|
      puts "Running iteration #{i + 1}" if (i + 1) % 1000 == 0
      times = benchmark_operations
      times.each { |key, time| total_times[key] += time }
    end

    average_times = total_times.transform_values { |time| time / iterations }
    print_results(average_times, iterations)
  ensure
    @conn.close if @conn
  end

  private

  def benchmark_operations
    times = {}

    times[:drop] = Benchmark.measure { @conn.exec("DROP TABLE IF EXISTS t") }.real
    times[:create] = Benchmark.measure { @conn.exec("CREATE TABLE t (id SERIAL PRIMARY KEY, name TEXT)") }.real
    times[:insert] = Benchmark.measure { @conn.exec("INSERT INTO t (name) VALUES ('jane')") }.real
    times[:select] = Benchmark.measure do
      result = @conn.exec("SELECT name FROM t WHERE id = 1")
      raise "Unexpected result" unless result[0]["name"] == "jane"
    end.real

    times
  end

  def print_results(times, iterations)
    total_time = times.values.sum

    puts "\nAVERAGE ELAPSED TIME (over #{iterations} iterations)"
    puts "drop    #{(times[:drop] * 1_000_000).round(2)} microseconds"
    puts "create  #{(times[:create] * 1_000_000).round(2)} microseconds"
    puts "insert  #{(times[:insert] * 1_000_000).round(2)} microseconds"
    puts "select  #{(times[:select] * 1_000_000).round(2)} microseconds"
    puts "TOTAL   #{(total_time * 1_000_000).round(2)} microseconds"
  end
end

if __FILE__ == $0
  connection_string = "<DB_CONNECTION_STRING>"
  benchmark = PostgresBenchmark.new(connection_string)
  benchmark.run(10_000)
end

Database Specifications and Setup

Digital Ocean

• Region: NYC3 data center
• Specs: 2 vCPU, 4GB Memory
• Price: $60 per month

Crunchy Data

• Provider: AWS
• Region: us-east-1
• Specs: 2 vCPU, 4GB Memory (hobby-4 plan)
• Price: $70 per month

We provisioned a Digital Ocean droplet in the NYC3 data center and invoked the benchmark script from the machine. The Digital Ocean database was also in the same NYC3 data center. For Crunchy Data, the availability zone selected was us-east-1 as it was the closest to NYC3.

Benchmarking results

Digital Ocean

AVERAGE ELAPSED TIME (over 10000 iterations)
drop    3448.07 microseconds
create  5048.39 microseconds
insert  891.81 microseconds
select  584.17 microseconds
TOTAL   9972.44 microseconds

Crunchy Data

AVERAGE ELAPSED TIME (over 10000 iterations)
drop    10097.89 microseconds
create  16818.63 microseconds
insert  8416.35 microseconds
select  7211.42 microseconds
TOTAL   42544.29 microseconds

Benchmarking analysis

The results of this benchmark do not come as a surprise. Digital Ocean is performing significantly better than Crunchy Data. This performance difference can be primarily attributed to network latency.

Network latency refers to the round-trip time (RTT) it takes for the data to travel from its source to its destination and back again across a network. In the context of database operations, it's the time taken for a query to be sent from the client to the database server and for the response to return to the client.

In our benchmarking, the Digital Ocean database and the client machine invoking the script were both located in the same data center (NYC3), resulting in minimal network latency. On the other hand, the Crunchy Data database was hosted in AWS us-east-1, and it had to communicate across a greater physical distance, adding to latency.

To get a more accurate value for the network latency, we can compare the average time taken to run the SELECT operation. The SELECT operation in the script is a point query. A point query refers to type of query that retrieves one or several rows based on a unique key.

In our script, it retrieves a single name value from the table t where the id is 1(which is the primary key), and is very fast to execute. Thus, the time taken to execute the SELECT operation can give us an approximate value for the network latency.

db_time = network_latency + query_execution_time

For point queries, the query_execution_time is almost zero so all the time taken is pretty much "network latency".

db_time ≈ network_latency

If we look at the benchmarking result, then we can see that for "select" operation time taken by Digital Ocean is "584 microseconds" and for Crunchy Data it is "7211 microseconds".

The difference in network latency is 6627 microseconds. That is 6.6 milliseconds.

This value over multiple queries can add up and can have a significant impact on the overall response time of your application. To put this into perspective, for a web application that makes 10 sequential database queries to render a page, this could add up to about 66 milliseconds to the page load time. Now, this could be an acceptable limit if your page loads in 3/4 seconds. However, if you are trying to load your page in 200 millisecond,s then you need to watch out.

Ensuring that the database is always up and it's properly backed up is a non-trivial problem. Latency notwithstanding, Crunchy Data takes care of running the database. This gives us peace of mind and allows us to exit the cloud one step at a time.

The Problem

Initially, when we shared a NeetoRecord video on Twitter, this is how the tweet would appear.

As we can see, the links are just plain text, lacking visual appeal and context. Users had to click away from Twitter and open a new tab to view the recording. This extra step not only disrupted the user's Twitter experience but also likely reduced the number of people who actually watched our videos.

The Solution: Twitter Cards

To address this issue, we turned to Twitter Cards. Similar to Facebook's Open Graph protocol, Twitter Cards allows us to showcase interactive media directly within tweets. While both protocols serve similar purposes, Twitter cards are designed to directly work in Twitter, so its crawler will look for these tags and only look for OG tags as a fallback.

What are Twitter Cards?

Twitter Cards are a set of meta tags that enable us to embed interactive content on a tweet. In our case, we want to provide users an embedded video player which they can use to view the NeetoRecord recording without leaving Twitter. It's not just about aesthetics. According to Twitter's own data, "tweets with cards have 43% higher engagement rates than regular tweets with links." This statistic underscores the impact of providing a seamless, visually appealing experience.

Implementing Twitter Cards for NeetoRecord

There are multiple types of cards that we can leverage to enhance our content such as summary cards, player cards & app cards. However, in this post, we are going to be discussing about Player cards, which is implemented in NeetoRecord. This card type allows us to embed an interactive video player directly in the tweet. Here's how it looks:

Clicking on this link preview will open our embedded content, in this case, a player to view the recording directly within Twitter.

As we can see, users can now watch our NeetoRecord videos without leaving Twitter. This seamless experience has several benefits:

• Higher Engagement: With the video right there in the tweet, more users are likely to watch it.
• Reduced Friction: No more clicking away or opening new tabs, keeping users engaged with our content and the Twitter conversation.
• Better Branding: The Player Card includes our logo and video title, reinforcing our brand with every view.

Technical Implementation

Implementing Twitter Cards is straightforward. We added meta tags to the <head> section of our web page. For our Player Card, we use tags as shown below.

<meta property="twitter:card" content="player" />
<meta
  property="twitter:url"
  content="https://oli.neetorecord.com/watch/864de7fb-2efb-4f2f-a60b-08dca64e4c3"
/>
<meta
  property="twitter:title"
  content="Introducing the new video player for NeetoRecord"
/>
<meta property="twitter:site" content="@NeetoRecord" />
<meta property="twitter:image" content="https://cdn.neeto.com/hycoe7" />
<meta
  property="twitter:player"
  content="https://oli.neetorecord.com/embeds/864de7fb-2efb-4f2f-a60b-08dca64e4c3"
/>
<meta property="twitter:player:width" content="1280" />
<meta property="twitter:player:height" content="720" />

These tags tell Twitter's crawler what to display in the card.

Let's break down the purpose and importance of each meta tag:

<meta property="twitter:card" content="player" />

This tag specifies the card type. Setting it to "player" tells Twitter that this is a Player Card, which is designed for video or audio content.

<meta
  property="twitter:url"
  content="https://oli.neetorecord.com/watch/864de7fb-2efb-4f2f-a60b-08dca64e4c3"
/>

This tag provides the URL of the web page that the card is describing. It should be the page where users can view the full content.

<meta
  property="twitter:title"
  content="Introducing the new video player for NeetoRecord"
/>

This tag sets the title of the card, which appears as the main headline.

<meta property="twitter:site" content="@NeetoRecord" />

This tag specifies the Twitter @username the card should be attributed to.

<meta property="twitter:image" content="https://cdn.neeto.com/hycoe7" />

This tag provides the image to be displayed in place of the player on platforms that don’t support iFrames or inline players.

<meta
  property="twitter:player"
  content="https://oli.neetorecord.com/embeds/864de7fb-2efb-4f2f-a60b-08dca64e4c3"
/>

This crucial tag specifies the URL of the video player. This should be an HTTPS URL to an iframe player that can play the content.

<meta property="twitter:player:width" content="1280" />
<meta property="twitter:player:height" content="720" />

These two tags define the width and height of the video player iframe, in pixels. They help ensure the video displays correctly in the Twitter feed.

Conclusion

Twitter Cards have improved how NeetoRecord videos are shared on Twitter. The embedded media experience reduces friction and increases engagement, while enhancing our brand visibility.

For our apps deployed on Heroku, we used the following command to get the latest commit hash as an environment variable in the Heroku server on each build:

heroku labs:enable runtime-dyno-metadata -a your-app-name

With NeetoDeploy, we didn't need this step since git is available in the console and we used git commands to set the latest commit hash as the Honeybadger revision. Honeybadger uses the revision to associate maps with the bundle when the bundle name doesn't change.

Steps to integrate Honeybadger:

1. Create a new project in the Honeybadger dashboard. We've made a video on setting it up following the React integration guide.

2. Set the environment variable HONEYBADGER_JS_API_KEY with the Honeybadger project's API key. The API key can be copied from Settings => API Keys.

In depth guides for Honeybadger integration & sourcemap upload:

• What is honeybadger
• Honeybadger integration basics
• Honeybadger integration in the Neeto ecosystem
• Sourcemaps explained
• Uploading sourcemap to honeybadger
• Sourcemaps in action within Honeybadger errors
• Handling CDN based application bundles

We've created a YouTube playlist of the above videos for easy access.

While writing the program, we realized that we'd need a switch-case that checked what data type each key was. We had to search for the get and set methods for each data type manually while writing the program. Late,r we used the redis-rb gem and was able to get the dump with OpenSSL::SSL::VERIFY_NONE, but we had still faced the issue of having to look up what the command was for different data types like zset and hash.

I thought this could be a good opportunity to write a blog post summarizing all the data types and the commands associated with them. The Redis documentation has a page about the different data types, but all the commands are not available at a single place.

Redis key-value database

When we say that Redis is a key-value database, there is more to it than what meets the eye. Redis keys can store values of several different data types. Redis has a set of commands for each of these different data types to do operations with them. In this post, we’ll go over the different data types, what they are and how we can work with them.

What are the different data types?

Redis has more than a couple of data types, which can be used to store different data based on your needs. These include the following:

• String - The basic data type we are all familiar with.
• List - An array of strings.
• Hash - A collection of key-value pairs, similar to a Ruby Hash.
• Set - A collection of unique strings.
• Sorted Set - A collection of unique strings maintaining by each string’s score.
• HyperLogLog - Probabilistic estimates of cardinality of large sets.
• Stream - An append only log.
• Geospatial Index - Data structure for storing geographic coordinates.

Checking data type of your keys

You can use the TYPE command to check what data type your key is. Once you know what type your key is, you can use the commands associated with your key's data type to interact with it.

127.0.0.1:6379> TYPE schedule
zset

Cheatsheet for Redis commands based on data type

Each Redis data type has its own set of commands for doing operations with the key and its value. Here's a quick overview of the basic data types you would encounter and some of the basic commands to set, retrieve and delete data from them.

Read more about the different data types in Redis and all the different commands that are available in their official documentation.

zprof

zprof is a utility that comes packaged with zsh, which you can use to profile your zsh script.

Add the following to the top of your .zshrc file to load zprof.

zmodload zsh/zprof

At the bottom of your .zshrc, add the following.

zprof

This would profile your zsh script and print a summary of all the commands run during your shell startup and the time it takes to execute them. Run exec zsh to apply the changes and restart your shell. Your shell will print something like this:

With this you can see which commands are taking the most time to load. Enabling profiling has helped pinpoint the issue and now you can look into fixing it. In the above example, you can see that nvm is taking up a considerable amount of time.

When it comes to actually using these products, you will be redirected to https://subdomain.neetocal.com. Here, "subdomain" would be the subdomain allocated to you when you signed up for Neeto.

We planned to add "Google sign in" feature to make it easier for folks to both signup and to log in. During testing, it all worked fine. However, when we asked "Google" to approve the app "NeetoCal" for "Google sign in" Google demanded that our users should be able to see the "Privacy Policy" and "Terms of conditions" on the website. In order to make Google happy, we added a redirection from "neetocal.com" to "neeto.com/neetocal".

However, Google was not happy with it. The users are logging into https://subdomain.neetocal.com so the "privacy policy" and "terms of service" should be visible on the domain "neetocal.com" itself.

We are using Cloudflare as our DNS provider. Using the tools provided to us by Cloudflare we decided to show the content of "neeto.com/neetocal" on "neetocal.com" without redirecting the user.

Note that in this case, if you type "neetocal.com", then you will see the URL change to "neetocal.com/neetocal" instantly. That's because the URL of the main marketing site is "neeto.com/neetocal".

Cloudflare provides Page rules which we will be using to achieve our goals. Below is a video of how it was done.

Watch on YouTube

SEO duplicate content issue

The Google search engine doesn't like it when we show exactly the same content on two different domains. Google thinks that the site is trying to cheat Google and Google will punish both sites.

We want Google to index our main marketing site https://neeto.com/neetocal and we want Google to ignore "neetocal.com". One way to tell Google not index the site is by adding a noindex meta tag.

<meta name="robots" content="noindex">

In the above example, we are asking all bots not to index the page containing the above meta tag.

We planned to inject this meta tag when a page is rendered for the URL "neetocal.com" and we will not inject this tag when the page is rendered for the URL "neeto.com".

Upon more research, we found that search engines also look at the response headers. Given below is the sequence that search engines follow for indexing the web pages.

• Crawler gets the raw page source as a response to the HTTP request.
• Crawler checks if x-robots-tag: noindex, nofollow header is present in the response.
• Crawler checks the meta tags to determine if the page needs to be indexed or not.

If a page has x-robot-tag: noidex, nofollow then the crawler will not index the page.

Based on this information, we decided to use Response Header Modification Rules feature of Cloudflare. Below is a video of how it was done.

Watch on YouTube

Watch on YouTube

We checked with SSLlabs and it said the same thing: the servers support TLS 1.0 and TLS 1.1.

TLS 1.0 was published in 1999, and TLS 1.1 was published in 2006. Microsoft and other companies don't support these two versions of TLS. Even Heroku doesn't support it.

All our Neeto applications are hosted on Heroku. If Heroku doesn't support TLS 1.0 and TLS 1.1, how come the server supports these older versions of TLS?

Solving the TLS issue using Cloudflare

We use Cloudflare as our DNS server for all Neeto products. Cloudflare allows us to proxy the request. It means that when the user hits neetocal.com, their request is not going to Heroku. Cloudflare will intercept the request, and then Cloudflare will make a request to the Heroku server on behalf of the user. When Cloudflare makes this request to Heroku will use its own SSL certificate.

Cloudflare allows us to have control over the "Minimum TLS version" to support. We configured Cloudflare to not support TLS 1.0 and TLS 1.1.

The following video goes into step-by-step detail on how we configured this in Cloudflare.

Watch on YouTube

CDN77 is the service we used in the video to check the TLS version.

We switched our pull-request review apps from Heroku to NeetoDeploy a couple of months ago, and it has been doing well. As the next step of the process, we are building features to target staging and production. We had an essential staging set up in March, and by the end of the month, we had migrated staging deployments of all Neeto from Heroku to NeetoDeploy. Everything was working fine for a week until I made a grave mistake.

What happened

Whenever a PR is opened or a new commit is pushed, NeetoDeploy receives a webhook call from GitHub. Review/staging apps are created/updated on NeetoDeploy in response to these webhook calls.

On 2023 April 5, a misconfiguration caused NeetoDeploy's webhook handler to malfunction for an hour. As a result, some review apps were not deleted even after the corresponding PRs were closed or merged. The solution was cross-checking review apps with the open PRs and deleting the unwanted apps. Using rails console, this could be done live on the server.

Here is what that solution looked like:

GithubRepository.find_each do |github_repository|
  access_token = github_repository.github_integration.access_token
  github_client = Octokit::Client.new(access_token:)

  open_pr_numbers = github_client
    .pull_requests(github_repository.name, state: :open)
    .pluck(:number)

  github_repository.project.apps.find_each do |app|
    next if open_pr_numbers.include?(app.pr_number)
    Apps::DestroyService.new(app).process!
  end
end

But there is a terrible mistake in the above code. See if you can spot that.

I'm going to wait...

A bit more waiting... Enough waiting; here's the mistake:

There is no filter in the apps that were picked to be destroyed. This snippet was written at a time when we only had review apps. So github_repository.project.apps was expected to return review apps. But we now also had staging apps in the database. And those staging apps weren't filtered out here. After running the snippet and noticing it took longer than expected, I realized the mistake and instantly pressedCTRL + C. Of course, it was taking time since it was deleting all the staging app databases and dynos 🤦.

In the end, out of 33 staging apps, only five remained. And thus started, the procedure to restore all of them.

The recovery

NeetoDeploy already had the feature to do manual DB exports but this wasn't being done routinely. We were only hosting review apps (whose data need not be persisted reliably), and staging had only started just a week before.

We had database backups from a week before (when we ultimately migrated staging apps off Heroku), and one by one, our small team of 4 brought back all the apps in 2 days. The next step is to try not to let this happen again; if it were to happen, we have a contingency plan. We thought of two types of contingency plans:

• Automatic scheduled backups
• Disk snapshots of the DB

Automatic scheduled backups

The idea is that the database would be exported at a particular time every day. Backups older than a month would be deleted automatically to save space.

We implemented this in a week. Every day at 12 AM UTC, all staging+production databases would be exported and uploaded to an S3 bucket.

While this feature was being implemented, I used the Rails console to manually export all the apps. The exported file URLs of each DB were manually copied to a text file. aria2c was then used to download them in parallel to a local folder:

aria2c -c --input-file export_urls.txt

aria2c is a smart downloader. It will resume interrupted downloads, wouldn’t duplicate downloads, and do everything in parallel.

Disk snapshots of the DB

The other contingency method is to do periodic snapshots of the volume holding the DB. We are working on this.

You can refer to this blog post of GitLab to know their recovery procedures when they faced a significant data loss in 2017.

Lessons

The core lesson here is to call destructive methods very carefully. Instead of calling the DestroyService instantly, there could have been an intermediate human check:

apps = []
GithubRepository.find_each do |github_repository|
  access_token = github_repository.github_integration.access_token
  github_client = Octokit::Client.new(access_token:)

  open_pr_numbers = github_client
    .pull_requests(github_repository.name, state: :open)
    .pluck(:number)

  github_repository.project.apps.review.find_each do |app|
    next if open_pr_numbers.include?(app.pr_number)
    apps.append(app)
  end
end

This would populate the list of apps to delete in apps variable, it can be displayed, verified and then we can destroy them individually:

apps.map do |app|
  Apps::DestroyService.new(app).process!
end

The other takeaway here is to have proper recovery mechanisms in place. Human/system errors are possible; we should be prepared when it happens.

NeetoDeploy is still not production-ready. However, if you want to give NeetoDeploy a try, then tweet to us at @neetoDeploy or send us an email at invite@neeto.com.

Not only can anyone send an email to anyone, but anyone can pretend to be anyone. I can send an email to you and the email will come to you as if it were sent by "Elon Musk" and the from address could be "elon@musk.com".

I can send an email to you pretending that the email is from your bank and you need to change your password. In one case, a hacker sent an email to a company's finance team telling them that the company's bank account number has changed. The hacker walked away with millions of dollars.

Hackers can also do damage to one's domain reputation.

Let's assume that there is a business called "PrixPapers" and they have thousands of customers all over the country. Malicious folks can start sending emails to people pretending to be "PrixPapers". When people receive an email like this they might buy the item. The item could be "fake goods". Ideally, no one should be able to send any email pretending to be someone else.

Since these malicious folks don't have the actual list of the customers of PrixPapers will in general spam people. Some people receiving this email will mark these emails as "spam". As more and more people mark these emails as "spam" the domain reputation of "prixpapers.com" will go down. It means that when "prixpapers.com" will send out a legitimate newsletter to its customers some of these emails will be marked as "spam" because of the nefarious work done by the previous scammer.

The challenge presented to the Internet authorities is that they want to keep email simple but safe at the same time. They came up with certain policies to make emails safe and secure. In this blog, we will see how SPF helps in keeping email secure. However, first, we need to know what is "Return-path". We'll later see how it is used in blocking fake emails.

Return-path

Let's say that we are dealing with a company called NeetoCal. They have the domain neetocal.com.

Let's say that brian@neetocal.com sent an email to peter@gmail.com. Let's pretend that for some reason Peter's mailbox is full and the email sent by Brian bounces. In that case, Brain will get an email that the message bounced. We all have seen these kinds of messages.

Now let's imagine that it's end of the year sale time and NeetoCal decided to send an email campaign to all its customers giving them a 10% discount. This email will go out to their 5000 customers. Some of these emails are bound to bounce. If all the bounced emails come to Brain then Brain's inbox would be filled with such bounced emails.

To solve this problem email protocol allows us to set a hidden field called Return-path. This value can be set in the email header. Return-path set in the email header indicates how to process bounced emails. Anytime an email server detects that an email can't be delivered for reasons like "mailbox is full" or "the email doesn't exist" then that email server can send an email to the email address mentioned in Return-path. The business owner can look at Return-path emails to analyze how many emails are bouncing and why.

If you are sending an email using "Gmail", "outlook", "yahoo" etc then the Return-path is set as your email so that you get to know if there is a bounced email. Email service providers allow us to customize Return-path in case one is running a big marketing campaign. For example here is a document from SendGrid describing how to set Return-path.

Why are we talking about how bounced emails are processed when we are dealing with the subject of "email deliverability". That's because Return-path plays a dual role as we will see next.

What is SPF record

Before we get into the SPF record, let's take a simple real-world example of how it works.

Let's say there's a gatekeeper at the NeetoCal office. This gatekeeper will only allow the people who work there. It means the gatekeeper has a list of approved people and for each person getting through the gate, the gatekeeper checks if the person is in the approved list or not.

SPF works similarly. SPF policy is a mechanism to tell the email server if the email is coming from a trusted source, then accept the email or reject the email. A sender policy framework (SPF) record is a type of DNS TXT record that lists all the servers authorized to send emails from a particular domain.

Let's take a real-world example. The SPF record of neetocal.com looks like v=spf1 include:spf.messagingengine.com -all. We can see this data using mxtoolbox.

How the SPF record is used

Let's look at the following case.

Step 1. notifications@neetocal.com sends an email to elon@gmail.com.

Step 2. This email is sent to Fastmail since NeetoCal is using the Fastmail services.

Step 3. The Fastmail email server receives this email as an outgoing email.

Step 4. Fastmail email server adds email header Return-path value to "notifiations@neetocal.com".

Step 5. The Fastmail email server sends this email to the gmail server.

Step 6. The Gmail server gets this email.

Step 7. The Gmail server extracts the Return-path key and finds that the domain is "neetocal.com".

Step 8. The Gmail server finds the TXT DNS records of "neetocal.com". This lists all the approved IP addresses.

Step 9. The Gmail server will check if the email server which sent the email is in the approved IP addresses or not.

Step 10. If the IP address is in the approved IP addresses list then the email is approved for further processing. Otherwise, the email is rejected.

We can go to https://dmarcian.com/spf-survey/ and enter "neetocal.com" here. They get the IP addresses published for "messagingengine.com" and then they show the list of the approved IPs.

Allowing the third party to send emails on your behalf

Let's say that Brian decides to use Mailerlite to send marketing emails. Now if Mailerlite sends an email then the receiving email server will notice that the IP address of Mailerlite is not in the approved list of ips and the email will be rejected.

A domain is allowed to have only one SPF record. So if NeetoCal wants to use Mailerlite for marketing then the spf records need to be updated. If you signup for a domain in Mailerlite then Mailerlite will check if that domain has an existing SPF record or not. If there is no SPF record then Mailerlite won't do anything. However, if there is an existing SPF record then Mailerlite will insist that first you update the SPF record to include Mailerlite so that the emails from Mailerlite are not rejected.

BigBinary uses Mailerlite to email newly published blogs to the subscribers. Given below is what the SPF record of BigBinary looks like. We can also see this result online.

v=spf1 include:_spf.mlsend.com include:_spf.google.com -all

BigBinary uses google workspace so the second include is for that reason. The first include is to ensure that Mailerlite is in the allowed ip list.

If we look at the result for "bigbinary.com" by visiting https://dmarcian.com/spf-survey/ then we will that all these includes are like regular programming language "imports". They allow the third party to include other third parties in the chain. The end result is that we have a finite list of approved IP addresses which can send email.

spf record standard

Let's take a look at NeetoCal's SPF record.

v=spf1 include:spf.messagingengine.com -all

v=spf1 tells the server that this record contains an SPF record. Every SPF record must begin with this string.

include:spf.messagingengine.com tells the server what third-party organizations are authorized to send emails on behalf of the domain. This tag signals that the content of the SPF record for the included domain (messagingengine.com in this case) should be checked and the IP addresses it contains should also be considered authorized. Multiple domains can be included within an SPF record.

-all tells the server that addresses not listed in the SPF record is not authorized to send emails and should be rejected. Alternative options here include ~all, which states that unlisted emails will be marked as insecure or spam but still accepted. +all signifies that any server can send emails on behalf of the domain.

Here is another format of SPF record.

v=spf1 ip4=192.0.2.0 ip4=192.0.2.1 include:messagingengine.email -all

In this example, the SPF record is telling the server that ip4=192.0.2.0 and ip4=192.0.2.1 are also authorized to send emails on behalf of the domain.

All this time, the AceInvoice website was not redirecting to NeetoInvoice. Today we did that using Cloudflare. The URL forwarding or redirecting with page rule is a neat feature of Cloudflare. Below are the screenshots of the steps taken to redirect the URLs. More details are covered in the video.

Handling www version

Handling no www version

Watch on YouTube

We didn't mention it in the video, but it's worth knowing that we need to have a CNAME record for a subdomain that needs forwarding. For examples let's say that we need to forward all traffic from https://videos.bigbinary.com to https://bigbinary.com/video. We can't add a page rule for this directly. For we need to add a DNS entry for subdomain videos and this entry must have cloudflare "proxy" checked so that you see Proxied next to it. If you see "DNS only" then that means Cloudflare will not be able to do any forwarding.

OAuth 2.0

OAuth 2.0 is an authorization framework that enables an application to access data from a server on behalf of a user. For example, OAuth enables applications to access data from Google, Facebook, GitHub, etc., on behalf of users having accounts in that service.

Let's say you have an application that requires access to a user's data on GitHub. The following are the steps involved in authorizing your application with GitHub:

1. The application requests authorization from GitHub. The following are some of the parameters this request should contain:

   • client_id - This is an ID used by GitHub for identifying the application. You need to register your application with GitHub to get this ID.
   • redirect_uri - The URI to which GitHub should send back a request once the authorization is approved.
   • login - This is a username on GitHub. The application requires access to data on behalf of the user with this username.
   • state - This should ideally be a string of random characters. Store this string in memory because it will be used in another step. This parameter is optional but highly recommended. We'll look into why later.

2. GitHub then asks the user to grant the authorization. The user might be prompted to log in to GitHub if they have not already logged in. Then GitHub displays information on the application and lists all the data the application wants to access. If the user denies authorization, the process ends here. If the user approves, we move on to the next step.

3. GitHub sends a request to the application through the URI passed as redirect_uri in the first step. This request will contain a code parameter that serves as a temporary authorization code and a state parameter.

4. The OAuth process must be dropped immediately if the value of the state parameter from the previous step does not match the random string stored in memory in step 1. This ensures that the OAuth process was initiated by your application. We'll look more into this later.

5. The application should send another request to GitHub to generate a permanent authentication token. This request should contain the code sent by GitHub in step 3.

6. GitHub responds with an authentication token that can be used by the application to access data on behalf of the user.

The state parameter

The state parameter plays a crucial role in ensuring that the GitHub OAuth process was initiated by your application or a trusted source. An unauthorized third party could pose as your application by using your client_id. The value of client_id is not exactly a secret. GitHub treats any OAuth process initiated using your ID as an OAuth process started by your application.

Let's see how the OAuth process would play out when initiated by an unauthorized third party:

1. Third-party requests authorization from GitHub. The third party would pass your ID as client_id. This would lead GitHub to believe the request is from your application. The third party may or may not pass a state parameter.

2. GitHub then asks the user to grant the authorization. If the third party uses one of their own accounts, they could grant the authorization. The third party could also convince a user to grant permission using social engineering. For example, they could perform a phishing attack using an email designed to trick the user into believing that the email was from your application.

3. GitHub sends a request to the application through the URI passed as redirect_uri in the first step. This request would contain the code parameter, and the state parameter if it was passed in step 1.

4. It would be clear that the process was not initiated by your application if the state parameter is missing. Even if there was a state parameter, it would not be found in memory. This is because the state parameter was generated by a third party and not your application. Only those state parameters generated by your application will be found in your memory.

JSON Web Tokens

JSON Web Tokens (JWT) is a structured format that contains header, payload, and signature components. Let's take a look at the significance of these components:

1. Header - The header is a JSON string that typically contains the signing algorithm used, such as HMAC, SHA256, or RSA.

2. Payload - The payload is a JSON string that contains claims and additional data. Claims can be used to enforce security constraints and validate the authenticity of the token and its contents. For example, a claims can specify the token expiration time. The expiration time, most commonly represented by exp, represents the date and time after which the token will no longer be considered valid.

3. Signature - The signature is used to verify that the sender of the JWT is who it says it is and to ensure that the token has not been tampered with along the way. Just like the hand signature of a person, it is hard to forge a JWT signature. (Digital signatures are significantly harder to forge compared to a person's hand signature)

Generating a JWT

The following steps outline the process of generating a JWT:

1. The header and the payload of the JWT are first encoded using Base64 encoding. So now we have two strings - the encoded header and the encoded payload.

2. The encoded header and the encoded payload are concatenated into a single string, with dots (.) separating each part. The concatenated string is signed using the signing algorithm specified in the header. The signing algorithm uses a secret key, that is known only to the issuer (your application), which ensures that only the issuer can generate a valid signature. The result of the signing process is a signature, which is also a string.

3. The encoded header, the encoded payload, and the signature are concatenated into a single string, with dots (.) separating each part. The resulting string is the JWT, which can be transferred securely between parties.

For example, let's say we have the following header:

{ "alg": "RS256" }

The value of alg contains the algorithm used for signing the JWT. Also, let's say we have the following payload:

{ "username": "sam@example.com", "exp": 1676263763 }

The username is data that needs to be transferred. exp contains the expiration time claim of the JWT.

When both these components are encoded, we get:

• Header - "eyJhbGciOiJSUzI1NiJ9"
• Payload - "eyJ1c2VyIjoic2FtQGV4YW1wbGUuY29tIiwiZXhwIjoxNjc2MjYzNzYzfQ"

When these are signed using the RS256 algorithm and a secret key, a signature is produced. The following signature was generated using the RS256 algorithm and a secret key (which will remain secret):

NlAT6awp68dCEcFXbDeeLTzZekqUmB3f6kr3jkGSFmrKa5zvLmFGeraWba_fUuQLVhRtcXUPZbRR1DKnKH0HVf1rRDvOqezwbhe-hR1wlz6vZkHuPjtYSCLx_aybGm7dy2ijfTQwYd14cD9ZiMI5vf6XcDDfE7mkhu0ogCOnqR1v3KOEWJkMkvGBHfHKuf9FKYbWltHtUE6bAEO1orq0JayD8UNUKxdGkElXA7mkuIEexmBuieG9PJ2ow_uo05QCsqDvxlzOCMMIe7WdT7gmz4myiZ7lVuUcL1V2-Y1PJqWDyqDZbKNxd4X_CwW0RLOF1pw9S2URgybqHZFG0murNw

So the final JWT would be:

The image above was captured from the decoding tool in jwt.io.

JWT is basically a Base64 encoded string with a signature attached to it. Having that signature component makes JWT a secure format for transferring data. It is possible to simply encode data with Base64 and transfer that string. From the example above, transferring the encoded payload "eyJ1c2VyIjoic2FtQGV4YW1wbGUuY29tIiwiZXhwIjoxNjc2MjYzNzYzfQ" can also get the data to the other party. However, the party that receives the data have no way of ensuring that the data was sent by a trusted source and that the data was not tampered with along the way.

Verifying JWT tokens

Anyone who has the secret key used to sign a JWT, can verify the integrity of the JWT. The steps involved in verifying the signature of a JSON Web Token (JWT) are:

1. Split the JWT into the encoded header, the encoded payload and the signature, using the dot (.) used to separate the three components.

2. Decode the encoded header and the encoded payload using Base64 to get the header and payload JSON strings.

3. The application recreates the signature by signing the encoded header and the encoded payload using the signing algorithm in the header and the secret key. If the signature created in this step is the same as the signature in the JWT, the JWT is valid.

4. The application validates the claims in the payload if there are any.

It can be verified that the JWT was generated by your application and has not been tampered with, if the signature is valid. If the JWT header or payload was tampered with, the signature produced while verifying would be different from the one in the JWT.

Using JWT for the state parameter

We can generate a JWT token and pass that as the state parameter in the GitHub OAuth authorizing process. Here's how the process will be different when using JWT:

1. The application requests authorization from GitHub. This time the state parameter will be a JWT signed using a secure algorithm and a secret key. The JWT need not be stored in memory.

2. GitHub gets the approval of the user.

3. GitHub sends a request to the application through the URI passed as redirect_uri in the first step. This request will contain the code and the state parameters. Here, the state parameter would be a JWT.

4. Validate the JWT from the state parameter. If the JWT is invalid, drop the authorization process immediately.

Advantages of using JWT for the state parameter

1. No Storage requirement - When using a random string for the state parameter, that string has to be stored, so that it can be used later for verification. However, JWTs can be verified without storing them once they are generated.

2. Ability to send additional data - The payload component of JWT can be used to send additional data such as user data, permissions data, etc.

3. Security - Using JWT can ensure that the OAuth process was initiated by your application or a trusted source. In addition, JWT also ensures the integrity of the data. This means that we can ensure that the payload in the JWT has not been tampered with.

4. Flexibility - The claims in the payload component of JWT can be used further enhance the security of the JWT by implementing custom security checks based on the claims in the payload.

5. Standardization - JWT is a widely used format for transferring data. Using JWT would be beneficial when there are different applications and services involved in the OAuth process.

References

1. OAuth 2.0 - https://www.digitalocean.com/community/tutorials/an-introduction-to-oauth-2

2. GitHub OAuth Authorization process - https://docs.github.com/en/developers/apps/building-oauth-apps/authorizing-oauth-apps

3. Phishing Attack - https://www.imperva.com/learn/application-security/phishing-attack-scam/

DNS management is a free service by Cloudflare. However, on first glance, it might not appear that it's a free service. Once we add a site, then we see a screen like this. Here, we need to remember to scroll down to see the free option.

Now let's see how we can map the DNS settings from Heroku to Cloudflare. We will look at both a standard domain name and then we will take a look at a wildcard domain name.

Standard domain name

We are hosting GitEmit using Heroku. We are letting Heroku manages the SSL for this domain.

After setting up domains in Heroku, here is what we see.

In Cloudflare, we can set it up using two CNAMEs. It looks like this.

Wild card domain name

We are hosting NeetoChat application using Heroku.

Since it's a wild card domain, we had to generate the certificates ourselves.

After setting up domains in Heroku, here is what we see.

In Cloudflar,e we can set it up using three CNAMEs. It looks like this.

In such scenarios, the baseUrl will be different in both environments. But cypress allows us to configure baseUrl only once in cypress.json . Then, how do we configure baseUrl specific for each of the environments? In this blog, we will see how to configure cypress step by step so that we can run cypress tests in multiple environments.

1. Add environment specific configuration

To configure Cypress for different environments, we can follow the two simple steps.

1. Create config folder in cypress.
2. Create separate files for each of the environments.

// cypress/config/cypress.development.json

{
  "baseUrl": "https://localhost:9006",
  "env": {
    "environment": "development"
  },
  "execTimeout": 18000,
  "defaultCommandTimeout": 300000,
  "requestTimeout": 10000,
  "pageLoadTimeout": 30000,
  "responseTimeout": 10000,
  "viewportWidth": 1200,
  "viewportHeight": 1200,
  "videoUploadOnPasses": false,
  "retries": {
    "runMode": 1,
    "openMode": 2
  }
}

// cypress/config/cypress.test.json

{
  "baseUrl": "https://test.example.com",
  "env": {
    "environment": "test"
  },
  "execTimeout": 300000,
  "defaultCommandTimeout": 60000,
  "requestTimeout": 20000,
  "pageLoadTimeout": 60000,
  "responseTimeout": 20000,
  "viewportWidth": 1200,
  "viewportHeight": 1200,
  "videoUploadOnPasses": true,
  "retries": {
    "runMode": 2,
    "openMode": 1
  }
}

// cypress/config/cypress.production.json

{
  "baseUrl": "https://live.example.com",
  "env": {
    "environment": "production"
  },
  "execTimeout": 300000,
  "defaultCommandTimeout": 60000,
  "requestTimeout": 20000,
  "pageLoadTimeout": 60000,
  "responseTimeout": 20000,
  "viewportWidth": 1200,
  "viewportHeight": 1200,
  "videoUploadOnPasses": true,
  "retries": {
    "runMode": 2,
    "openMode": 1
  }
}

2. Initialize config files

After adding config folder, we need to tell cypress about those config files. We can do that by updating plugins/index.js with following code. This file gets executed after we start the cypress server.

// cypress/plugins/index.js

const fs = require("fs-extra");
const path = require("path");

const fetchConfigurationByFile = file => {
  const pathOfConfigurationFile = `config/cypress.${file}.json`;

  return (
    file && fs.readJson(path.join(__dirname, "../", pathOfConfigurationFile))
  );
};

module.exports = (on, config) => {
  const environment = config.env.configFile || "development";
  const configurationForEnvironment = fetchConfigurationByFile(environment);

  return configurationForEnvironment || config;
};

In the above code, Cypress loads the configuration file based on the environment. When we run Cypress we can pass environment variables. In this case, we need to pass configFile as an environment variable. If we don't pass configFile, by default Cypress will consider development as the current environment.

3. Setup scripts

Cypress can accept command-line arguments. We can set the environment by passing these arguments in the cypress run or cypress open command e.g. cypress open --env configFile=test. This command looks lengthy. Also, sometimes we need to pass more command-line arguments along with configFile. We can create short and handy commands by configuring the package.json.

For example: yarn run cy:open:dev.

// cypress/package.json

"cy:open:dev": "cypress open --env configFile=development",
"cy:open:dev:chrome": "cypress open --browser chrome --env configFile=development",
"cy:run:dev": "cypress run --env configFile=development",
"cy:open:staging": "cypress open --env configFile=test",
"cy:run:staging": "cypress run --env configFile=test",

4. Precedence of configuration

By default, we have cypress.json in Cypress where some config can be added. The precedence of configuration is higher for the files in the config folder than the cypress.json. So if we have some key execTimeout defined in both cypress.json and cypress.test.json. The value of execTimeout in the cypress.test.json will be considered.

Generally, the rule of thumb is that a common configuration for all the environments can be kept in the cypress.json. The environment-specific config can be kept in config/ folder.

Conclusion

We saw that how we can configure the cypress in multiple environments. In this way we can run Cypress for each environment without much hassle. This definitely helps us in running Cypress in both local as well as in the production environment.

Cookies can be a very safe method for storage on the client side. They can be set as:

• HTTP only: cannot be accessed through client-side JavaScript, saving it from any third party client-side scripts or web extensions.
• Secure: The web browser ensures that the cookies are set only on a secure channel.
• Signed: We can sign the content to make sure it isn't changed on the client side.
• Same Site: Make sure that the cookie is sent only if the site matches your domain/subdomain (details)

Prerequisites

• Postgraphile - Generates an instant GraphQL API from a Postgres database
• Express - Minimalistic backend framework for NodeJS

Setup

We will start off with a base Express setup generated with express-generator.

const createError = require("http-errors");
const express = require("express");
const path = require("path");
const cookieParser = require("cookie-parser");
const logger = require("morgan");

const app = express();

require("dotenv").config();

app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(express.static(path.join(__dirname, "public")));

// Use secret key to sign the cookies on creation and parsing
app.use(cookieParser(process.env.SECRET_KEY));

// Catch 404 and forward to error handler
app.use(function (req, res, next) {
  next(createError(404));
});

// Error handler
app.use(function (err, req, res) {
  // Set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get("env") === "development" ? err : {};

  // Render the error page
  res.status(err.status || 500);
  res.render("error");
});

module.exports = app;

From Postgraphile's usage library page for adding Postgraphile to an express app:

app.use(
  postgraphile(
    process.env.DATABASE_URL || "postgres://user:pass@host:5432/dbname",
    "public",
    {
      watchPg: true,
      graphiql: true,
      enhanceGraphiql: true,
    }
  )
);

Now for the table setup. We need a private user_accounts table and a method named authenticate_user that will return a JWT token of the form:

{
  token: 'jwt_token_here',
  username: '',
  ...anyOtherDetails
}

We will not be detailing table creation or authentication as there are many ways to go about it. But if you need help, Postgraphile security is the page to rely on.

Adding the Plugin library

To attach a cookie to the request, we will use the @graphile/operation-hooks library which is open-sourced on Github.

npm install @graphile/operation-hooks
# OR
yarn add @graphile/operation-hooks

To add the library to the app:

const { postgraphile, makePluginHook } = require("postgraphile");

const pluginHook = makePluginHook([
  require("@graphile/operation-hooks").default,
  // Any more PostGraphile server plugins here
]);

app.use(
  postgraphile(
    process.env.DATABASE_URL || "postgres://user:pass@host:5432/dbname",
    "public",
    {
      watchPg: true,
      graphiql: true,
      enhanceGraphiql: true,
      pluginHook,
      appendPlugins: [
        // You will be adding the hooks here
      ],
    }
  )
);

Adding the Plugin

The plugin allows for two different types of hooks:

1. SQL Hooks
2. JavaScript Hooks

Since accessing cookies is a JavaScript operation, we will be concentrating on the second type.

To hook the plugin into the build system, we can use the addOperationHook method.

module.exports = function OperationHookPlugin(builder) {
  builder.hook("init", (_, build) => {
    // Register our operation hook (passing it the build object):
    // setAuthCookie is a function we will define later.
    build.addOperationHook(useAuthCredentials(build));

    // Graphile Engine hooks must always return their input or a derivative of
    // it.
    return _;
  });
};

If this is contained in a file named set-auth-cookie.js, then the plugin can be added to the append plugins array as follows:

{
  appendPlugins: [
    require('./set-auth-cookie.js'),
  ],
}

Designing the hook

The function to be executed receives two arguments: build process and the current fieldContext.

The fieldContext consists of fields that can be used to narrow down the mutation or query that we want to target; e.g. if the hook is to run only on mutations, we can use the fieldContext.isRootMutation field.

const useAuthCredentials = build => fieldContext => {
  const { isRootMutation } = fieldContext;
  if (!isRootMutation) {
    // No hook added here
    return null;
  }
};

To direct the system on usage of the plugin, we have to return an object with before, after or error fields. Here is how these keywords can be used:

(comments are from the example repository)

return {
  // An optional list of callbacks to call before the operation
  before: [
    // You may register more than one callback if you wish. They will be mixed in with the callbacks registered from other plugins and called in the order specified by their priority value.
    {
      // Priority is a number between 0 and 1000. If you're not sure where to put it, then 500 is a great starting point.
      priority: 500,
      // This function (which can be asynchronous) will be called before the operation. It will be passed a value that it must return verbatim. The only other valid return is `null` in which case an error will be thrown.
      callback: logAttempt,
    },
  ],

  // As `before`, except the callback is called after the operation and will be passed the result of the operation; you may return a derivative of the result.
  after: [],

  // As `before`; except the callback is called if an error occurs; it will be passed the error and must return either the error or a derivative of it.
  error: [],
};

Since we want our action to happen after we get result from the mutation, we will add it to the after array.

const useAuthCredentials = build => fieldContext => {
  const { isRootMutation, pgFieldIntrospection } = fieldContext;
  if (!isRootMutation) {
    // No hook added here
    return null;
  }

  if (
    !pgFieldIntrospection ||
    // Name of the mutation is authenticateUser
    pgFieldIntrospection.name !== "authenticateUser"
  ) {
    // narrowing the scope down to the mutation we want
    return null;
  }

  return {
    before: [],
    after: [
      {
        priority: 1000,
        callback: (result, args, context) => {
          // The result is here, so we can access accessToken and username.
          console.log(result);
        },
      },
    ],
    error: [],
  };
};

Since the functionality is inside the plugin hook, we do not have the express result to set the cookie 😞.

But we do have an escape hatch with the third argument: context. Postgraphile allows us to pass functions or values into the context variable from the postgraphile instance.

app.use(
  postgraphile(process.env.DATABASE_URL, "public", {
    async additionalGraphQLContextFromRequest(req, res) {
      return {
        // Function to set the cookie passed into the context object
        setAuthCookie: function (authCreds) {
          res.cookie("app_creds", authCreds, {
            signed: true,
            httpOnly: true,
            secure: true,
            // Check if you want to include SameSite cookies here, depending on your hosting.
          });
        },
      };
    },
  })
);

We can now set the cookie inside the plugin hook.

{
  priority: 1000,
  callback: (result, args, context) => {
    // This function is passed from additionalGraphQLContextFromRequest as detailed in the snippet above
    context.setAuthCookie(result);
  }
}

Reading from the Cookie 🍪

We have already added the cookieParser with SECRET_KEY, so express will parse the cookies for us.

But we probably want them to be accessible inside SQL functions for Postgraphile. That is how we can determine if the user is signed in or what their permissions are. To do that, Postgraphile provides a pgSettings object.

app.use(
  postgraphile(process.env.DATABASE_URL, "public", {
    pgSettings: async req => ({
      user: req.signedCookies["app_creds"],
    }),
  })
);

Inside an SQL function, the variables passed from settings can be accessed like this:

current_setting('user')

That's all 🎉. We can store any details in cookies, retrieve them on the Express end and use them inside Postgres functions for authentication or authorization.

Check out operation-hooks plugin for more details.

First, let's see Authentication vs Authorization.

Authentication is determining whether a user is logged in or not. Authorization is then deciding what the users has permission to do or see.

Comparing the implementation of a blog application using Postgraphile vs REST

Suppose we have to build a blog application with the below schema.

Features of the blog application.

• Display published blogs with is_published = true to all users.

• Display unpublished blogs with is_published = false to its creator only.

REST Implementation

The REST implementation with JavaScript and sequelize can be like below.

The client requests the blogs using an endpoint, it also attaches the access token received from the authentication service.

const getBlogs = () =>
  requestData({
    endpoint: `/api/blogs`,
    accessToken: "***",
  });

The backend code in the server receives the request, finds the current logged in user from the access token, and requests the data based on the current logged in user from the database.

const userEmail = findEmail(accessToken);
const blogs = await models.Blogs.findAll({
  where: { [Op.or]: [{ creatorEmail: userEmail }, { isPublished: true }] },
});

res.send(blogs);

Here, the backend code finds the user’s email from the access token, then requests the database to give the list of blogs that have creatorEmail matching to the current user's email or the field isPublished is true.

The database will return whatever data the server requests.

Similarly, for creating, editing, and deleting blogs, we can have different end-points to handle the authorization logic in the backend code.

Postgraphile Implementation

The postgraphile implementation can be like below.

The client requests the blogs using a GraphQL query. It also attaches the access token received from the authentication service.

const data = requestQuery({
 query: "allBlogs {
         nodes {
            content
            creatorEmail
            visiblityType
           }
        }"
 accessToken: '***'
})

In the server, we configure Postgraphile to pass the user information to the database.

export postgraphile(DATABASE_URL, schemaName, {
  pgSettings: (req) => {
     const userEmail = findEmail(accessToken);
     return({
         'current_user_email': userEmail
     })
  }
})

We can pass a function as Postgraphile’s pg Settings property, whose return value will be accessible from the connected Postgres database by calling the current_setting function.

In the database, the row-level security policies can be defined to control the data access.

Row-level security policies are basically just SQL that either evaluates to true or false. If a policy is created and enabled for a table, that policy will be checked before doing an operation on the table.

create policy blogs_policy_select
on public.blogs for select to users
USING (
 isPublished OR
 creator_email = current_setting('current_user_email')
);
ALTER TABLE blogs ENABLE ROW LEVEL SECURITY;

Here the policy named blogs_policy_select will be checked before selecting a row in the table public.blogs. A row will be selected only if the isPublished field is true or creator_email matches with the current user's email.

Similarly, for creating, editing, and deleting blogs, we can have row level security policies for INSERT, UPDATE, and DELETE operations on the table.

Conclusion

The REST implementation does the authorization on the server level but the Postgraphile does it on the database level. Each implementation has its own advantages and disadvantages, which is a topic for another day.

Although, sorting based on associated table’s columns or adding a custom sort can be achieved via plugins. In this blog we will explore two such plugins.

Using pg-order-by-related plugin

pg-order-by-related plugin allows us to sort query result based on associated table's columns. It does that by adding enums for all associated table's columns. Here's what we need to do to use this plugin.

Installation

npm i @graphile-contrib/pg-order-by-related

Adding the plugin

const express = require("express");
const { postgraphile } = require("postgraphile");
const PgOrderByRelatedPlugin = require("@graphile-contrib/pg-order-by-related");

const app = express();

app.use(
  postgraphile(process.env.DATABASE_URL, "public", {
    appendPlugins: [PgOrderByRelatedPlugin],
  })
);

Using associated table column enum with orderBy argument

query getPostsSortedByUserId {
  posts: postsList(orderBy: AUTHOR_BY_USER_ID__NAME_ASC) {
    id
    title
    description
    author: authorByUserId {
      id
      name
    }
  }
}

pg-order-by-related plugin is useful only when we want to sort data based on first level association. If we want to apply orderBy on second level table columns or so, we have to use makeAddPgTableOrderByPlugin.

Using makeAddPgTableOrderByPlugin

makeAddPgTableOrderByPlugin allows us to add custom enums that are accessible on specified table's orderBy argument. We can write our custom select queries using this plugin.

We will use a complex example to understand the use-case of custom orderBy enum.

In our posts list query, we want posts to be sorted by author's address. Address has country, state and city columns. We want list to be sorted by country, state and city in the same order.

Here's how we can achieve this using makeAddPgTableOrderByPlugin.

plugins/orderBy/orderByPostAuthorAddress.js

import { makeAddPgTableOrderByPlugin, orderByAscDesc } from "graphile-utils";

export default makeAddPgTableOrderByPlugin(
  "public",
  "post",
  ({ pgSql: sql }) => {
    const author = sql.identifier(Symbol("author"));
    const address = sql.identifier(Symbol("address"));
    return orderByAscDesc(
      "AUTHOR_BY_USER_ID__ADDRESS_ID__COUNTRY__STATE__CITY",
      ({ queryBuilder }) => sql.fragment`(
            SELECT
              CONCAT(
                ${address}.city,
                ', ',
                ${address}.state,
                ', ',
                ${address}.country
              ) AS full_address
            FROM public.user as ${author}
            JOIN public.address ${address} ON ${author}.address_id = ${address}.id
            WHERE ${author}.id = ${queryBuilder.getTableAlias()}.user_id
            ORDER BY ${address}.country DESC, ${address}.state DESC, ${address}.city DESC
            LIMIT 1
          )`
    );
  }
);

Export all custom orderBy plugins

plugins/orderBy/index.js

export { default as orderByPostAuthorAddress } from "./orderByPostAuthorAddress";

Append custom orderBy plugins to postgraphile

const express = require("express");
const { postgraphile } = require("postgraphile");
import * as OrderByPlugins from "./plugins/orderby";

const app = express();

app.use(
  postgraphile(process.env.DATABASE_URL, "public", {
    appendPlugins: [...Object.values(OrderByPlugins)],
  })
);

Using custom enum with orderBy argument

query getPostsSortedByAddress {
  posts: postsList(
    orderBy: AUTHOR_BY_USER_ID__ADDRESS_ID__COUNTRY__STATE__CITY
  ) {
    id
    title
    description
    author: authorByUserId {
      id
      name
      address {
        id
        country
        state
        city
      }
    }
  }
}

Please head to pg-order-by-related and makeAddPgTableOrderByPlugin pages for detailed documentation.

The ssl certificate in this example was bought from namecheap but the same process would apply for other vendors too.

The video of the whole process is available here.

Watch on YouTube

Script to generate keys

openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr

When the prompt asks for Common name(full qualified host name) then enter *.yourdomainname.com. Since we are setting up a wild card certificate it's important that the common name starts with a *. Otherwise later we are going to get an error.

Except the above mentioned question the answer to other questions do not matter at all. You can enter junk values and the SSL will work just fine.

Hit enter when a challenge password is requested.

Script to generate ssl bundle

$ cat __neetohelp_net.crt __neetohelp_net.ca-bundle > ssl-bundle.crt

Note that the order of the crt and bundle files matters when combining them.

Secondly, as shown in the video, we might have to split the combined line. Now let's examine the contents of the combined file.

$ cat ssl-bundle.crt

If we see a line like the one below:

-----END CERTIFICATE----------BEGIN CERTIFICATE-----

Then we need to split the line such that END and BEG align vertically like so:

-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----

We recently got a chance to see workspaces of our colleagues and everyone shared photos of environments they work in on Slack and it was fun seeing everyone's desk and the setup they have. We thought it would be fun to share a peek at the home offices we have. Here we go.

Akhil Gautam

Amit Choudhary

Chimed Palden

Chirag Shah

Ershad Kunnakkadan

Mohit Natoo

Navaneeth PK

Neeraj Singh

Nitin Kalasannavar

Paras Bansal

Pranav Raj

Prathamesh Sonpatki

Rahul Mahale

Rishi Mohan

Shibin Madassery

Sony Mathew

Sunil Kumar

Tyler and Naiara

Unnikrishnan KP

Vishal Telangre

We achieved this to date without having any outbound marketing and sales strategy.

• We have never sent a cold email.
• We have never sent a cold LinkedIn message.
• The only time we advertised was a period of two months when we tried Google advertisements, with no outcomes.
• We do not sponsor any podcast.
• We have not had a sales person.
• We have not had a marketing person.

We have kept our head down and have focused on what we do best, such as designing, developing, debugging, devops, and blogging.

This is what has worked out for us so far:

• We contribute to the community through blog posts and open source.
• We sponsor community events like Rails Girls and Ruby Conf India.
• We sponsor many React and Ruby meetups.
• We focus on keeping our existing clients happy.

Over the years I have come across many people who aspire to be freelancers. While it is not for everyone, I encourage them to give freelancing a try.

The greatest hindrance I have seen is that they stress over sales and marketing, and as it should be. Being a freelancer means constant need to find your next client.

I'm not here to say what others ought to do. I'm here to say what has worked out for BigBinary over the last 7 years.

While we plan to experiment with new forms of marketing, networking, and sales channel as we grow, it is not the end-all-be-all for freelancers. While marketing, networking, and sales may be effective for some, it was not how we started BigBinary and may not be how you want to start as well.

For us at BigBinary, it has been writing blogs. When we come across a potentially intriguing blog topic, we save the topic by creating a Github issue. When we have downtime, we pick up a topic from our issues list. It’s as simple as that and has been our primary driver of growth thus far.

While you should experiment to find out what works best for you, you need to find out what suits your personality. If you are good at teaching through videos, consider creating your own YouTube channel. If you contribute to open source, try creating a blog about your efforts and learnings. If you are good at concentrating on a niche technology, build your marketing and business around that.

I can confidently say that majority of people I met and who want to be freelancer would do fine if they simply share what they are learning. Most of these people do technical work. Some of them already blog and others can blog. A blog is a decent start nearly everybody will say. I'm saying that it is a good end too.

If you do not want to do any other form of marketing then that's fine too. Just blogging will work out fine for you just like it has worked out fine for us at BigBinary.

Just because you are going to be a freelancer you don’t have to change who you are. If you don't like sending cold emails then don't. If you do not like networking then that’s alright as well. Write personal emails, dump corporate talk, show compassion and be genuine.

So go on and do some freelancing. It would teach you a lot about software development, business, life, managing money, creating value and capturing value. It will be rough at times. And it would be hard at times. But it would also be a ton of fun.

The "demo" database is actually a live client where sales team does demo. This ensures that the data that is copied is fresh and not stale.

We implemented a solution where we simply listed all the tables in namespace and used activerecord-import to copy the table data. We used activerecord-import gem to keep code agnostic of underlying database as we used different databases in development from production. Production is "SQL Server" and development database is "PostgreSQL". Why this project ended up having different database in development and in production is worthy of a separate blog.

When we started using the above mentioned strategy then we quickly ran into a problem. Inserts for some tables were failing.

insert or update on table "dependent_table" violates foreign key constraint "fk_rails"
Detail: Key (column)=(1) is not present in table "main_table".

The issue was we had foreign key constraints on some tables and "dependent" table was being processed before the "main" table.

So initially we thought of simply hard coding the sequence in which to process the tables. It means if any new table is added then we will have to update the service to include the newly added table. So we needed a way to identify the foreign key dependencies and determine the sequence to copy the tables at runtime. To resolve this issue, we thought of using Topological Sorting.

Topological Sorting

To get started we need the list of dependencies of "main" and "dependent" tables. In Postgresql, this sql query fetches the table dependencies.

SELECT
    tc.table_name AS dependent_table,
    ccu.table_name AS main_table
FROM
    information_schema.table_constraints AS tc
    JOIN information_schema.key_column_usage AS kcu
      ON tc.constraint_name = kcu.constraint_name
      AND tc.table_schema = kcu.table_schema
    JOIN information_schema.constraint_column_usage AS ccu
      ON ccu.constraint_name = tc.constraint_name
      AND ccu.table_schema = tc.table_schema
WHERE constraint_type = 'FOREIGN KEY'
and (tc.table_name like 'namespace_%' or ccu.table_name like 'namespace_%');

=> dependent_table  | main_table
-----------------------------------
   dependent_table1 | main_table1
   dependent_table2 | main_table2

The above query fetches all the dependencies for only the tables have namespace or the tables we are interested in. The output of above query was [[dependent_table1, main_table1], [dependent_table2, main_table2]].

Ruby has a TSort module that for implementing topological sorts. So we needed to run the topological sort on the dependencies. So we inserted the dependencies into a hash and included the TSort functionality into the hash. Following is the way to include the TSort module into hash by subclassing the Hash.

require "tsort"

class TsortableHash < Hash
  include TSort

  alias tsort_each_node each_key

  def tsort_each_child(node, &block)
    fetch(node).each(&block)
  end
end
# Borrowed from https://www.viget.com/articles/dependency-sorting-in-ruby-with-tsort/

Then we simply added all the tables to dependency hash, as below

tables_to_sort = ["dependent_table1", "dependent_table2", "main_table1"]
dependency_graph = tables_to_sort.inject(TsortableHash.new) {|hash, table| hash[table] = []; hash }

table_dependency_map = fetch_table_dependencies_from_database
=> [["dependent_table1", "main_table1"], ["dependent_table2", "main_table2"]]

# Add missing tables to dependency graph
table_dependency_map.flatten.each {|table| dependency_graph[table] ||= [] }

table_dependency_map.each {|constraint| dependency_graph[constraint[0]] << constraint[1] }

dependency_graph.tsort
=> ["main_table1", "dependent_table1", "main_table2", "dependent_table2"]

The output above, is the dependency resolved sequence of tables.

Topological sorting is pretty useful in situations where we need to resolve dependencies and Ruby provides a really helpful tool TSort to implement it without going into implementation details. Although I did spend time in understanding the underlying algorithm for fun.

If we start puppeteer tests and do nothing in our laptop (just watch the tests being executed) then all the tests will pass.

However if we are doing our regular work in our laptop while tests are running then tests would fail randomly. This was quite puzzling.

Debugging such flaky tests is hard. We first suspected that the test cases themselves needed more of implicit waits for element/text to be present/visible on the DOM.

After some debugging using puppeteer protocol logs, it seemed like the browser was performing certain actions very slowly or was waiting for the browser to be active ( in view ) before performing those actions.

Chrome starting with version 57 introduced throttling of background tabs for improving performance and battery life. We execute one test per browser meaning we didn't make use of multiple tabs. Also tests failed only when the user was performing some other activities while the tests were executing in other background windows. Pages were hidden only when user switched tabs or minimized the browser window containing the tab.

After observing closely we noticed that the pages were making requests to the server. The issue was the page was not painting if the page is not in view. We added flag --disable-background-timer-throttling but we did not notice any difference.

After doing some searches we noticed the flag --disable-renderer-backgrounding was being used in karma-launcher. The comment states that it is specifically required on macOS. Here is the code responsible for lowering the priority of the renderer when it is hidden.

But the new flag didn't help either.

While looking at all the available command line switches for chromium, we stumbled upon --disable-backgrounding-occluded-windows. Chromium also backgrounds the renderer while the window is not visible to the user. It seems from the comment that the flag kDisableBackgroundingOccludedWindowsForTesting is specifically added to avoid non-deterministic behavior during tests.

We have added following flags to chromium for running our integration suite and this solved our problem.

const chromeArgs = [
  "--disable-background-timer-throttling",
  "--disable-backgrounding-occluded-windows",
  "--disable-renderer-backgrounding",
];

References

• Background tabs & offscreen frames
• Mac Window Occlusion API Use

Recently we started working with Elm in a couple of our projects - APISnapshot and AceHelp.

Tools like prettier and rubocop have really helped us take a load off our mind with regards to formatting code. And one of the very first things we wanted to sort out when we started doing Elm was pretty printing our Elm code.

elm-format created by Aaron VonderHaar formats Elm source code according to a standard set of rules based on the official Elm Style Guide.

Automatic code formatting

Let's setup git hook to automatically take care of code formatting. We can achieve this much like how we did it in our previous post, using Husky and Lint-staged.

Let's add Husky and lint-staged as dev dependencies to our project. And for completeness also include elm-format as a dev dependency.

npm install --save-dev husky lint-staged elm-format

Husky makes it real easy to create git hooks. Git hooks are scripts that are executed by git before or after an event. We will be using the pre-commit hook which is run after you do a git commit command but before you type in a commit message.

This way we can change and format files that's about to be committed by running elm-format using Husky.

But there is one problem here. The changed files do not get added back to our commit.

This is where Lint-staged comes in. Lint-staged is built to run linters on staged files. So instead of running elm-format on a pre-commit hook we would run lint-staged. And we can configure lint-staged such that elm-format is run on all staged elm files.

We can also include Prettier to take care of all staged JavaScript files too.

Lets do this by editing our package.json file.

{
  "scripts": {
    "precommit": "lint-staged"
  },
  "lint-staged": {
    "*.elm": ["elm-format --yes", "git add"],
    "*.js": ["prettier --write", "git add"]
  }
}

All set and done!

Now whenever we do a git commit command, all our staged elm and JavaScript files will get properly formatted before the commit goes in.

So, we thought about automating it with CircleCI, similar to continuous deployment.

We are using the following configuration in circle.yml to continuously release chrome extensions from the master branch.

workflows:
  version: 2
  main:
    jobs:
      - test:
          filters:
            branches:
              ignore: []
      - build:
          requires:
            - test
          filters:
            branches:
              only: master
      - publish:
          requires:
            - build
          filters:
            branches:
              only: master

version: 2
jobs:
  test:
    docker:
      - image: cibuilds/base:latest
    steps:
      - checkout
      - run:
          name: "Install Dependencies"
          command: |
            apk add --no-cache yarn
            yarn
      - run:
          name: "Run Tests"
          command: |
            yarn run test
  build:
    docker:
      - image: cibuilds/chrome-extension:latest
    steps:
      - checkout
      - run:
          name: "Install Dependencies"
          command: |
            apk add --no-cache yarn
            apk add --no-cache zip
            yarn
      - run:
          name: "Package Extension"
          command: |
            yarn run build
            zip -r build.zip build
      - persist_to_workspace:
          root: /root/project
          paths:
            - build.zip

  publish:
    docker:
      - image: cibuilds/chrome-extension:latest
    environment:
      - APP_ID: <APP_ID>
    steps:
      - attach_workspace:
          at: /root/workspace
      - run:
          name: "Publish to the Google Chrome Store"
          command: |
            ACCESS_TOKEN=$(curl "https://accounts.google.com/o/oauth2/token" -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&refresh_token=${REFRESH_TOKEN}&grant_type=refresh_token&redirect_uri=urn:ietf:wg:oauth:2.0:oob" | jq -r .access_token)
            curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -X PUT -T /root/workspace/build.zip -v "https://www.googleapis.com/upload/chromewebstore/v1.1/items/${APP_ID}"
            curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -H "Content-Length: 0" -X POST -v "https://www.googleapis.com/chromewebstore/v1.1/items/${APP_ID}/publish"

We have created three jobs named as test, build and publish and used these jobs in our workflow to run tests, build the extension, and publish them to the chrome store, respectively. Every step requires the previous step to run successfully.

Let's check each job one by one.

test:
  docker:
    - image: cibuilds/base:latest
  steps:
    - checkout
    - run:
        name: "Install Dependencies"
        command: |
          apk add --no-cache yarn
          yarn
    - run:
        name: "Run Tests"
        command: |
          yarn run test

We use cibuilds docker image for this job. First, we do a checkout to the branch and then use yarn to install dependencies. Alternatively, we can use npm to install dependencies as well. Then, as the last step, we are use yarn run test to run tests. We can skip this step if running tests is not needed.

build:
  docker:
    - image: cibuilds/chrome-extension:latest
  steps:
    - checkout
    - run:
        name: "Install Dependencies"
        command: |
          apk add --no-cache yarn
          apk add --no-cache zip
          yarn
    - run:
        name: "Package Extension"
        command: |
          yarn run build
          zip -r build.zip build
    - persist_to_workspace:
        root: /root/project
        paths:
          - build.zip

For building chrome extensions, we use the chrome-extension image. Here, we also first do a checkout and then, install dependencies using yarn. Note, we are install zip utility along with yarn because we need to zip our chrome extension before publishing it in next step. Also, we are not generating version numbers on our own. The version number will be picked from the manifest file. This step assumes that we have a task named build in package.json to build our app.

The Chrome store rejects multiple uploads with the same version number. So, we have to make sure to update the version number, which should be unique in the manifest file before this step.

In the last step, we use persist_to_workspace to make build.zip available for the next step, publishing.

publish:
  docker:
    - image: cibuilds/chrome-extension:latest
  environment:
    - APP_ID: <APP_ID>
  steps:
    - attach_workspace:
        at: /root/workspace
    - run:
        name: "Publish to the Google Chrome Store"
        command: |
          ACCESS_TOKEN=$(curl "https://accounts.google.com/o/oauth2/token" -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&refresh_token=${REFRESH_TOKEN}&grant_type=refresh_token&redirect_uri=urn:ietf:wg:oauth:2.0:oob" | jq -r .access_token)
          curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -X PUT -T /root/workspace/build.zip -v "https://www.googleapis.com/upload/chromewebstore/v1.1/items/${APP_ID}"
          curl -H "Authorization: Bearer ${ACCESS_TOKEN}" -H "x-goog-api-version: 2" -H "Content-Length: 0" -X POST -v "https://www.googleapis.com/chromewebstore/v1.1/items/${APP_ID}/publish"

For publishing of the chrome extension, we use the chrome-extension image.

We need APP_ID, CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN/ACCESS_TOKEN to publish our app to the chrome store.

APP_ID needs to be fetched from Google Webstore Developer Dashboard. APP_ID is unique for each app whereas CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN/ACCESS_TOKEN can be used for multiple apps. Since APP_ID is generally public, we specify that in the yml file. CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN/ACCESS_TOKEN are stored as private environment variables using CircleCI UI. For cases when our app is unlisted in the chrome store, we need to store APP_ID as a private environment variable.

CLIENT_ID and CLIENT_SECRET need to be fetched from Google API console. There, we need to select a project and then click on the credentials tab. If there is no project, we need to create one and then access the credentials tab.

REFRESH_TOKEN needs to be fetched from Google API. It also defines the scope of access for Google APIs. We need to refer to Google OAuth2 for obtaining the refresh token. We can use any language library.

In the first step of the publish job, we are attaching a workspace to access build.zip, which was created previously. Now, by using all the required tokens obtained previously, we need to obtain an access token from Google OAuth API (Link is not available), which must be used to push the app to the chrome store. Then, we make a PUT request to the Chrome store API to push the app, and then use the same API again, to publish the app.

Uploading via API has one more advantage over manual upload. Manual upload generally takes up to 1 hour to show the app in the chrome store. Whereas uploading using Google API generally reflects the app within 5-10 minutes, considering app does not go for a review by Google.

Apple announced iOS 12 at WWDC 2018 a few days back. Being honest, it was a bit disappointing to see some of the most requested features not being there in iOS 12. Like users have been asking for Dark mode since before iOS 11, and the ability to set default apps. It’s more of an update focussed on performance improvements, usability, and compatibility. The fact that iOS 12 is also available for the iPhones Apple released 5 years back is a great effort from Apple to keep users happy. And unlike last couple of years, this time we decided to calm our curiosities and installed iOS 12 beta 1 on our devices right away after Apple released it for developers. This blog is based on our experience with iOS 12 on iPhone 8 Plus.

Installing iOS 12 on your iPhone

First things first, make sure you have iPhone 5s or newer. And before getting started, plug-in your phone to iTunes and take a full-backup in case your phone gets bricked while installing iOS 12, which is very unlikely.

Once done, download and install beta profile{:target="_blank"} for iOS 12 and then download and update from “Software Update” section just like you install a regular iOS update. It’s a straightforward OTA update process which you’re already familiar with.

Note: This beta profile is from a third-party developer and is not officially from Apple. Apple will officially release public beta in around a month.

We’ve been running iOS 12 since last week now and here are our thoughts on the additions and changes introduced in iOS 12.

iOS 12 is fast

The performance improvements are significant and definitely noticeable. Previously on iOS 11, while accessing spotlight by swiping down from home screen, it used to lag. And not just that, sometimes keyboard didn’t even used to show up and we had to repeat the same action to make it work. But things are faster and better in iOS 12. The keyboard gets up as soon as spotlight shows up.

Another thing that we’ve noticed is the multitasking shortcut for switching to the last app by 3d touching on left wasn't that reliable in iOS 11, so much that it was easy to ignore the feature altogether than to use it, but in iOS 12 the same shortcut is very smooth. Although there are times when it still doesn’t work well, but that's very rare.

Spotlight widgets load faster in iOS 12 than they used to in iOS 11. Apart from this, 3d touch feels pretty smooth too. It’s good to see Apple squeezing out the power to improve the already good performance in iOS.

Notifications

Notifications in iOS 11 are a mess, there’s no grouping, there’s no way to quickly control notifications, there’s no way to set priorities. Apple has added notifications grouping and better notification management in iOS 12, so now notifications from the same app are grouped together and you can control how often you want to get notifications from the apps right from the notification.

We think the implementation can be a whole lot better. For the notifications that are grouped, you get to see only the last notification from that app, a better way would’ve been to show two or three notifications and cut the rest. There’s no Notification pinning or snoozing which could've been very useful features.

Screen time and App limits

There’s a new feature in iOS 12 called Screen Time which is more like bird's-eye view of your phone usage. There’s a saying that you can’t improve something that you can’t measure. Screen Time is a feature that’s going to be very useful for everyone who wants to cut down time on Social apps or overall phone usage. It shows you every little detail of how many apps you use and for how much time and at what times. Not only this, it also keeps track of how many times you picked up your phone, and how many notifications you receive from the apps you have on your phone.

Other useful sub-feature of Screen time is App limits, which allows you to set limit on app usage based on app or category. So let’s say you don’t want to use WhatsApp for more than 30 mins a day, you can do that through App limits. It works for app categories including Games, Social Networking, Entertainment, Creativity, Productivity, Education, Reading & Reference, Health & Fitness. So you can limit it via category which works across apps. Plus, it syncs across your other iOS devices, so you can’t cheat that way.

Siri and Shortcuts app

In iOS 12, you can assign custom shortcuts to Siri to trigger specific actions, which not only works for System apps but also with third-party apps. So now if you want to send a specific message to someone on WhatsApp, you can assign a command for that to Siri and you can trigger that action just from Siri using that command.

Apple has also introduced a new app in iOS 12 called Shortcuts. Shortcuts app lets you group actions and run those actions quickly. Although Shortcuts app isn’t there in iOS 12 beta 1 but we think it’s one of best addition in iOS 12.

Updated Photos app

Photos app now has new section called "For you", where it shows all the new Albums, your best moments, share suggestions, photos and effect suggestions. This is more like the Assistant tab of Google Photos app. Also you can share selected photos or albums with your friends right from the Photos app.

The Album tab in Photos app is redesigned for easier navigation. Also there's a new tab for Search which has been advanced, so you can now search photos using tags like "surfing" and "vacation".

It's good to see Apple paying attention to up the Photos app but we still think Google Photos is a better option for average person considering it lets you store photos in Cloud for free. Also photo organization in Google Photos is much better than in new Photos app in iOS 12.

Enhanced Do Not Disturb Mode

Do Not Disturb in iOS 12 is enhanced to be more flexible. You can now enable Do Not Disturb mode to end automatically in an hour, or at night, or according to your Calendar events, or even based on your location.

Not just that, Do Not Disturb has a new Bedtime mode enabling which will shut all your notifications during bed time and dim your display. And when you will wake up, it'll show you a welcome back message along with weather details on the lock screen.

Conclusion

There are other updates and under the hood improvements as well, like new Measure app, redesigned iBooks app, tracking prevention, group FaceTime etc. Overall, we think it’s an okay update considering there are not as many bugs as there should be according to Apple standards. The force touch in the keyboard to drag cursor doesn’t work, Skype and some other apps crash, but for the most part, it’s good enough to be installed on your primary device.

One of the features of ApiSnapshot is support for Basic Authentication.

While we were rebuilding the whole application in Elm, we had to port the "Add Basic Authentication" feature. This feature can be accessed from the "More" drop-down on the right-hand side of the app and it lets user add username and password to the request.

Let's see how the Model of our Elm app looks.

type alias Model =
{ request : Request.MainRequest.Model
, response : Response.MainResponse.Model
, route : Route
}

Here is the Model in Request.MainRequest module.

type alias APISnapshotRequest =
{ url : String
, httpMethod : HttpMethod
, requestParameters : RequestParameters
, requestHeaders : RequestHeaders
, username : Maybe String
, password : Maybe String
, requestBody : Maybe RequestBody
}

type alias Model =
{ request : APISnapshotRequest
, showErrors : Bool
}

username and password fields are optional for the users so we kept them as Maybe types.

Note that API always responds with username and password whether user clicked to add Basic Authentication or not. The API would respond with a null for both username and password when a user tries to retrieve a snapshot for which user did not fill username and password.

Here is a sample API response.

{
  "url": "http://dog.ceo/api/breed/affenpinscher/images/random",
  "httpMethod": "GET",
  "requestParams": {},
  "requestHeaders": {},
  "requestBody": null,
  "username": "alanturning",
  "password": "welcome",
  "assertions": [],
  "response": {
    "response_headers": {
      "age": "0",
      "via": "1.1 varnish (Varnish/6.0), 1.1 varnish (Varnish/6.0)",
      "date": "Thu, 03 May 2018 09:43:11 GMT",
      "vary": "",
      "cf_ray": "4151c826ac834704-EWR",
      "server": "cloudflare"
    },
    "response_body": "{\"status\":\"success\",\"message\":\"https:\\/\\/images.dog.ceo\\/breeds\\/affenpinscher\\/n02110627_13221.jpg\"}",
    "response_code": "200"
  }
}

Let's look at the view code which renders the data received from the API.

view : (Maybe String, Maybe String) -> Html Msg
view usernameAndPassword =
case usernameAndPassword of
(Nothing, Nothing) -> text ""
(Just username, Nothing) -> basicAuthenticationView username ""
(Nothing, Just password) -> basicAuthenticationView "" password
(Just username, Just password) -> basicAuthenticationView username password

basicAuthenticationView : String -> String -> Html Msg
basicAuthenticationView username password =
[ div [ class "form-row" ]
[ input
[ type_ "text"
, placeholder "Username"
, value username
, onInput (UpdateUsername)
]
[]
, input
[ type_ "password"
, placeholder "Password"
, value password
, onInput (UpdatePassword)
]
[]
, a
[ href "javascript:void(0)"
, onClick (RemoveBasicAuthentication)
]
[ text "×" ]
]
]

To get the desired view we apply following rules.

1. Check if both the values are string.
2. Check if either of the values is string.
3. Assume that both the values are null.

This works but we can do a better job of modelling it.

What's happening here is that we were trying to translate our API responses directly to the Model . Let's try to club username and password together into a new type called BasicAuthentication.

In the model add a parameter called basicAuthentication which would be of type Maybe BasicAuthentication. This way if user has opted to use basic authentication fields then it is a Just BasicAuthentication and we can show the input boxes. Otherwise it is Nothing and we show nothing!

Here is what the updated Model for Request.MainRequest would look like.

type alias BasicAuthentication =
{ username : String
, password : String
}

type alias APISnapshotRequest =
{ url : String
, httpMethod : HttpMethod
, requestParameters : RequestParameters
, requestHeaders : RequestHeaders
, basicAuthentication : Maybe BasicAuthentication
, requestBody : Maybe RequestBody
}

type alias Model =
{ request : APISnapshotRequest
, showErrors : Bool
}

Elm compiler is complaining that we need to make changes to JSON decoding for APISnapshotRequest type because of this change.

Before we fix that let's take a look at how JSON decoding is currently being done.

import Json.Decode as JD
import Json.Decode.Pipeline as JP

decodeAPISnapshotRequest : Response -> APISnapshotRequest
decodeAPISnapshotRequest hitResponse =
let
result =
JD.decodeString requestDecoder hitResponse.body
in
case result of
Ok decodedValue ->
decodedValue

            Err err ->
                emptyRequest

requestDecoder : JD.Decoder APISnapshotRequest
requestDecoder =
JP.decode Request
|> JP.optional "username" (JD.map Just JD.string) Nothing
|> JP.optional "password" (JD.map Just JD.string) Nothing

Now we need to derive the state of the application from our API response .

Let's introduce a type called ReceivedAPISnapshotRequest which would be the shape of our old APISnapshotRequest with no basicAuthentication field. And let's update our requestDecoder function to return a Decoder of type ReceivedAPISnapshotRequest instead of APISnapshotRequest.

type alias ReceivedAPISnapshotRequest =
{ url : String
, httpMethod : HttpMethod
, requestParameters : RequestParameters
, requestHeaders : RequestHeaders
, username : Maybe String
, password : Maybe String
, requestBody : Maybe RequestBody
}

requestDecoder : JD.Decoder ReceivedAPISnapshotRequest

We need to now move our earlier logic that checks to see if a user has opted to use the basic authentication fields or not from the view function to the decodeAPISnapshotRequest function.

decodeAPISnapshotRequest : Response -> APISnapshotRequest
decodeAPISnapshotRequest hitResponse =
let
result =
JD.decodeString requestDecoder hitResponse.body
in
case result of
Ok value ->
let
extractedCreds =
( value.username, value.password )

                    derivedBasicAuthentication =
                        case extractedCreds of
                            ( Nothing, Nothing ) ->
                                Nothing

                            ( Just receivedUsername, Nothing ) ->
                                Just { username = receivedUsername, password = "" }

                            ( Nothing, Just receivedPassword ) ->
                                Just { username = "", password = receivedPassword }

                            ( Just receivedUsername, Just receivedPassword ) ->
                                Just { username = receivedUsername, password = receivedPassword }
                in
                    { url = value.url
                    , httpMethod = value.httpMethod
                    , requestParameters = value.requestParameters
                    , requestHeaders = value.requestHeaders
                    , basicAuthentication = derivedBasicAuthentication
                    , requestBody = value.requestBody
                    }

            Err err ->
                emptyRequest

We extract the username and password into extractedCreds as a Pair from ReceivedAPISnapshotRequest after decoding and construct our APISnapshotRequest from it.

And now we have a clean view function which just takes a BasicAuthentication type and returns us a Html Msg type.

view : BasicAuthentication -> Html Msg
view b =
[ div [ class "form-row" ]
[ input
[ type_ "text"
, placeholder "Username"
, value b.username
, onInput (UpdateUsername)
]
[]
, input
[ type_ "password"
, placeholder "Password"
, value b.password
, onInput (UpdatePassword)
]
[]
, a
[ href "javascript:void(0)"
, onClick (RemoveBasicAuthentication)
]
[ text "×" ]
]
]

We now have a Model that better captures the business logic. And should we change the logic of basic authentication parameter selection in the future, We do not have to worry about updating the logic in the view .

We built APISnapshot for two reasons.

We wanted to work with Elm.

We wanted to have a tool that is easy to use and that will help us capture what response we are getting from the API in a format that is easy to share in github issue, in slack or in an email. As a consulting company we work with various teams around the world and during development phase either API is unstable or they do not do what they should be doing.

We originally built this tool using React. Elm compiler is quite strict and forced us to take into consideration all possibilities. This lead us to notice a few bugs which were still present in React code. In this way Elm compiler helped us produce "correct" software by eliminating some of the bugs that we would have found later.

JSON encoding/decoding is a hard problem in Elm in general. In most of the cases we know the shape of the API response we are going to get.

In the case of APISnapshot we do not know the shape of the JSON response we will get. Because of that it took us a bit longer to build the application. However, this forced us to really dig deep into JSON encoding/decoding issue in Elm and we learned a lot.

We would like to thank all the contributors to the project. Special shout out goes to Jasim for the discussion and the initial work on parsing the JSON file.

We like the combination of Elm and Ruby on Rails so much so that we are building AceHelp using the same technologies. AceHelp is open source from day one.

Since puppeteer scripts execute so fast certain tests fail when they should be passing. Debugging those tests can be a challenge.

Chrome devtools comes with filmstrip feature. In "Performance" tab we can see screenshots of the site as they change over time.

We wanted puppeteer to generate similar filmstrip so that we can visually identify the source of the problem.

It turns out that puppeteer makes it very easy. Here is the full code.

import puppeteer from "puppeteer";

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  await page.setViewport({ width: 1280, height: 1024 });

  await page.tracing.start({ path: "trace.json", screenshots: true });

  await page.goto("https://www.bigbinary.com");
  await Promise.all([
    page.waitForNavigation(),
    page.click("#navbar > ul > li:nth-child(1) > a"),
  ]);

  await page.tracing.stop();

  await browser.close();
})();

If we execute this script then it will generate a file called trace.json. This file has images embedded in it which are base64 encoded.

To see the filmstrip drag the trace.json file to "Performance" tab in the Chrome devtool. Here is a quick video explaining this.

Watch on YouTube

An identity function is a very basic function that

• takes one argument
• returns the argument

f(x) = x;

This seems like the most useless function in the world. We never needed any function like this while building any application. Then what's the big deal about this identity function.

In this blog we will see how this identity concept is used in the real world.

For the implementation we will be using Ramda.js. We previously wrote about how we, at BigBinary, write JavaScript code using Ramda.js.

Again please note that in the following code R stands for Ramda and not for programming language R.

Example 1

Here is JavaScript code.

if (x) return x;
return [];

Here is same code using Ramda.js.

R.ifElse(R.isNil, () => [], R.identity);

try it

Example 2

Here we will use identity as the return value in the default case.

R.cond([
  [R.equals(0), R.always("0")],
  [R.equals(10), R.always("10")],
  [R.T, R.identity],
]);

try it

Example 3

Get the unique items from the list.

R.uniqBy(R.identity, [1, 1, 2]);

try it

Example 4

Count occurrences of items in the list.

R.countBy(R.identity, ["a", "a", "b", "c", "c", "c"]);

try it

Example 5

Begin value from zero all the way to n-1.

R.times(R.identity, 5);

try it

Example 6

Filter truthy values.

R.filter(R.identity, [
  { a: 1 },
  false,
  { b: 2 },
  true,
  "",
  undefined,
  null,
  0,
  {},
  1,
]);

try it

After researching we found that JWPlayer makes an AJAX request to load the m3u8 file. To fix the issue, we needed to enable CORS and for that we needed to make changes to S3 and Cloudfront configurations.

S3 configuration changes

We can configure CORS for the S3 bucket by allowing requests originating from specified hosts. As show in the image below we can find the CORS configuration option in Permissions tab of the S3 bucket. Here is the official documentation on configuring CORS for S3.

S3 bucket will now allow requests originating from the specified hosts.

Cloudfront configuration changes

Cloudfront is a CDN service provided by AWS which uses edge locations to speed up the delivery of static content. Cloudfront takes content from S3 buckets and caches it at edge locations and delivers it to the end user. For enabling CORS we need to configure Cloudfront to allow forwarding of required headers.

We can configure the behavior of Cloudfront by clicking on Cloudfront Distribution's "Distribution Settings". Then from the "Behaviour" tab click on "Edit". Here we need to whitelist the headers that need to be forwarded. Select the "Origin" header to whitelist which is required for CORS, as shown in the image below.

After attending the conference, I can say that Elm community is growing strong. The conference was attended by around 350 people and many were using Elm in production. More number of people wanted to try Elm in production.

There was a lot of enthusiasm about starting new Elm meetups. As a Ruby on Rails and React meetup organizer myself, I was genuinely interested in hearing experiences of seasoned meetup organizers. In general Evan and Richard prefer meetup to be a place where people form small groups and hack on something rather than one person teaching the whole group something.

I liked all the talks. There was variety in the topics and the speakers were all seasoned. Kudos to the organizers for putting up a great program. Below is a quick summary of my thoughts from the conference.

Keynote by Evan

Evan talked about the work he has been doing for the upcoming release of Elm. He discussed the optimization work related to code splitting, code generation and minification for speeding up building and delivering single page apps using Elm. He made another interesting point that he changed the codegen which generates the JS code from Elm code twice but nobody noticed it. Things like this can give a huge opportunity to change and improve existing designs which he has been doing for the upcoming release.

In the end he mentioned that his philosophy is not to rush things. It's better to do things right than doing it now.

After the keynote, he encouraged people to talk to him about what they are working on which was really nice.

Accessibility with Elm

Tessa talked about her work around adding accessibility support for Elm apps. She talked about design decisions, prior art and some of the challenges she faced while working on the library like working with tabs, interactive elements and images. There was a question at the end about whether this will be incorporated into Elm core but Evan mentioned that it might take some time.

Putting the Elm Platform in the Browser

Luke, the creator of Ellie - a way to easily share your elm code with others online - talked about how he started with Ellie. He talked about the problems he had to face for implementing and sustaining Ellie through ads. During the talk, he also open sourced the code, so we can see it on Github now.

Luke mentioned how he changed the architecture of Ellie from mostly running on the server to running in the browser using service workers. He discussed future plans about sustaining Ellie, building an Elm editor instead of using Codemirror, getting rid of ads and making Ellie better for everyone.

The Importance of Ports

In other frameworks like PureScript and BuckleScript invoking native JavaScript functions is easy. In Elm one has to use "Ports". Using Ports requires some extra stuff. In return we get more safety.

Murphy Randle presented a case where he was using too many ports which was resulting in fragmented code. He discussed how port is based on Actor Model and once we get that then using port would be much easier. He also showed refactored code.

Murphy also runs Elm Town Podcast (Link is not available). Listen to episode 13 to know more about Ports.

Keynote by Richard Feldman

Richard talked about his experiences in teaching beginners about Elm. He has taught Elm a lot. He has done an extensive Elm course on Front end masters. He is currently writing Elm in Action book.

He talked about finding motivation to teach using the SWBAT technique. It helped him in deciding the agenda and finding the direct path for teaching. He mentioned that in the beginning being precise and detailed is not important. This resonated with me as the most important thing for anyone who is getting started is getting started with the most basic things and then iterating over it again and again.

Parting thoughts

Elm community is small, tight, very friendly and warm. Lots of people are trying a lot of cool things. Elm Slack came in the discussions again and again as a good place to seek out help for beginners.

When I heard about Elm first, it was about good compiler errors and having run time safety. However after attending the conference I am mighty impressed with the Elm community.

Big props to Brian and Luke for organizing the conference!

All the videos from the conference are already getting uploaded here.

Elm FAQ has an answer to this question. However I could not fully understand the answer.

This is my attempt in explaining it.

What is type

In Elm everything has a type. Fire up elm-repl and you will see 4 is a number and "hello" is a String.

> 4
4 : number

> "hello"
"hello" : String

Let's assume that we are working with users records and we have following attributes of those users.

• Name
• Age
• Status (Active or Inactive)

It's pretty clear that "Name" should be of type "String" and "Age" should be of type "number".

Let's think about a moment what is the type of "Status". What is "Active" and "Inactive" in terms of type.

Active and Inactive are two valid values of Status. In other programming languages we might represent Status as an enum.

In Elm we need to create a new type. And that can be done as shown here.

type Status = Active | Inactive

Second thing we are doing is that we are stating that the valid values for this new type are Active and Inactive.

When I discussed this code with my team members they asked me to show where is Active and Inactive defined. Good question.

The simple answer is that they are not defined anywhere. They do not need to be defined. What needs definition is the new type that is being created.

What makes understanding it a bit hard for people coming from Ruby, Java and such background is that these people (including me) are looking at Active and Inactive as a class or a constant which is not the right way to look at.

Active and Inactive are the valid values for type Status.

> Active
-- NAMING ERROR ----------

Cannot find variable `Active`

3|   Active
     ^^^^^^

As you can see repl is not sure what Active is.

We can solve this by pasting following code in repl.

type Status = Active | Inactive

Now we can run the same code again. This time no error.

> Active
Active : Repl.Status

What is type alias

Let's see a simple application which just prints name and age of a single user.

Here is the code. I'm posting screenshot of the same below with certain part highlighted.

As you can see { name : String, age : Int } is repeated at four different places. In a bigger application it might get repeated more often.

This is what type alias does. It removes repetition. It removes verbosity.

As the name suggests this is just an alias. Note that type creates a new type whereas type alias is literally saving keystrokes. type alias does not create a new type.

Now if you read the FAQ answer again then hopefully it will make morse sense now.

Here is modified code using type alias.

Why use type alias Username : String

While browsing Elm code in general, I came across following code.

type alias Username = String

Question is what does code like this buy us. All it does is that instead of String I can now type Username.

First let's see how it might be used.

Let's assume that we have a function which returns Status of a user for the given username.

The function might have implementation as shown below.

getUserStatus username =
  make_db_call_and_return_user_status

Now let's think about what the type annotation (rubyist think of it as method signature ) of function getUserStatus might look like.

It takes username as input and returns user record.

So the type annotation might look like

getUserStatus : String -> Status

This works. However the issue is that String is not expressive enough. It can be made more expressive if the signature were

getUserStatus : Username -> Status

Now that we know about type alias all we need to do is

type alias Username = String

This makes code more expressive.

No recursion with type alias

An example of where we might need recursion is while designing commenting system. A comment can have sub-comments. However since type alias is just a substitution and recursion does not work with it.

> type alias Comment = { message : String, responses : List Comment }

This type alias is recursive, forming an infinite type!

2| type alias Comment = { message : String, responses : List Comment }
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When I expand a recursive type alias, it just keeps getting bigger and bigger.
So dealiasing results in an infinitely large type! Try this instead:

    type Comment
        = Comment { message : String, responses : List Comment }

This is kind of a subtle distinction. I suggested the naive fix, but you can
often do something a bit nicer. So I would recommend reading more at:
<https://github.com/elm-lang/elm-compiler/blob/0.18.0/hints/recursive-alias.md>

Hint for Recursive Type Aliases discusses this issue in greater detail and it also has solution to the problem of recursion.

Dual role of type alias as constructor and type

Let's say that we have following code.

type alias UserInfo =
    { name : String, age : Int }

Now we can use UserInfo as a constructor to create records.

> type alias UserInfo = { name : String, age : Int }
> sam = UserInfo "Sam" 24
{ name = "Sam", age = 24 } : Repl.UserInfo

In the above case we used UserInfo as a constructor to create new user records. We did not use UserInfo as a type.

Now let's see another function.

type alias UserInfo =
    { name : String, age : Int }


getUserAge : UserInfo -> Int
getUserAge userinfo =
    userinfo.age

In this case UserInfo is being used in type annotation as type and not as constructor.

Which one to use type or type alias

Both of them serve different purpose. Let's see an example.

Let's say that we have following code.

type alias UserInfo =
    { name : String, age : Int }

type alias Coach =
    { name : String, age : Int, sports : String }

Now let's write a function that gets age of the given userinfo.

getUserAge : UserInfo -> Int

getUserAge UserInfo =
    UserInfo.age

Now let's create two types of users.

sam = UserInfo "Sam" 24
charlie = Coach "Charlie" 52 "Basketball"

Now let's try to get age of both of these people.

getUserAge sam
getUserAge charlie

Here is the complete version if you want to run it.

Please note that elm-repl does not support type annotation so you can't test this code in elm-repl.

The main point here is that since we used type alias, function getUserAge works for both UserInfo as well as Coach. It would be a stretch to say that this sounds like "duck typing in Elm" but it comes pretty close.

Yes Elm is statically typed language and it enforces type. However the point here is the type alias is not exactly a type.

So why did this code work.

It worked because of Elm's support for pattern matching for records.

As mentioned earlier type alias is just a shortcut for typing the verbose version. So let's expand the type annotation of getUserAge.

If we were not using type alias UserInfo then it might have looked like as shown below.

getUserAge : { name : String, age : Int } -> Int

Here the argument is a record. Here is official guide on Records. While dealing with records Elm looks at the argument and if that argument is a record and has all the matching attributes then Elm will not complain because of its support for pattern matching.

Since Coach has both name and age attribute getUserAge charlie works.

You can test it by removing the attribute age from Coach and then you will see that Compiler will complain.

In summary if we want strict type enforcement then we should go for type. If we need something so that we do not need to type all the attributes all the time and we want pattern matching then we should go for type alias.

It's here and it looks like this.

length : String -> Int

> String.length "Hello World"
11 : Int

If we look at the similar feature in Ruby world then we get length method.

length -> integer

Method signature in ruby's documentation and Elm's documentation is quite similar. Both return an integer.

In Elm's world method definitions are called "Type Annotations". Going forward that's what I'm going to use in this blog.

Now let's look at method definition of slice method in Ruby.

It looks like this.

slice(start, length) -> new_str or nil

irb(main):006:0> "snakes on a plane!".slice(0,6)
=> "snakes"

In Elm world it looks like this.

slice : Int -> Int -> String -> String

> String.slice  0  6 "snakes on a plane!"
"snakes" : String

Questions is what's up with all those arrows.

$ netstat -nlt
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:3737          0.0.0.0:*               LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN

As we can see above port 5432 is bound to 127.0.0.1. It means any attempt to connect to the postgresql server from outside the machine will be refused. We can try hitting the port 5432 by using telnet.

$ telnet 107.170.11.79 5432
Trying 107.170.11.79...
telnet: connect to address 107.170.11.79: Connection refused
telnet: Unable to connect to remote host

Configuring postgresql.conf

In order to fix this issue we need to find postgresql.conf. In different systems it is located at different place. I usually search for it.

$ find / -name "postgresql.conf"
/var/lib/pgsql/9.4/data/postgresql.conf

Open postgresql.conf file and replace line

listen_addresses = 'localhost'

with

listen_addresses = '*'

Now restart postgresql server.

$ netstat -nlt
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:2812          0.0.0.0:*               LISTEN
tcp6       0      0 ::1:11211               :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::5432                 :::*                    LISTEN
tcp6       0      0 ::1:25                  :::*                    LISTEN

Here we can see that "Local Address" for port 5432 has changed to 0.0.0.0.

Configuring pg_hba.conf

Let's try to connect to remote postgresql server using "psql".

$ psql -h 107.170.158.89 -U postgres
psql: could not connect to server: Connection refused
	Is the server running on host "107.170.158.89" and accepting
	TCP/IP connections on port 5432?

In order to fix it, open pg_hba.conf and add following entry at the very end.

host    all             all              0.0.0.0/0                       md5
host    all             all              ::/0                            md5

The second entry is for IPv6 network.

Do not get confused by "md5" option mentioned above. All it means is that a password needs to be provided. If you want client to allow collection without providing any password then change "md5" to "trust" and that will allow connection unconditionally.

Restart postgresql server.

$ psql -h 107.170.158.89 -U postgres
Password for user postgres:
psql (9.4.1, server 9.4.5)
Type "help" for help.

postgres=# \l

You should be able to see list of databases.

Now we are able to connect to postgresql server remotely.

Please note that in the real world you should be using extra layer of security by using "iptables".

We added more clients. We increased our team size. We wrote more blogs, spoke at more conferences, and made even more videos !

Here is the breakdown.

We love conferences

We presented at 10 conferences across 7 countries, on topics from Rails to ReactJS.

• Garden City RubyConf, India
• RubyConf, Philippines
• RubyConf, India
• Reddot RubyConf, Singapore
• Deccan RubyConf, India
• Full Stack Fest, Spain
• RubyConf, Taiwan
• Rocky Mountain Ruby, Colorado
• RubyConf, Colombia

We are all in ReactJS

We at BigBinary adopted ReactJS pretty early. Early in the year we published series of videos titled Learn ReactJS in steps which takes a "Hello World" app into a full TODO application using ReactJS in incremental steps.

Vipul and Prathamesh are currently authoring a book on ReactJS. Check it out at ReactJS by Example- Building Modern Web Applications with React Book

We also started an ios app using React Native. It's coming along pretty good and soon we should see it in app store.

We authored many blogs

We love sharing our experiences via blogs on various topics like ReactJS, Rails, React Native and Robot framework etc. Here are some of our blogs from 2015.

• How to obtain current time from a different timezone in Selenium IDE using javascript, Prabhakar, Jan 2015
• Author information in jekyll blog, Neeraj, Jan 2015
• Phone verification using SMS via Twilio, Santosh, Jan 2015
• Blue border around JWPLAYER video, Prathamesh, Feb 2015
• Gotcha with after_commit callback in Rails, Prathamesh, March 2015
• Voice based phone verification using twilio, Santosh, March 2015
• Verifying PubSub Services from Rails using Redis, Vipul, May 2015
• Using ReactJS with Rails Action Cable, Vipul, July 2015
• How to test React Native App on a real iPhone, Chirag, Aug 2015
• Optimize JavaScript code using BabelJS, Prathamesh, Aug 2015
• Configuring Pycharm IDE to run a Robot Framework test suite or a single test script, Prabhakar, Oct 2015
• Migrating rails app from postgresql to sql server, Rohit, Oct 2015
• Getting around Apple iTunesConnect account activation issue, Prathamesh, Oct 2015
• Rails 5 allows setting custom HTTP Headers for assets, Vipul, Oct 2015
• Using Stripe API in React Native with fetch, Chirag, Nov 2015
• How constant lookup and resolution works in Ruby on Rails, Mohit, Nov 2015
• Explicitly ssh into vagrant machine, Neeraj, Dec 2015
• ApplicationRecord in Rails 5, Prathamesh, Dec 2015

Video Summary

• Learn Ruby on Rails
  • Use uuid, X-Request-Id and tagged logging to debug rails application.
  • Rails development using vagrant.
  • Using ES6 in Rails application.
• Learn ReactJS in step
• Keep up with ReactJS
• Learn JavaScript
  • Refactor JavaScript code using module pattern
  • A review of tools to test ES6
• Learn Selenium

Open Source

Apart from our team members contributing to various OpenSource projects, we also support some projects from our team. This year, we added and helped build following projects-

• Wheel : Wheel is our Rails template for new Ruby on Rails projects, with sane defaults and setups for different environments, and common functionalities like image uploaders, debugging, etc.
• Mail Interceptor : Interception, Forwarding emails in Ruby on Rails application
• Handy : A collection handy tools and Rails tasks for your Project.
• Learn ReactJS in Steps : Collection of examples from Learn ReactJS in step video series.
• Doctsplit Chef(Link is not available) : Check cookbox for docsplit ruby gem
• Fixtures Dumper(Link is not available) : Dump your Rails data to fixtures easily to a database to populate data.

Community Engagement

Along with speaking at various conferences, we also helped organize, our Fun edition of Pune's regional RubyConf, DeccanRubyConf, and supported other Indian Conferences, including RubyConfIndia, GardenCity RubyConf.

We also help Pune's local ruby meetup, which had a splendid engagement this year.

Overall we are super excited about what we accomplished in year 2015. We are looking forward to an exciting year of 2016!

vagarnt ssh

While working with chef I needed to explicitly ssh into the vagrant machine. It took me sometime to figure it out.

The key is command vagrant ssh-config. The output might look like this.

$ vagrant ssh-config
Host vmachine
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/nsingh/code/vagrant-machine/.vagrant/machines/vmachine/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL
  ForwardAgent yes

Open ~/.ssh/config and paste the output at the end of the file and save the file.

Now I can ssh into vagrant machine using ssh command as shown below.

ssh vmachine

If you are wondering from where the name vmachine came then this is the name I had given to my vagrant machine .

Configuration to run a single test script

Running a single test script

Configuration to run a particular test suite

Running a particular test suite

This is because of the CSS class jwplayer-tab-focus.

The blue border around currently selected video allows to identify which instance of JWPlayer is in focus.

But with a single JWPlayer instance, it can be annoying.

To remove this blue border, we can override the default JWPlayer CSS as follows.

.jw-tab-focus:focus {
  outline: none;
}

To keep all the overridden CSS in once place, we can add this change in a separate file such as jwplayer_overrides.css.

Short summary

In year 2014 we presented at following 8 conferences across 6 countries.

• RubyConf Goa, India
• RubyConf Philippines
• RedDotRubyConf Singapore
• DeccanRubyConf Pune, India
• Madion+Ruby Wisconsin, USA
• RubyConf Brazil
• RubyKaigi Tokyo, Japan
• Golden Gate Ruby Conference San Francisco, USA

Rails and Ruby Conferences

View the embedded content

At the start of our travel, we visited RubyConfIndia 2014. The conference took place at an amazing beach resort in Goa. The two days of conference were full of fun and interactions with the best ruby people around India. During the conference our team announced launch of Ruby India to help spread ideas and experiments from the Ruby Indian Community, as well as highlight content from people.

Soon after, I visited Philippines, to conduct a workshop on "Contributing to Rails" at RubyConf Philippines. It was amazing to meet the growing Philippines community. I was happy to spend time with some amazing Rubyists, like Nick Sutterer, Zachary Scott, Konstantin Hasse, Koichi Sasada San, PJ Hagerty, Andre Arko and so on.

After that I and Prathamesh went to Singapore to speak at RedDotRubyConf. We spoke on Arel and ActiveRecord. "RedDorRubyConf" was our first joint talk together at a conference. Again we met a lot of awesome people like Piotr Solnica, Grzegorz Witek, Yinquan Teo, Sayanee Basu, Chinmay Pendharkar and Winston Teo Yong Wei. We also visited Marina Sand Bay and Sentosa Island.

Back in Pune, we hosted the first ever DeccanRubyConf. Our team was busy working on tasks right from building the website, inviting speakers, planning, and other arrangements. The conference had good talks and some really useful workshops. it was a fun one day conference, with attendance of over 170+ people.

The conference also saw our team announce the launch of RubyIndia Podcast, which does regular podcast interviews with notable people from the Ruby Community and Indian Community.

Prathamesh and I, then left on around a one and a half month travel, to attend and speak at multiple conferences.

We started with Madison+Ruby, in Madison, WI. After several missed flights, and a storm, we visited our first US conference after a travel of 48 hours. MadisonRuby was a conference like no other. Several topics touched the humane side of Ruby and the community. We spoke on 'Building an own ORM using ARel'. Set in the cultural town of Madison, we immensely enjoyed the cheese-curds, farmers markets and game night arranged by the Conf team. A huge thanks to Jim and Jennifer Remsik, for hosting such an amazing event. Thanks also to Scott Ruttencutter for giving us space to work from his office and giving us a tour of the state capital.

We then visited Sao Paulo, Brazil for RubyConf Brazil and presented a talk on Building an ORM using ARel. It was a pleasure to meet Fabio Akita and the CodeMiner team. We made friends with Celso Fernandes and Paulo who were kind enough to help us around, since in Brazil primarily portuguese is spoken. We also met Rafael Franca and Carlos Antonio da Silva who have helped us a lot with Rails issue tracker.

Next Prathamesh headed to RubyKaigi, being held in Tokyo, Japan. He presented on Fixtures in Rails. He met Matz, creator of Ruby on his first day in Japan. Mostly all the core Ruby contributors attended RubyKaigi. He got to interact with Koichi Sasada San, Jonan Scheffler, Akira Matsuda, Godfrey Chan, Richard Schneeman and lot of awesome Rubyists. He also met with his JRuby Core Tom Enebo for the first time. Thanks to Yasuo Honda, Michael Reinsch for helping with Japanese food.

From Brazil, I first visited Miami, and was happy to visit The Lab Miami, WynCode and interact with Rubyists from Miami.

Before heading to San Francisco, for GoGaRuco, I was able to make a quick stop in Boston and visit AlterConf Boston. The theme of the conference was around diversity in tech and gaming industry.

My latest conference was in the amazing city on San Francisco. I presented about 'Building an ORM', at GoGaRuCo, which incidentally was the last ever GoGaRuCo. The conference taking place in San Francisco, saw an amazing turnout of crowd. I was able to interact with Sarah Allen, Yehuda Katz, Pat Allen, Sarah Mei, Ryan Davis. I spent most of the time along with Nathan Long, Randy Coulman, and Nathan's friend Michael Gundlach, who is the creator of popular plugin Adblock. I also ran into Chris Eppstein, creator of compass. All around it was one of the most amazing interactions I had in a conference.

2014, was an amazing year for our team. Together we presented or were part of 8 conferences, launched RubyIndia Newsletter as well as the RubyIndia Podcast, started with 6 new video series on topics from ReactJS to Rubymotion to Selenium, published numerous blogs, and contributed to a number of OpenSource projects.

2015, starts with our team presenting at GardenCityRuby Conf. We hope to get more such chances to interact and help out the community. Onwards to a new year!

Here are the specific steps I took which brought the running time to under 15 minutes.

Set to run at the maximum speed

setSpeed command takes Target value in milliseconds. By setting the value to zero, I set the speed to maximum and the tests indeed ran fast. However, now I had lots of tests failing which were previously passing.

What happened.

In our tests real firefox browser is fired up and real elements are clicked. The application does make round trip to the rails server hosted on heroku.

By setting the selenium tests to the maximum speed the selenium tests started asserting for elements on the page even before the pages were fully loaded by the browser.

I needed sets of instructions using which I could tell selenium how long to wait for before asserting for elements.

Selenium provides a wonderful suite of commands which helped me fine tune the test run. Here I'm discussing some of those commands.

waitForVisible

This command is used to tell selenium to wait until the specified element is visible on the page.

In the below mentioned case, the Selenium IDE will wait until the element css=#text-a is visible on the page.

waitForText

This command is used to tell selenium to wait until a particular text is visible in the specified element.

In the case mentioned below, Selenium IDE will wait until the text violet is displayed in the element css=#text-a.

The difference between waitForVisible and waitForText is that waitForVisible waits until the specified element is visible on the page while waitForText waits until a particular text is visible in the specified element on the page.

waitForElementPresent

This command is used to tell Selenium to wait until the specified element is displayed on the page.

In the below mentioned case, the Selenium IDE will wait until the element css=a.button is displayed on the page.

waitForVisible and waitForElementPresent seem very similar. It seems both of these commands do the same thing. There is a subtle difference though.

waitForVisible waits until the specified element is visible. Visibility of an element is manipulated by the settings of CSS properties. For example using display none; one can make an element not be visible at all.

In contrast the command waitForElementPresent waits until the specified element is present on the page in the form of html markup. This command does not give consideration to css settings.

refreshAndWait

This command is used to tell Selenium to wait until the page is refreshed and the targeted element is displayed on the web page.

In the example mentioned below, the Selenium IDE will wait until the page is refreshed and the targeted element css=span.button is displayed on the page.

clickAndWait

This command is used to tell selenium to wait until a particular button is clicked for submitting the form and the page starts reloading. The subsequent commands are paused until, the page is reloaded after the element is clicked on the page.

In the case mentioned below, Selenium IDE will wait until the page is reloaded after the specified element css=input#edit is clicked.

Selenium IDE commands used above and more are available at Selenium documentation .

Time to time we rebase the branch. And after rebasing we need to do force push to send the changes to github. And once in a while someone force pushes into master by mistake. We recommend to set push.default to current to avoid such issues but still sometimes force push does happen in master.

In order to prevent such mistakes in future we are using pre-push hook. This is a small ruby program which runs before any git push command. If you are force pushing to master then it will reject the push like this.

*************************************************************************
Your attempt to FORCE PUSH to MASTER has been rejected.
If you still want to FORCE PUSH then you need to ignore the pre_push git hook by executing following command.
git push master --force --no-verify
*************************************************************************

Requirements

pre-push hook was added to git in version 1.8.2. So you need git 1.8.2 or higher. You can easily upgrade git by executing brew upgrade git .

$ git --version
git version 1.8.2.3

Setting up hooks

In order for these hooks to kick in they need to be setup.

First step is to clone the repo to your local machine. Now open ~/.gitconfig and add following line.

[init]
  templatedir= /Users/neeraj/code/tiny_scripts/git-hooks

Change the value /Users/neeraj/code/tiny_scripts/git-hooks to match with the directory of your machine.

Making existing repositories aware of this hook

Now pre-push hook is setup. Any new repository that you clone will have the feature of not being able to force push to master.

But existing repositories do not know about this git-hook. To make existing repositories aware of this hook execute following command on all repositories.

$ git init
Reinitialized existing Git repository in /Users/nsingh/dev/projects/streetcommerce/.git/

Now if you look into the .git/hooks directory of your project you should see a file called pre-push.

$ ls .git/hooks/pre-push
.git/hooks/pre-push

It means this project is all set with pre-push hook.

New repositories

When you clone a repository then git init is invoked automatically and you will get pre-push already copied for you. So you are all set for all future repositories too.

How to ignore pre-push hook

To ignore pre-push hook all you need to do is

# Use following command to ignore pre-push check and to force update master.
git push master --force --no-verify

The hook is here .

git clone git@github.com:neerajsingh0101/rails.git

Now cd rails and execute git remote -v . This is what I see.

origin git@github.com:neerajsingh0101/rails.git (fetch)
origin git@github.com:neerajsingh0101/rails.git (push)

Now I will add upstream remote by executing following command.

git remote add upstream git@github.com/rails/rails.git

After having done that when I execute git remote -v then I see

origin git@github.com:neerajsingh0101/rails.git (fetch)
origin git@github.com:neerajsingh0101/rails.git (push)
upstream git://github.com/rails/rails.git (fetch)
upstream git://github.com/rails/rails.git (push)

Now I want to make some changes to the code. After all this is why I forked the repo.

Let's say that I want to add exception handling to the forked code I have locally. Then I create a branch called exception-handling and make all your changes in this branch. The key here is to not to make any changes to master branch. I try to keep master of my forked repository in sync with the master of the original repository where I forked it.

So now let's create a branch and I will put in all my changes there.

git checkout -b exception-handling

In the Gemfile I will use this code like this

gem 'rails', github: 'neerajsingh0101/rails', branch: 'exception-handling'

A month has passed. In the meantime rails master has tons of changes. I want those changes in my exception-handling branch. In order to achieve that first I need to bring my local master up-to-date with rails master.

I need to switch to master branch and then I need to execute following commands.

git checkout master
git fetch upstream
git rebase upstream/master
git push

Now the master of forked repository is in-sync with the master of rails/rails. Now that master is up-to-date I need to pull in the changes in master in my exception-handling branch.

git checkout exception-handling
git rebase master
git push -f

Now my branch exception-handling has my fix on top of rails master.