Understanding Pub/Sub

Most  people use “Messaging” technology to get & send updates between the server and client/IOT devices.

Ever wondered how that works?. Well, we will try to explain how we have implemented it in one of our applications.

Pub/Sub is one of the most popular and effective messaging protocol/infrastructure used for communication between browsers, mobiles and IOT Apps supported devices. It gives the flexibility to get and send event-driven responses to the server.

It also helps us to expand our application support for mobile devices and all browsers. Using Cloud Pub/Sub we can build a system to sustain more than 40K concurrent users.

How does using Pub/Sub help manage Gmail Notification?

I was working on a Gmail chrome extension for one of my clients. The extension allows the users to manage their emails on Gmail under multiple ‘Trello’ style boards. Being a chrome extension it only supports activity happenings on the extension level. If any user changes his data from other browser or from a device it does not reflect instantly on the application. Moreover there was the requirement of sharing boards where users of a group can work together in parallel on the shared board. So, any changes on emails should get reflected on the shared board immediately.

Google Cloud Pub/Sub helps synchronizing multiple boards and allows runtime changes happening on the gmail reflect immediately on the boards. It also helps build scalable architecture required to provide good service to our paid as well as free customers.  

Old System Architecture:

As we were only managing boards for a single user i.e. the board owner, every time when the owner lands on the board, email information would be fetched from Gmail and displayed on the board.

Story to deliver:

As a team, all members can perform an action on the shared board, like add or remove emails from the board. Every member should get recent changes/activity done by the other team members on their screen, so that they can work without stepping on someone’s toes.

Solution Proposed/Designed:

Rendering board information for the board owner is a bit easy as we are using client-side API’s to fetch the information. But for the other team members to fetch the information from Gmail as well as to synchronize their boards with the recent activity was a bit complex and time taking. It was taking more than a minute to load the board for the other members of the board.

To overcome this problem, we created a POC in which we stored required information of the emails into our database and used I/O socket to synchronize team member boards. This solved our problem of synchronization and loading time. But still updating the database with the latest changes on the email was pending. We had to update our database whenever a team member performs any activity on the board like adding new email or removing existing email. So, we were looking out for a mechanism which gives us information about the recent changes happening on the board and we could synchronize all members’ boards using I/O socket.

That’s where the Gmail Pub/Sub came into the picture. It is a messaging paradigm where Senders (publisher) send messages to specific channels and Receivers (subscribers) who are interested in multiple channels get a notification whenever a publisher sends a message to a channel.

Cloud Pub/Sub provides one-to-one as well as many-to-many asynchronous messaging that allows secure and highly available communication between senders and receivers. It helped us to update the database after any activity happening on the shared emails. It has also helped us to give support to all the activity that has happened outside of the chrome extension.

Our next challenge was to implement this architecture to support more than 40K concurrent user requests. To solve this problem, we have designed the following architecture which helped us to solve all our performance and functional challenges.

We used a separate set of EC2 cluster to handle all Gmail push notifications for free and paid users. So in the future, we can manage or support more concurrent users. Also, we can give better Cloud Computing Service to our existing paid and potential paid customers.

As soon as Gmail sends a notification, we updated our database according to the latest changes and informed other members of the board using socket I/O. If a team member did any changes from the extension, they get reflected into the database and using socket I/O we have updated other members’ boards with the new changes.

Bottom Line 

You can enhance your application using Pub/Sub mechanism and can effectively manage your communication/synchronization between cross browsers and devices as well. You can manage your application loosely coupled, so that you can scale your application as per your need.