You know how you’re scrolling through a listing on a matchmaking app and this one profile just catches your eye?
You click to check out the full profile. You’re Intrigued! You want to know more.
And then there are times when you feel nothing, so you just keep scrolling?
At shaadi.com we were wondering if these little hints of Intrigue and Ignore could allow us to craft a better matchmaking experience for you.
But this would involve us tracking billions of events per day and drawing inferences about what made your heart beat a little faster. An impossible task, certainly.
Impossible, you say? Well, we gave it a shot anyway! 
So What was the Problem?
Before we got TrueView to sort our profile matches, only Full Profile Views were considered as Viewed profiles.
But with TrueView we could capture the actual intent of the user irrespective of a profile view or a list view and then utilize it to resurface and sort profiles based on their intent.
We wanted to mark certain user actions on the profiles as Intrigued or Ignored signals and then utilise it to sort their Matches as per the following order of pools –
- Not-viewed profiles – profiles on which the user has not taken any action or the user has not yet discovered the profile
- Intrigued profiles – profiles that appear intriguing to the user
- Ignored profiles – profiles which are ignored by the user
We had some rules to follow..
- Intrigued signal should override an Ignored signal immediately
- Ignored signal should override an Intrigued signal only after 1 day (24 hours)
- Profiles should be removed from Intrigued and Ignored list after certain days of marking it as Intrigued or Ignored
And some challenges to overcome
Before jumping to the solution, let me talk about some of the challenges we faced in solving this problem.
- How to manage and process the large volumes of user actions and events
- How to mark user actions as Intrigued or Ignored
- Where do all the profiles marked as intrigued or ignored go
- How to remove the profiles from the users’ list of Intrigued and Ignored after certain days of marking it
- How to sort the search result for finding Matches utilising these events
Now to solve these challenges for achieving the desired result, we needed to design a system considering the following important points –
- Our systems should be robust and scalable which can easily manage the load to process and store large volumes of user actions
- They should be fast enough to iterate and support multiple use cases and changes of direction
- They should be consistently good in performance and should be reliable
Solution
You can have a look at the architecture diagram to get an overview of how we implemented the solution to achieve the desired result. I will explain the role and purpose of each component.
Elasticsearch
At Shaadi.com, for all the search features we use ElasticSearch. To know what Elasticsearch is and what it does, read here
Function score query
To sort users’ matches as per the desired order of pools, we have made a use of Elasticsearch’s function score query. The function score allows you to modify the score of documents that are retrieved by a query. To use function_score, we need to define a query and one or more functions that compute a new score for each document returned by the query.
One can optionally choose to apply the function only if a document matches a given filtering query. To get a detailed explanation on how to use function score query, read here.
This is how our function score query looks like –
"function_score": {
"query": {
// Query to get User's Matches as per their preferences
},
"functions": [
{
"weight":3,
"filter": {
"bool": {
"must_not": [
{
"terms": {
"profileid": [<All the profile ids of the profiles which are intriguing to the user>]
}
},
{
"terms": {
"profileid": [<All the profile ids of the profiles which are ignored by the user>]
}
}
]
}
}
},
{
"weight": 2,
"filter": {
"bool": {
"filter": {
"terms": {
"profileid": [<All the profile ids of the profiles which are intriguing to the user>]
}
}
}
}
},
{
"weight": 1
}
],
"score_mode": "sum",
"boost_mode": "replace"
}
As you can see, three functions are defined with the filtering queries –
- The profiles which are not yet marked as Intrigued or Ignored are scored 3 by the first function
- The profiles which are marked as Intrigued are scored 2 by the second function
- All the profiles retrieved by a query are scored 1 by the third and a last function
Then, the scores computed by the above functions are summed. Thus, all the Not-viewed profiles will be scored 4, Intrigued profiles will be scored 3 and Ignored profiles will be scored 1. Profiles are sorted by the descending order of the final scores computed.
This will sort the users’ Matches as per the desired order of pools.
But I have not explained the most important part of this problem, yet. And that is –
- From where are the users’ Intrigued and Ignored profiles fetched, to use it in a query
- How are all user actions processed and marked as Intrigued or Ignored
- How the business rules mentioned above are handled
So let’s begin!
Kafka
Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. To know more about Kafka, read here.
At Shaadi.com, we publish all the user actions and events to Kafka. Engineers write their own consumers and process the stream of events according to their use cases.
I will cover how we processed these events from the Kafka queue and how we put it to use for sorting in Matches.
Role of a Kafka consumer
Consumer identifies the events as intrigued or ignored on the basis of event type, calculates the TTL for the events according to the business rules and records it into the Dynamodb table.
Before inserting the event, it queries the table to check if the user has previously taken any action on a profile, If yes, it overrides previous event if required according to the rules.
Why Dynamodb? And how it solved our challenges ?
Large volume of events
With 55k events to process in a minute and store it into the table which has approximately 460 Million records already present at any point of time, we needed a database which can easily manage this heavy reads and writes on such a large table in a quick response time.
Dynamodb guarantees to provide consistent, single-digit millisecond response times at this scale. It guarantees to handle more than 20 million requests per second.
Number of events processed by the consumer throughout the day. (980 requests per second)
Checkout the Dynamodb’s put latency (write latency) at this scale which is just 3 milliseconds per write request.
Serverless and Autoscaling
Dynamodb is serverless with no servers to provision or manage and it automatically scales tables up and down to adjust for capacity and maintain performance. Also we wanted something with no overhead of managing and scaling.
Thus, it was an ideal tool for this use case and it helped us in solving our scaling challenges.
See how Dynamodb auto scales the provisioned write capacity throughout the day.
Built In TTL support
As per the business rules mentioned above, we wanted to delete the event record after a particular period. Dynamodb’s TTL feature was icing on the cake for us. We just set the TTL on every event inserted into the table and it gets automatically deleted from the table when expired.
The need to write a cron script and manage it which would have deleted these expired records was avoided.
Read more about Dynamodb here.
Cron’s Job
The cron script periodically fetches the users who have logged into Shaadi recently. It then fetches each user’s events from Dynamodb and updates their Intrigued and Ignored profiles into the users’ document of Elasticsearch index.
WHY IS THE CRON REQUIRED?
And Why update it into the Elasticsearch index ?
In the above function score query, we provide the lists of Intrigued and Ignored profileids to the filtering query of the functions. And we have hundreds of users’ intrigued and ignored profiles which are to be provided. That is a lot of data to pass over the wire.
Additionally, on each and every request to find Matches, we have to first query a Dynamodb table to get the users’ events, segregate the result into Intrigued and Ignored, extract the profileids from the segregated results before providing it to the Elasticsearch query.
Not only do we need to perform two requests – a Dynamodb query and a search, but we have to shuffle a potentially large list of profileids across the wire twice. The Elasticsearch’s terms lookup feature allows you to bypass this inefficiency.
Terms lookup fetches the field values of an existing document. To get a detailed explanation on how to use terms lookup, read here.
This is how intrigued and ignored list is provided to the filtering query of the functions using terms lookup
{
"terms": {
"profileid": {
"index": "<index_name>",
"id": "<profile id of the user for whom the search is performed>",
"path": "<ignored|intrigued>"
}
}
}
This is why the cron is required. It updates the ignored and intrigued field of the users document in advance after every session of the user
Conclusion
The impact on our business was an 8% lift in Interests sent and 6% lift in UIS (Unique Interest Senders) by implementing this sort. And this lift was across both males and females users.
To design a solution which can be consistently good in performance and scalable and at the same time, can support multiple use cases and changes of direction, we have made the use of multiple technology services such as Dynamodb, Kafka, Elasticsearch etc.
This is how selecting the right tool for the right use cases can solve nearly all technical challenges.
Excellent architecture decisions have the ability to deploy pretty much every product requirement we can think of!
