Building a Siamese Network for Duplicate Question Detection

Understanding How Siamese Networks Learn Similarity

So far, you’ve learned about embeddings, similarity scores, and loss functions. Now it’s time to see how they all fit together.

In this post, you’ll build a cost function, apply gradient descent, and understand how batch-based learning helps detect duplicate questions — like “What is your age?” and “How old are you?”

As promised, you'll see how everything fits together right now. You will start by building a cost function and then you will use gradient descent to optimize this cost function.

Step 1: Preparing Batches of Data

Imagine you have pairs of similar questions such as:

Here, each row represents a pair of duplicate questions. None of the questions within the same column are duplicates of each other — only the questions across columns are.

This setup gives us a batch size of 4, often denoted as b = 4.

Let's take a look at how this works. To compute the cost, begin by preparing the data in batches. Here, you have the questions, what is your age and how old are you? You can see these are duplicates because they mean the same thing. Can you see me and are you seeing me are also duplicates. Where are thou and where are you are duplicates too as are when is the game and what time is the game. With four pairs, you have batch size of four. Here, we will use the letter b to stand for batch size. Something that's very important to note is that each question has its corresponding duplicates to the left or right of it. That is in each row, all of the sentences in the columns are duplicates but you'll notice that each question has no duplicates above or below it. That is for any column, none of the rows and that's column contain a sentence that is a duplicate of another sentence in that column. This is how you prepare the batches.

Step 2: Passing Batches Through the Model

When you feed this batch into the model, it produces embedding vectors for each question.

Let’s say each embedding has 5 dimensions — you’ll get:

• A vector output for each question in Batch 1, noted as v_1.

• A matching set of vectors for Batch 2, noted as v_2.

Since your batch size is 4, you’ll end up with matrices like:

Now, let me show you how you will want to organize the data in this way. Given the first batch, you're going to run it through this model to get a vector v_1 with dimensions 1 row by 5 columns. The number of columns shown in this matrix is equal to the dimension of your embedding layer, which in this case is five. I'll refer to this dimension of the embedding layer as the model for each question in the batch. I haven't talked about the dimension of the embedding layer yet, but don't worry, it will become more clear once you're working with the code. The important takeaway is that the dimension of the embedding, the model is a parameter that determines the dimensions of the weights through each layer in the model and thus determines the size of the outputs vector. The model is running a batch size that is greater than 1, so the v_1 output is actually a matrix of stacked vectors like this. In this visual example there are four rows in this matrix to indicate that there are four observations in this batch, and the batch size is four. I'll subscript the observations in the batch as v_11, v_12, and so on corresponding to the vector outputs for each question in the batch. You'll do the same thing for the batch of v_2 vectors. Each question in the Batch 1, is a duplicate of its corresponding question in Batch 2 but none of the questions in Batch 1 are duplicates of each other, and the same applies to Batch 2. Here for example, the 11 is a duplicate of v_21, as are the rest of the respective row pairs but v_11 is not a duplicates of any other rows in v_1.

Step 3: Calculating Similarity

The Siamese network combines these outputs by computing similarity between every pair of v_1 and v_2.

This gives a similarity matrix — where:

• Diagonal values show similarity scores for duplicates (positive examples).

• Off-diagonal values show scores for non-duplicates (negative examples).

The diagonal scores should be higher if the model is learning correctly

The last step is to combine the two branches of the Siamese network by calculating the similarity between all vector pair combinations of v_1 with v_2. For this example with a batch size of 4, you might get a matrix of similarities that looks like this. The diagonal is a key feature here. These values are the similarities for all your positive examples, the question duplicates. Notice that all the values are generally greater than the numbers in the off-diagonals. The model is performing as you would expect for duplicates questions because you would expect the question duplicates to have higher similarity compared to the non duplicates. In the upper right and lower left, you have the similarities for all the negative examples. These are the results for the non duplicates pairs. Notice that most of these numbers are lower than the similarities that are along the diagonal. Also notice that you can have negative example question pairs that still have a similarity greater than zero. The range of similarity ranges from negative 1 to positive 1 but there isn't any special requirements that a similarity greater than zero indicates duplicates, or that's a similarity or less than zero indicates not duplicates. What matters for a properly functioning model is that it generally finds that duplicate have a higher similarity relative to non duplicates. Creating non duplicate pairs like this removes the need for additional non duplicates examples in the input data, which turns out to be a big deal. Instead of needing to set up specific batches with negative examples, your model can learn from them into existing question duplicates, batches.

Step 4: Using Triplet Loss

Once similarities are computed, you can calculate the triplet loss, which ensures that duplicates are closer in the embedding space than non-duplicates.

Mathematically, the overall cost is the sum of all triplet losses across the dataset.

But there’s an even smarter trick — hard negative mining, which helps the model learn better by focusing on the most challenging non-duplicate pairs.

Now, you could just stop here and use these similarities with the triplet loss-function you already know shown here. Then the overall costs for your Siamese network will be the sum of these individual losses over the training sets. Here, you can see that superscripts i refers to a specific training example and there are m observations, but there are more techniques available. This can vastly improve upon model performance. I'll show you those next. You have now seen how hard negative mining is used when computing the cost and during training. You only need to have two similar texts which you put on the diagonals, and you use the off-diagonals as the non-similar examples.