🔍 Building a Siamese Network for Duplicate Question Detection with Python

👋 Introduction

Welcome to the final post of the week!
In this tutorial, we’ll explore how to train and test a Siamese neural network — a special type of deep learning model used to measure similarity between two inputs.

We’ll focus on a real-world application: detecting duplicate questions using the Quora Question Pairs dataset.

By the end of this guide, you’ll understand:

• How the dataset is structured

• How the Siamese model is built

• How cosine similarity helps determine whether two questions mean the same thing

🧠 What Is a Siamese Network?

A Siamese network consists of two identical subnetworks that share the same weights.
Each subnetwork processes one input — in our case, one question — and converts it into a vector embedding.

Then, a similarity function (like cosine similarity) compares these two vectors to decide whether the inputs are duplicates or not.

In short:

Siamese networks learn to tell how similar two inputs are — not what they are.

Welcome to the last blog of the week. You're going to see what the datasets would look like for a Siamese network. I'll show you how you can train your model and then you can use that model to test your Siamese network. Let's take a look at how you can do this.

📊 The Quora Question Pairs Dataset

The Quora Question Pairs dataset contains pairs of questions and a Boolean label indicating whether the two are duplicates.

Here’s what it looks like:

This dataset provides thousands of examples that help the model learn how similar meaning can appear in different wording.

You'll be using the Quora question duplicate datasets for this week's programming assignments and it looks like this. It consists of a collection of question pairs within is duplicates Boolean for each question. For example, for question 1 and question 2, what is your age, and how old are you? Is_duplicate equals true because these two questions are duplicates? Where are you from and where are you going are not duplicates, so it's false and so on. These datasets gives your model plenty of examples to learn from.

🧩 How the Data Is Prepared

Before training, we prepare batches of paired questions:

• Each batch contains two sets of questions (q1 and q2).

• Corresponding questions in q1 and q2 are duplicates.

• No two questions within the same batch are duplicates of each other.

For example:

• q1_a = “What is your age?”

• q2_a = “How old are you?” → (duplicate)

• q1_b = “Where are you from?”

• q2_b = “Where were you born?” → (duplicate)

But note: q1_a and q1_b are not duplicates.
This clean pairing ensures the model learns meaningful relationships between similar questions.

First, you will process the dataset so that it looks like this. You will preprocess the data into batches of size b. The corresponding questions from each batch are duplicates. For example, the first question in Batch 1, what is your age, is a duplicate of the first question in Batch 2. How old are you? The second question in Batch 1 is a duplicate of the second question in Batch 2 and so on. Note, however, that there are no duplicates within an individual batch. If I call this q1_a and this q2_a, then q1_a and q2_a are duplicates. If this was q1_b and this was q2_b, then q1_b and q2_b are duplicates. However, q1_a and q1_b are not duplicates. Similarly, q2_a and q2_b are not duplicates. I'll show you how to prepare the batches in such a way that no question within the same batch is duplicated. Finally, you'll use these inputs to get outputs vectors for each batch. Then you can calculate the cosine similarity between each pair of outputs vectors.

🧬 Architecture of a Siamese Network

A typical Siamese network for text similarity works like this:

1. Embedding Layer — Converts words into numerical vectors (e.g., using Word2Vec or pretrained embeddings).

2. LSTM Layer — Processes sequences and captures contextual meaning.

3. Vector Output (v₁, v₂) — The output representation of each question.

4. Cosine Similarity — Measures how close the two embeddings are.

5. Threshold (τ) — If similarity > τ → classify as duplicate; otherwise, not.

Both subnetworks are identical and share the same parameters.
So effectively, you’re training one model, not two.

This is the Siamese model that you'll be implementing in the assignments. You'll create a subnetwork, which is then duplicated, and drawn in parallel. In each subnetwork, you get the embedding, run it through the LSTM, take your vector outputs, and then use them to find the cosine similarity. An important note here is that the learned parameters of the subnetworks are exactly the same between the two subnetworks. You are actually only training one set of weights, not two.

⚙️ Testing the Siamese Network (One-Shot Learning)

During testing, the model performs one-shot learning — it compares two unseen questions and predicts whether they mean the same thing.

Steps:

1. Convert each question into a numerical vector.

2. Feed both through the subnetworks.

3. Compute cosine similarity between output vectors v₁ and v₂.

4. Compare similarity score with a threshold τ.

If similarity > τ → classify as duplicate, otherwise not duplicate.

The values of τ and margin α (from the contrastive loss function) are tunable hyperparameters that affect performance.

When testing the model, you will perform one-shot learning. The goal is to find a similarity score between two inputs questions. First, convert each input into an array of numbers, feed these into your model, compare the subnetwork outputs v_1 and v_2 using cosine similarity for a similarity score. Then test the score against some threshold Tau and if the cosine similarity is greater than Tau, then the questions are classified as duplicates. Note that both Tau and the margin Alpha from the loss function are tunable hyperparameters. Congratulations. You now know how to train your Siamese network and you know how to test it. In this week's programming exercise, you'll be using a Siamese network to identify whether a question is a duplicate or not. Specifically, you'll be using the Quora question duplicate datasets, and using that you'll be able to get a very good accuracy.

🧠 Key Takeaways

• Siamese networks are ideal for similarity-based tasks like duplicate detection, face recognition, and signature verification.

• Both subnetworks share the same weights — you’re training a single model to measure similarity.

• Cosine similarity and thresholding are key to classification decisions.

• Hyperparameters like τ (threshold) and α (loss margin) need tuning for best results.