Kadam Parikh

Projects / Built · Case study

Reproducing a landmark question-answering model from the paper

A from-scratch implementation of BiDAF, a pre-transformer machine-comprehension network, that reached 83.5 exact match on SQuAD v1.1, within about a point of BERT’s reference result.

Context
Research reproduction, from the era before LLMs
My role
Implementation, tuning, and mentoring the team
Benchmark
SQuAD v1.1, dev set
Focus
Reproducing a paper from scratch

The problem

To answer a question from a passage, a model has to read the passage with the question in mind. The dominant approach at the time squeezed the passage into one fixed-size vector early, before attention was applied, which throws away the fine detail needed to pinpoint an exact answer. BiDAF’s idea is to let attention flow between passage and question at every step and pass it forward, rather than collapsing it.

The task: given a Wikipedia passage and a question, return the answer as a contiguous, verbatim span of the passage. Nothing is generated, and every question is answerable.

Result

83.5 EMMy implementation, SQuAD v1.1 dev set (exact match)
84.2 EMBERT’s contemporary reference result (91.1 F1)

Exact match requires the normalized prediction to equal a reference answer exactly, so it is the stricter of the two standard metrics. I did not record F1 for this run, so I only state the exact-match figure.

What I did

  • Translated the paper into a from-scratch TensorFlow and Keras implementation of its five layers and trained it for answer-span prediction on SQuAD v1.1.
  • Diagnosed and resolved convergence and accuracy bottlenecks through iterative experimentation and hyperparameter tuning.
  • Built the data preprocessing pipeline so training and evaluation saw consistent inputs.
  • Mentored teammates on attention mechanisms and machine-comprehension design, and ran knowledge-sharing sessions on deep learning and question answering.

The five layers

  1. 1 · Word embedding

    Pretrained GloVe vectors for every passage and question token.

  2. 2 · Contextual encoding

    A shared bidirectional LSTM turns each sequence into context-aware representations, passage and question separately.

  3. 3 · Attention flow

    The core contribution. A passage-question similarity matrix gives attention in both directions: which question words matter for each passage word, and which passage words matter most given the question. These flow forward at every position instead of being pooled into one vector.

  4. 4 · Modeling

    A second bidirectional LSTM over the attention output, so it models interactions that attention has already shaped rather than raw embeddings.

  5. 5 · Output

    Two heads give a probability for every passage position being the start of the answer and for being the end. Inference picks the start and end pair with the highest combined probability and returns the text between them.

Limits

  • My implementation uses GloVe word embeddings only. The paper also adds character-level embeddings to handle rare words, so it is not an exact replica.
  • It handles only questions that have an answer in the passage, and only one passage at a time, with no reasoning across passages. Later benchmarks added unanswerable questions, which this architecture does not address.
  • Recurrent encoders read sequentially, so they train more slowly than the fully parallel transformer encoders that followed.