BERT

blogs

blog-image
The Heads Hypothesis: A Unifying Statistical Approach Towards Understanding Multi-Headed Attention in BERT
BERT has been a top contender in the space of NLP models. With its sucess, a parallel stream of research, named BERTology, has emerged, that tries to understand how does BERT work so well. With the similar objective, this blog explains a novel technique for analysing the behaviour of BERT’s attention heads. Read this for interesting insights which unravel the inner working of BERT!
By Madhura Pande
• Sep 3, 2021
Know More

publications

Publication Image
Perturbation CheckLists for Evaluating NLG Evaluation Metrics

Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, Mitesh M. Khapra,

In Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2021)

https://arxiv.org/abs/2109.05771
Publication Image
The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT

Madhura Pande, Aakriti Budhraja, Preksha Nema, Pratyush Kumar, Mitesh Khapra,

In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 15, pp. 13613-13621)

https://ojs.aaai.org/index.php/AAAI/article/view/17605
Publication Image
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining

Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora and Mitesh M. Khapra,

Transactions of the Association for Computational Linguistics

https://doi.org/10.1162/tacl_a_00347
Publication Image
On the weak link between importance and prunability of attention heads

Aakriti Budhraja, Madhura Pande, Preksha Nema, Pratyush Kumar, Mitesh M. Khapra,

In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics.

http://dx.doi.org/10.18653/v1/2020.emnlp-main.260
Publication Image
Towards Interpreting BERT for Reading Comprehension Based QA

Sahana Ramnath, Preksha Nema, Deep Sahni, Mitesh M. Khapra,

In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

http://dx.doi.org/10.18653/v1/2020.emnlp-main.261
Publication Image
On Incorporating Structural Information to improve Dialogue Response Generation

Nikita Moghe, Priyesh Vijayan, Balaraman Ravindran, Mitesh M. Khapra,

arXiv:2005.14315

https://arxiv.org/pdf/2005.14315.pdf