- What do you think are the three biggest open problems in NLP at the moment?
- What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
- What, if anything, has led the field in the wrong direction?
- What advice would you give a postgraduate student in NLP starting their project now?
I made a more readable document, and shared it here.
What I gather from the experts’ responses is the following:
What do you think are the three biggest open problems in NLP at the moment?
- Current models don’t understand language as humans do.
- Generalization (related to domain adaptation, transfer learning, few-shot learning, zero-shot learning).
- Evaluation. Current evaluation approaches are often misused and misinterpreted.
- Reliance on biased, giant datasets for models to “learn” anything.
- Related to point 2: Adapt current models / data to low-resource settings (languages with few data).
- Instrospection / interpretability.
What would you say is the most influential work in NLP in the last decade, if you had to pick just one?
- A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning by Collobert and Weston, 2008.
- Natural Language Processing (Almost) from Scratch by Collobert, Weston, et al., 2011.
- word2vec, aka Distributed Representations of Words and Phrases and their Compositionality, by Mikolov et al., 2013.
- Seq2seq (Bahdanau et al., 2014, Sutskever et al., 2014)
What, if anything, has led the field in the wrong direction?
- Distance between computational methods and linguistics.
- Learning how to solve datasets instead of understanding and trying to solve the big picture.
- Blindly trying to beat suboptimal benchmarks or the state of the art in specific datasets. This leads to architecture hacking and “graduate student descent” (also see this tweet).
What advice would you give a postgraduate student in NLP starting their project now?
There are lots of good advice and I would definitely recommend reading them all. Some personal favorites:
George Dahl: Learn how to tune your models, learn how to make strong baselines, and learn how
to build baselines that test particular hypotheses. Don’t take any single paper too
seriously, wait for its conclusions to show up more than once. In many cases, you
can take even pretty solid published work and make much stronger baselines that
give some more context on the results.
Karen Livescu: Collaborate a lot. Do internships. Find multiple mentors and learn to weight their
advice. Take courses on fundamental methods, even if they don’t seem relevant right
now. Learn the history of the field.
Kyunghyun Cho: I believe scientific pursuit is meant to be full of failures. 99 out of 100 ideas you come up with are supposed to fail. If every idea works out, it’s either a) you’re not ambitious enough, b) you’re subconsciously cheating yourself, or c) you’re a genius, the last of which I heard happens only once every century or so. So, don’t despair!
Share your own thoughts and opinions!