Daily summary:

Healthy:

• Workout: 4 Km

• Self-desciplined diet, reach a new low of 77.05 Kg.
• Start to sleep earlier and try to get up early. (7:30 get up, 11:30 to bed today).

Research:

• Pushing our C project.

• Revising CV and PS for NUS PhD application due 15th June.

  • Meeting with Wenqiang tmr for his advice on revision.

A Survey on Contextual Embeddings:

Link: https://arxiv.org/abs/2003.07278

Pretraining can be divided into 2 types:

• Un-supervised: LM
• Supervised: NLI / Machine Translation

ELMo can be degraded into BiLSTM?

No, because ELMo is pretrained to be like that.

ELMo is essentially a Bi-LSTM, the representation for each layer is the concatenation of forward and backward.

ELMo has many layer, it is the weighted sum. the representation is task dependent: by the weight.

GPT model:

Transformer based.

Task specific fine-tuning, GPT applies task-specific input adaptations motivated by traversal approach. Preprocess each text input as a single contiguous sequence of tokens.

• START
• DELIM
• EXTRACT

GPT2: authors argue that, training on very large datasets, the model starts to learn some common supervised NLP tasks.

BERT:

The final hidden state of [CLS] is used for sentence-level tasks and the final hidden state of each token is used for token-level tasks.

TODO: For partial insight on this, we refer the readers to (Raffel et al., 2019) for a controlled comparison between unidirectional and bidirectional models,

Ways to use contextual embeddings for downsream:

• Feature based
• Fine tuning methods
• Adapters

Catastrophic forgetting：

• Freezing layers
• Adaptive Learning Rates
  • Lower level tends to capture general language knowledge

Knowledge Distillation

Analyzing Contextual Embeddings:

• Probing task
• Visualization

Probe classifiers. A large body of work studies contextual embeddings using probes. These are constrained classifiers designed to explore whether syntactic and semantic information is en-coded in these representations or not.

• Liu et al. (2019a) design a series of token labelling, segmentation, and pairwise relation tasks for studying the effectiveness of contextual embeddings. Contextual embeddings achieve competitive results compared to the state-of-the-art models on most tasks, yet fail on some fine-grained linguistic tasks (e.g. conjunct identification).
• Hewitt and Manning (2019) propose a structural probe for finding syntax in contextual embeddings. The model attempts to learn a linear transformation under which the L2 distances between tokens encode the distances between these tokens in syntactic parsing trees like dependency trees.
• Tenney et al. (2019a) find that BERT rediscovers the traditional NLP pipeline in an interpretable and localizable way. Specifically, it is capable at POS tagging, parsing, NER, semantic roles, and coreference, and these are learned in order.
• Jawahar et al. (2019) use ten sentence-level probing tasks (e.g. SentLen, TreeDepth) and find that BERT captures phrase-level information in earlier layers and long-distance dependency information in deeper layers.

A Structural Probe for Finding Syntax in Word Representations

Link: https://nlp.stanford.edu/pubs/hewitt2019structural.pdf

What I can learn for our C project is that: we can use the syntactic tree as a probing task! e.g. syntactic tree distance, syntactic tree patterns …