Metaphor Detection Using Contextual Word Embeddings From Transformers(Liu, J., O’Hara, N., Rubin, A., Draelos, R., & Rudin, C. 2020)

Liu, J., O’Hara, N., Rubin, A., Draelos, R., & Rudin, C. (2020, July). Metaphor detection using contextual word embeddings from transformers. In Proceedings of the Second Workshop on Figurative Language Processing (pp. 250-255).

 

<선행연구>

●구 automatic metaphor detection: hand-creafting informative features에 집중& supervised machine learning algorithm 사용

    -features: POS tag, concreteness, iageabilitym semantic distributions, semantic classes(by wordNet etc)

●deep learning models: LSTM+word embedding+features(ex. concrete rates)

●contextual embedding의 필요성 존재: bank robber vs river bank 구분

    -transformer-based embeddings 사용: BERT, GPT2, XLNet

 

<연구대상>

●metaphor detections

 

 

<실험방법>

1. 임베딩

● contextualized embeddings 생성

    -pretrained model에 feature-based로 생성

 

2. 실험 architecture

●BERT: huggingface, 12-lyrs, 768차원, 12head, 110Mparameters, pretrained

    -VUA, TOEFL dataset 학습, tokenizer도 BERT꺼로(subword 가능)

● XLNet: huggingface, VUA, TOEFL사용, pretrained

● 두 모델 concatenate: 1536 차원의 word embedding

    -catch contextual informations

    -POS tag도 사용(stanford parser 사용)

●Neural Network: Bi-LSTM; long-range relationships 캐치

    -dense lyr, acti.는 sigmoid 사용, binary cross entropy

    -ensemble strategy(Wu et al.2018): 다르게 initialization된 4개의 Bi-LSTM을 훈련

이후 결과의 평균

●서로 다른 문맥에 쓰인 같은 단어들에 비슷한 probabilities 부여 경향

    -> prediction is higher than the average prediction means that words was a good indicator of the presence of metaphor.: test 때 이 개념 사용

 

<결론>

●dataset이 작을수록(TOEFL < VUA) F1score 감소

●bi-directional relationships btw words play a crucial role in metaphor detection. (XLNet > GPT2)

●K-Nearest Neighbours combined

   -전제: whether a specific use of a specific word is more likely to be metaphorical.

   -lemmatized

   -LSTM과 KNN은 각자 다른 information을 catch