A Chinese Dimensional Valence-Arousal-Irony Detection on Sentence-level and Context-level Using Deep Learning Model

Paper Info

#

What Changed from the Conference Version

#

The dataset is the same CDVAI (1,004 ironic sentences, 843 ironic contexts); the annotation guidelines, annotator agreement, and score distribution match the ROCLING version and are not repeated here. What is new is the modeling:

The paper also frames irony detection explicitly as sentiment detection: irony has a humorous effect in conversation and brings specific emotional patterns, so beyond “is it ironic?”, the valence and arousal behind it matter.

Model Architecture

#

The experiments come in two parts.

Part 1: human-annotated context span

#

A pre-trained BERT encodes the input and a linear layer predicts the scores. Three ways to combine sentence and context features:

M1 is the baseline; M2 asks whether the span alone is enough; M3 asks whether seeing both lets the model pick up the sentiment contrast between them.

Part 2: predicting the context span

#

In practice nobody marks the context for you, so Part 2 trains a context-span predictor: BERT encodes the sentence and a linear layer on each token decides whether it belongs to the context.

The predicted span then replaces the human-annotated one, and Part 1’s M2 and M3 are rerun to see how much an imperfect span hurts VA prediction.

Experimental Setup

#

The splits follow the ROCLING version: stratified 7:3 train/test, with one tenth of train held out for validation (sentence level 632 / 71 / 301, context level 531 / 59 / 253).

The context carries less information than the sentence, and span prediction is harder still, so both try smaller learning rates.

Pre-trained models (all Chinese):

Part 1 Results

#

Sentence-level VAI (MAE, lower is better)

#

Bold marks the lowest value for each indicator.

• Valence: about 0.4 everywhere, and M1 is best. The paper argues valence is easy to judge, so richer features only hurt.
• Arousal: about 0.6, harder than valence. M1 is not always best; PM3–PM6 improve with M2 or M3.
• Irony: about 0.5–0.6, a little better than arousal, which suggests scoring irony by the keyword–context gap works. PM2, PM4, PM5, and PM6 improve with M2 or M3.
• Overall the sentence level does well, but M2 and M3 do not clearly help it.

Context-level VA (MAE)

#

Bold marks each model’s best setup for that indicator.

The picture is the opposite of the sentence level:

• Valence: M3 is best for 5 of the 6 models, down to 0.389.
• Arousal: M2 wins by a wide margin, cutting MAE from about 0.8 with M1 to about 0.6. The paper reads this as: to judge how aroused the context is, the context’s own information matters more than the whole sentence.
• The conclusion: understanding the true sentiment of an ironic context requires combining it with the sentence, which is exactly why CDVAI annotates the context separately.

Part 2 Results

#

Context-span prediction

#

PM1–PM6 were not good enough, so the paper adds PM7 (DeBERTa-v2), which still tops out at an F1 of 0.405. Finding the boundary of an ironic context is a hard task on its own.

Rerunning M2 / M3 with predicted spans

#

The spans predicted by PM7 replace the human annotation and the experiments are rerun.

Sentence level

• Sentence valence with M2 actually improves (PM1 from 0.421 to 0.337). The paper suspects the predicted spans, though wrong, happen to include more emotion words.
• Arousal and irony move both ways; for example arousal with M3 drops from 0.639 to 0.585 for PM1 but rises from 0.596 to 0.648 for PM3.

An interesting takeaway: at the sentence level, what matters may be a few key words in the context rather than the whole context.

Context level

The context level is less lucky. Valence is about the same as Part 1, but arousal with M2 falls back from 0.60–0.65 to 0.80–0.87, erasing almost all of M2’s gain. Context sentiment depends on the right span: once the span is wrong, the model can no longer learn the context’s own arousal.

Error Analysis

#

The paper examines one error from PM3, which did well in Part 1:

“Very good… even the speakers are broken X-(”, with the context “even the speakers are broken”

• The model treats “even” and “broken” as strongly negative and rates valence too low, but a broken speaker is usually not that upsetting; the model lacks that common sense.
• Irony is over-predicted: the model reacts to the gap between “very good” and the negative context, but the sentence has no other ironic span, so the gold score is low.

Conclusion and Limitations

#

1. Sentence level: M2 and M3 do not clearly help, though Part 2 hints that key words in the context may matter more than the full context.
2. Context level: M2 and M3 clearly help, so the sentiment of an ironic context only makes sense together with the sentence.
3. The span has to be right: automatic span prediction reaches only 0.405 F1, and with predicted spans the context-arousal gain disappears.

The limitations match the ROCLING version: the dataset is small and does not cover every ironic pattern, so it is best used as a starting point for more samples or annotation guidelines. The span predictor also has plenty of room to improve.