A Dimensional Valence-Arousal-Irony Dataset for Chinese Sentence and Context

Paper Info

#

In January 2022 I wrote the first draft of this work as first author (then called VAI); co-authors carried it forward into the ROCLING and journal versions. Annotation used the Irony Extension labeling website I built in 2021.

This paper is about the dataset itself: how irony is defined, how scores are given, how consistent the annotators are, and a BERT baseline showing the data can be learned. The journal version runs a fuller set of model experiments.

Motivation

#

The defining feature of irony is that the literal meaning is the opposite of the intended one. The most common pattern in Chinese wraps a negative event in a positive keyword:

“Great, it’s raining and I didn’t bring an umbrella…”

The context “raining without an umbrella” is negative, the keyword “Great” is positive, and the gap between them is what makes it ironic. So understanding irony requires looking at the context.

Earlier datasets mostly could not support that:

Studies also show the three indicators are correlated: stronger irony tends to come with lower valence (more negative) and higher arousal. The paper therefore argues for annotating V, A, and I together, and for scoring the sentence and its context separately, so a model can learn the sentiment shift inside irony.

Dataset: CDVAI

#

CDVAI (Chinese Dimensional Valence-Arousal-Irony) extends the NTU Irony Corpus, the only Chinese corpus that marks both ironic sentences and their context, by adding sentiment scores on top.

After filtering, 1,004 sentences remain, 843 of which have an ironic context to annotate.

• Sentence level: valence, arousal, and irony intensity.
• Context level: valence and arousal intensity (the context has no ironic keyword, so it gets no irony score).

Defining irony

#

Following Li & Huang (2020), irony is “an expression in which the true meaning is the opposite of the literal meaning”. In the NTU corpus the context is usually a negative description and carries the true meaning, while the ironic keyword is usually positive and flips the literal meaning.

Annotation guidelines (1–5)

#

Valence

• 1–2: negative
• 3: neutral, or cannot tell
• 4–5: positive

Arousal

Irony

Irony intensity depends on the gap between how positive the ironic keyword is and how negative the context is. The paper ranks the ironic keywords in the corpus as wonderful > great > very good > good.

Annotation quality

#

Each sentence was scored independently by three annotators (graduate students and an undergraduate, all native Chinese speakers aged 20–25), and the average of the three is the gold label.

The paper does not use Cohen’s kappa, because the labels are continuous intensities: the guidelines set score baselines rather than asking everyone to pick the same class. It measures agreement as the MAE between annotators instead:

All differences stay within one point, so the guidelines work. Irony and context arousal disagree the most, which makes sense since irony intensity is the most subjective.

An annotation example

#

The paper walks through a post complaining about a factory colleague. It opens with “Very good (applause)”, complains that the colleague has worked there for years but cannot send a meeting invite in Outlook, then swears and adds “give me your salary and I’ll send it for you”.

• Valence 1: the swearing and complaints are highly negative.
• Arousal 5: clear abusive and manic language.
• Irony 4: “very good” is only weakly positive, but the incident upset the writer a lot, so the gap is large; “give me your salary and I’ll send it for you” is a second ironic span, which pushes the score up.
• Context: the context carries the same sentiment as the whole sentence, so it also gets V 1 and A 5.

Score distribution

#

Averaged scores, rounded:

• Every sentence is ironic, so sentence valence is always negative (1–2).
• Sentence arousal sits mostly at 2–4, since most sentences carry emotion.
• Context arousal is clearly lower than sentence arousal: removing the ironic keyword also removes the exaggerated tone, which shows the keyword itself raises arousal.
• Irony is spread more evenly, but high scores are rare because they mean a large factual gap.

Model Architecture

#

To check that the annotations are learnable, the paper uses BERT as an encoder to predict sentence-level VAI and context-level VA.

Sentence level: the sentence goes through BERT and a linear layer outputs valence, arousal, and irony.

Context level: the context span is already marked in each sentence, so three setups are compared:

Experimental Setup

#

Splits: stratified sampling into train and test (7:3), with one tenth of train held out for validation.

Hyperparameters: Adam, 50 epochs. Sentence-level learning rates 4e-4, 4e-5, 4e-6; the context carries less information, so it tries smaller rates of 4e-5, 4e-6, 4e-7.

Six pre-trained models:

Results

#

Sentence level (MAE, lower is better)

#

• Valence is the easiest: every model is below 0.4.
• Arousal and irony both stay above 0.5. The paper’s explanation: arousal is judged from how serious the described event is, not just word intensity; and irony can come from several spans in one sentence, not a single keyword. Neither is easy to learn from surface wording.

Context level (MAE)

#

• Context valence: around 0.45 whatever the setup. M1 is still best for most models and M2 never beats it; the paper suspects that focusing on the span or concatenating adds noise. The best result is PM5 with M3 (0.388).
• Context arousal: much harder, around 0.8. Here M3 wins for most models, so seeing the sentence and the context together helps judge arousal; M2 barely helps.

Error analysis

#

“Very good… even the speakers are broken X-(”, with the context “even the speakers are broken”

• The model rates valence too negative, perhaps treating “even” and “broken” as strongly negative words. A broken speaker is usually not that upsetting, which takes common sense the model lacks.
• Irony is over-predicted: the model sees a large gap between “very good” and the negative valence, but the sentence has no other ironic span, so the gold score is low.

Overall, BERT’s main difficulties are inferring the speaker’s intent and judging how serious the event is.

Conclusion and Limitations

#

• CDVAI is the first Chinese dataset to annotate sentiment intensity for ironic context, which lets a model learn the sentiment contrast between a sentence and its context.
• The baseline shows the data is learnable, and M3 (sentence plus context) helps context arousal.
• The dataset is still small for machine learning and cannot cover every domain or ironic pattern. It works best as seed data for collecting more samples, as an annotation-guideline template, or combined with other irony corpora.

The journal version applies M1/M2/M3 at the sentence level too, switches to a different set of Chinese pre-trained models, and adds automatic context-span prediction; see the journal post.