DVA-BERT: Sentiment-Based Masked Language Modeling for Valence–Arousal Prediction

Paper Info

#

This was one of the first research topics I started in the NLP Lab. In early 2020 I wrote the first version in the IJCAI-PRICAI 2020 format (A Novel Dimensional Valence-Arousal Prediction Model Using Masked Sentiment Words Approach), then extended it into a journal paper for Applied Intelligence. Per the paper’s CRediT statement, I was responsible for conceptualization, methodology, experiments, and writing.

Motivation

#

Why valence–arousal

#

Most sentiment analysis stops at positive versus negative, which says nothing about how strong a feeling is. Dimensional valence–arousal (DVA) uses two continuous values instead:

• Valence: the direction of the emotion, positive or negative.
• Arousal: how excited the emotion is, calm or agitated.

This matters for mental-health text. Comments that are negative and highly aroused, for example, show up in depression-related posts and may signal unstable emotions or negative behavior; a binary label cannot capture that.

BERT’s random masking ignores sentiment

#

BERT is pre-trained with masked language modeling (MLM): mask random tokens, then predict them back. That teaches the model to read context, but every token has the same chance of being masked, no matter how much it matters to the sentiment.

Earlier work that applied BERT to sentiment analysis mostly swapped in new input data and a new output layer and fine-tuned, without treating the domain’s sentiment words specially. This paper’s idea: since masking forces the model to understand the context around the masked position, mask the sentiment words specifically, and the model will focus on them and how they are used.

Model Architecture: DVA-BERT

#

DVA-BERT has two learning tasks:

• VAIE (valence–arousal intensity estimation, the main task): predict the sentence’s valence and arousal.
• RMSWP (random masked sentiment word prediction, the auxiliary task): replace BERT’s random masking with intensity-based masking of sentiment words, then predict which sentiment word was masked.

In the figure, “hate” in “I hate you very much” is a sentiment word and becomes [MASK]. The pipeline has five steps.

1. Word segmentation

#

The CKIP tagger splits sentence s_i into words X = {x1, …, xN}.

2. Sentiment intensity → masking probability

#

The sentiment lexicon is CVAW, where every word has a valence and an arousal score on a 1–9 scale.

For valence, first take the distance from the midpoint:

TI_V(sw) = | I_V(sw) − mid(I_V) |

The more positive or negative a word, the larger TI_V. Then min–max normalize and reverse with |norm − 1|:

P_V(sw) = | (TI_V(sw) − min(TI_V)) / (max(TI_V) − min(TI_V)) − 1 |

After reversing, the weaker the sentiment, the higher the masking probability. The paper’s reasoning: low-intensity sentiment words are the ones most likely to confuse the model about how strong an emotion they carry, so the model should be pushed to learn them.

For arousal, min–max normalize directly and reverse the same way:

P_A(sw) = | (I_A(sw) − min(I_A)) / (max(I_A) − min(I_A)) − 1 |

3. Building the masked sequence

#

Only sentiment words can be masked; every other word has probability 0. Each sentiment word gets one Bernoulli draw for valence and one for arousal, and is masked only if both hit:

P_mask(x_n) = Bernoulli(β · P_V(x_n)) × Bernoulli(β · P_A(x_n))

β is the initial masking probability, a hyperparameter that stops too many sentiment words in a sentence from being masked, which would leave the model unable to learn the original meaning. The final input is:

X' = [<CLS>, mn_1, …, mn_N, <EOS>]

4. Output layers for the two tasks

#

X' goes through BERT to produce hidden states H = [h_CLS, h_1, …, h_N, h_EOS].

5. Training

#

The two losses are combined with a weight:

L = α · l_VAI + (1 − α) · l_RMSW

α controls how much the model leans toward the main task. With α = 1 the auxiliary loss is ignored, but the input is still the masked sentence. All BERT parameters are fine-tuned with Adam.

Data

#

PsychPark depression corpus

#

The data come from PsychPark, an online psychiatric clinic run by volunteer psychiatrists, psychologists, and social workers. 2,300 sentences from depression-related posts fall into six categories:

Annotation: at least 10 annotators scored each sentence’s valence and arousal on a 1–9 scale independently. Scores beyond ±1.5 standard deviations from the mean were dropped as outliers before averaging.

Depression-related sentences are generally calm: arousal never goes above 6. Among the six categories, “suicide” has the lowest mean valence (1.97) and the highest mean arousal (5.77).

Sentiment word coverage

#

If a sentence has no lexicon word, there is nothing to mask. The paper counts how many sentences contain at least one CVAW sentiment word:

Coverage ranges from 70.9% to 91.9%, so each Bernoulli draw has plenty of different sentiment words to choose from and the model sees many different masks.

Experimental Setup

#

Splits: 7:1:2 train/validation/test with five-fold cross-validation; reported numbers are averages over the five test folds.

Hyperparameters:

Every model starts from Google’s pre-trained Chinese BERT. The best model is chosen by the best α–β combination.

Metrics: RMSE, MAE, MdAE (median absolute error), and R².

The 11 masking setups

#

Results

#

Comparing the 11 setups

#

• DVA-BERT has the lowest RMSE and the best standardized sum over the four metrics.
• Reversed setups win: MRV, MRA, and RVAU beat the unreversed MV, MA, and VAU. Low-intensity sentiment words are more frequent and used in more ways, so masking them teaches finer distinctions in sentiment.
• DVA-BERT-MS is the worst: it treats every sentiment word as a candidate regardless of intensity, masks too many words, and breaks the meaning of short sentences.
• DVA-BERT-VAI and RVAU also do well, so choosing masks from both dimensions beats choosing from one.
• The valence-only MRV is strong too, which suggests valence words carry much of a sentence’s sentiment.

Sentences with and without sentiment words

#

Test sentences split into those with and without a sentiment word (excerpt):

• Every model is more accurate on sentences with sentiment words, so those words drive sentence sentiment.
• On those sentences DVA-BERT clearly beats the unmasked NM on all four metrics, so learning to guess sentiment words helps.
• Interestingly, DVA-BERT also has the best RMSE, MAE, and R² on sentences without sentiment words: what it learns from masking transfers.

Loss weight α

#

• DVA-BERT reaches its lowest MAE at α = 0.8, i.e. 80% main task and 20% auxiliary task.
• MS, MA, and VAU, which mask many words, improve as α grows (leaning toward the main task); RM and VAI go the other way; the rest show no stable trend.
• The paper’s inference: when few words are masked, guessing them helps the main task; when too many are masked, guessing becomes a burden.
• Even at α = 1, with the auxiliary loss ignored, different masking setups still give different results, so which words are masked in the input already shapes what the model learns.

Initial masking probability β

#

The paper plots the average number of masked words during training against β:

• Models that already mask many words (RM, MS, VAU, MRA) do best with low β, which cuts the mask count.
• Models that start with few masked words (MV, RVAU, DVA-BERT) can afford a high β, choose where to mask more flexibly, and end up best.

Effect of pre-trained weights

#

Without pre-trained weights RMSE falls back from 1.33 to 1.79. Sentiment-word masking builds on BERT’s existing language ability.

Against sequence models

#

The three bidirectional sequence models use the same 768 hidden and embedding size as BERT, with the same learning rate and epochs. The BERT models are clearly better; among the sequence models the simplest, Bi-RNN, beats Bi-LSTM and Bi-GRU because the sentences are short and need no long-range memory.

Public dataset

#

To check that the method is not tied to depression text, the paper also tests on the Chinese sentence VA dataset of Yu et al., built from forum comments where most sentences contain sentiment words:

DVA-BERT beats random masking and no masking on all four metrics, so the masking scheme carries over to other domains.

Discussion and Limitations

#

• DVA-BERT works because it uses both V and A intensity to decide which sentiment words to mask, which focuses the model on the context around them.
• The union setups (VAU, RVAU) bring no clear gain: they mask so many sentiment words that the auxiliary task becomes too hard.
• Limitation: the training data are single sentences, so masking even one word can change the meaning and keep the model from learning the original sentiment. Future work could use multi-sentence text for more and more varied mask candidates, and apply the scheme to other attention-based models.

Code and Resources

#

Each Models/VA_BERT_* folder in the repo is one experiment. The README has a table mapping each folder and training script to the paper’s model names (for example, DVA-BERT is VA_BERT_mask_sentiment_prob_Intersection_reverse, trained with run_train_MSPRIN_kfold*.sh).

The repo also has sense_data/ (positive and negative sentiment words; add your own), data/ (five-fold splits and masking-coverage statistics), result_statistic/ (collects metrics into CSV), notebooks/ (plotting code for the paper figures), and Graph/ (the figures).