FinSentKH–CafeF: Aspect-Based Sentiment Analysis for Vietnamese Financial Headlines

This repository implements and validates FinSentKH, an Aspect-Based Sentiment Analysis (ABSA) model for financial news, and tests whether its sentiment signal is associated with abnormal stock returns on the Vietnamese market (CafeF headlines, 2019–2022).

The work has two parts:

1. Intrinsic evaluation — train/evaluate FinSentKH on the SEntFiN 1.0 benchmark and compare against baselines (finBERT, FinABSA) and zero-shot LLMs (Qwen3-8B, Gemma4-31B).
2. Extrinsic (market) validation — apply FinSentKH zero-shot to Vietnamese CafeF headlines, link entities to tickers, and test the relationship between predicted sentiment and abnormal returns via event study and panel regression.

This is an academic study of statistical association, not an investment or trading system.

Background

A single headline often mentions multiple companies with different sentiment ("HPG rose sharply, while SSI declined"). Sentence-level sentiment models assign one label per sentence and can misattribute signal across tickers. ABSA solves this by predicting sentiment for one target entity at a time.

FinABSA (baseline) does this by masking only the target entity ([TGT]), leaving other company names visible — which means predictions can still be influenced by the model's prior knowledge about those other companies (non-stationary knowledge baked into the pretrained model).

FinSentKH (this work) fine-tunes T5-Large (770M params) on SEntFiN 1.0 using Target–Other masking: the target entity is replaced with Target and every other entity is replaced with Other. This removes company-name information the model could otherwise lean on, giving more consistent predictions independent of which company appears in the sentence.

Original: "Gold shines on seasonal demand; Silver dull."
Masked:   "Target shines on seasonal demand; Other dull."

FinSentKH is evaluated zero-shot on Vietnamese CafeF headlines with no additional fine-tuning on Vietnamese data.

Research questions

• RQ1 (intrinsic): How does FinSentKH perform on SEntFiN test (English) and on Vietnamese CafeF headlines (zero-shot), versus baselines and large LLMs?
• RQ2 (market): Does the predicted sentiment score predict abnormal returns after a news event? Which aggregation method is strongest, and how do results hold up with firm and date fixed effects?
• RQ3 (robustness): Do results hold across lag structures (0, 1, 2, 3, 5, 10 sessions), entity-linking strictness (strict vs. broad), return-adjustment method (market-model vs. market-adjusted), and out-of-time splits?

Pipeline

CafeF sitemap → parsing → entity linking → target masking → FinSentKH
   → ticker–day aggregation → statistical tests (event study / panel regression)

Three separated responsibilities:

• Entity linking: company name → ticker.
• FinSentKH: sentiment classification for a specified target ticker only (not NER, not ticker prediction).
• Market pipeline: ticker–day → abnormal return, event study, robustness checks.

Method summary

• Abnormal return: market model R_i,t = α_i + β_i·R_m,t + ε_i,t, AR_i,t = R_i,t − Ê(R_i,t), fit on ≥20 pre-event daily observations, with VNINDEX as the market proxy.
• Event windows: [0,0], [0,+1], [0,+3] trading sessions after the first session following article publication.
• Inference: one-sample t-test, sign-flip permutation test, bootstrap 95% CI; news-day vs. no-news-day contrasts; sentiment-group contrasts; ticker/date two-way fixed-effects panel regression; lag robustness at 0, 1, 2, 3, 5, 10 sessions.

Data

Sample covers four October periods (2019–2022), 5,133 articles total, 839 strict-confidence (≥0.95) entity-ticker predictions, 796 of which link to price data. Label distribution is imbalanced (759 neutral / 54 positive / 26 negative), so confidence intervals on group-level tests are wide.

Key results (summary)

• Intrinsic: FinSentKH (Acc 0.972 / Macro-F1 0.975) outperforms finBERT, FinABSA, and zero-shot LLMs (Qwen3-8B, Gemma4-31B) on SEntFiN test, and scores the same on zero-shot CafeF evaluation.
• Market association: Same-day CAR is not statistically different from zero in any of the four periods. [0,+3] CAR is negative and nominally significant in 2020 and 2021 only. Fixed-effects sentiment coefficients are significant (and positive) only in 2020; sign and magnitude are unstable across years. Lag robustness shows no stable monotonic delayed effect.
• Conclusion: the pipeline is complete and reproducible, and a few period-specific estimates reach nominal significance, but sign and magnitude are not stable across years, event windows, or lags. Results do not establish a causal effect, a stable predictive relationship, or a viable trading strategy.

See slide.pdf / main (2).pdf for the full set of tables and figures.

Repository layout

• run_pipeline.py — end-to-end CafeF pipeline entry point.
• data/sentfin/ — SEntFiN source, transformed inputs, benchmark sample.
• data/cafef_oct2022/ — CafeF articles, predictions, market data, analysis.
• results/finabsa/ — benchmark evaluation outputs and detailed predictions.
• results/reports/ — generated Markdown and PDF reports.
• docs/ — dataset card, experiment notes, evaluation methodology, and the original FinABSA library README.
• tests/ — pipeline and CLI contract tests.

Key scripts:

• prepare_sentfin_data.py, train.py, evaluate_intrinsic.py — SEntFiN fine-tuning and intrinsic evaluation of FinSentKH.
• prepare_cafef_data.py, finish_cafef_fetch.py, download_market_data.py — CafeF news and market-data collection.
• prepare_model_inputs.py, run_finabsa_on_cafef.py, normalize_model_output.py, validate_predictions.py — Target–Other masking, zero-shot inference on CafeF, and prediction validation.
• audit_dataset.py, audit_market_data.py — data-quality checks.
• run_experiments.py, run_robustness.py — event study, panel regression, and robustness checks (lag structure, entity-linking strictness, return adjustment method).
• generate_report.py, build_report.py, md_2_pdf.py — report generation.
• FinABSA.py, zsABSA.py — baseline model wrappers.

Setup

Install the CafeF and analysis dependencies:

python3 -m pip install -r requirements_cafef.txt

Model training/evaluation additionally requires PyTorch, Transformers, and the libraries used by the selected FinABSA/FinSentKH checkpoint.

Common commands

python3 -m unittest tests/test_pipeline.py
python3 audit_dataset.py
python3 run_pipeline.py --skip-market

The full pipeline expects CafeF model predictions at data/cafef_oct2022/model_predictions.csv. Use --model or --raw-output through the pipeline to create or normalize them. Full methodology and output descriptions are in docs/README_CAFEF_PIPELINE.md.

Limitations

• Single-month (October) sample per year; regime dependence remains high.
• CafeF is a single news source; article selection is not random.
• Entity linking has false positives/negatives; strong class imbalance in labels.
• Daily prices provide no intraday timing; sparse panels require HC1 covariance fallback where HC3 is not well defined.
• Many statistical tests are run without multiple-testing correction — nominal p-values should be read cautiously.

Future work

1. Independent gold labels with adjudication, F1/MCC/kappa, and error analysis.
2. Store logits/probabilities for calibration and uncertainty-aware aggregation.
3. Extend time coverage, broaden entity linking, add sector/regime interactions.
4. Pre-registered tests, placebo/factor models, multiple-testing correction, out-of-time forecasting.

License

Apache-2.0 (see LICENSE).

Authors

Lê Khắc Quang Huy, Phùng Hữu Khoa, Nguyễn Nam Khánh, Lê Tuấn Duy —- Natural Language Processing course project.