Appearance
Applying sentiment analysis to historical text is defensible only when you (1) validate any modern lexicon or model against a hand-coded sample from your own period, (2) measure and control for OCR noise before scoring, and (3) report sentiment as a distribution with uncertainty bands rather than a single trend line. The single biggest failure mode is treating a present-day tool as period-neutral. Below is the checklist I use on every project, with the reasoning behind each step.
Why can't I just run VADER and trust the numbers?
Off-the-shelf tools encode modern valence. "Awful" meant awe-inspiring into the 18th century; "nice" once meant foolish; "let" could mean to hinder. A lexicon scores these on today's polarity, so a corpus full of period-shifted vocabulary produces systematically wrong scores that look perfectly clean. The numbers are precise and false — the worst combination.
The fix is not to abandon lexicons but to calibrate them. Pull 200 random passages, hand-code each as negative / neutral / positive, then compare to the tool. If raw agreement (Cohen's kappa) is below about 0.4, the tool is not measuring what you think.
A reproducible scoring pipeline
Keep the pipeline explicit and version-controlled so a reviewer can rerun it. A minimal lexicon pass in Python:
python
import pandas as pd
from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
df = pd.read_parquet("corpus_clean.parquet") # one row per document
df["compound"] = df["text"].apply(lambda t: sia.polarity_scores(t)["compound"])
# bin by decade and summarise as a DISTRIBUTION, not a single mean
summary = (df.groupby(df["year"] // 10 * 10)["compound"]
.agg(["mean", "std", "count"]))
print(summary)Note: compound ranges from -1 to 1. Document length affects it, so normalise by sampling equal-length windows or report per-sentence scores aggregated up.
How do I stop OCR noise from skewing results?
Garbled tokens (tlie, bſt, rnodern) never match the lexicon, so noisy documents drift toward neutral and look artificially calm. Always:
- Compute character error rate (CER) on a ground-truth sample.
- Drop or down-weight documents above your CER threshold (I use 10%).
- Re-run the validation set through the same noisy pipeline so accuracy reflects real conditions.
| Approach | Transparency | Period sensitivity | Setup cost |
|---|---|---|---|
| Modern lexicon (VADER) | High | Low (needs calibration) | Minutes |
| Custom period lexicon | High | High | Days |
| Fine-tuned transformer | Low | High (if labels are period-specific) | Weeks + labels |
Building a period-aware lexicon
If calibration fails, derive valence from your own corpus. Seed a small list of clearly positive/negative period words, then expand using nearest neighbours in a word-embedding model trained on the same corpus (see word vectors below). Hand-check every added term — embeddings cluster by topic as much as by sentiment.
Reporting sentiment honestly
A single descending line is rarely the truth. Bootstrap-resample documents within each time bin 1,000 times and plot the 2.5th-97.5th percentile band. If your "decline in optimism" disappears inside that band, you have noise, not a finding. State your lexicon version, CER threshold, sample size and kappa in the methods — these four numbers make the work reproducible.
Genres that don't carry sentiment
Before any of this, ask whether the genre expresses affect at all. Parish registers, customs ledgers and ship logs are deliberately flat; measuring their "sentiment" produces meaningless drift dominated by formulaic phrasing. Sentiment analysis fits letters, diaries, pamphlets, reviews and editorials — not bookkeeping.
Key Takeaways
- Never trust a modern lexicon on historical text without calibrating it on a hand-coded sample from your own period.
- Validate on 200-300 passages and report Cohen's kappa; below ~0.4 the tool is unreliable.
- Measure OCR character error rate first; noisy tokens bias scores toward false neutrality.
- Prefer transparent lexicons over black-box models unless you have thousands of period-specific labels.
- Report distributions with bootstrap confidence bands, not single trend lines.
- Confirm your genre actually carries affect before measuring it.
- Record lexicon version, CER threshold, sample size and kappa for reproducibility.
Frequently Asked Questions
Can I use modern sentiment lexicons like VADER on 18th-century text?
Only with caution. VADER and AFINN encode present-day word valence, so historically shifted words such as "awful" (once meaning awe-inspiring) or "gay" will be mis-scored. Always validate against a hand-coded sample from your own period.
How big should my validation set be?
Hand-code at least 200-300 randomly sampled passages against your tool's output and compute agreement. Below ~150 your confidence interval on accuracy is too wide to publish.
Lexicon or fine-tuned model — which should I pick?
Start with a lexicon for transparency and speed, then move to a fine-tuned transformer only if you have a few thousand period-specific labels and the lexicon's accuracy on your validation set is unacceptable.
Does OCR noise affect sentiment scores?
Yes, badly. Garbled tokens silently drop out of lexicon matches, biasing scores toward neutral. Measure character error rate first and clean or filter before scoring.
How do I report uncertainty in sentiment trends?
Bootstrap-resample documents within each time bin and plot the confidence band, not just the mean line. A trend that vanishes inside the band is not a finding.
Is sentiment even a coherent concept for historical genres?
Not always. Legal records, ship logs and parish registers carry little affect by design. Confirm your genre actually expresses sentiment before measuring it.