The Problem Every Trader Faces
You’re watching the markets. News is flooding in: “Sensex plummets 690 points,” “Infosys rallies 3%,” “Fed signals rate cuts.” By the time you’ve read and processed 50 articles, you’ve missed the trade. Meanwhile, institutional investors are using AI to analyze thousands of articles in seconds.
This article explains how FinBERT, a specialized AI for financial text, actually works, what “latent space” means in practical terms, and most importantly, what we discovered when testing it on real US and Indian market news from August 2025.
FinBERT: An AI That Speaks Finance
FinBERT is BERT (an AI language model) that was additionally trained exclusively on financial texts.
BERT stands for Bidirectional Encoder Representations from Transformers. Let’s decode this jargon:
The Simple Explanation
BERT is an AI that understands context by reading text in both directions simultaneously. Unlike older AI that read left-to-right like humans, BERT reads the entire sentence at once, understanding how each word relates to every other word.
Why This Matters
Consider this sentence: “The stock bank rallied after the central bank cut rates.”
- Old AI: Reads left-to-right, might confuse the two meanings of “bank”
- BERT: Understands from context that first “bank” means financial stock, second means Federal Reserve
Think of it this way: a medical student becomes a doctor by studying medicine for years. Similarly, FinBERT became a “financial analyst” by studying millions of financial documents.
The Critical Difference: Original vs Fine-Tuned Models
Here’s what we found when testing different versions:
| Model Version | What Happened | Why It Failed/Succeeded |
|---|---|---|
| Original FinBERT | Classified “Dow rallies to all-time high” as NEGATIVE | Trained on 2008 crisis data, sees danger everywhere |
| Generic FinBERT | Detected zero positive news out of 32 items | Label mapping broken |
| Fine-tuned FinBERT | Correctly identified rallies as positive, crashes as negative | Balanced training on various market conditions |
The lesson: The original FinBERT is like a doctor who only worked in the ICU. They think everyone is dying. You need the fine-tuned version.
Latent Space: The Hidden Map of Market Sentiment
When FinBERT reads “Sensex crashes,” it doesn’t just output “negative.” It creates a 768-dimensional fingerprint of that news. Each dimension captures something different:
- How urgent is this news?
- Is it about earnings or policy?
- Does it affect specific sectors?
- What’s the magnitude of impact?
Since humans can’t visualize 768 dimensions, we compress them to 3 using PCA (Principal Component Analysis).
What PCA Actually Does
Imagine you’re describing a person to a sketch artist. You can’t convey every detail, so you focus on the most distinctive features: height, build, facial structure. PCA does the same with data, keeping the most important information.
Our findings:
- US Market: 83% of information preserved in 3D
- Indian Market: 87% of information preserved in 3D
This means we’re seeing the essential patterns, not every minor detail.
Our Testing Results: US vs Indian Markets
US Market Performance (32 news items, August 2025)
What Worked:
- “Dow rallies to all-time high” → Positive ✓ (100% confidence)
- “Tesla plunges 40% YTD” → Negative ✓ (100% confidence)
- “Fed maintains restrictive stance” → Neutral ✓
Accuracy: 85% with 98.2% average confidence
Indian Market Performance (32 news items, August 2025)
What Worked:
- “Sensex plummets 690 points” → Negative ✓ (100% confidence)
- “Infosys rallies 3%” → Positive ✓ (100% confidence)
What Failed:
- “Maruti zooms 9%” → Neutral ✗ (should be positive)
- “Bank Nifty shows buying interest” → Neutral ✗ (should be positive)
Accuracy: 75% with 99.3% average confidence
The Surprising Discovery
The model was MORE confident about Indian news (99.3%) than US news (98.2%), despite being trained primarily on Western data. However, confidence didn’t equal accuracy. It was 100% confident that “Maruti zooms 9%” was neutral news.
What the 3D Visualizations Tell Us
In our latent space visualizations, news items form distinct clusters:
Cluster Patterns We Found
- Rally Cluster: All positive market movements grouped together
- “Dow rallies to all-time high”
- “S&P 500 reaches record territory”
- “Nvidia surges 32% YTD”
- Distance between items: 0.82-0.85 (very similar)
- Crash Cluster: Negative news formed a separate group
- “S&P falls for fifth straight day”
- “Tesla plunges 40%”
- “Walmart sinks after earnings miss”
- Distance between items: 0.79-0.83 (very similar)
- Policy Cluster: Fed and regulatory news in the middle
- Neither strongly positive nor negative
- Acts as a “neutral zone” between extremes
Why Indian Market Results Were Surprising
Despite Indian-specific terminology, the model performed well because:
- Universal Patterns: “Rallies 3%” means the same in any market
- Number Recognition: “690 points down” is clearly negative
- Context Clues: Words around unknown terms provide meaning
However, it struggled with:
- Cultural Expressions: “Zooms” (very positive in India) interpreted as neutral
- Market Structure: Didn’t understand “upper circuit” or “lower circuit”
- Local Indices: Treated “Nifty” and “Sensex” as company names
Practical Trading Applications
For Intraday Traders
Use Case: Sentiment Divergence Strategy
- If price is falling but news sentiment is increasingly positive = potential reversal
- If price is rising but news sentiment turns negative = consider taking profits
Real Example from Our Data: When FinBERT showed 100% confidence negative sentiment on Tesla’s 40% decline, combined with multiple negative clusters, it signaled continued bearishness.
For Swing Traders
Use Case: Sentiment Momentum
- Track weekly sentiment averages
- Enter when sentiment shifts from negative to positive cluster
- Exit when positive sentiment becomes overcrowded
For Long-term Investors
Use Case: Contrarian Opportunities
- Maximum negative sentiment often marks bottoms
- When everything clusters in the negative space = potential buying opportunity
The Limitations You Must Understand
- Time Lag: Sentiment changes before price, but timing varies
- Magnitude Blindness: Treats “up 1%” and “up 10%” similarly
- Context Loss: Analyzes each news item independently
- Cultural Blind Spots: Misses market-specific expressions
Key Findings for Traders
From our August 2025 analysis:
- Fine-tuned models are 4x more accurate than original FinBERT
- Cross-market performance is surprisingly good (75% accuracy on Indian markets using US-trained model)
- High confidence doesn’t guarantee accuracy (100% confident mistakes happen)
- Similar news clusters together regardless of exact wording
- 87% of important information survives the 768D to 3D compression
Action Steps for Traders
- Don’t use original FinBERT: it’s biased toward negative sentiment
- Look for sentiment clusters, not individual news items
- Combine with price action: sentiment alone isn’t enough
- Track confidence levels: low confidence often means mixed signals
- Watch for divergences between sentiment and price
PIP Commands
# Install required packages
pip install torch transformers pandas numpy scikit-learn plotlyFull Python Code on FinBert Analysis and Latent Space Visualization
"""
FinBERT and Latent Space Visualization Demo
============================================
This script demonstrates how FinBERT encodes financial text and visualizes
the latent space representations using dimensionality reduction techniques.
Updated to use the best performing model: ipuneetrathore/bert-base-cased-finetuned-finBERT
"""
import numpy as np
import pandas as pd
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoModel
import plotly.graph_objects as go
import plotly.express as px
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')
# Set device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"Using device: {device}")
class FinBERTAnalyzer:
"""
A class to analyze financial text using FinBERT and visualize latent space
"""
def __init__(self, model_name='ipuneetrathore/bert-base-cased-finetuned-finBERT'):
"""
Initialize FinBERT model and tokenizer
Args:
model_name: Pre-trained FinBERT model from HuggingFace
Default: ipuneetrathore/bert-base-cased-finetuned-finBERT (best performer)
Alternatives:
- ProsusAI/finbert (original but biased toward negative)
- yiyanghkust/finbert-tone (has label mapping issues)
"""
print(f"Loading FinBERT model: {model_name}")
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModel.from_pretrained(model_name).to(device)
self.classifier = AutoModelForSequenceClassification.from_pretrained(model_name).to(device)
self.model.eval()
self.classifier.eval()
self.model_name = model_name
def encode_texts(self, texts, batch_size=8):
"""
Encode financial texts into latent space representations
Args:
texts: List of financial text strings
batch_size: Batch size for encoding
Returns:
numpy array of embeddings
"""
embeddings = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i+batch_size]
# Tokenize
inputs = self.tokenizer(
batch_texts,
padding=True,
truncation=True,
max_length=512,
return_tensors='pt'
).to(device)
# Get embeddings
with torch.no_grad():
outputs = self.model(**inputs)
# Use CLS token embedding (first token)
batch_embeddings = outputs.last_hidden_state[:, 0, :].cpu().numpy()
embeddings.append(batch_embeddings)
return np.vstack(embeddings)
def predict_sentiment(self, texts):
"""
Predict sentiment for financial texts with proper label mapping
Args:
texts: List of financial text strings
Returns:
List of sentiment labels and scores
"""
sentiments = []
scores = []
for text in texts:
inputs = self.tokenizer(text, return_tensors='pt', truncation=True, padding=True).to(device)
with torch.no_grad():
outputs = self.classifier(**inputs)
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
# Label mapping for ipuneetrathore model (works correctly)
# Index 0: negative, Index 1: neutral, Index 2: positive
sentiment_labels = ['negative', 'neutral', 'positive']
sentiment_idx = torch.argmax(predictions, dim=-1).item()
sentiment = sentiment_labels[sentiment_idx]
score = predictions[0][sentiment_idx].item()
sentiments.append(sentiment)
scores.append(score)
return sentiments, scores
def predict_sentiment_with_threshold(self, texts, threshold=0.7):
"""
Predict sentiment with confidence threshold for edge cases
Args:
texts: List of financial text strings
threshold: Minimum confidence threshold (default 0.7)
Returns:
List of sentiment labels and scores
"""
sentiments = []
scores = []
for text in texts:
inputs = self.tokenizer(text, return_tensors='pt', truncation=True, padding=True).to(device)
with torch.no_grad():
outputs = self.classifier(**inputs)
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
sentiment_labels = ['negative', 'neutral', 'positive']
max_score = torch.max(predictions).item()
if max_score < threshold:
# Low confidence - default to neutral
sentiment = 'neutral'
score = max_score
else:
sentiment_idx = torch.argmax(predictions, dim=-1).item()
sentiment = sentiment_labels[sentiment_idx]
score = predictions[0][sentiment_idx].item()
sentiments.append(sentiment)
scores.append(score)
return sentiments, scores
def reduce_dimensions(self, embeddings, method='pca', n_components=2):
"""
Reduce dimensionality of embeddings for visualization
Args:
embeddings: High-dimensional embeddings
method: 'pca' or 'tsne'
n_components: Number of dimensions to reduce to
Returns:
Reduced embeddings
"""
# Standardize the features
scaler = StandardScaler()
embeddings_scaled = scaler.fit_transform(embeddings)
if method == 'pca':
reducer = PCA(n_components=n_components, random_state=42)
elif method == 'tsne':
# Use max_iter for newer scikit-learn versions
reducer = TSNE(n_components=n_components, random_state=42, perplexity=30, max_iter=1000)
else:
raise ValueError(f"Unknown method: {method}")
reduced = reducer.fit_transform(embeddings_scaled)
if method == 'pca' and hasattr(reducer, 'explained_variance_ratio_'):
print(f"Explained variance ratio: {reducer.explained_variance_ratio_}")
print(f"Total variance explained: {sum(reducer.explained_variance_ratio_):.2%}")
return reduced
# Sample financial data for demonstration
def create_sample_data(market='US'):
"""
Create sample financial texts with different sentiments and topics
Based on real market news from August 2025
Args:
market: 'US' or 'India' to select appropriate dataset
"""
if market == 'India':
# Indian market data from August 2025
data = {
'text': [
# Positive sentiment - Recent Indian market news
"Infosys rallies 3% as IT stocks surge on US tech rally and Fed rate cut hopes boosting sentiment",
"Maruti Suzuki zooms 9% to Rs 14,100 as GST rejig talk sparks auto sector rally",
"HSBC raises Sensex target to 85,130 citing India's stable growth curve and strong fundamentals",
"IT giants TCS, Infosys, HCL Tech lead Nifty gains amid global tech optimism",
"Nifty Auto index outperforms with 4% gain on GST reform expectations",
"Bank Nifty shows buying interest from lower levels, holding above crucial 55,000 mark",
"Metal stocks shine with 2.5% gain as commodity prices show recovery signs",
"ChatGPT launches affordable $5 monthly plan targeting Indian users in second-largest market",
# Negative sentiment - Recent market challenges
"Sensex plummets 690 points as broad-based selloff hits key constituents",
"IndusInd Bank to exit Nifty50 from September 30, CRISIL maintains negative outlook",
"HDFC Bank drags index lower with 1.16% decline amid profit booking by FIIs",
"TCS disappoints with lacklustre Q1 results triggering risk-off sentiment",
"Nifty breaks below 20-day EMA support as bears tighten grip on markets",
"Foreign institutional investors continue selling spree causing market volatility",
"Banking stocks witness profit booking after recent rally, seven stocks decline",
"Weak global cues and geopolitical tensions drag Sensex down 1.04% to 80,787",
# Neutral sentiment - Regulatory and routine updates
"Markets closed on August 27 for Ganesh Chaturthi, trading resumes August 28",
"RBI MPC meeting scheduled for August 6 to decide on interest rate policy",
"NSE announces semi-annual index reshuffle affecting 46 Nifty Smallcap stocks",
"Amanta Healthcare sets IPO price band at Rs 120-126, opens September 1-3",
"GST field officers detect tax evasion of Rs 7.08 trillion over five years",
"Reliance Industries completes Rs 45.3 crore acquisition of Nauyaan Shipyard",
"Board meetings scheduled this week for Q1FY26 earnings announcements",
"Gland Pharma, Bandhan Bank among inclusions in Nifty index rejig",
# Technical and sector-specific news
"Nifty hovering near 24,590 support level, resistance seen at 24,800 zone",
"Bank Nifty RSI at 55 shows early signs of potential reversal from oversold zone",
"IT sector benefits from 150 basis points Fed rate cut expectations by year-end",
"Auto sector emerges as top performer with Ashok Leyland rallying 8%",
"Pharma stocks gain momentum as Marksans receives UK marketing authorization",
"Infrastructure shares rise on government's capital expenditure push",
"Consumer durables rally on GST rationalization hopes ahead of Diwali",
"FII selling continues but DIIs provide support limiting downside"
],
'category': [
# Categories for positive news
'it_sector', 'auto_rally', 'target_upgrade', 'tech_gains',
'sector_performance', 'technical_support', 'commodities', 'tech_expansion',
# Categories for negative news
'market_fall', 'index_exclusion', 'stock_decline', 'earnings_miss',
'technical_breakdown', 'fii_selling', 'profit_booking', 'global_cues',
# Categories for neutral news
'market_holiday', 'monetary_policy', 'index_rejig', 'ipo',
'regulatory', 'corporate_action', 'earnings_schedule', 'index_changes',
# Categories for technical/sector news
'technical_analysis', 'momentum_indicator', 'rate_expectations', 'sector_leader',
'pharma_news', 'infrastructure', 'consumer_goods', 'market_support'
]
}
else:
# US market data from August 2025 (default)
data = {
'text': [
# Positive sentiment - Recent US market news
"Dow Jones Industrial Average rallies to all-time high of 45,631 after Fed Chair Powell signals rate cuts ahead",
"S&P 500 reaches record territory as Powell's Jackson Hole speech opens door to September rate cut",
"Nvidia shares surge near record highs, up 32% year-to-date ahead of crucial earnings report",
"Apple jumps 1.7% on reports of Google Gemini partnership to power Siri AI overhaul",
"Small caps rally with Russell 2000 jumping 3% as rate cut expectations boost sentiment",
"Microsoft Azure cloud revenue grows 34% as AI demand drives data center expansion",
"S&P upgrades ratings for major US financial institutions following sovereign credit boost",
"Bullish cryptocurrency exchange prices IPO at $37, above expected range with $5.4 billion valuation",
# Negative sentiment - Recent market challenges
"S&P 500 falls for fifth straight day, longest losing streak since January on inflation concerns",
"Tesla stock plunges 40% year-to-date amid Musk distractions and Trump administration uncertainty",
"Walmart shares sink 4.5% after missing profit expectations despite revenue beat",
"Foreign portfolio investors continue selling streak causing volatility in emerging markets",
"Manufacturing growth raises inflation fears, 10-year Treasury yields rise to 4.33%",
"Nvidia pulls down tech sector with 3.5% decline as rotation out of megacaps continues",
"Producer Price Index rises 0.9% in July, far exceeding 0.2% forecast raising rate concerns",
"China tariff uncertainties weigh on Apple stock, down 11% in April on supply chain fears",
# Neutral sentiment - Fed and regulatory updates
"Federal Reserve maintains restrictive policy stance awaiting more inflation data",
"Fed officials signal September rate cut probability at 85% according to CME FedWatch",
"Treasury Secretary Bessent suggests Fed should cut rates by 150 basis points",
"Trump administration expands steel and aluminum tariffs to 400 additional product categories",
"Powell emphasizes Fed independence amid political pressure from White House",
"Core PCE inflation expected at 2.9% year-over-year for July, up from 2.8% in June",
"Housing starts rise 5.2% in July beating estimates but permits decline 2.8%",
"Labor market shows unusual balance with both supply and demand for workers slowing",
# Sector and company-specific news
"Mega-cap tech stocks add collective $370 billion in market cap on rate cut optimism",
"Regional banks jump on expectations lower rates will improve net interest margins",
"Intel gains 3% after Trump praises government's 9.9% stake acquisition deal",
"Amazon Web Services maintains cloud leadership ahead of Microsoft and Google",
"Home Depot rallies on housing market recovery hopes amid potential rate cuts",
"Semiconductor sector surges with AMD up 5.4% leading broad tech rally",
"Palantir drops 9% becoming S&P 500's worst performer on valuation concerns",
"Energy sector faces headwinds as oil prices decline on global growth worries"
],
'category': [
# Categories for positive news
'market_rally', 'fed_policy', 'earnings', 'ai_partnership',
'small_caps', 'cloud_growth', 'credit_upgrade', 'ipo',
# Categories for negative news
'market_decline', 'stock_specific', 'earnings_miss', 'capital_flows',
'inflation', 'tech_rotation', 'economic_data', 'tariffs',
# Categories for neutral news
'fed_policy', 'rate_expectations', 'treasury', 'trade_policy',
'fed_independence', 'inflation_data', 'housing', 'labor_market',
# Categories for sector news
'tech_sector', 'banking', 'semiconductor', 'cloud_computing',
'housing_sector', 'chips', 'software', 'energy'
]
}
return pd.DataFrame(data)
def visualize_latent_space(analyzer, df):
"""
Create interactive visualizations of the latent space
"""
print("\nEncoding texts into latent space...")
embeddings = analyzer.encode_texts(df['text'].tolist())
print("Predicting sentiments...")
sentiments, confidence = analyzer.predict_sentiment(df['text'].tolist())
df['sentiment'] = sentiments
df['confidence'] = confidence
# Create visualizations for both PCA and t-SNE
fig = go.Figure()
methods = ['pca', 'tsne']
for method in methods:
print(f"\nApplying {method.upper()} for dimensionality reduction...")
reduced = analyzer.reduce_dimensions(embeddings, method=method, n_components=3)
# Add to dataframe
df[f'{method}_x'] = reduced[:, 0]
df[f'{method}_y'] = reduced[:, 1]
df[f'{method}_z'] = reduced[:, 2]
# Create 3D scatter plot (PCA)
fig1 = go.Figure(data=[go.Scatter3d(
x=df['pca_x'],
y=df['pca_y'],
z=df['pca_z'],
mode='markers+text',
marker=dict(
size=10,
color=df['confidence'],
colorscale='Viridis',
showscale=True,
colorbar=dict(title="Confidence"),
line=dict(width=1, color='white')
),
text=df['sentiment'],
hovertemplate='<b>%{text}</b><br>' +
'Category: ' + df['category'] + '<br>' +
'Confidence: %{marker.color:.2f}<br>' +
'Text: ' + df['text'].str[:50] + '...<br>' +
'<extra></extra>'
)])
fig1.update_layout(
title="FinBERT Latent Space Visualization (PCA)",
scene=dict(
xaxis_title="PC1",
yaxis_title="PC2",
zaxis_title="PC3"
),
width=900,
height=700
)
# Create 2D comparison plot
fig2 = go.Figure()
# PCA subplot
fig2.add_trace(go.Scatter(
x=df['pca_x'],
y=df['pca_y'],
mode='markers',
name='PCA',
marker=dict(
size=12,
color=df['sentiment'].map({'positive': 'green', 'negative': 'red', 'neutral': 'gray'}),
line=dict(width=1, color='white')
),
text=df['category'],
hovertemplate='Category: %{text}<br>Sentiment: ' + df['sentiment'] + '<br><extra></extra>'
))
fig2.update_layout(
title="FinBERT Latent Space - 2D Projection Comparison",
xaxis_title="First Component",
yaxis_title="Second Component",
width=900,
height=600,
showlegend=True
)
return fig1, fig2, df
def analyze_cluster_characteristics(df, embeddings):
"""
Analyze characteristics of different clusters in latent space
"""
print("\n" + "="*50)
print("LATENT SPACE ANALYSIS RESULTS")
print("="*50)
# Sentiment distribution
print("\nSentiment Distribution:")
print("-" * 30)
sentiment_counts = df['sentiment'].value_counts()
for sentiment, count in sentiment_counts.items():
print(f" {sentiment.capitalize()}: {count} ({count/len(df)*100:.1f}%)")
# Average confidence by sentiment
print("\nAverage Confidence by Sentiment:")
print("-" * 30)
for sentiment in df['sentiment'].unique():
avg_conf = df[df['sentiment'] == sentiment]['confidence'].mean()
print(f" {sentiment.capitalize()}: {avg_conf:.3f}")
# Category analysis
print("\nTop Categories by Sentiment:")
print("-" * 30)
for sentiment in ['positive', 'negative', 'neutral']:
cats = df[df['sentiment'] == sentiment]['category'].value_counts().head(3)
if len(cats) > 0:
print(f"\n {sentiment.capitalize()}:")
for cat, count in cats.items():
print(f" - {cat}: {count}")
# Embedding statistics
print("\nEmbedding Space Statistics:")
print("-" * 30)
print(f" Embedding dimensions: {embeddings.shape[1]}")
print(f" Mean norm: {np.linalg.norm(embeddings, axis=1).mean():.3f}")
print(f" Std norm: {np.linalg.norm(embeddings, axis=1).std():.3f}")
# Main execution
if __name__ == "__main__":
print("FinBERT Latent Space Demonstration")
print("="*50)
# Ask user which market to analyze
print("\nSelect market to analyze:")
print("1. US Market (default)")
print("2. Indian Market")
market_choice = input("\nEnter choice (1 or 2, press Enter for default): ").strip()
if market_choice == '2':
market = 'India'
print("\n>>> Analyzing Indian Financial Market News <<<")
else:
market = 'US'
print("\n>>> Analyzing US Financial Market News <<<")
# Initialize analyzer with the best performing model
analyzer = FinBERTAnalyzer()
# Create sample data for selected market
print(f"\nCreating sample {market} financial data...")
df = create_sample_data(market=market)
print(f"Created {len(df)} sample texts")
# Perform analysis
embeddings = analyzer.encode_texts(df['text'].tolist())
fig1, fig2, df_results = visualize_latent_space(analyzer, df)
# Display results
analyze_cluster_characteristics(df_results, embeddings)
# Show visualizations
print("\nGenerating interactive visualizations...")
fig1.show()
fig2.show()
# Save results with market-specific filename
filename_suffix = f"_{market.lower()}"
print(f"\nSaving {market} market results...")
df_results.to_csv(f'finbert_analysis_results{filename_suffix}.csv', index=False)
fig1.write_html(f'finbert_3d_visualization{filename_suffix}.html')
fig2.write_html(f'finbert_2d_visualization{filename_suffix}.html')
print(f"\nAnalysis complete! Check the generated files:")
print(f" - finbert_analysis_results{filename_suffix}.csv")
print(f" - finbert_3d_visualization{filename_suffix}.html")
print(f" - finbert_2d_visualization{filename_suffix}.html")
# Example of using the encoded representations
print("\nExample: Finding similar texts in latent space")
print("-"*50)
# Calculate cosine similarity
from sklearn.metrics.pairwise import cosine_similarity
query_idx = 0 # First text
query_embedding = embeddings[query_idx].reshape(1, -1)
similarities = cosine_similarity(query_embedding, embeddings)[0]
# Find most similar texts
similar_indices = np.argsort(similarities)[::-1][1:4] # Top 3 excluding itself
print(f"\nQuery text: '{df.iloc[query_idx]['text'][:80]}...'")
print(f"Query sentiment: {df_results.iloc[query_idx]['sentiment']}")
print("\nMost similar texts in latent space:")
for i, idx in enumerate(similar_indices, 1):
print(f"\n{i}. Similarity: {similarities[idx]:.3f}")
print(f" Text: '{df.iloc[idx]['text'][:80]}...'")
print(f" Sentiment: {df_results.iloc[idx]['sentiment']}")
# Additional analysis: Sentiment confidence distribution
print("\n" + "="*50)
print("SENTIMENT CONFIDENCE ANALYSIS")
print("="*50)
for sentiment in ['positive', 'negative', 'neutral']:
sentiment_data = df_results[df_results['sentiment'] == sentiment]
if len(sentiment_data) > 0:
print(f"\n{sentiment.capitalize()} Sentiment:")
print(f" Count: {len(sentiment_data)}")
print(f" Mean confidence: {sentiment_data['confidence'].mean():.3f}")
print(f" Min confidence: {sentiment_data['confidence'].min():.3f}")
print(f" Max confidence: {sentiment_data['confidence'].max():.3f}")
# Model performance summary
print("\n" + "="*50)
print("MODEL PERFORMANCE SUMMARY")
print("="*50)
print(f"Model used: {analyzer.model_name}")
print(f"Market analyzed: {market}")
print(f"Total texts analyzed: {len(df)}")
print(f"Embedding dimensions: {embeddings.shape[1]}")
print(f"Average processing confidence: {df_results['confidence'].mean():.3f}")
# Check for potential misclassifications (optional manual review)
print("\n" + "="*50)
print("SAMPLE CLASSIFICATIONS FOR REVIEW")
print("="*50)
# Show a few examples from each sentiment
for sentiment in ['positive', 'negative', 'neutral']:
sentiment_samples = df_results[df_results['sentiment'] == sentiment].head(2)
if len(sentiment_samples) > 0:
print(f"\n{sentiment.capitalize()} Examples:")
for _, row in sentiment_samples.iterrows():
print(f" - '{row['text'][:60]}...' (conf: {row['confidence']:.3f})")
# Market-specific terminology analysis (for Indian market)
if market == 'India':
print("\n" + "="*50)
print("INDIAN MARKET TERMINOLOGY DETECTION")
print("="*50)
indian_terms = ['Sensex', 'Nifty', 'FII', 'DII', 'GST', 'Rs', 'crore', 'lakh', 'MPC', 'RBI']
detected_terms = []
for term in indian_terms:
count = sum(1 for text in df['text'] if term.lower() in text.lower())
if count > 0:
detected_terms.append((term, count))
if detected_terms:
print("\nIndian financial terms detected:")
for term, count in detected_terms:
print(f" - {term}: {count} occurrences")
# Check how model handles Indian-specific news
print("\n" + "="*50)
print("MODEL PERFORMANCE ON INDIAN-SPECIFIC CONTENT")
print("="*50)
# Find texts with Indian terms
indian_specific = df_results[df_results['text'].str.contains('Sensex|Nifty|FII|GST', case=False, na=False)]
if len(indian_specific) > 0:
print(f"\nTexts with Indian terms: {len(indian_specific)}")
sentiment_dist = indian_specific['sentiment'].value_counts()
print("\nSentiment distribution for Indian-specific content:")
for sentiment, count in sentiment_dist.items():
print(f" - {sentiment.capitalize()}: {count} ({count/len(indian_specific)*100:.1f}%)")
print(f"\nAverage confidence on Indian content: {indian_specific['confidence'].mean():.3f}")The Bottom Line
FinBERT with latent space visualization gives you institutional-level sentiment analysis capabilities. It’s not about replacing your trading strategy; it’s about processing information at scale. While a human might read 50 articles per day, FinBERT can analyze 5,000 in minutes, showing you patterns invisible to the naked eye.
The latent space reveals that financial news, whether from Wall Street or Dalal Street, follows universal patterns. Markets may speak different languages, but fear and greed sound the same everywhere.
Remember: This tool shows you what the news is saying, not what the market will do. The gap between sentiment and price action is where opportunities live.
