Data Pipeline / NLP
Market News Scraping + Sentiment Analysis
A news monitoring system that scrapes CNBC Indonesia and Bloomberg Technoz, runs sentiment analysis with IndoBERT, and visualizes trends.
Overview
This system scrapes financial news from CNBC Indonesia and Bloomberg Technoz, performs sentiment analysis using IndoBERT (a Bahasa Indonesia BERT model), and presents sentiment trends over time through a dashboard.
Problem
Market participants need to track how news sentiment shifts over time, but reading and categorizing articles manually doesn't scale across multiple sources.
Goal
Build an automated pipeline that collects, analyzes, and visualizes news sentiment so trends can be spotted without manual reading.
Architecture
- Web scraper for CNBC Indonesia and Bloomberg Technoz.
- Text cleaning and preprocessing pipeline.
- IndoBERT-based sentiment classification.
- Time-series storage for trend analysis.
- Dashboard for sentiment visualization.
System Flow
Input
Scheduler triggers scrapers at configured intervals.
Process
Scraper extracts article text, timestamp, and metadata.
AI Layer
Text is cleaned and passed to the sentiment model.
Storage/API
Results are stored with article reference and timestamp.
Review
Dashboard queries aggregated sentiment for trend display.
Tech Stack
Key Features
- Scheduled scraping from multiple news sources.
- Bahasa Indonesia sentiment analysis using IndoBERT.
- Sentiment trend visualization over time.
- Article-level detail view with sentiment breakdown.
AI / ML Component
- IndoBERT fine-tuned for financial news sentiment.
- Text preprocessing optimized for Indonesian news text.
- Aggregated sentiment scoring per source and time period.
Data Flow
- 1Scheduler triggers scrapers at configured intervals.
- 2Scraper extracts article text, timestamp, and metadata.
- 3Text is cleaned and passed to the sentiment model.
- 4Results are stored with article reference and timestamp.
- 5Dashboard queries aggregated sentiment for trend display.
Challenges
- News site structure changes breaking the scraper.
- Handling mixed-language content (Indonesian + English financial terms).
- Domain-specific sentiment nuances in financial news.
Solution / Trade-off
- Prioritize scraper maintainability over coverage depth.
- Use a general IndoBERT model rather than costly domain fine-tuning for MVP.
- Store raw article text alongside sentiment for audit.
Result
Working scraping pipeline and sentiment analysis model. Dashboard displays time-series sentiment trends.
Screenshot / Demo Placeholder
/images/news-sentiment-placeholder.png
Replace this area with real screenshots, dashboard captures, architecture diagrams, or a short demo video once the asset is ready.
GitHub / Live Link Placeholder
What I Would Improve
- Add more news sources.
- Build anomaly detection for sudden sentiment shifts.
- Add topic clustering to see which subjects drive sentiment.