Case Study
Fraudulent Job Posting Detection
NLP fraud classifier on 17,880 EMSCAD job postings. 0.80 precision on the fraud class and 0.66 PR-AUC against a 0.048 base rate. 19 tests.
Executive Summary
A classifier that flags job postings as legitimate or fraudulent from hybrid text and metadata features, with a real-time Dash dashboard for probability visualization. Evaluated on the 17,880-posting EMSCAD dataset with a leakage-safe train/test split and 19 automated tests.
Problem & Constraints
Job platforms face rising scam volumes. Fraud is rare (only ~4.8% of EMSCAD postings), so raw accuracy is a poor headline. The system has to catch fraud (precision/recall on the minority class) on a heavily imbalanced corpus, keep scores interpretable, and never leak label-correlated fields into features.
Results
On the held-out EMSCAD test set the model reaches 96.9% accuracy with 0.80 precision on the fraud class and 0.66 PR-AUC against a 0.048 base rate, a ~14× lift over random on the metric that matters for rare-event detection.
Methodology
- Trained and evaluated on all 17,880 EMSCAD postings with a leakage-safe split (a prior leakage bug that inflated scores was identified and fixed)
- Engineered hybrid TF-IDF vectorization combined with VADER sentiment scoring on text + metadata features
- Optimized for fraud-class precision/recall and PR-AUC rather than accuracy alone, given the 4.8% positive base rate
- Deployed a real-time Dash dashboard for fraud probability visualization
- Locked behavior with 19 automated tests
Results & Metrics
| Metric | Result |
|---|---|
| Dataset | 17,880 EMSCAD postings |
| Accuracy | 96.9% |
| Fraud-class precision | 0.80 |
| PR-AUC (vs 0.048 base rate) | 0.66 |
| Features | TF-IDF + VADER + metadata |
| Tests | 19 |
| Code | GitHub repo |
Tech Stack
Python, Scikit-Learn, NLTK/VADER, TF-IDF, Dash, Plotly
Future Work
Deep learning text classifiers, multi-language support, API integration for job boards.