Researchers release 160,000-article Brazilian Portuguese financial news dataset

To evaluate performance, the team annotated subsets of the corpus both manually and using a Large Language Model (LLM). They tested three similarity-based methods: cosine similarity, K-Nearest Neighbors (KNN), and FAISS-based nearest-neighbor search, utilizing multilingual semantic embeddings. The experiments highlighted that the classification task is highly sensitive to label quality and class distribution.

Results showed that KNN achieved the best performance on the manually labeled subset, with 47.3% accuracy and a 31.9% F1-score. In contrast, while accuracy reached approximately 50% on the LLM-annotated subset, F1-scores dropped to near 20%. This discrepancy underscores the negative impact of label imbalance and noisy weak annotations, emphasizing the critical need for high-quality manual annotations in future research.