Heimdall: an architecture for online machine learning through imbalanced data

Abstract

Machine Learning (ML) algorithms have been increasingly applied to domain areas where data is available for process automation. However, in the case of imbalanced data applications, the training process is challenging since ML algorithms intrinsically learn from balanced distributions. This research proposes Heimdall, a resourceful architecture for online ML through imbalanced data. Designed as a service for prediction and analysis requests, Heimdall serves existing applications from external systems, extending artificial intelligence capabilities and automated processes to traditional applications supervised by experts. The architecture focuses on efficiently solving imbalance and improving performance through a set of good practices compiled from mapped studies – such as probability threshold optimization, high-performance sampling, and ensemble learning. Furthermore, Heimdall proposes and evaluates the efficiency of novel functionalities. Firstly, a new performance metric corrects precision-recall balance according to the application’s needs, enhancing probability threshold optimization. Secondly, the architecture independently automates data management and training pipelines through two rule-based reactive agents constantly monitoring data changes and model degradation to trigger processes. These reactive agents compose a strategy for adaptive efficiency, enabling better and more stable performance by sacrificing efficiency in warm-up conditions, and maintaining excellent performance and efficiency in hot conditions. To adequately evaluate the architecture, this study implemented a prototype for one well-studied and severely imbalanced application – Credit Card Fraud Detection (CCFD). Isolating the improvement of each proposed functionality, the analysis evaluated performance over time and overall performance against related works through five scenarios. Namely, the results indicated that the prototype achieved excellent performance even with few anomalies, and improved systemic efficiency over time. Finally, the overall performance achieved comparable results to the best-performing related works.