LogAI is a free library for log analytics and intelligence that supports various log analytics and intelligence tasks. It’s compatible with multiple log formats and has an interactive graphical user interface. LogAI provides a unified model interface for popular statistical, time-series, and deep-learning models, making it easy to benchmark deep-learning algorithms for log anomaly detection.

Logs generated by computer systems contain essential information that helps developers understand system behavior and discover issues. Traditionally, log evaluation was done manually, but AI-based log evaluation automates tasks similar to log parsing, summarization, clustering, and anomaly detection, making the method more efficient. Different roles in academia and industry have various requirements for log evaluation. For instance, machine learning researchers must quickly benchmark experiments against public log datasets and reproduce results from other research groups to develop latest log evaluation algorithms. Industrial data scientists must run existing log evaluation algorithms on their log data and choose one of the best algorithm and configuration combination as their log evaluation solution. Unfortunately, no existing open-source libraries can meet all of those requirements. Due to this fact, LogAI is introduced to handle these needs and higher conduct log evaluation for various academic and industrial use cases.

The absence of comprehensive AI-based log evaluation in log management platforms creates challenges for unified evaluation as a consequence of the necessity for a unified log data model, redundancy in preprocessing, and a workflow management mechanism. Reproducing experimental results is difficult, requiring customized evaluation tools for various log formats and schemas. Different log evaluation algorithms are implemented in separate pipelines, adding to the complexity of managing experiments and benchmarking.

LogAI comprises two primary components, namely LogAI core library and LogAI GUI. The LogAI GUI module allows users to hook up with log evaluation applications within the core library and interactively visualize evaluation results through a graphical user interface. However, the LogAI core library comprises 4 distinct layers: 

The Data Layer in LogAI consists of information loaders and a unified log data model defined by OpenTelemetry. It also offers various data loaders to convert raw log data into LogRecordObjects in a standardized format.

The Preprocessing Layer of LogAI cleans and partitions logs using preprocessors and partitioners. Preprocessors extract entities and separate records into unstructured loglines and structured log attributes while partitioners group logs into events for machine learning models. Customized preprocessors and partitioners can be found for specific open-log datasets and could be prolonged to support other log formats.

The Information Extraction Layer of LogAI converts log records into vectors for machine learning. It has 4 components: log parser, log vectorizer, categorical encoder, and have extractor. 

The Evaluation Layer incorporates modules for conducting evaluation tasks, with a unified interface for multiple algorithms.

LogAI uses deep learning models like CNN, LSTM, and Transformer for log anomaly detection and may benchmark them on popular log datasets. Results show it performs equally or higher than deep-loglizer, with a supervised bidirectional LSTM model providing one of the best performance.

Take a look at the Github and Blog. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 18k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.