Log management is the process of generating, transmitting, storing, accessing, and disposing of log data. Log data (or logs) is composed of entries (records), and each entry contains information related to a specific event that occurred within an organization's computing assets, including physical and virtual platforms, networks, services, and cloud environments. Logging events and logging transactions can help form an audit trail.
The process of log management generally breaks down into:
Log collection - a process of capturing actual data from log files, application standard output stream (stdout), network socket and other sources.
Log aggregation (centralization) - a process of putting all the log data together in a single place for the sake of further analysis or/and retention.
Log storage and retention - a process of handling large volumes of log data according to corporate or regulatory policies (compliance).
Log monitoring and log analysis - a process that helps operations and security team to handle system performance issues and security incidents
Overview
The primary drivers for log management implementations are concerns about security, system and network operations (such as system or network administration) and regulatory compliance. Logs are generated by nearly every computing device, and can often be directed to different locations both on a local file system or remote system.
Effectively analyzing large volumes of diverse logs can pose many challenges, such as:
Volume: log data can reach hundreds of gigabytes of data per day for a large organization. Simply collecting, centralizing and storing data at this volume can be challenging.
Normalization: logs are produced in multiple formats. The process of normalization is designed to provide a common output for analysis from diverse sources.
Velocity: The speed at which logs are produced from devices can make collection and aggregation difficult
Veracity: Log events may not be accurate. This is especially problematic for systems that perform detection, such as intrusion detection systems.
Users and potential users of log management may purchase complete commercial tools or build their own log-management and intelligence tools, assembling the functionality from various open-source components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it.
Logging can produce technical information usable for the maintenance of applications or websites. It can serve:
to define whether a reported bug is actually a bug
to help analyze, reproduce and solve bugs
to help test new features in a development stage
Deployment
Organizations can use different log-analyzers for analyzing the logs of devices in the security perimeter, typically aiming to identify patterns of attack on the perimeter infrastructure of the organization. Organizations may mandate logging to identify the access and usage of confidential data within the security perimeter.
A log analyzer can also track and monitor the performance and availability of systems at the level of the enterprise, especially of those information assets whose availability organizations regard as vital. Organizations may integrate the logs of various business applications into an enterprise log manager for a better value proposition.
