Log management
Log management (LM) comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc.). LM covers:[1]
- log collection
- centralized aggregation
- long-term retention
- log rotation
- log analysis (in real-time and in bulk after storage)
- log search and reporting.
Concerns about security,[2] system and network operations (such as system or network administration) and regulatory compliance drive log management.
Effectively analyzing large volumes of diverse logs can pose many challenges — such as:
- huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization)
- log-format diversity
- undocumented proprietary log-formats (that resist analysis)
- the presence of false log records in some types of logs (such as intrusion-detection logs)
Users and potential users of LM can build their own log-management and intelligence tools, assemble the functionality from various open-source components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it.[3]
Lately, more and more the suggestion is made to change the definition of logging. This change would keep matters both more pure and more easily maintainable:
- Logging would then be defined as all instantly discardable data on the technical process of an application or website, as it represents and processes data and user input.
- Auditing, then, would involve data that is not immediately discardable. In other words: data that is assembled in the auditing process, is stored persistently, is protected by authorization schemes and is, always, connected to some end-user functional requirement.
Logging can produce technical information usable for the maintenance of applications or websites. It can serve:
- to define whether a reported bug is actually a bug
- to help analyze, reproduce and solve bugs
- to help test new features in a development stage
Deployment life-cycle
One view of assessing the maturity of an organization in terms of the deployment of log-management tools might use successive categories such as:
- Level 1: in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.
- Level 2: with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.
- Level 3: at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the enterprise — especially of those information-assets whose availability organizations regard as vital.
- Level 4: organizations integrate the logs of various business-applications into an enterprise log manager for better value proposition.
- Level 5: organizations merge the physical-access monitoring and the logical-access monitoring into a single view.
Solutions
See also
- Audit trail
- Common Base Event
- Common Log Format
- DARPA PRODIGAL and Anomaly Detection at Multiple Scales (ADAMS) projects.
- Data logging
- Log analysis
- Log management knowledge base
- Security information and event management
- Server log
- Syslog
- Web counter
- Web log analysis software
References
- Chris MacKinnon: "LMI In The Enterprise". Processor November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10
- MITRE: Common Event Expression (CEE) Proposed Log Standard. Online at http://cee.mitre.org, retrieved 2010-03-03
- NIST 800-92: Guide to Security Log Management. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf, retrieved 2010-03-03