Log Management

Log management (LM) comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc). LM covers log collection, centralized aggregation, long-term retention and log analysis (in real-time and in bulk after storage).

Systems administrators usually perform LM analysis for reasons of security, of operations (such as system or network administration) or of regulatory compliance.

Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs).

Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source components, or acquire (sub-)systems from commercial vendors.

Log Management Functions

Log management infrastructures typically perform several functions that assist in the storage, analysis, and disposal of log data. These functions are normally performed in such a way that they do not alter the original logs.

The following items describe common log management infrastructure functions:

General

• Log parsing is extracting data from a log so that the parsed values can be used as input for another logging process. A simple example of parsing is reading a text-based log file that contains 10 comma-separated values per line and extracting the 10 values from each line. Parsing is performed as part of many other logging functions, such as log conversion and log viewing.
• Event filtering is the suppression of log entries from analysis, reporting, or long-term storage because their characteristics indicate that they are unlikely to contain information of interest. For example, duplicate entries and standard informational entries might be filtered because they do not provide useful information to log analysts. Typically, filtering does not affect the generation or short-term storage of events because it does not alter the original log files.
• In event aggregation, similar entries are consolidated into a single entry containing a count of the number of occurrences of the event. For example, a thousand entries that each record part of a scan could be aggregated into a single entry that indicates how many hosts were scanned. Aggregation is often performed as logs are originally generated (the generator counts similar related events and periodically writes a log entry containing the count), and it can also be performed as part of log reduction or event correlation processes, which are described below.

Storage

• Log rotation is closing a log file and opening a new log file when the first file is considered to be complete. Log rotation is typically performed according to a schedule (e.g., hourly, daily, weekly) or when a log file reaches a certain size. The primary benefits of log rotation are preserving log entries and keeping the size of log files manageable. When a log file is rotated, the preserved log file can be compressed to save space. Also, during log rotation, scripts are often run that act on the archived log. For example, a script might analyze the old log to identify malicious activity, or might perform filtering that causes only log entries meeting certain characteristics to be preserved. Many log generators offer log rotation capabilities; many log files can also be rotated through simple scripts or third-party utilities, which in some cases offer features not provided by the log generators.
• Log archival is retaining logs for an extended period of time, typically on removable media, a storage area network (SAN), or a specialized log archival appliance or server. Logs often need to be preserved to meet legal or regulatory requirements. There are two types of log archival: retention and preservation. Log retention is archiving logs on a regular basis as part of standard operational activities. Log preservation is keeping logs that normally would be discarded, because they contain records of activity of particular interest. Log preservation is typically performed in support of incident handling or investigations.
• Log compression is storing a log file in a way that reduces the amount of storage space needed for the file without altering the meaning of its contents. Log compression is often performed when logs are rotated or archived.
• Log reduction is removing unneeded entries from a log to create a new log that is smaller. A similar process is event reduction, which removes unneeded data fields from all log entries. Log and event reduction are often performed in conjunction with log archival so that only the log entries and data fields of interest are placed into long-term storage.
• Log conversion is parsing a log in one format and storing its entries in a second format. For example, conversion could take data from a log stored in a database and save it in an XML format in a text file. Many log generators can convert their own logs to another format; third-party conversion utilities are also available. Log conversion sometimes includes actions such as filtering, aggregation, and normalization.
• In log normalization, each log data field is converted to a particular data representation and categorized consistently. One of the most common uses of normalization is storing dates and times in a single format. For example, one log generator might store the event time in a twelve-hour format (2:34:56 P.M. EDT) categorized as Timestamp, while another log generator might store it in twenty-four (14:34) format categorized as Event Time, with the time zone stored in different notation (-0400) in a different field categorized as Time Zone.24 Normalizing the data makes analysis and reporting much easier when multiple log formats are in use. However, normalization can be very resource-intensive, especially for complex log entries (e.g., typical intrusion detection logs).
• Log file integrity checking involves calculating a message digest for each file and storing the message digest securely to ensure that changes to archived logs are detected. A message digest is a digital signature that uniquely identifies data and has the property that changing a single bit in the data causes a completely different message digest to be generated. The most commonly used message digest algorithms are MD5 and Secure Hash Algorithm 1 (SHA-1).25 If the log file is modified and its message digest is recalculated, it will not match the original message digest, indicating that the file has been altered. The original message digests should be protected from alteration through FIPS-approved encryption algorithms, storage on read-only media, or other suitable means.

Analysis

• Event correlation is finding relationships between two or more log entries. The most common form of event correlation is rule-based correlation, which matches multiple log entries from a single source or multiple sources based on logged values, such as timestamps, IP addresses, and event types. Event correlation can also be performed in other ways, such as using statistical methods or visualization tools. If correlation is performed through automated methods, generally the result of successful correlation is a new log entry that brings together the pieces of information into a single place. Depending on the nature of that information, the infrastructure might also generate an alert to indicate that the identified event needs further investigation.
• Log viewing is displaying log entries in a human-readable format. Most log generators provide some sort of log viewing capability; third-party log viewing utilities are also available. Some log viewers provide filtering and aggregation capabilities.
• Log reporting is displaying the results of log analysis. Log reporting is often performed to summarize significant activity over a particular period of time or to record detailed information related to a particular event or series of events.

Disposal

• Log clearing is removing all entries from a log that precede a certain date and time. Log clearing is often performed to remove old log data that is no longer needed on a system because it is not of importance or it has been archived.

Log Management Software

Assuria Log Manager (ALM) has achieved CESG CCTM approval and securely collects and manages audit logs to comply with regulations. The small footprint ALM agents are available for Windows, UNIX and Linux servers, databases, applications, network devices, firewalls, routers, access control systems and many more. Collection from new log sources can be added via agent plug-ins. Collected logs are stored in their original format in a standard file / folder structure with log data integrity ensured through digital signatures and cryptographic hashes.

For more details on Assuria Log Manager

Log Management Appliances

LogRhythm is an enterprise-class application that seamlessly combines Log & Event Management, File Integrity Monitoring and Endpoint Monitoring & Control into a single integrated solution.  It is highly reliable, cost effective and easily scalable across any size enterprise.  With LogRhythm, you can invest in a single solution to address needs and challenges throughout your organization, whether they are related to compliance, security or IT operations.

For more details on LogRhythm Single Integrated Appliances

LogLogic Open Log Management Collect, normalize, index, store, and search log data automatically with our easy-to-deploy appliances or hosted solutions. Rapidly drill down into log details and create detailed reports with our built-in templates. All LogLogic appliances are run on hardened linux appliances that are designed to offer full log processing and archive based on the amount of Event Per Second (EPS) design. 

For more details on LogLogic Log Management Appliances

Log Management SaaS

 Assuria provide a hosted Log Management solution.

For more details on Assuria Log Manager