The importance of logs

Devices produced data in the form of events.
 As a refresher, events are observable occurrences that happen on a network  system or device.
 This data provides visibility into an environment.
 Logs are one of the key ways security professionals detect unusual or  malicious activity.
 A log is a record of events that occur within an organization's systems.
 System activity is recorded in what's known as a log file or  commonly called logs.
 Almost every device or system can generate logs.
 Logs contain multiple entries which detail information about a specific event or  occurrence.
 Logs are useful to security analysts during incident investigation since  they record details of what, where, and when an event occurred on the network.
 This includes details like date, time, location, the action made, and  the names of the users or systems who performed the action.
 These details offer valuable insight, not only for  troubleshooting issues related to system performance,  but most importantly, for security monitoring.
 Logs allow analysts to build a story and  timeline around various event occurrences to understand what exactly happened.
 This is done through log analysis.
 Log analysis is the process of examining logs to identify events of interest.
 Since there are different sources available to get logs,  an enormous volume of log data can be generated.
 It's helpful to be selective in what we log, so that we can log efficiently.
 For example, web applications generate a high volume of log messages, but  not all of this data may be relevant to an investigation.
 In fact, it may even slow things down.
 Excluding specific data from being logged  helps reduce the time spent searching through log data.
 You may recall our discussion on SIEM technology.
 SIEM tools provide security professionals with a high-level overview  of what happens in a network.
 SIEM tools do this by first collecting data from multiple data sources.
 Then, the data gets aggregated or centralized in one place.
 Finally, the diverse log formats get normalized or  converted into a single preferred format.
 SIEM tools help process large log volumes from multiple data sources in real-time.
 This allows security analysts to quickly search for log data and  perform log analysis to support their investigations.
 So how do logs get collected?
 Software known as log forwarders collect logs from various sources and  automatically forward them to a centralized log repository for storage.
 Since different types of devices and systems can create logs,  there are different log data sources in an environment.
 These include network logs, which are generated by devices such as proxies,  routers, switches, and firewalls, and  system logs, which are generated by operating systems.
 There's also application logs, which are logs related to software applications,  security logs, which are generated by security tools like IDS or IPS,  and lastly authentication logs, which record login attempts.
 Here's an example of a network log from a router.
 There are a couple of log entries here, but we'll focus on the first line.
 Here, we can observe a number of fields.
 First, there's an action specifying ALLOW.
This means that the router's  firewall settings allowed access from a specific IP address to google.com.
 Next, there's a field specifying the source, which lists an IP address.
 So far, the information from this log entry is telling us that network traffic to  google.com from this source IP address is allowed.
 The last field specifies the timestamp,  which is one of the most essential fields in a log.
 We can identify the exact date and time of an action that's occurred.
 This is useful for  correlating multiple events to develop a timeline of the incident.
 There you have it!
You've analyzed your first network log.
Coming up,  we'll continue our discussion on logs and explore log formats.
 Devices produced data in the form of events.
As a refresher, events are observable occurrences that happen on a network system or device.
This data provides visibility into an environment.
Logs are one of the key ways security professionals detect unusual or malicious activity.
A log is a record of events that occur within an organization's systems.
System activity is recorded in what's known as a log file or commonly called logs.
Almost every device or system can generate logs.
Logs contain multiple entries which detail information about a specific event or occurrence.
Logs are useful to security analysts during incident investigation since they record details of what, where, and when an event occurred on the network.
This includes details like date, time, location, the action made, and the names of the users or systems who performed the action.
These details offer valuable insight, not only for troubleshooting issues related to system performance, but most importantly, for security monitoring.
Logs allow analysts to build a story and timeline around various event occurrences to understand what exactly happened.
This is done through log analysis.
Log analysis is the process of examining logs to identify events of interest.
Since there are different sources available to get logs, an enormous volume of log data can be generated.
It's helpful to be selective in what we log, so that we can log efficiently.
For example, web applications generate a high volume of log messages, but not all of this data may be relevant to an investigation.
In fact, it may even slow things down.
Excluding specific data from being logged helps reduce the time spent searching through log data.
You may recall our discussion on SIEM technology.
SIEM tools provide security professionals with a high-level overview of what happens in a network.
SIEM tools do this by first collecting data from multiple data sources.
Then, the data gets aggregated or centralized in one place.
Finally, the diverse log formats get normalized or converted into a single preferred format.
SIEM tools help process large log volumes from multiple data sources in real-time.
This allows security analysts to quickly search for log data and perform log analysis to support their investigations.
So how do logs get collected?
Software known as log forwarders collect logs from various sources and automatically forward them to a centralized log repository for storage.
Since different types of devices and systems can create logs, there are different log data sources in an environment.
These include network logs, which are generated by devices such as proxies, routers, switches, and firewalls, and system logs, which are generated by operating systems.
There's also application logs, which are logs related to software applications, security logs, which are generated by security tools like IDS or IPS, and lastly authentication logs, which record login attempts.
Here's an example of a network log from a router.
There are a couple of log entries here, but we'll focus on the first line.
Here, we can observe a number of fields.
First, there's an action specifying ALLOW.
This means that the router's firewall settings allowed access from a specific IP address to google.com.
Next, there's a field specifying the source, which lists an IP address.
So far, the information from this log entry is telling us that network traffic to google.com from this source IP address is allowed.
The last field specifies the timestamp, which is one of the most essential fields in a log.
We can identify the exact date and time of an action that's occurred.
This is useful for correlating multiple events to develop a timeline of the incident.
There you have it!
You've analyzed your first network log.
Coming up, we'll continue our discussion on logs and explore log formats.
Devices produced data in the form of events.
As a refresher, events are observable occurrences that happen on a network system or device.
This data provides visibility into an environment.
Logs are one of the key ways security professionals detect unusual or malicious activity.
A log is a record of events that occur within an organization's systems.
System activity is recorded in what's known as a log file or commonly called logs.
Almost every device or system can generate logs.
Logs contain multiple entries which detail information about a specific event or occurrence.
Logs are useful to security analysts during incident investigation since they record details of what, where, and when an event occurred on the network.
This includes details like date, time, location, the action made, and the names of the users or systems who performed the action.
These details offer valuable insight, not only for troubleshooting issues related to system performance, but most importantly, for security monitoring.
Logs allow analysts to build a story and timeline around various event occurrences to understand what exactly happened.
This is done through log analysis.
Log analysis is the process of examining logs to identify events of interest.
Since there are different sources available to get logs, an enormous volume of log data can be generated.
It's helpful to be selective in what we log, so that we can log efficiently.
For example, web applications generate a high volume of log messages, but not all of this data may be relevant to an investigation.
In fact, it may even slow things down.
Excluding specific data from being logged helps reduce the time spent searching through log data.
You may recall our discussion on SIEM technology.
SIEM tools provide security professionals with a high-level overview of what happens in a network.
SIEM tools do this by first collecting data from multiple data sources.
Then, the data gets aggregated or centralized in one place.
Finally, the diverse log formats get normalized or converted into a single preferred format.
SIEM tools help process large log volumes from multiple data sources in real-time.
This allows security analysts to quickly search for log data and perform log analysis to support their investigations.
So how do logs get collected?
Software known as log forwarders collect logs from various sources and automatically forward them to a centralized log repository for storage.
Since different types of devices and systems can create logs, there are different log data sources in an environment.
These include network logs, which are generated by devices such as proxies, routers, switches, and firewalls, and system logs, which are generated by operating systems.
There's also application logs, which are logs related to software applications, security logs, which are generated by security tools like IDS or IPS, and lastly authentication logs, which record login attempts.
Here's an example of a network log from a router.
There are a couple of log entries here, but we'll focus on the first line.
Here, we can observe a number of fields.
First, there's an action specifying ALLOW.
This means that the router's firewall settings allowed access from a specific IP address to google.com.
Next, there's a field specifying the source, which lists an IP address.
So far, the information from this log entry is telling us that network traffic to google.com from this source IP address is allowed.
The last field specifies the timestamp, which is one of the most essential fields in a log.
We can identify the exact date and time of an action that's occurred.
This is useful for correlating multiple events to develop a timeline of the incident.
There you have it!
You've analyzed your first network log.
Coming up, we'll continue our discussion on logs and explore log formats.