A Security Information and Event Management, a SIEM system, is a comprehensive tool that
acts as a nerve center of an organization's security operations.
It collects logs and security-related data from a wide array of sources across the
digital infrastructure, including endpoints, servers, network devices, cloud services,
and various security tools like firewalls or antivirus systems.
The main advantage of a SIEM is its ability to centralize this data, which is crucial
for effective monitoring and analysis. This allows security teams to have a unified
view of all security events and logs enabling faster detection,
investigation, and response to potential threats.
One of the primary capabilities of a SIEM is centralized log management.
It aggregates logs from different sources such as Syslog, Windows event logs,
and cloud-based services, streamlining the process of managing large volumes
of data. This centralized approach not only
simplifies data storage but also enhances the efficiency
of security operations as analysts can quickly access and correlate data from
various systems. SIEM systems also offer significant threat
visibility. By correlating events from multiple
sources they can detect suspicious patterns that indicate potential
threats such as lateral movement within a network
or privileged escalation attempts within a host.
For example, if a user with limited access rights
suddenly tries to access sensitive data, a SIEM would flag this as a potential
security incident. Moreover, SIEM systems play a crucial
role in helping organizations maintain security compliance.
They can be configured to meet various regulatory standards
like PCI DSS, NIST, or ISO 27001. Ensuring that the security practices
aligned with the industry requirements would be this compliance.
These requirements often mandate detailed logging
and real-time monitoring, both of which are core functions of the SIEM systems.
For example, a retail company may use a SIEM system to monitor its payment
processing systems. A system could be configured to send an alert
if login attempts are made outside of business hours
from a foreign IP address, signaling a potential security breach.
This automated alert allows the security team to investigate the
situation and take action before a full-scale attack occurs.
SIEMs begin their operation by collecting logs from a variety of data
sources within an organization infrastructure.
These sources include endpoint servers, switches,
routers, firewalls, and security appliances,
all of which generate logs in different formats.
To make sense of this disparate data, SIEMs employ a process of log
normalization. This process involves converting logs from various formats
into a unified structure, allowing them to be analyzed and
correlated efficiently. The technical process of log collection
typically involves collectors or agents such as NXLog,
Winlogbeat, or Beats from ELK Stack, Fluentd, or others
that pull data from these resources. These agents are responsible for gathering
the logs and sending them to the SIEM for processing.
Once collected, the logs are normalized using predefined parsing rules and
field mappings. This step converts the data into a consistent
schema like the Common Event Format (CEF) or JSON.
These formats standardize the structure of the logs, making them
easier to analyze and search across the entire system,
regardless of the original source. Another critical aspect of normalization
is timestamp unification. Since logs are generated in different
might be generated in different time zones or may experience clock drifts,
it's essential to align the timestamps to ensure that events are
accurately correlated across the systems.
Without timestamp unification, logs from different sources might appear out of
sequence, making it difficult to track the flow of
events and identify security incidents. For example, imagine a SIEM
system that receives logs from a Cisco ASA
firewall and a set of Linux servers. Each of these sources may generate logs in
different formats. The SIEM system will standardize
these logs so a single search can look for SSH
login attempts across both systems regardless of whether the logs came
from a firewall or the servers. This normalization enables analysts to
quickly and efficiently correlate events leading to faster
identification of potential security threats.
In summary, SIEMs rely on log collection and normalization
to transform diverse data into a structured format
that can be effectively analyzed and correlated. This process ensures that
security teams can quickly identify and respond to incidents
regardless of the systems generating the data.
The technical capabilities of SIEMs enable them to perform
the type of sophisticated analysis. Correlation engines
are at the heart of this functionality. These engines match patterns using
predefined rules or by establishing behavioral
baselines. In doing so, the SIEM can detect anomalies
that deviate from the normal behavior or of a system
or a user such as an unusual high number of failed logins in a short
period. Thresholds and heuristics are then
applied to determine when these patterns should trigger an alert,
helping to reduce noise and focusing on the attention of events that are most
likely to indicate a security incident. SIEMs also offer the flexibility to
customize alert rules for specific needs.
For example, you can create rules based on known attack signatures
like specific network traffic patterns associated with malware
or on behavior anomalies such as unusual login times
or activity outside of normal working hours.
Additionally, many SIEM platforms support custom rule scripting
in languages such as Lucene, SPL, Search Processing Language which is in
Splunk, Sigma or
Kibana Query Language (KQL) and other, allowing security teams to tailor their
detection capabilities to their specific environment.
An example of how SIEM correlation works in practice could be when a user logs in
from two geographically distant locations such as New York
and Singapore within a matter of minutes.
This event might be flagged by the SIEM system using a custom geolocation
velocity rule. This rule compares the user's login patterns
against the known geographical locations and flags a rapid change as a
potential indication of credential compromise or account takeover.
By correlating these events, the SIEM system can provide security
analysts with actionable alerts that require further investigation.
In summary, SIEMs use data correlation to link events and uncover complex
attacks patterns. By combining predefined rules, behavioral
analysis and customizable alert conditions, SIEM systems can help
organizations detect and respond to security incidents more
effectively and efficiently.
SIEM dashboards are essential tools for security analysts providing
real-time visualization of network and system activities.
These dashboards help security teams to monitor key performance
indicators such as failed logging attempts,
firewall denials and virus detections. By displaying this information in an easy
to understand format, SIEM dashboards allow
analysts to quickly identify and act on
potential security threats, reducing the time it takes to respond to
incidents. One of the key features of SIEM dashboards
is the ability to display event volume,
severity levels and active alerts in real time.
This dynamic view helps analysts track the flow of security data and
prioritize the response based on the criticality of each event.
For instance, if there is a surge in failed logging attempts or a sudden
spike in suspicious network activity, it will immediately be
visible to the dashboard, prompting further
investigation. Additionally, many SIEM
dashboards include the drill-down views.
This allows analysts to click on specific anomalies or alerts and examine
the details such as the source of the activity,
the associated user or device and other contextual
information. This capability speeds up the investigation
process, enabling analysts to make informed decisions quickly.
SIEM systems also provide reporting capabilities,
offering pre-built templates that can be customized to meet organizational
needs or regulatory requirements. These reports help ensure compliance with
standards such as PCI DSS or GDPR and they are often used for audits
or executive summaries. The ability to schedule these reports ensures that
stakeholders stay informed about security posture and that compliance
is continuously monitored. For example, a security operation center,
a SOC team, might use the SIEM's dashboard to monitor SSH login
attempts across production servers. If more than 10 failed login attempts occur
within five minutes on any server, the system can trigger an alert.
This allows the SOC teams to investigate potential brute force attack
attempts, quickly taking action to prevent a
successful intrusion. In summary, SIEM dashboards are very crucial
for providing real-time visibility into the security events
and streamlining the investigative process. By combining live monitoring
features, drill-down analysis and automated reporting, SIEMs empower
security teams to respond faster and maintain compliance within the
organizational and regulatory standards.
So, as we can see, one of the key capabilities of the SIEM
is the detection of unusual behavior. For example, if a user logs in
at odd hours, like 3 o'clock in the morning, or from an unfamiliar
geolocation, the SIEM can flag this activity as suspicious.
It's like User Behavior Analytics (UBA). These anomalies can be indicative of
compromised credentials or unauthorized access attempts.
Another powerful feature is the ability to correlate multiple events into a
single consolidated alert. For example, a SIEM may detect a series of
failed login attempts, followed by a successful login and subsequent
privilege escalation. When these events occur in close succession,
the SIEM links them together, triggering an alert that helps security teams
identify potential attacks like brute force login attempts
or lateral movement within the network. Additionally, SIEMs can
integrate with threat intelligence feeds, to match activity against
known indicators of compromise, such as IP addresses,
file hashes or domain names associated with
malicious behavior. This enables them to flag traffic
or actions that align with known attack patterns,
adding another layer of context and precision to the detection process.
The SIEMs can also extract threat intelligence and
indicators of compromise in order to help other SIEMs to respond
to such activities. Let's see, for example, imagine a SIEM that detects a
user account login at 3 o'clock in the morning
from a foreign IP address. If the user then accesses
sensitive files, the system can trigger an alert based on
preconfigured correlation rules that link the
unusual login behavior with the subsequent access to sensitive data.
So it's like a kill chain. This will be identified by the SIEM.
This early detection helps security teams respond swiftly,
minimizing the risk of a data breach.
Modern SIEMs have advanced capabilities that go beyond
just detecting threats. They can also trigger predefined actions
in response to specific alerts, which significantly reduces the need for
manual intervention and accelerates the incident
containment process. This automation allows security
teams to respond to potential threats faster and more efficiently,
minimizing the impact of security breaches. For example, a SIEM can automatically
block IP addresses, of course, sending a firewall rule to the specific
firewall that are involved in malicious activities,
such as when being in a brute force login attempt or DDoS attacks.
When multiple failed login attempts were detected, the SIEM can trigger an
action to block the suspicious IP address,
preventing further unauthorized access attempts.
Another action SIEMs take is to automatically disable compromised accounts.
If the system detects unusual login behavior, such as logins from
other locations or weird hours, it can instantly disable
the affected account to prevent any potential data breach or escalation.
So it can be integrated the SIEM with firewalls or identity access controls
in order to revoke rights and actually respond to the
to the alert. In addition, SIEMs can send real-time alerts to security
teams via various channels such as email, Slack,
or integration with Security Orchestration, Automation
and Response, as we call SOAR tools. This ensures that the right people
are notified immediately, allowing them to take swift action.
Let's give an example. Imagine a SIEM detects multiple failed SSH login
attempts, followed by a successful login from an
unfamiliar source. Upon detecting this pattern, the SIEM
could automatically block the suspicious IP address
through the firewall to prevent further access attempts.
Simultaneously, it could send an email notification to the Security Operation
Center, to the SOC team, with the incident details for further
investigation. Let's see a specific tool, Splunk.
Splunk is a commercial SIEM solution known for its scalability,
rich visualization and powerful search capabilities.
It ingests logs and machine data from a wide variety of sources, enabling
real-time monitoring, threat detection and compliance reporting.
One of the key features of Splunk is its Search Processing Language,
also as we call it SPL. SPL is a domain-specific
query language that enables complex searches, statistical analysis and the
creations of visual dashboards. Analysts use SPL from Splunk
to dive into data and identify potential threats.
It provides a highly customizable search experience,
helping analysts identify threats and outliers
in large datasets, such as repeated failed login attempts
or unusual network traffic patterns. Another powerful tool in Splunk is the
Machine Learning Toolkit, which allows users to build predictive models
for anomaly detection and User Behavior Analytics,
UEBA. This toolkit helps detect unknown threats by identifying
unusual patterns in data, which might not be flagged by traditional
rule-based detection methods. It enables Splunk to go beyond basic searches
and provide insights into future behavior, helping the analysts
to detect advanced threats like insider threats or zero-day attacks.
Splunk's Enterprise Security module adds an additional layer of
functionality tailored to security operations.
It includes security-specific dashboards, threat intelligence integration
and incident review features. This module streamlines the detection and
management of security incident by offering specialized tools for threat
hunting and incident response, making it easier for analysts
to investigate and mitigate potential threats across the infrastructure.
For example, a security analyst might use SPL
to search for failed login attempts across various systems in the network.
The query could be something like index equals
auth sourcetype equals linux_secure action equals failed
and then stats count by user and source.
This allows the analyst to quickly spot abnormal behavior such as a large
number of failed logins from a single user or IP address,
which could indicate a brute force attack. By using SPL, the analyst can
immediately take actions such as locking accounts or
blocking malicious IP addresses to prevent further escalation
of the threat. The Splunk's combination of advanced querying, machine
learning and tailored security modules makes it an
essential tool for large-scale security operations.
It enables organizations to detect and respond to threats
faster, providing visibility and actionable
insights into the security posture of the network.
Another tool, a SIEM, is the ELK Stack, which is based on Elasticsearch,
Logstash and Kibana. This is an open-source alternative that has gained
significant traction in SIEM deployments lately.
While it demands more setup and configuration compared to commercial
solutions like Splunk, it provides organizations with
flexibility and cost control. This makes it particularly
attractive for businesses within-house DevOps or SecOps teams.
The ELK Stack offers powerful capabilities for log aggregation, analysis and
visualization, which is key for maintaining effective
security monitoring. At the heart of the ELK Stack is Elasticsearch,
a fast and distributed search engine designed to index and search vast
amounts of log data. It allows more real-time querying,
making it ideal for quickly retrieving and analyzing log data.
Elasticsearch excels in handling large datasets, ensuring rapid response
times even when dealing with complex searches.
This ensures that security teams can query data efficiently
and identify patterns or anomalies in a timely manner.
Next, Logstash handles the data ingestion
and transformation. It pulls log from data, the log data from various resources,
parse it and then sends the processed data to the Elasticsearch.
Logstash's role is critical because logs are often generated in different
formats across multiple systems. It ensures that data is standardized,
parsed correctly and then enriched with additional context where necessary,
making it ready for analysis. Without this step logs would be difficult to work
with and would hinder any meaningful investigation.
Finally, Kibana serves as the visualization layer of the ELK Stack.
It enables users to create interactive dashboards and run queries using
Lucene-based syntax. They support Kibana Query Language (KQL), Elastic Query
Language and so on. Kibana allows security analysts to visualize
data trends, identify potential threats and track
key metrics in real time. Through customizable dashboards, users can
easily monitor anomalies, create reports and respond to emerging security
events with a clear visual overview that the Kibana provides.
For example, when monitoring for SSH brute force attacks,
Logstash can be set up to parse the SSH logs
from the folder /var/log/auth.log where login attempts are recorded on a Linux
system. These logs are processed and indexed by
Elasticsearch and NoSQL database, making it easy to search and query.
Kibana can then visualize the failed login attempts over the time.
Displaying trends and mapping the source IPs.
This setup allows security teams to quickly identify abnormal patterns,
such as multiple failed attempts from a single IP, which may signal
a brute force attack.
When comparing Splunk and the ELK Stack, there are several key differences in
terms of features licensing and scalability.
Understanding these differences can help organizations
choose the right solution for their security monitoring and log management
needs. Licensing is one of the most
significant contrasts between the two. Splunk operates on a commercial
tiered pricing model, meaning that organization must pay based on the
amount of data ingested. While this model offers robust enterprise
support, it can become very costly as data volume increases.
On the other hand, the ELK Stack is open source, which means it's free to use.
However, the organizations that require additional support,
Elastic, the company behind the ELK Stack,
offers paid support options, providing the flexibility
to scale as needed without a significant upfront
investment in licensing. In terms of query language, Splunk uses its own
proprietary Search Processing Language, SPL. SPL is a highly
flexible and powerful, allowing users to perform complex searches,
statistical analysis, and even create custom dashboards.
While SPL offers advanced capabilities, it can also have a steeper
learning tier for users unfamiliar with these decision
blocks. The ELK Stack, in contrast, uses
Lucene and Kibana Query Language, both of which are more standardized
and easier to learn, but less advanced compared to SPL.
For those who are already familiar with SQL, like
query language, Kibana Query Language offers a more accessible entry point.
It feels similar to the SQL, but it may support,
might not support, the level of complex querying that SPL does.
When it comes to setup complexity, Splunk is known for being easy to deploy,
especially with its enterprise support options.
This makes it ideal for organizations that need a quick
out-of-the-box solution with minimal configuration.
The ELK Stack, however, requires more manual configuration and tuning.
It can be set up to meet specific requirements, but this also means more
effort is needed for deployment, especially when scaling it
across multiple environments. Integration capabilities also vary between the two
solutions. Splunk offers extensive built-in
integrations and add-ons, making it easier to incorporate into an existing
environment. These pre-built integrations with
popular systems and tools help streamline the deployment processes.
Meanwhile, the ELK Stack is highly extensible,
offering the flexibility to integrate with many systems through plugins
and Beats, lightweight data shippers. The agents on the ELK
is called Beats. This extensibility allows the ELK Stack to
be customized and tailored for specific use cases.
However, it might require additional configurations
to ensure seamless integration with the systems.
Finally, scalability is an area where both solutions differ.
Splunk is cloud-native, offering scaling capabilities that are managed by
Splunk cloud itself. This makes it easier for organizations
to scale up as needed without the burden of managing the infrastructure.
On the other hand, ELK Stack requires more manual cluster management for
scaling. While it can scale to handle large datasets,
this process demands more hands-on effort, including managing
nodes, handling data distribution, and ensuring high availability of the
system itself. Graylog is an open-source log management
platform commonly used for mid-sized SIEM deployments.
It is built on top of Elasticsearch and MongoDB,
offering a user-friendly, flexible, and extensible solution
for log management and analysis. Known for its simplicity, Graylog is
particularly suited for environments where ease of setup and customization
are crucial, yet it maintains powerful features that meet the
demands of security teams. One of the standout features of Graylog
is its modular architecture. It uses components like inputs,
extractors, and pipelines to process, filter, and enrich logs,
which provides flexibility in how log data is ingested, parsed, and analyzed.
This architecture allows organizations to tailor their log processing pipelines
to fit specific needs, making Graylog an adaptable
solution for various use cases. Graylog also provides
stream-based filtering, which helps users route logs into specific streams
based on their content. This enables customized alerting and
dashboarding based on the log types that are most important
to the security team. For example, if there are certain logs associated
with critical systems or particular threats,
Graylog can be configured to prioritize these events and trigger
alerts accordingly. Alerting and correlation are core components
of Graylog, offering real-time monitoring capabilities.
Users can configure alert conditions based on thresholds or specific patterns
within the stream of logs. For instance, if a predefined threshold
is exceeded, such as multiple failed login attempts,
in a short period of time, Graylog can trigger an alert,
notifying the security team of potential threats.
An example, a network administrator sets up a stream to monitor failed SSH
logins. The stream filters logs where event
dot action equals SSH login failed. And if more than 10
failures occur within one minute from the same IP address, Graylog triggers an
alert, helping identify potential brute-force attacks.
From a technical perspective, Graylog supports several
ingestion methods, including Syslog,
GELF, which is Graylog Extended Log Format, and REST APIs is supported,
making Graylog compatible with a wide range of log sources.
The platform uses Elasticsearch again for indexing,
which provides fast search and query capabilities across the
large datasets. Additionally, Graylog plugin ecosystem is
another highlight, offering integrations for geolocation,
threat intelligence, and other advanced use cases.
This flexibility allows organizations to extend Graylog functionality
and adapt to their specific security and operational needs.
IBM QRadar is a comprehensive enterprise-level SIEM platform,
widely used in regulated industries, particularly where robust
security and compliance requirements are essential.
Known for its advanced correlation engine and integration capabilities,
QRadar is designed to handle complex security environments with ease,
providing deep insights and real-time detection of threats.
One of QRadar's most notable features is its
auto-correlation engine, which uses a rule-based
logic to detect security patterns across multiple log sources.
This engine automatically correlates events from various systems,
such as firewalls, endpoints, and servers, to identify suspicious activities.
Another key feature of QRadar is its asset modeling capability.
The platform dynamically builds an asset inventory based on network traffic
observations, giving security teams a clear view
of their assets' vulnerabilities. This asset modeling helps them to
identify critical systems that are being targeted
and enables organizations to prioritize responses accordingly.
Integrated threat feeds are another significant advantage of QRadar.
It connects seamlessly with external threat intelligence,
with providers like IBM X-Force and other third-party feeds,
enriching the detection and correlation process.
This integration allows QRadar to cross-reference incoming log data
with known attack indicators and evolving threat intelligence,
improving the accuracy of alerts and helping security teams respond more
proactively. Let's see an example use. QRadar
might correlate repeated failed logins from multiple users on the same subnet,
followed by a successful login from one of the other users.
This sequence suggests a credential stuffing attack, where attackers use
stolen credentials to gain unauthorized access. QRadar's auto correlation
engine will generate a high severity offense,
alerting the security team to potential credential compromise.
From a technical perspective, QRadar is highly scalable,
supporting log ingestion from thousands of sources, including firewalls,
endpoints, and cloud platforms. This makes QRadar well suited for
large complex environments that require centralized log management and threat
detection. Additionally, QRadar has built-in flow
analytics through QFlow that allows it to inspect
Layer 7 traffic, providing visibility into application level activity,
which is essential for detecting more advanced attacks.
QRadar also leverages the Ariel database for high-speed searching across
terabytes of log data, enabling fast query performance,
even in large-scale deployments. This allows security analysts to quickly
search through extensive logs and identify critical security events,
which is vital for maintaining real-time monitoring capabilities.
Microsoft Sentinel is a cloud-native SIEM solution that is tightly
integrated with Azure, offering scalability,
powerful analytics, and automated response capabilities.
It leverages Microsoft's AI, machine learning,
and security graph intelligence to deliver smart threat detection
and streamlined security operations, making it particularly suited for
enterprises already using Azure or other Microsoft services.
One of the Sentinel's key strengths is its use of a
Kusto Query Language, KQL, a fast and flexible query language
optimized for large-scale log analysis.
KQL allows security analysts to quickly search and analyze vast
amounts of log data, making it an essential tool for identifying
threats in real time. The language is powerful enough
to handle complex queries while being efficient enough
to work at the scale of enterprise environments.
Another important capability of Sentinel's
is its behavior analytics, which includes User Behavior Analytics,
UEBA. This feature analyzes user and the
system behavior, to find data anomalies that might indicate malicious activity.
By continuously monitoring user and entity actions,
Sentinel can identify deviations from normal behavior,
such as unusual login times, atypical access patterns,
or unauthorized resource access, which are often the signs of insider threats
or compromised accounts. Playbooks in Microsoft Sentinel,
which are based on SOAR, Security Orchestration, Automation and Response,
enable organizations to automate incident response.
Using Azure Logic Apps, playbooks can execute predefined
workflows to respond to security incidents automatically.
These playbooks help reduce the response time to incidents,
ensuring a quick and consistent reaction to common security threats.
Let's see an example. The Sentinel threat detection capabilities
is its ability to flag impossible logins.
Suppose a user logs from Germany and then from the US, within a two-minute
window, Sentinel would use KQL queries on
sign-in logs to flag this as an anomaly. By querying the
GeoIP and timestamp data, Sentinel can immediately identify and alert
on the impossible scenario, indicating a potential credential compromise
or an unauthorized access attempt. From a technical perspective, Microsoft
Sentinel provides seamless integration with the
Microsoft services, like Microsoft 365, Defender and Azure
activity logs. This tight integration helps security
teams gain comprehensive visibility across their
cloud and on-premises environments. Additionally,
Sentinel supports a wide range of third-party
and connectors such as AWS, Cisco and Palo Alto,
which allow organizations to ingest data, to ingest
data from multiple sources into a centralized platform
for more complete threat detection and analysis.
To begin utilizing Splunk, for example, for log data analysis, it's
important to understand each step of the process for ingesting and
configuring log data. The first stage is to navigate to
settings and add data, which opens a simple and
intuitive workflow for adding your data sources.
This process is critical as it ensures that your Splunk instance
can properly collect data from a variety of sources,
be it from files or real-time monitoring. The first action
here is select monitor files and directories.
This allows you to specify which log files to track for
real-time collection. For instance, if you want to
monitor authentication logs from a Linux system, you might
choose files like /var/log/auth.log or /var/log/secure kernel or
whatever. By choosing the files and directories
option, you are telling Splunk to look for new log entries
that appear within these files or directories,
which will then be indexed for later searching.
This is key because it means Splunk is constantly aware of new events,
making it easier to detect issues as they arise in real-time.
Next, one of the most crucial decisions is in the setup process
to select a source type for the log data.
The source type defines how Splunk should interpret
and parse the raw log data. Without a proper source type,
Splunk wouldn't be able to extract meaningful fields
or structure the data correctly. For example, Linux authentication logs
typically use linux_secure source type.
By assigning the appropriate source type,
you ensure that Splunk understands the log format
and can correctly break down the logs into fields such as user names,
timestamps, login as a success, failure, statuses and so on.
After defining the source type, the next step is to configure an index.
The index acts like a container for logs
and is essential for organizing the large amounts of data
that may be ingested into Splunk. For example, security logs could be placed
in an index called "security_logs", which would make it easy
to later filter and search through only the security-related events in your
system. By grouping logs into different indexes,
Splunk can more efficiently retrieve the data needed for analysis without
sifting through irrelevant information, improving
performance and ensuring that search queries
are faster and more relevant. Once you've completed the source type and
index configuration, Splunk will automatically begin
ingesting the log data from the specified files into the database.
As it ingests those logs, Splunk parses the raw log entries into
structured events according to the source type.
Each event contains important metadata fields such as time,
host, source and source type which helps categorize the data and make it
searchable. For example, the timestamp field
last_time will allow you to search logs by date and time.
Host will tell you which machine the log originated from
and source type will help you identify the type of log,
if it's a system log, application log, security log and so on.
These fields make the data far more usable and allow you to quickly find
relevant entries in a large dataset. Once the logs have been ingested
and parsed into the events, the final step is to validate the data by
running searches. This is where the power of Splunk
really shines. By querying specific indexes and source types,
you can rapidly filter through logs to uncover valuable insights.
For instance, to track failed login attempts, you could search for all failed
password entries in the security logs using a
query like index=security_logs sourcetype=linux_secure
and then the message "failed password".
This query will return all entries matching that search string
which would be incredibly helpful for identifying potential security
breaches, brute force attacks or misconfigurations in the system.
Once your logs are ingested into Splunk the next step
is to analyze them using the SPL, the Search Processing
Language. One of the most fundamental queries in SPL
is using the stats command to calculate metrics across different fields in your
logs. For instance, if you want to analyze how
many events are coming from different log sources,
you can use the following SPL query index=security_logs
| stats count by source. This will provide a breakdown of event
counts from each log source helping to identify which sources are most active.
This analysis is important for security teams to monitor systems
as unusual spikes in events from specific sources could indicate a
potential security incident. Another key feature of SPL is the
top command which identifies the most frequent
frequent values in a specific field. For example, if you want to see which source
types are most common in your logs, you could use the query
index=security_logs | top sourcetype.
Source types are essential categories that Splunk assigns to logs based on
their format. By identifying the most
frequent source types you can assess whether the log sources are
appropriately categorized and if there are any unexpected
log types presented. These unexpected source types might point
to misconfigured systems or in the worst case potential
security risks like a new undetected source of data.
With these commands you can easily dive into your log data and start
uncovering valuable insights. The SPL language supports also
more advanced features such as subsearches,
field extractions and time-based analytics.
For example, if you want to focus on logs from a speccy
specific time range or filter out specific events that aren't relevant
to your investigation, SPL allows you to refine
your queries according to the time. This makes it versatile tool for not only
security incident response but also for ongoing operational
monitoring. To detect brute force attempts, one
effective method using Splunk is to examine logs for repeated
failed login messages. Brute force attacks often involve an
attack repeatedly trying different passwords in an attempt to gain unauthorized
access to the system. These failed attempts are typically
recorded in logs such as SSH or authentication logs
which makes them essential for detecting malicious activity.
By analyzing those logs you can identify patterns that indicate brute force
attacks in progress. One way to detect a brute force attack is by
using queries in log analysis. So index=security_logs "failed
password" so this is the message that should appear in the logs
and then | stats count by source. This command is specifically designed
to find instances where login attempts fail. Here's how it works.
index=security_logs specifies that the query should search within the
security logs index which likely contains entries
related to the login attempts. "failed password" searches for events
in the logs that contain the phrase "failed password"
which is commonly used to log unsuccessful login attempts.
| stats count by source is a statistical function that counts the
occurrences how many times for example this occurrence appeared
grouped by the source IP address. For example it should be
an IP address and then how many failed passwords
was on a specific IP. Essentially this part of the command aggregates the
data to show how many failed login attempts came from
each unique IP address. The result from this query could look
like this IP address and the IP address
how many failed login attempts were appeared
20 for example index=security_logs "failed password" | top user
this work this query works slightly different but still aims to identify
the brute force attack. Again the security logs is the
index and the failed password is the phrase inside the logs.
The | top user is a function that returns the most frequently
targeted user names. It aggregates the data based on the
number of times each username appears in the failed login attempts
essentially ranking the user names based on how many times they were
targeted so it will provide the highest number of
user names that were tried for these login attempts.
So that's the top command. We'll explain like that the result will be like
username root which is very common 15 attempts username admin 10
attempts username super user whatever
number of attempts. This means that the user names root and admin were
specifically targeted. Such results suggest that attackers
are focusing on file value accounts from the root of the admin
and they are actually trying to do brute force attacks on the systems.
Visualizing log data is essential for quickly spotting suspicious activities
or abnormal behavior in real time. Dashboards in SIEM tools like Splunk
provide an interactive interface to track and analyze trends
such as spikes in failed login attempts or repeated attacks.
These visualizations help security analysts monitor potential threats
without needing to shift through the raw log data manually
allowing for more efficient and effective response.
To track brute force attempts or other login failures
over time you can use the query like the one that we explained security logs
"failed password" and then | timechart span=1h count by source.
This query generates a time chart the
tracks failed login attempts by source IP address over a specific time period
in this case hourly each hour. By breaking down the data into time-based
increments you can visualize how login attempts evolve and identify sudden
spikes or patterns that might indicate an ongoing attack.
To create a visual representation of this query in Splunk
close the steps navigate to the dashboard section and select create new
dashboard. Choose the type of chart you want to use
such as a line chart or a column chart use the query
index=security_logs "failed password"
| timechart span=1h count by source as the data source for your chart.
Set a meaningful time range such as the last 24 hours to provide the relevant
view of the activity you are monitoring. The expected output is a time-based
graph where you can track failed login attempts by
source IP. The graph will display which IP addresses
are attempting logins at certain times and you can easily spot any IP addresses
with an unusual high frequency of failed attempts.
These are likely to be sources of brute force attacks.
With this visualization you can quickly identify the most active and
potentially malicious IP addresses allowing you to
take immediate action such as blocking the IP
or further investigating the activity.
Logstash is a powerful tool in the ELK Stack designed for ingesting processing
and forwarding log data from multiple resources.
When you configure Logstash to handle logs from a firewall
it plays a vital role in structuring raw data
raw log data in meaningful searchable formats. The process starts with the
installation of Logstash on your server which can be done
using package manager or by directly downloading the binaries.
Once installed you move on to the configuring the Logstash pipeline
which consists of three main sections input filter and output.
In the input section Logstash is set up to pull logs from your firewall's
log file or syslog output. For example firewall logs are often stored in
files like /var/log/firewall.log
and Logstash needs to be told where to find them.
To start position beginning option ensures that Logstash processor logs
starting from the beginning of the log file.
Once Logstash has adjusted the logs the filter section comes into play.
Here you can use filters like Grok to parse and extract
specific data fields from the raw logs. For example you might use the Grok
filter to pull out the source IP destination IP and action taken by
the firewall. This step is essential for
transforming unstructured log data into structured
fields that can be indexed and analyzed. Finally the output section
of the configuration defines where the parsed data will go.
In this case you can configure Logstash to forward the processed data to the
Elasticsearch where it will be indexed and made available for
search and analysis. The logs are usually stored in an
index format such as "firewall_logs-date" or whatever name
allowing for easy organization and querying based on the date the
logs were created or upon a different tactic and methodology.
Once the setup is complete and Logstash is actively processing the logs
the output will consist of structured data that can be queried and
visualized using visualization tools like Kibana.
Its firewall entries such as one of the IP address
destination and source with action of accept will be broken into
different fields making it easy to search for trends, identify patterns
and gain insights into the network security events.
This integration of Logstash and Elasticsearch enables you to create a robust
log management system that enhances your ability to monitor and
secure your network. Once the logs are indexed into Elastic
search you can use its powerful querying capabilities to detect
potential security threats. By constructing targeted queries you
can identify issues live such as unauthenticated access
and brute force attempts. This query allow for efficient
monitoring of network activity making it easier to identify
security incidents. One way to detect suspicious activity is
searching for traffic from a specific source IP address.
For example using the query source_ip:
and then the IP address AND action:deny
will filter the logs where traffic from IP address
the specific address was denied by the firewall.
This can help you identify a particular source if it's attempting
unauthorized access or repeatedly trying to breach the firewall.
A typical output might show logs indicating that this IP address
has been denied access multiple times to different destinations
suggesting potential malicious activity. These types of logs are
useful for detecting early signs of attack allowing timely
intervention. Another important query focuses on detecting unusual access
patterns for specific ports. For example destination_port:80
AND action:allow searches for allowed traffic targeting port 80.
Since port 80 is commonly used for web traffic monitoring it for
unusual activity can reveal if there are abnormal access patterns
such as frequent connections that may not match regular web browsing
behavior. The results might show a list of allowed
traffic entries but it's the frequency and context of these connections that
will help determine whether the activity is a legitimate or
indicative of an exploit attempt. By regularly reviewing those patterns
network administrators can spot the irregularities and respond.
To detect brute force attacks you can search for multiple
login failures like we saw on Splunk like source_ip AND action:deny
| stats count by source_ip. This query helps identify repeated
denied login attempts for a specific IP address a common sign of
brute force attack. The results might show that the IP
address has made a high number of failed login attempts
indicating an automated attack or a security misconfiguration that requires
attention. Such queries are particularly useful
for spotting high-risk activities and of course
ELK Stack can be modeled can be customized
to use the best the best queries that are possible for the infrastructure.
As we said the Kibana is the powerful tool for visualizing the data stored in the
Elasticsearch on the ELK Stack and it allows you to create
interactive dashboards to track and analyze the events.
By using Kibana security teams can easily identify the trends anomalies
and monitor key security metrics in real time.
To begin you can open Kibana and navigate to the dashboard section.
From there you can create a new dashboard specifically for your security
event monitoring needs. Adding various visualizations to the dashboard
will help you to track important firewall events and gain
different insights into network activity. One of the first visualizations
you might want to add is the bar chart for top
denied IP addresses. This chart will show you the source IP
addresses that have been mostly frequently
denied accessed by the firewall. To configure this you will aggregate
data by the source IP field and display the count of deny actions.
This will give you a quick view of which IP addresses are attempting access
and being blocked most often or potentially highlighting sources of
suspicious activity. Another useful visualization is the
pie chart for action breakdown. This will break down the actions taken
by the firewall such as allow versus deny. The pie chart can be configured
to aggregate data by the action field providing an easy to read representation
to the distribution of traffic types. This visualization is particularly
useful for understanding the general behavior of the network of the
firewall activity for example. Finally you can create a line graph
for detailed denied access. This visualization will allow you to
track the number of denied connections per day
which can help you to spot trends over time.
Configuring this line graph will with a time range of 24 hours or five or
seven days will allow you to monitor if there are any spikes in denied traffic
that might indicate ongoing or recurring attempts to breach
the firewall or the systems.