Log Analysis Tools and Techniques

Log Analysis Tools and Techniques

Effective log analysis transforms raw data into security insights. Various tools provide different analysis capabilities, from simple pattern matching to advanced correlation and machine learning. Understanding tool strengths helps select appropriate solutions for specific analysis requirements.

PowerShell provides powerful Windows log analysis capabilities:

# Analyze failed logon patterns
$FailedLogons = Get-WinEvent -FilterHashtable @{LogName='Security'; ID=4625} -MaxEvents 10000 |
    Select-Object @{n='Time';e={$_.TimeCreated}},
                  @{n='Account';e={$_.Properties[5].Value}},
                  @{n='Source';e={$_.Properties[19].Value}},
                  @{n='Reason';e={$_.Properties[8].Value}}

# Group by source IP to identify brute force attempts
$FailedLogons | Group-Object Source | Where-Object {$_.Count -gt 10} |
    Select-Object Count, Name | Sort-Object Count -Descending

# Detect privilege escalation
Get-WinEvent -FilterHashtable @{LogName='Security'; ID=4672} |
    Where-Object {$_.Message -notmatch "SYSTEM|LOCAL SERVICE|NETWORK SERVICE"} |
    Select-Object TimeCreated, @{n='User';e={$_.Properties[1].Value}},
                  @{n='Privileges';e={$_.Properties[4].Value}}

# Timeline analysis of security events
$StartTime = (Get-Date).AddDays(-1)
$Events = Get-WinEvent -FilterHashtable @{LogName='Security'; StartTime=$StartTime} |
    Where-Object {$_.ID -in 4624,4625,4634,4672,4720,4732} |
    Select-Object TimeCreated, Id, Message |
    Sort-Object TimeCreated

Linux log analysis with built-in tools and scripts:

#!/bin/bash
# Comprehensive security log analysis script

# SSH analysis
echo "=== SSH Brute Force Detection ==="
grep "Failed password" /var/log/auth.log | 
    awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# Sudo usage analysis
echo -e "\n=== Sudo Command Analysis ==="
grep "COMMAND" /var/log/auth.log | 
    awk -F'COMMAND=' '{print $2}' | sort | uniq -c | sort -rn

# System modifications
echo -e "\n=== System File Modifications ==="
aureport -f -i --summary

# User activity timeline
echo -e "\n=== User Activity Timeline ==="
last -F | head -50

# Failed service starts
echo -e "\n=== Failed Service Starts ==="
journalctl -p err -S "24 hours ago" | grep "Failed to start"

# Unusual process execution
echo -e "\n=== Unusual Process Execution ==="
ausearch -m execve -ts today | aureport -x --summary

Advanced analysis using ELK Stack (Elasticsearch, Logstash, Kibana):

# Logstash configuration for security log processing
input {
  file {
    path => "/var/log/auth.log"
    type => "auth"
    start_position => "beginning"
  }
  
  file {
    path => "/var/log/audit/audit.log"
    type => "audit"
    start_position => "beginning"
  }
}

filter {
  if [type] == "auth" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{PROG:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:message}" }
    }
    
    if [program] == "sshd" and [message] =~ /Failed password/ {
      grok {
        match => { "message" => "Failed password for( invalid user)? %{USERNAME:username} from %{IP:source_ip} port %{INT:source_port}" }
      }
      mutate {
        add_tag => [ "ssh_brute_force" ]
      }
    }
  }
  
  if [type] == "audit" {
    grok {
      match => { "message" => "type=%{WORD:audit_type} msg=audit\(%{NUMBER:timestamp}:%{NUMBER:serial}\): %{GREEDYDATA:audit_message}" }
    }
  }
  
  geoip {
    source => "source_ip"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "security-%{+YYYY.MM.dd}"
  }
}