Log Collection and Centralization

3 min read Infrastructure & DevOps Security

Log Collection and Centralization

Centralized log collection enables comprehensive security analysis across distributed environments. Implementing log aggregation requires careful architecture planning to ensure reliable collection, secure transmission, and efficient storage. Modern log collection systems support various protocols and formats while providing filtering and enrichment capabilities.

Windows Event Forwarding (WEF) provides native log centralization:

# Configure collector server
wecutil qc /q

# Create subscription for security events
$xml = @'
<QueryList>
  <Query Id="0" Path="Security">
    <Select Path="Security">*[System[(EventID=4624 or EventID=4625 or EventID=4634 or EventID=4647 or EventID=4648 or EventID=4672 or EventID=4697 or EventID=4698 or EventID=4699 or EventID=4700 or EventID=4701 or EventID=4702 or EventID=4719 or EventID=4720 or EventID=4722 or EventID=4723 or EventID=4724 or EventID=4725 or EventID=4726 or EventID=4728 or EventID=4729 or EventID=4730 or EventID=4731 or EventID=4732 or EventID=4733 or EventID=4734 or EventID=4735 or EventID=4737 or EventID=4738 or EventID=4739 or EventID=4740 or EventID=4741 or EventID=4742 or EventID=4743 or EventID=4744 or EventID=4745 or EventID=4746 or EventID=4747 or EventID=4748 or EventID=4749 or EventID=4750 or EventID=4751 or EventID=4752 or EventID=4753 or EventID=4754 or EventID=4755 or EventID=4756 or EventID=4757 or EventID=4758 or EventID=4759 or EventID=4760 or EventID=4761 or EventID=4762 or EventID=4763 or EventID=4764 or EventID=4767 or EventID=4768 or EventID=4769 or EventID=4770 or EventID=4771 or EventID=4772 or EventID=4773 or EventID=4774 or EventID=4775 or EventID=4776 or EventID=4777 or EventID=4778 or EventID=4779 or EventID=4780 or EventID=4781 or EventID=4782 or EventID=4783 or EventID=4784 or EventID=4785 or EventID=4786 or EventID=4787 or EventID=4788 or EventID=4789 or EventID=4790 or EventID=4793 or EventID=5136 or EventID=5137 or EventID=5138 or EventID=5139 or EventID=5140 or EventID=5141)]]</Select>
  </Query>
</QueryList>
'@

wecutil create-subscription SecurityEvents /cf:Events /l:custom /q:"$xml" /cm:custom /ca:"O:BAG:BAD:(A;;0xf0005;;;BA)(A;;0x5;;;NS)" /de:true

Linux centralized logging with rsyslog:

# Configure rsyslog server for log reception
cat >> /etc/rsyslog.conf << EOF
# Load modules for network reception
module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Template for organizing logs by host
\$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs

# High-performance queue for reliable delivery
main_queue(
    queue.type="LinkedList"
    queue.filename="main_queue"
    queue.maxdiskspace="1g"
    queue.saveonshutdown="on"
    queue.dequeuebatchsize="1000"
    queue.highwatermark="90000"
    queue.lowwatermark="5000"
)
EOF

# Configure clients to forward logs
cat >> /etc/rsyslog.conf << EOF
# Forward authentication logs
auth,authpriv.* @@logserver.company.com:514

# Forward audit logs
\$ModLoad imfile
\$InputFileName /var/log/audit/audit.log
\$InputFileTag audit:
\$InputFileStateFile audit-log
\$InputFileFacility local6
\$InputRunFileMonitor
local6.* @@logserver.company.com:514
EOF

Implement secure log transmission using TLS:

# Generate certificates for rsyslog TLS
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/rsyslog.d/ca-key.pem -out /etc/rsyslog.d/ca-cert.pem

# Configure TLS on server
cat >> /etc/rsyslog.conf << EOF
# Load TLS driver
module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.AuthMode="anon")

# Certificate settings
global(
    DefaultNetstreamDriverCAFile="/etc/rsyslog.d/ca-cert.pem"
    DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server-cert.pem"
    DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server-key.pem"
)

# TLS listener
input(type="imtcp" port="6514")
EOF