Centralized Logging for Agility
July 6, 2017
Centralized Logging of Events from Systems to a Central Point
The overall goal is to centralized critical events (defined specific events) into a centralized solution within the Agility platform. Review/IR can be performed from this centralized solution and will allow for forwarding of events into secondary systems for further review/analysis.
Centralized Syslog Repositories
For teams comfortable with using Syslog, modern implementations such as syslog-ng and rsyslog allow for a centralized Syslog repository to accept Syslog entries from other servers.
Since the entries are centralized, they are aggregated easily and log entry loss is limited in the case of a server shutdown.
Client Log Shipper
NXLog is one of the most popular Windows log shippers.
Scale/Distributing Syslog Repositories
TODO Need to determine how to scale the centralized repository(s)
- front-end load-balancer
- statically assign customers to specific endpoint
- number of customer/endpoints per repository
Network location of Syslog Repositories
TODO Need to determine location of centralized Syslog servers
Currently for the POC the Syslog server is located in VLAN 3. I will recommend creating a new separate network located off the Cisco Nexus 7000 or maybe as a Customer Tenant network to reduce the amount of impact on infrastructure.
See the infrastructure impact diagram
Diagram of Centralized Syslog POC
.
Syslog server
System Requirements
To set up a Linux host as a central log server, we need to create a separate /var partition, and allocate a large enough disk size or create a LVM special volume group. That way, the Syslog server will be able to sustain the exponential growth of collected logs over time.
Allow firewall ports from server
- Based upon Centos 7 server
- Default port number for syslog is 514
[root@server ~]# firewall-cmd --permanent --add-port=514/tcp
[root@server ~]# firewall-cmd --permanent --add-port=514/udp
[root@server ~]# firewall-cmd --reload
Enable Rsyslog Daemon
rsyslog daemon comes pre-installed on Centos Linux distributions, but is not enabled by default. To enable rsyslog daemon to receive external messages, edit its configuration file located in /etc/rsyslog.conf.
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
Note that both TCP and UDP can be set on the server simultaneously to listen on TCP/UDP connections.
Open /etc/rsyslog.conf with a text editor, and append the following template before the GLOBAL DIRECTIVES block. Received messages from remote clients written to a single file named after their IP address, use the following template.
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
$MaxMessageSize 64k
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
$ModLoad immark # provides --MARK-- message capability
$ModLoad imfile
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
# set some access rights to written log files
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
# do NOT escape control chars
$EscapeControlCharactersOnReceive off
# Filter duplicated messages
$RepeatedMsgReduction on
$template DynaFile,"/var/log/remote/%HOSTNAME%.log"
*.* -?DynaFile
*.* @10.16.72.38:514
Restart the service
sudo systemctl restart rsyslog
Check if process is listening
sudo ss -pl | grep rsyslog
udp UNCONN 0 0 *:syslog *:* users:(("rsyslogd",pid=18546,fd=3))
udp UNCONN 0 0 :::syslog :::* users:(("rsyslogd",pid=18546,fd=4))
tcp LISTEN 0 25 *:shell *:* users:(("rsyslogd",pid=18546,fd=5))
tcp LISTEN 0 25 :::shell :::* users:(("rsyslogd",pid=18546,fd=6))
Logrotate
logrotate will rename or compress the main log when a condition is met so that the next event is recorded on an empty file.
In addition, it will remove “old” log files and will keep the most recent ones.
TODO We get to decide what “old” means and how often we want logrotate to clean up the logs for us and what/if a retention period is required
TODO Backup and archival from server
- what
- how
- when
- how long
Installing Logrotate in Linux
[root@server ~]# yum update && yum install logrotate
Options
[root@server ~]# cd /etc/logrotate.d/
[root@server ~]# touch remotelogs
/var/log/remote/*/messages.log
{
daily
copytruncate
rotate 5
size 10M
compress
dateext
dateformat -%Y-%m-%d
missingok
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
The first line indicates that the directives inside the block apply to all directories within /var/log/remote/ with the specific file:
- weekly means that the tool will attempt to rotate the logs on a weekly basis. Other possible values are daily and monthly.
- rotate 3 indicates that only 3 rotated logs should be kept. Thus, the oldest file will be removed on the fourth subsequent run.
- size=10M sets the minimum size for the rotation to take place to 10M. In other words, each log will not be rotated until it reaches 10MB.
- compress and delaycompress are used to tell that all rotated logs, with the exception of the most recent one, should be compressed.
Perform a dry-run
Use the -d option followed by the configuration file (you can actually run logrotate by omitting this option):
[root@server ~]# logrotate -d /etc/logrotate.d/remotelogs.conf
Collecting Logs from a Windows System
This topic provides procedures for installing NXLog:
Installing NXLog
NXLog (generic server configuration)
1. Download the newest stable NXLog-community version of NXLog, available at http://nxlog.co/products/NXLog-community-edition/download.
2. Follow the instructions to install the file.
3. For security, make a backup copy of C:\Program Files (x86)\nxlog\conf\nxlog.conf, and give it another name. (You can delete it later.)
4. Delete the contents in the new copy of nxlog.conf and replace it with the following:
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Input eventlog>
Module im_msvistalog
SavePos FALSE
ReadFromLast FALSE
Query <QueryList>\
<Query Id="0" Path="System">\
<Select Path="System">*[System[(Level=4 or Level=0) and (EventID=4740 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4735 or EventID=4628)]]</Select>\
<Select Path="Microsoft-Windows-Security-Audit-Configuration-Client/Operational">*[System[(Level=4 or Level=0) and (EventID=4740 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4735 or EventID=4628)]]</Select>\
<Select Path="System">*[System[Provider[@Name='LsaSrv'] and (EventID=40960)]]</Select>\
</Query>\
<Query Id="1" Path="Application">\
<Select Path="Application">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=1000 or EventID=1002 or EventID=1001 or EventID=1 or EventID=2)]]</Select>\
<Select Path="System">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=1000 or EventID=1002 or EventID=1001 or EventID=1 or EventID=2)]]</Select>\
<Select Path="Application">*[System[Provider[@Name='.NET Runtime'] and (EventID=1026)]]</Select>\
</Query>\
<Query Id="2" Path="Microsoft-Windows-AppLocker/EXE and DLL">\
<Select Path="Microsoft-Windows-AppLocker/EXE and DLL">*[System[(EventID=8004 or EventID=8003 or EventID=8006 or EventID=8007)]]</Select>\
<Select Path="Microsoft-Windows-AppLocker/MSI and Script">*[System[(EventID=8004 or EventID=8003 or EventID=8006 or EventID=8007)]]</Select>\
</Query>\
<Query Id="3" Path="Security">\
<Select Path="Security">*[System[(Level=4 or Level=0) and (EventID=104 or EventID=1102)]]</Select>\
<Select Path="System">*[System[(Level=4 or Level=0) and (EventID=104 or EventID=1102)]]</Select>\
<Select Path="Security">*[System[Provider[@Name='Microsoft-Windows-Eventlog'] and (Level=4 or Level=0) and (EventID=6005)]]</Select>\
<Select Path="Setup">*[System[Provider[@Name='Microsoft-Windows-Eventlog'] and (Level=4 or Level=0) and (EventID=6005)]]</Select>\
<Select Path="System">*[System[Provider[@Name='Microsoft-Windows-Eventlog'] and (Level=4 or Level=0) and (EventID=6005)]]</Select>\
</Query>\
<Query Id="4" Path="System">\
<Select Path="System">*[System[(EventID=43 or EventID=400)]]</Select>\
</Query>\
<Query Id="5" Path="System">\
<Select Path="System">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=5038 or EventID=6281 or EventID=3001 or EventID=3002 or EventID=3003 or EventID=3004 or EventID=3010 or EventID=3023 or EventID=219)]]</Select>\
<Select Path="Microsoft-Windows-CodeIntegrity/Operational">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=5038 or EventID=6281 or EventID=3001 or EventID=3002 or EventID=3003 or EventID=3004 or EventID=3010 or EventID=3023 or EventID=219)]]</Select>\
</Query>\
<Query Id="6" Path="Security">\
<Select Path="Security">*[System[(Level=4 or Level=0) and (EventID=4624 or EventID=4625)]]</Select>\
<Select Path="Microsoft-Windows-NTLM/Operational">*[System[(Level=4 or Level=0) and (EventID=4624 or EventID=4625)]]</Select>\
</Query>\
<Query Id="7" Path="Application">\
<Select Path="Application">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5) and (EventID=64004 or EventID=4688 or EventID=4697 or EventID=4698 or EventID=4657 or EventID=7035 or EventID=7036 or EventID=7040)]]</Select>\
<Select Path="Security">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5) and (EventID=64004 or EventID=4688 or EventID=4697 or EventID=4698 or EventID=4657 or EventID=7035 or EventID=7036 or EventID=7040)]]</Select>\
<Select Path="System">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5) and (EventID=64004 or EventID=4688 or EventID=4697 or EventID=4698 or EventID=4657 or EventID=7035 or EventID=7036 or EventID=7040)]]</Select>\
</Query>\
<Query Id="8" Path="Application">\
<Select Path="Application">*[System[(Level=4 or Level=0) and (EventID=6 or EventID=7045 or EventID=1022 or EventID=1033 or EventID=903 or EventID=904 or EventID=905 or EventID=906 or EventID=907 or EventID=908)]]</Select>\
<Select Path="System">*[System[(Level=4 or Level=0) and (EventID=6 or EventID=7045 or EventID=1022 or EventID=1033 or EventID=903 or EventID=904 or EventID=905 or EventID=906 or EventID=907 or EventID=908)]]</Select>\
<Select Path="Microsoft-Windows-Application-Experience/Program-Inventory">*[System[(Level=4 or Level=0) and (EventID=6 or EventID=7045 or EventID=1022 or EventID=1033 or EventID=903 or EventID=904 or EventID=905 or EventID=906 or EventID=907 or EventID=908)]]</Select>\
</Query>\
<Query Id="9">\
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
</Query>\
<Query Id="10" Path="System">\
<Select Path="System">*[System[(Level=2) and (EventID=7000 or EventID=7011 or EventID=7022 or EventID=7023 or EventID=7024 or EventID=7026 or EventID=7030 or EventID=7031 or EventID=7032 or EventID=7034 or EventID=7043)]]</Select>\
</Query>\
</QueryList>
</Input>
<Output out>
Module om_udp
Host [SYSLOG SERVER IP]
Port 514
Exec $EventTime = strftime($EventTime, '%Y-%m-%d %H:%M:%S, %z');
Exec $Message = to_json(); to_syslog_bsd();
</Output>
<Route 1>
Path eventlog, internal, IIS_Logs => out
</Routedefine ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Input eventlog>
Module im_msvistalog
SavePos FALSE
ReadFromLast FALSE
Query <QueryList>\
<Query Id="0" Path="System">\
<Select Path="System">*[System[(Level=4 or Level=0) and (EventID=4740 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4735 or EventID=4628)]]</Select>\
<Select Path="Microsoft-Windows-Security-Audit-Configuration-Client/Operational">*[System[(Level=4 or Level=0) and (EventID=4740 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4735 or EventID=4628)]]</Select>\
<Select Path="System">*[System[Provider[@Name='LsaSrv'] and (EventID=40960)]]</Select>\
</Query>\
<Query Id="1" Path="Application">\
<Select Path="Application">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=1000 or EventID=1002 or EventID=1001 or EventID=1 or EventID=2)]]</Select>\
<Select Path="System">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=1000 or EventID=1002 or EventID=1001 or EventID=1 or EventID=2)]]</Select>\
<Select Path="Application">*[System[Provider[@Name='.NET Runtime'] and (EventID=1026)]]</Select>\
</Query>\
<Query Id="3" Path="Security">\
<Select Path="Security">*[System[(Level=4 or Level=0) and (EventID=104 or EventID=1102)]]</Select>\
<Select Path="System">*[System[(Level=4 or Level=0) and (EventID=104 or EventID=1102)]]</Select>\
<Select Path="Security">*[System[Provider[@Name='Microsoft-Windows-Eventlog'] and (Level=4 or Level=0) and (EventID=6005)]]</Select>\
<Select Path="Setup">*[System[Provider[@Name='Microsoft-Windows-Eventlog'] and (Level=4 or Level=0) and (EventID=6005)]]</Select>\
<Select Path="System">*[System[Provider[@Name='Microsoft-Windows-Eventlog'] and (Level=4 or Level=0) and (EventID=6005)]]</Select>\
</Query>\
<Query Id="4" Path="System">\
<Select Path="System">*[System[(EventID=43 or EventID=400)]]</Select>\
</Query>\
<Query Id="5" Path="System">\
<Select Path="System">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=5038 or EventID=6281 or EventID=3001 or EventID=3002 or EventID=3003 or EventID=3004 or EventID=3010 or EventID=3023 or EventID=219)]]</Select>\
<Select Path="Microsoft-Windows-CodeIntegrity/Operational">*[System[(Level=2 or Level=3 or Level=4 or Level=0) and (EventID=5038 or EventID=6281 or EventID=3001 or EventID=3002 or EventID=3003 or EventID=3004 or EventID=3010 or EventID=3023 or EventID=219)]]</Select>\
</Query>\
<Query Id="6" Path="Security">\
<Select Path="Security">*[System[(Level=4 or Level=0) and (EventID=4624 or EventID=4625)]]</Select>\
<Select Path="Microsoft-Windows-NTLM/Operational">*[System[(Level=4 or Level=0) and (EventID=4624 or EventID=4625)]]</Select>\
</Query>\
<Query Id="7" Path="Application">\
<Select Path="Application">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5) and (EventID=64004 or EventID=4688 or EventID=4697 or EventID=4698 or EventID=4657 or EventID=7035 or EventID=7036 or EventID=7040)]]</Select>\
<Select Path="Security">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5) and (EventID=64004 or EventID=4688 or EventID=4697 or EventID=4698 or EventID=4657 or EventID=7035 or EventID=7036 or EventID=7040)]]</Select>\
<Select Path="System">*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0 or Level=5) and (EventID=64004 or EventID=4688 or EventID=4697 or EventID=4698 or EventID=4657 or EventID=7035 or EventID=7036 or EventID=7040)]]</Select>\
</Query>\
<Query Id="8" Path="Application">\
<Select Path="Application">*[System[(Level=4 or Level=0) and (EventID=6 or EventID=7045 or EventID=1022 or EventID=1033 or EventID=903 or EventID=904 or EventID=905 or EventID=906 or EventID=907 or EventID=908)]]</Select>\
<Select Path="System">*[System[(Level=4 or Level=0) and (EventID=6 or EventID=7045 or EventID=1022 or EventID=1033 or EventID=903 or EventID=904 or EventID=905 or EventID=906 or EventID=907 or EventID=908)]]</Select>\
<Select Path="Microsoft-Windows-Application-Experience/Program-Inventory">*[System[(Level=4 or Level=0) and (EventID=6 or EventID=7045 or EventID=1022 or EventID=1033 or EventID=903 or EventID=904 or EventID=905 or EventID=906 or EventID=907 or EventID=908)]]</Select>\
</Query>\
<Query Id="10" Path="System">\
<Select Path="System">*[System[(Level=2) and (EventID=7000 or EventID=7011 or EventID=7022 or EventID=7023 or EventID=7024 or EventID=7026 or EventID=7030 or EventID=7031 or EventID=7032 or EventID=7034 or EventID=7043)]]</Select>\
</Query>\
</QueryList>
</Input>
<Output syslog_server>
Module om_tcp
Host 10.2.0.250
Port 514
Exec to_syslog_snare();
</Output>
<Route 1>
Path eventlog, internal => syslog_server
</Route>
5. Replace [SYSLOG SERVER IP] with the IP address of the syslog server
6. Save the file.
7. Open Windows Services and restart the NXLog service.
8. SSH into syslog servers can check /var/log - the source IP address of the client
Note: If you need to debug NXLog, open C:\Program Files (x86)\nxlog\data\nxlog.log.
(OPTIONAL) Collecting Logs from a Windows System IIS
(OPTIONAL) Installing Sysmon & Collecting Logs
(OPTIONAL) Sysmon is an advanced data collection service that monitors in the background on Windows, recording security-related events for use in intrusion detection and forensics.
Sysmon is a free Windows Sysinternals tool from Sysinternals/Microsoft.
After installation and configuration on a system, Sysmon acts as a Windows service to log any system activity to the Windows event log. NXLog collects this audit log data and forwards it to USM Anywhere over the Syslog protocol on UDP port 514.
On Microsoft’s Windows Vista and more recent Windows operating system versions, events are stored in Applications and Services !Logs/Microsoft/Windows/Sysmon/Operational.
On older operating systems, Windows writes events to the System event log.
To install Sysmon
1. Download the Sysmon ZIP file and unzip it in the target system.
2. Save the unzipped configuration file (shown) in a folder with the name sysmon_config.xml.
Sysmon schemaversion="3.10">
<HashAlgorithms>md5</HashAlgorithms>
<EventFiltering>
<!-- Do not log process termination -->
<ProcessTerminate onmatch="include"/>
<!-- Exclude connection to Amazon metadata -->
<NetworkConnect onmatch="exclude">
<DestinationIp>169.254.169.254</DestinationIp>
</NetworkConnect>
<!-- Exclude ImageLoad with signed files -->
<ImageLoad onmatch="exclude">
<Signed>true</Signed>
</ImageLoad>
<RawAccessRead onmatch="include"/>
</EventFiltering>
</Sysmon>
This is my recommend/tested use of this configuration file to drastically reduce the number of events recorded at the host level, for example:
- Process termination.
- Connection to 169.254.169.254
- ImageLoad events that are valid signed files.
- RawAccessRead events (very noisy).
3. Install Sysmon in the Windows system and execute the following command:sysmon.exe -accepteula -h md5 -n -l -i sysmon_config.xml Sysmon starts logging the information to the Windows Event Log.
4. Edit the NXLog configuration file under under EventLog > Querylist.
Look for the <Input eventlog> tag and add the line:
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
See below:
<Input eventlog>
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>\
</Query>\
</QueryList>
</Input>
5.Save the file.
6. Apply the last configuration by opening Windows Services and restarting NXLog.
7. SSH into syslog servers can check /var/log - goto the source directory (ip address) and review the logs to confirm
(OPTIONAL) Microsoft Windows DNS Server
These tasks are subtasks of Collecting Logs from a Windows System. They enable a Windows DNS server to send log data to a syslog server
| Device | Details |
|---|---|
| Device Vendor | Microsoft |
| Device Type | Server |
| Transport Type | Syslog |
| Links | https://www.microsoft.com/en-US/download/details.aspx?id=53314 and Collecting Logs from a Windows System. |
Integration Prerequisites
- IP Address of the syslog server
Integrating Microsoft Windows DNS Server
Here are some best practices:
- When you set the debug logging options, select log packets for debugging. You may want to adjust these selections for performance.
-
The most useful debug logging output comes from selecting at least three options:
- One option under Packet direction
- One option under Transport protocol
- At least one more option in another category
In addition to selecting events for the DNS debug log file, you can specify the file name, location, and a maximum file size for the file. However, in most cases, the default selections are adequate.
-
Consider limiting the traffic captured by logging between your server and the DNS server.
- Select Filter packets by IP address.
- Add the appropriate IP addresses by clicking Filter.
Enabling Windows DNS Debug Logging
To enable Windows DNS debug logging
1. From the Windows Start Menu, select All Programs > Administrative Tools.
2. Select DNS.
3. From the console tree, right-click the applicable DNS server; then click Properties.
4. Click the Debug Logging tab.
5. Select log packets for debugging, then select the events that you want the DNS server to record for debug logging.
6. Add the following lines to the NXLog configuration file:
<Input DNS_Logs>
Module im_file
File "C:\\Windows\\Sysnative\\dns\\dns.log"
SavePos TRUE
InputType LineBased
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
$Message = $raw_event: \
$SourceName = "DNS"; \
$raw_event = to_json(); \
}
</Input>
Setting Up Windows Event Collection Forwarding
If you plan to implement Windows Event Collection and Forwarding (WEF) to collect Windows logs, add the following line in the collector’s NXLog configuration file:
Nxlog conf
<Select Path="ForwardedEvents">*</Select>