Parsing Timestamps from Windows and Cisco devices from Props.conf

[MikeyG]

I have one Sourcetype [syslog-ng] with a myriad of devices all sending (IDS; CISCO; Linux; Windows Servers) through it. When I added the Cisco devices discovered that %FWSM events do not have a timestamp and indexing slowed WAY down .. apparently Splunk is parsing everything looking for a timestamp. Given the following which parses windows events correctly, skips the first timestamp, and retrieves the correct timestamp:

[syslog]
MAX_TIMESTAMP_LOOKAHEAD =128
TIME_PREFIX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s[^\s]*\s
TRANSFORMS-strip-syslog = syslog-header-stripper-ts-host
TRANSFORMS-ip_address = basic_ip_address
TRANSFORMS-win_snare = windows-snare
REPORT-snare = windows-snare, windows-snare-group-mod
TRANSFORMS-sfire_ids = sourcefire
REPORT-ids_evnts = sourcefire
TRANSFORMS-nms_data = NMS

Can I add the following to grab the first timestamp in the event?

# Cisco Firewall Service Modules - Where %FWSM messages do not contain a timestamp
[host :: XXX-FW-*]
MAX_TIMESTAMP_LOOKAHEAD = 32
TIME_FORMAT = %b %d %H:%M:%S

[rforsythe]

Also make sure you have 'logging timestamp' enabled on your FWSM module (and other Cisco devices) to include that in messages. Cisco logging is very flexible, but that means the default config isn't always how you'd like it.

[gkanapathy · Accepted solution]

Possibly. When [host::myhostname] and [mysourcetype] settings both apply, the settings will all be merged, and in case of conflict, the [host::myhostname] settings win. You may also need to modify the TIME_PREFIX as Splunk will not look for a timestamp until after that match.

However, note that [host::myhostname] settings will only apply if you are not relying on a tranformed host name. If your data is coming in to Splunk via UDP, then host value will be the IP address from which it was received. If it is coming in via a Splunk forwarder, it will be whatever the forwarder reports. It will not be the value extracted by syslog-header-stripper-ts-host, since the host has not been set to that value at the time the rules are selected.

Another alternative is to create a custom datetime.xml file with the two formats listed in appropriate priority.