Hi all,
I've been absolutely stumped with a problem now for two days. I can't seem to get event breaks working for when a file is forwarded from a server, even though it seems to work with an exact copy of the same file as long as it's on the same machine as the indexer. Could anyone please take a look to see where I went wrong?
Sample Data:
(ORD_Area_Date) INFO(19May13@21:33:12:646) Updated entrydate: SYBRCH
(ORD_Area_Date) INFO(19May13@21:33:12:731) Loading market
(ORD_Area_Date) INFO(19May13@21:33:12:747) Historical market not there - use default
(ORD_Area_Date) INFO(19May13@21:33:12:836) Loading current market
I want each event to break at (ORD_Area_Date)
(this pattern is used for multiple applications, so I cannot hard code to ORD_Area_Date
- it needs to regex match a set of brackets with some words in the middle.
Machine 1: Forwarder - Unix
inputs.conf:
[monitor:///logs/areaDateService/*.log*]
disabled = false
index = tarsan_dev
sourcetype = tarsan_dby_HMSf
Machine 2: Indexer - Windows
props.conf:
[tarsan_dby_HMSf]
BREAK_ONLY_BEFORE = \(([A-Za-z1-9_\s]+))\s
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %d%b%y@%H:%M:%S:%f
TIME_PREFIX = [A-Z]\(
pulldown_type = 1
Screenshots:
Image 1 - sourcetype stanza used in props.conf above works fine in Data Preview with the local copy
Image 2 - Event breaking works fine on local copy
Image 3 - Event breaking not working on forwarded logs
Other Notes:
Any help would be greatly appreciated. Please let me know if additional information is required.
That's correct, it should be \s
- thanks Kristian!
dart, jonathon, it seems that the regex has an 'S' instead of 's', which would cause it to fail, i.e. you're trying to match non-whitespace characters instead of whitespace (between the (ORD_Area_Date)
and INFO
.)
/K
Hi,
Yes, the forwarding machine is using a Universal Forwarder. I have the props.conf on the Indexing machine, with the inputs.conf on the forwarder specifying the sourcetype.
I tried using the stanza above in the Data Preview, but it was unable to break up the events properly.
Thanks,
Jonathan
Hi Jonathon,
Is the forwarder a Universal Forwarder?
If it's not, line breaking happens on the forwarder, if it is, line breaking happens on the indexer.
I usually do this with a LINE_BREAKER and SHOULD_LINEMERGE being false.
[tarsan_dby_HMSf]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\([^\) ]+\)\S+\w+\(\d+
I got it working in the end, but not through the props.conf. I ended up just adding the regex for the timestamp in to datetime.xml and did not specify a sourcetype in the inputs.conf file.
Well, there might be a good reason for your action, but all events must have a sourcetype - and I fail to see why you should get Splunk to guess it, if you already know what the sourcetype is/should be.
A lot of things are dependent on sourcetype (field extractions, searches, eventtypes etc).
Are these all single line events? If so, I would suggest that you set your props accordingly;
[tarsan_blah]
SHOULD_LINEMERGE = false
TIME_FORMAT = %d%b%y@%H:%M:%S:%3N
TIME_PREFIX = [A-Z]\(
The NO_BINARY_CHECK
is an inputs-phase related setting, so you could try copying it to the forwarder as well. Although the %f
seems to work, I haven't seen it in official Splunk docs.
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings
Hope this helps,
Kristian
Hi Kristian,
Thanks for the suggestion. Unfortunately, I tried that and it didn't work. Even on auto break by timestamp, it still ends up clustering events together.
The annoying thing is it works fine if I don't try and specify the timestamp format, but it can't parse the date format and gets that wrong.