Background information about my environment:
Distributed environment with CM server, clustered indexers(two indexers), two search heads(not clustered)
We have the F5 Network Apps that helps with the ingestion of our BigIP logs. We are experiencing extensive line breaking issues. Splunk isn't breaking each event into one log entry. We have tried countless LINE_BREAKING attributes. Our current props.conf attributes are as followed:
[f5:bigip:apm:syslog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ^(\w{3})\s(\d{2})\s(\d{2}):(\d{2}):(\d{2})
Splunk uses the first capture group in the LINE_BREAKER
attribute and discards the match for that group. The regex should identify some text (which can be empty) that falls between events. In your example, it seems unlikely a timestamp will separate events. Perhaps this will work (if you haven't tried it already).
LINE_BREAKER = ^()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
Splunk uses the first capture group in the LINE_BREAKER
attribute and discards the match for that group. The regex should identify some text (which can be empty) that falls between events. In your example, it seems unlikely a timestamp will separate events. Perhaps this will work (if you haven't tried it already).
LINE_BREAKER = ^()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
Hi,
I'm not that familiar with F5, but when I look at your LINE_BREAKER, do you need brackets? Do you want to break before something like 'Oct 24 08:34:00' ? Then I think this should work:
LINE_BREAKER=^\w{3}\s\d*\s\d{2}\W\d{2}\W\d{2}
Good luck!
Cheers,
Ruud
@ruud_umbrio & @richgalloway, Yes the beginning of the log file follows "Oct 24 08:34:00" format. I've tried many different combinations; unfortunately, some events are still showing multiple log entries. The props.conf file in the app's local directory reads:
[f5:bigip:apm:syslog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
I ran the following search: index=network sourcetype="f5:bigip:apm:syslog" linecount!=1. The results yielded 366 events in a day.
I'm stomped on this one.
Please provide a sanitized example of an event that is not broken as expected.
Is it possible to provide an email address for me to send to? We have really strict policies.
Having more eyes looking the sample data will improve the chances of successfully resolving your problem. Feel free to mask sensitive data before posting sample events.
Unfortunately, I'm not able to post even a sanitized example of our events. I can give you all the pertinent information:
Search- index=network sourcetype="f5:bigip:apm:syslog" linecount!=1
Time Event
10/26/17 Oct 26 10:04:58 bigip1 (Sensitive data)......
10:04:58:000am Oct 26 10:05:00 bigip2 (Sensitive data)........
Oct 26 10:05:02 bigip3 (Sensitive data)..........
This is an example of one event with three log entries. The desired result is to have one event per log entry. I know this isn't ideal, but I hope you can get an idea of what I'm trying to convey.
What you have there is three different time formats that a single regex can't cover easily. Consider using the default LINE_BREAKER
and BREAK_ONLY_BEFORE_DATE
settings and update your DATETIME_CONFIG.XML file to match the date formats in your data.
Thank you for your help on this. My previous comment didn't clearly reflect the issue we are having. I attempted to imitate the actual Splunk forward. I hope this will articulate exactly what's going on.
SEVERAL LOGS IN ONE EVENT AS FOLLOWS:
EVENT 1
Oct 26 10:04:58 bigip1 (rest of the log)......end of log
Oct 26 10:05:00 bigip2 (rest of the log).......end of log
Oct 26 10:05:02 bigip3 (rest of the log)........end of log
These three log entries would all fit in one event.
The desired output is to have one log entry per event.
EVENT 1
Oct 26 10:04:58 bigip1 (rest of the log)........end of log
EVENT 2
Oct 26 10:05:00 bigip2 (rest of the log).........end of log
EVENT 3
Oct 25 10:05:02 bigip3 (rest of the log)..........end of log
Hmm... That should work.
Have you tried these props.conf settings?
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
I've attempted every iteration of the appropriate regex LINE_BREAKER. The current props.conf settings are as follows:
LINE_BREAKER=^\w{3}\s\d*\s\d{2}\W\d{2}\W\d{2}
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
Our events are still showing as:
EVENT 1
Oct 26 10:04:58 bigip1 (rest of the log)......end of log
Oct 26 10:05:00 bigip2 (rest of the log).......end of log
Oct 26 10:05:02 bigip3 (rest of the log)........end of log
These three log entries would all fit in one event.
Hi, don't know if my previous comment came through. You could try this:
TIME_PREFIX =
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOAKAHEAD = 15
LINE_BREAKER = ([\r\n]+)\w{3}\s\d+\s\d{2}\W\d{2}\W\d{2}
SHOULD_LINEMERGE = false
TRUNCATE =
This regex string may cover all three date formats. It's not perfect, but should get you headed in the right direction if you don't want to mess with DATETIME_CONFIG.XML.
((\d+\/\d+\/\d\d\s\w+\s\d+\s\d+:\d\d:\d\d)|(\d+:\d\d:\d\d:\d{3}[ap]m\s\w+\s\d+)|(\w+\s\d+\s\d+:\d\d:\d\d))