All Apps and Add-ons

Splunk Add-on for F5 BIG-IP: linebreaking issues

BailsandBrayDad
Explorer

Background information about my environment:
Distributed environment with CM server, clustered indexers(two indexers), two search heads(not clustered)
We have the F5 Network Apps that helps with the ingestion of our BigIP logs. We are experiencing extensive line breaking issues. Splunk isn't breaking each event into one log entry. We have tried countless LINE_BREAKING attributes. Our current props.conf attributes are as followed:
[f5:bigip:apm:syslog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ^(\w{3})\s(\d{2})\s(\d{2}):(\d{2}):(\d{2})

1 Solution

richgalloway
SplunkTrust
SplunkTrust

Splunk uses the first capture group in the LINE_BREAKER attribute and discards the match for that group. The regex should identify some text (which can be empty) that falls between events. In your example, it seems unlikely a timestamp will separate events. Perhaps this will work (if you haven't tried it already).

LINE_BREAKER = ^()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk uses the first capture group in the LINE_BREAKER attribute and discards the match for that group. The regex should identify some text (which can be empty) that falls between events. In your example, it seems unlikely a timestamp will separate events. Perhaps this will work (if you haven't tried it already).

LINE_BREAKER = ^()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
---
If this reply helps you, Karma would be appreciated.
0 Karma

ruud_umbrio
New Member

Hi,
I'm not that familiar with F5, but when I look at your LINE_BREAKER, do you need brackets? Do you want to break before something like 'Oct 24 08:34:00' ? Then I think this should work:

LINE_BREAKER=^\w{3}\s\d*\s\d{2}\W\d{2}\W\d{2}

Good luck!
Cheers,
Ruud

0 Karma

BailsandBrayDad
Explorer

@ruud_umbrio & @richgalloway, Yes the beginning of the log file follows "Oct 24 08:34:00" format. I've tried many different combinations; unfortunately, some events are still showing multiple log entries. The props.conf file in the app's local directory reads:
[f5:bigip:apm:syslog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ()\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
I ran the following search: index=network sourcetype="f5:bigip:apm:syslog" linecount!=1. The results yielded 366 events in a day.
I'm stomped on this one.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please provide a sanitized example of an event that is not broken as expected.

---
If this reply helps you, Karma would be appreciated.
0 Karma

BailsandBrayDad
Explorer

Is it possible to provide an email address for me to send to? We have really strict policies.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Having more eyes looking the sample data will improve the chances of successfully resolving your problem. Feel free to mask sensitive data before posting sample events.

---
If this reply helps you, Karma would be appreciated.
0 Karma

BailsandBrayDad
Explorer

Unfortunately, I'm not able to post even a sanitized example of our events. I can give you all the pertinent information:
Search- index=network sourcetype="f5:bigip:apm:syslog" linecount!=1
Time Event
10/26/17 Oct 26 10:04:58 bigip1 (Sensitive data)......
10:04:58:000am Oct 26 10:05:00 bigip2 (Sensitive data)........
Oct 26 10:05:02 bigip3 (Sensitive data)..........
This is an example of one event with three log entries. The desired result is to have one event per log entry. I know this isn't ideal, but I hope you can get an idea of what I'm trying to convey.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What you have there is three different time formats that a single regex can't cover easily. Consider using the default LINE_BREAKER and BREAK_ONLY_BEFORE_DATE settings and update your DATETIME_CONFIG.XML file to match the date formats in your data.

---
If this reply helps you, Karma would be appreciated.
0 Karma

BailsandBrayDad
Explorer

Thank you for your help on this. My previous comment didn't clearly reflect the issue we are having. I attempted to imitate the actual Splunk forward. I hope this will articulate exactly what's going on.
SEVERAL LOGS IN ONE EVENT AS FOLLOWS:
EVENT 1
Oct 26 10:04:58 bigip1 (rest of the log)......end of log
Oct 26 10:05:00 bigip2 (rest of the log).......end of log
Oct 26 10:05:02 bigip3 (rest of the log)........end of log
These three log entries would all fit in one event.

The desired output is to have one log entry per event.
EVENT 1
Oct 26 10:04:58 bigip1 (rest of the log)........end of log
EVENT 2
Oct 26 10:05:00 bigip2 (rest of the log).........end of log
EVENT 3
Oct 25 10:05:02 bigip3 (rest of the log)..........end of log

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Hmm... That should work.

Have you tried these props.conf settings?

LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
---
If this reply helps you, Karma would be appreciated.
0 Karma

BailsandBrayDad
Explorer

I've attempted every iteration of the appropriate regex LINE_BREAKER. The current props.conf settings are as follows:
LINE_BREAKER=^\w{3}\s\d*\s\d{2}\W\d{2}\W\d{2}
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
Our events are still showing as:
EVENT 1
Oct 26 10:04:58 bigip1 (rest of the log)......end of log
Oct 26 10:05:00 bigip2 (rest of the log).......end of log
Oct 26 10:05:02 bigip3 (rest of the log)........end of log
These three log entries would all fit in one event.

0 Karma

ruud_umbrio
New Member

Hi, don't know if my previous comment came through. You could try this:

TIME_PREFIX =
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOAKAHEAD = 15
LINE_BREAKER = ([\r\n]+)\w{3}\s\d+\s\d{2}\W\d{2}\W\d{2}
SHOULD_LINEMERGE = false
TRUNCATE =

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This regex string may cover all three date formats. It's not perfect, but should get you headed in the right direction if you don't want to mess with DATETIME_CONFIG.XML.

((\d+\/\d+\/\d\d\s\w+\s\d+\s\d+:\d\d:\d\d)|(\d+:\d\d:\d\d:\d{3}[ap]m\s\w+\s\d+)|(\w+\s\d+\s\d+:\d\d:\d\d))
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...