Splunk Enterprise

MAX_EVENTS issue on local Splunk Enterprise application

allisonwalther
Path Finder

Hello all,

I am having an problem with a Splunk application I am making on my local instance of Splunk Enterprise 6.6.3.

The applicaiton essentially kicks off a script then monitors .log files that contain the script's output. I want each file to be its own event, even if the file is 9000+ lines long. In order to ensure that the event does not break at 256 lines, I added MAX_EVENTS=10000 to my application's props.conf in the default directory. Even though I added that, one of events are getting broken up where it shouldn't be.

One .log file is 5425 lines long and it gets broken down into 9 events ranging from 14 lines long to 4161 lines long. It appears that a new event is created every time Splunk encounters a string of numbers (the portion bolded in the example listed below. This does not happen when I take out the bolded portion of the event. Why would Splunk split my events when it sees those numbers? Is that what's happening?

Example of event data:

NEW EVENT
75,%KERN-5: hw.chassis.startup_time update to 1492671208.525734. [local5.notice]
76,BGP Peer Connection Connected ( BGP Peer: 184.21.61.254 )
77,Link Down ( xe-3/2/0 )
78,Link Up ( xe-3/2/0.0 )
79,Link Up ( xe-3/2/0 )
Show all 38 lines

NEW EVENT
113,%KERN-5: hw.chassis.startup_time update to 1492670267.410601. [local5.notice]
114,%DAEMON-4: /usr/sbin/sshd[61495]: exited status 255. [local5.warning]
115,%DAEMON-4-RPD_MPLS_LSP_CHANGE: MPLS LSP SLC_BRDR01_TO_SEA_INAR01_LSP02 change on primary() Route 10.73.251.41(flag 0x29) 10.73.255.234(flag 9 Label 607940) 10.71.251.41(flag 0x29) 10.71.254.5(flag 9 Label 303452) 10.71.251.43(flag 0x29) 10.71.255.241(flag
116,BFD Session Down ( bfdSessEntry.230 )
117,ISIS Adjacency State Changed To Down ( Circuit ID: 566 )
Show all 37 lines

NEW EVENT
150,%KERN-5: hw.chassis.startup_time update to 1492671208.526210. [local5.notice]
151,%DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2600:6800:5:3::fffe (External AS 11344) changed state from EstabSync to Established (event RsyncAck) (instance master). [local5.warning]
152,%DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 184.21.61.254 (External AS 11344) changed state from EstabSync to Established (event RsyncAck) (instance master). [local5.warning]
153,%DAEMON-4-RPD_MPLS_LSP_CHANGE: MPLS LSP SLC_BRDR01_TO_RTV_EXAR01_LSP02 change on primary() Route 10.73.251.41(flag 0x29) 10.73.255.234(flag 9 Label 506516) 10.71.251.41(flag 0x29) 10.71.254.5(flag 9 Label 542523) 10.71.251.44(flag 0x21) 10.71.255.238(flag
154,%DAEMON-4-RPD_MPLS_LSP_CHANGE: MPLS LSP SLC_BRDR02_TO_RTV_INAR02_LSP01 change on primary() Route 10.71.251.41(flag 0x29) 10.71.254.5(flag 9 Label 541803) 10.71.251.44(flag 0x21) 10.71.255.238(flag 1 Label 428324) 10.95.236.2(flag 0x20) 10.95.240.42(Label
Show all 423 lines

Thoughts???

0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

The numbers, such as 1492671208.525734 look like timestamps. Are you specifying the timestamp format in your props.conf? If not, then it is probably assuming that it is a new event due to the existence of the timestamp.

Are you giving a specific way for it to eventbreak? There are several to choose from. If specifying the timestamp isn't the problem, then that is your next thing to look at.

If you already have both of those, post your props.conf information here for the data.

View solution in original post

cpetterborg
SplunkTrust
SplunkTrust

The numbers, such as 1492671208.525734 look like timestamps. Are you specifying the timestamp format in your props.conf? If not, then it is probably assuming that it is a new event due to the existence of the timestamp.

Are you giving a specific way for it to eventbreak? There are several to choose from. If specifying the timestamp isn't the problem, then that is your next thing to look at.

If you already have both of those, post your props.conf information here for the data.

allisonwalther
Path Finder

I don't specify a timestamp, I just want it to use the current time of indexing as the timestamp. I guess Splunk is just assuming that's a timestamp and splitting my events there. I am also not using a delimiting character to specify when to break events, but I'll test that out and see if that fixes my problem (I think it will). Thank you!

And extra info to those reading, here is my current props.conf:
[events]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
disabled=false
MAX_EVENTS=10000

[matrix]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
disabled=false
MAX_EVENTS=10000

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...