Splunk File Monitoring Line Breaking not working

[ariel_esh]

Hello, I was trying to ingest snmptrapd logs with self file monitoring (Only one Splunk Instance in my environment)

Here is the log format:

<UNKNOWN> - 2025-01-13 10:55:44
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:17:26:51.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osDiskFreeSpaceNotification
CYBER-ARK-MIB::osDiskDrive "C:\\"
CYBER-ARK-MIB::osDiskPercentageFreeSpace "71.61"
CYBER-ARK-MIB::osDiskFreeSpace "58221"
CYBER-ARK-MIB::osDiskTrapState "Alert"
<UNKNOWN> - 2025-01-13 10:55:44
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:17:26:51.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13524732
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3660968
CYBER-ARK-MIB::osMemoryTrapState "Alert"
<UNKNOWN> - 2025-01-13 10:55:44
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:17:26:51.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osSwapMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13524732
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3660968
CYBER-ARK-MIB::osMemoryTrapState "Alert"

I tried to use "<UNKNOWN>" as the line breaker, but it does not work at all and the event is broke in a weird way(sometimes it works, most of the time it doesn't)

Please find the props.conf setting as below:

[cyberark:snmplogs]

LINE_BREAKER = \<UNKNOWN\>

NO_BINARY_CHECK = true

SHOULD_LINEMERGE = true

category = Custom

pulldown_type = 1

BREAK_ONLY_BEFORE = \<UNKNOWN\>

MUST_NOT_BREAK_BEFORE = \<UNKNOWN\>

disabled = false

LINE_BREAKER_LOOKBEHIND = 2000

Line Breaking Result in Splunk:

[ariel_esh]

@PickleRick  @isoutamo @kiran_panchavat  Thank you for the replies!

I think I should provide more information about the log. It is from snmp traps, and I have a script that will export the trap line by line to the log file that will be monitored by Splunk. 

The props.conf @PickleRick  helped to amend works well if I use 'add data' to add a static log file instead of file monitoring, but If I use file monitoring (new lines of snmp traps will be written around every 10 minutes), the line breaking went wrong.

So I was thinking is the problem due to the file being updated? But the snmp traps were written almost at the same time (as seen in the timestamps), if I would like to fix it, what configurations can I change?

[PickleRick]

Do you mean that separate lines are written with 10 minute intervals or every 10 minutes a whole multiline event is written? Anyway, if it's a UF it might help to add EVENT_BREAKER_ENABLE=true and set EVENT_BREAKER to the same value as LINE_BREAKER.

[ariel_esh · Accepted solution]

I set the following in inputs.conf and seems it is working fine now.

multiline_event_extra_waittime = true
time_before_close = 120

I will monitor it for a while and see if the successful event breaking is stable. Thank you for your help!

[kiran_panchavat]

@ariel_esh 

[PickleRick]

Don't use SHOULD_LINEMERGE=true. It's a very very rarely useful option.

In your case it will be probably just

LINE_BREAKER=([\r\n]+)<UNKNOWN>

You might need to escape < and > and maybe enclose <UNKNOWN> in a non-capturing group.

[ariel_esh]

Thank you for the reply. 

I have changed the props.conf to 

[cyberark:snmplogs]

LINE_BREAKER = ([\r\n]+)\<UNKNOWN\>

NO_BINARY_CHECK = true

category = Custom

pulldown_type = 1

disabled = false

However, the line breaking is still wrong. Sometimes, Splunk even only ingest the first line for that event (16:04:48). Do you have any idea on the reason behind this?

Actual log file:

<UNKNOWN> - 2025-01-13 16:04:48
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:35:56.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osDRServiceNameNotification
CYBER-ARK-MIB::osServiceName "CyberArk Vault Disaster Recovery"
CYBER-ARK-MIB::osServiceStatus "Stopped"
CYBER-ARK-MIB::osServiceTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:17
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osDiskFreeSpaceNotification
CYBER-ARK-MIB::osDiskDrive "C:\\"
CYBER-ARK-MIB::osDiskPercentageFreeSpace "71.56"
CYBER-ARK-MIB::osDiskFreeSpace "58183"
CYBER-ARK-MIB::osDiskTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:17
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osSwapMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13521168
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3651932
CYBER-ARK-MIB::osMemoryTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:18
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osCpuUsageNotification
CYBER-ARK-MIB::osCpuUsage "0.000000"
CYBER-ARK-MIB::osCpuTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:18
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13521168
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3651932
CYBER-ARK-MIB::osMemoryTrapState "Alert"

[isoutamo]

Hi

Based on your sample data and if your props.conf is just what you have shown to us this should be work as @PickleRick told.

Quite probably you have something else for those event in your input file. Can you found those problematic events and one before and after from it? Then add those inside editors </> -block, so we can be sure that there haven't been any editor changes when you are posting those into this thread.

r. Ismo

[PickleRick]

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)<UNKNOWN>

 This should do the trick. Of course you need to set your timestamp recognition as well but that's another story.