topic Re: Splunk File Monitoring Line Breaking not working in Getting Data In

Splunk File Monitoring Line Breaking not working

ariel_esh — Mon, 13 Jan 2025 03:05:54 GMT

Hello, I was trying to ingest snmptrapd logs with self file monitoring (Only one Splunk Instance in my environment)

Here is the log format:

<UNKNOWN> - 2025-01-13 10:55:44
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:17:26:51.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osDiskFreeSpaceNotification
CYBER-ARK-MIB::osDiskDrive "C:\\"
CYBER-ARK-MIB::osDiskPercentageFreeSpace "71.61"
CYBER-ARK-MIB::osDiskFreeSpace "58221"
CYBER-ARK-MIB::osDiskTrapState "Alert"
<UNKNOWN> - 2025-01-13 10:55:44
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:17:26:51.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13524732
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3660968
CYBER-ARK-MIB::osMemoryTrapState "Alert"
<UNKNOWN> - 2025-01-13 10:55:44
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:17:26:51.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osSwapMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13524732
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3660968
CYBER-ARK-MIB::osMemoryTrapState "Alert"

I tried to use "<UNKNOWN>" as the line breaker, but it does not work at all and the event is broke in a weird way(sometimes it works, most of the time it doesn't)

Please find the props.conf setting as below:

[cyberark:snmplogs]

LINE_BREAKER = \<UNKNOWN\>

NO_BINARY_CHECK = true

SHOULD_LINEMERGE = true

category = Custom

pulldown_type = 1

BREAK_ONLY_BEFORE = \<UNKNOWN\>

MUST_NOT_BREAK_BEFORE = \<UNKNOWN\>

disabled = false

LINE_BREAKER_LOOKBEHIND = 2000

Line Breaking Result in Splunk:

Re: Splunk File Monitoring Line Breaking not working

PickleRick — Mon, 13 Jan 2025 07:39:06 GMT

Don't use SHOULD_LINEMERGE=true. It's a very very rarely useful option.

In your case it will be probably just

LINE_BREAKER=([\r\n]+)<UNKNOWN>

You might need to escape < and > and maybe enclose <UNKNOWN> in a non-capturing group.

Re: Splunk File Monitoring Line Breaking not working

ariel_esh — Mon, 13 Jan 2025 08:13:03 GMT

Thank you for the reply.

I have changed the props.conf to

[cyberark:snmplogs]

LINE_BREAKER = ([\r\n]+)\<UNKNOWN\>

NO_BINARY_CHECK = true

category = Custom

pulldown_type = 1

disabled = false

However, the line breaking is still wrong. Sometimes, Splunk even only ingest the first line for that event (16:04:48). Do you have any idea on the reason behind this?

Actual log file:

<UNKNOWN> - 2025-01-13 16:04:48
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:35:56.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osDRServiceNameNotification
CYBER-ARK-MIB::osServiceName "CyberArk Vault Disaster Recovery"
CYBER-ARK-MIB::osServiceStatus "Stopped"
CYBER-ARK-MIB::osServiceTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:17
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osDiskFreeSpaceNotification
CYBER-ARK-MIB::osDiskDrive "C:\\"
CYBER-ARK-MIB::osDiskPercentageFreeSpace "71.56"
CYBER-ARK-MIB::osDiskFreeSpace "58183"
CYBER-ARK-MIB::osDiskTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:17
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osSwapMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13521168
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3651932
CYBER-ARK-MIB::osMemoryTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:18
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osCpuUsageNotification
CYBER-ARK-MIB::osCpuUsage "0.000000"
CYBER-ARK-MIB::osCpuTrapState "Alert"
<UNKNOWN> - 2025-01-13 16:06:18
UDP: [10.0.216.39]:53916->[10.0.214.14]:162
SNMPv2-SMI::mib-2.1.3.0 30:22:37:25.00
SNMPv2-SMI::snmpModules.1.1.4.1.0 CYBER-ARK-MIB::osMemoryUsageNotification
CYBER-ARK-MIB::osMemoryTotalKbPhysical 16776172
CYBER-ARK-MIB::osMemoryAvailKbPhysical 13521168
CYBER-ARK-MIB::osMemoryTotalKbSwap 19266540
CYBER-ARK-MIB::osMemoryAvailKbSwap 3651932
CYBER-ARK-MIB::osMemoryTrapState "Alert"

Re: Splunk File Monitoring Line Breaking not working

PickleRick — Mon, 13 Jan 2025 08:39:15 GMT

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)<UNKNOWN>

This should do the trick. Of course you need to set your timestamp recognition as well but that's another story.

Re: Splunk File Monitoring Line Breaking not working

isoutamo — Mon, 13 Jan 2025 08:40:59 GMT

Based on your sample data and if your props.conf is just what you have shown to us this should be work as @PickleRick told.

Quite probably you have something else for those event in your input file. Can you found those problematic events and one before and after from it? Then add those inside editors </> -block, so we can be sure that there haven't been any editor changes when you are posting those into this thread.

r. Ismo

Re: Splunk File Monitoring Line Breaking not working

kiran_panchavat — Mon, 13 Jan 2025 17:59:45 GMT

@ariel_esh

Re: Splunk File Monitoring Line Breaking not working

ariel_esh — Tue, 14 Jan 2025 01:05:16 GMT

@PickleRick @isoutamo @kiran_panchavat Thank you for the replies!

I think I should provide more information about the log. It is from snmp traps, and I have a script that will export the trap line by line to the log file that will be monitored by Splunk.

The props.conf @PickleRick helped to amend works well if I use 'add data' to add a static log file instead of file monitoring, but If I use file monitoring (new lines of snmp traps will be written around every 10 minutes), the line breaking went wrong.

So I was thinking is the problem due to the file being updated? But the snmp traps were written almost at the same time (as seen in the timestamps), if I would like to fix it, what configurations can I change?

Re: Splunk File Monitoring Line Breaking not working

PickleRick — Tue, 14 Jan 2025 08:30:19 GMT

Do you mean that separate lines are written with 10 minute intervals or every 10 minutes a whole multiline event is written? Anyway, if it's a UF it might help to add EVENT_BREAKER_ENABLE=true and set EVENT_BREAKER to the same value as LINE_BREAKER.

Re: Splunk File Monitoring Line Breaking not working

ariel_esh — Tue, 14 Jan 2025 15:08:56 GMT

I set the following in inputs.conf and seems it is working fine now.

multiline_event_extra_waittime = true time_before_close = 120

I will monitor it for a while and see if the successful event breaking is stable. Thank you for your help!