topic Re: possible file input defect in Getting Data In

possible file input defect

jgauthier — Fri, 03 Jun 2011 14:40:14 GMT

Greetings,

I've noticed that one of my input files was not reading into splunk entirely. (ver 4.2 96430)
After doing some troubleshooting, it appears that the file is read short, and I can reproduce this based on the contents of the file.

For instance, I have inputting this data:

Host=SERVERNAME
Source=Broadcom BCM5709S NetXtreme II GigE [NDIS VBD Client]
BytesReceivedPerSec="3362"
BytesSentPerSec="0"
BytesTotalPerSec="3362"
Description=""
Frequency_Object=""
Frequency_PerfTime=""
Frequency_Sys100NS=""
OutputQueueLength="0"
CurrentBandwidth="1000000000"
PacketsOutboundDiscarded="0"
PacketsOutboundErrors="0"
PacketsPerSec="24"
PacketsReceivedDiscarded="0"
PacketsReceivedErrors="0"
PacketsReceivedNonUnicastPerSec="13"
PacketsReceivedPerSec="24"
PacketsReceivedUnicastPerSec="10"
PacketsReceivedUnknown="0"
PacketsSentNonUnicastPerSec="0"
PacketsSentPerSec="0"
PacketsSentUnicastPerSec="0"
Timestamp_Object=""
Timestamp_PerfTime=""
Timestamp_Sys100NS=""

Splunk only shows the data up until the line that reads:
CurrentBandwidth="1000000000"

The reading is aborted at the line. If I move the line up, or down, it follows the line.
If I change the data in the line, even removing a single 0, then splunk reads the data correctly in its entirety.

I'm pretty sure this is a defect, but I don't know how to open a bug report. Hopefully this is sufficient.

Other data that might be relevant is that I am putting this into a directory that is a sinkhole, s it's batch inputted.

Let me know what other information may be useful to supply.

Re: possible file input defect

Simeon — Fri, 03 Jun 2011 16:45:04 GMT

What happens when you "view source" for the event?

Re: possible file input defect

jgauthier — Fri, 03 Jun 2011 16:52:18 GMT

It shows the truncated data as well.

Re: possible file input defect

jgauthier — Fri, 03 Jun 2011 17:19:01 GMT

This is definitely a defect. I have been able to reproduce this repeatedly on large numbers starting with a 1. for instance, all of these lines, in different files cause splunk to stop processing:
Timestamp_PerfTime="1309871324971"
CommitLimit="103056531456"
AvgDiskQueueLength="18446743927822183616"

However, if I change the leading digit to a 2, or other number the data loads correctly.

I've put in a bug, but didn't realize this was the problem, so I am going to try and update it.

Re: possible file input defect

gkanapathy — Mon, 28 Sep 2020 09:38:32 GMT

actually it's probably getting split into a new event based on the BREAK_ONLY_BEFORE_DATE default rule. The number is read as an epoch timestamp, and the data is getting indexed but with a time range outside where you're looking. a solution would be a to specify an explicit TIME_FORMAT, or DATETIME_CONFIG = CURRENT, or if you're trying to index the entire file as a single event (vs one line per event), set SHOULD_LINEMERGE = false and LINE_BREAKER = (?!)

Re: possible file input defect

jgauthier — Fri, 03 Jun 2011 17:49:02 GMT

Wow. It was reading it as a time stamp. That's really interesting. Those adjustments allowed the data to process correctly. Thanks for the advice.

Re: possible file input defect

jrodman — Mon, 28 Sep 2020 11:29:12 GMT

I would not recommend LINE_BREAKER = (?!)
Instead, please use something like LINE_BREAKER = ()zqsxtp-will-not-be-present In other words, a paren pair, and text which will not be present in your event, preferably starting with a rare character.