Getting Data In

Why can't Splunk index my entire log file?

chustar
Path Finder

I am trying to index a somewhat long log file (about 38805 bytes according to the tailing processor).
This log file contains 417 lines, but Splunk only indexed 47 lines.

I thought it might be the TRUNCATE default of 10000 bytes, but looking at the logs, I noticed that it successfully indexed all the log files below 18181 bytes in size (except for one log file that is 4124 bytes, but I'm not sure if that's important)

My log's inputs.conf is configured as such:

[monitor://\\path\to\our\internal\network]
whitelist = WhiteListPattern
initCrcLength = 2048
sourcetype = generic_single_line
disabled = false

Anyone have any idea what's going on here?

0 Karma
1 Solution

chustar
Path Finder

The issue here seems to be a combination of several things:
1. The events do not have timestamps.
2. One of the events has a value of 1319553808 which splunk automatically used as _time of 10/25/11 6:44:48.000 AM.
3. Once it made this conversion, all following events also received that same timestamp (i'm guessing because the following values didn't convert as nicely.
4. All our queries have an automatic window of 30 days, and as such these "old" events weren't picked up.

View solution in original post

0 Karma

chustar
Path Finder

The issue here seems to be a combination of several things:
1. The events do not have timestamps.
2. One of the events has a value of 1319553808 which splunk automatically used as _time of 10/25/11 6:44:48.000 AM.
3. Once it made this conversion, all following events also received that same timestamp (i'm guessing because the following values didn't convert as nicely.
4. All our queries have an automatic window of 30 days, and as such these "old" events weren't picked up.

0 Karma

Richfez
SplunkTrust
SplunkTrust

You are ingesting a file using a UNC pathname. While that normally works, I suspect you may be having issues because of that coupled with something in your environment. Could be exact flavor of Windows, perhaps something in your Windows Sharing setup, repeated very tiny network hiccups ... the available ways for that to go wrong are limitless.

To test this, create a temporary index in Splunk and create a temporary folder on your Splunk Indexer. Copy a bunch of those files from their existing location and drop them into that temporary folder. Then create an input just like your existing one (obviously, you'll use a different path!) and see if those get ingested properly.

If there are no problems, I'd recommend installing the Universal Forwarder on the system involved and using that to read the files locally and forward them. I suspect you'll have no problems out of that method.

If there are still problems ingesting the entire file we can continue investigating, but at least we'll have ruled out quite a few things that might have been happening.

FYI, just the past week I ingested a 7 GB log file with no issues. I needed two or three lines out of it but was having problem opening it in any editor and searching, so I decided to ingest all 7 GB and search with Splunk which worked perfectly.

0 Karma

chustar
Path Finder

I should note we also have other very large files on the same network share and those ones haven't had any issues. It just seems to only crop up with these specific files.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Compare Splunk events with the log file directly, find what the last thing is that it reads properly and then examine the next few lines - could be something in there.

0 Karma

chustar
Path Finder

I've actually figured out the "what" if not the "why". I'll add it as an answer.

0 Karma

Jeremiah
Motivator

When you say long log file, do you mean a long event in a file? If truncate is the issue, you should see truncate messages in your _internal logs.

0 Karma

chustar
Path Finder

No, I mean a long file. The actual events lines are rather short.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...