Security

Why is Splunk not indexing all the data from the log file?

zealvora
Engager

Hi

We have a script that stores all the user commands to a log file named commands.log and entire file is monitored by splunk. Initially for few minutes it seems to be working well however there are certain lines within commands.log which do not show up in splunk.

Sample commands.log file from the server :

Feb 28 12:01:41  ec2-user: ec2-user [12024]: date [0]
Feb 28 12:04:52  ec2-user: ec2-user [12024]: cat butter.txt [1]
Feb 28 12:05:02  ec2-user: ec2-user [12024]: cat pease.txt [1]
Feb 28 12:05:44  ec2-user: ec2-user [12024]: touch butter01.txt [0]
Feb 28 12:05:49  ec2-user: ec2-user [12024]: touch pease01.txt [0]
Feb 28 12:06:40  ec2-user: ec2-user [12024]: nslookup pingdom.com [0]
Feb 28 12:06:44   ec2-user: ec2-user [12024]: ping c1 motorola.com [2]

Splunk Events:

Feb 28 12:06:44  ec2-user: ec2-user [12024]: ping c1 motorola.com [2]
Feb 28 12:06:40  ec2-user: ec2-user [12024]: nslookup pingdom.com [0]
Feb 28 12:01:41  ec2-user: ec2-user [12024]: date [0]

So basically what has happened is four command.log file lines (cat butter.txt, cat pease.txt, touch butter01.txt, touch pease01.txt) are not indexed in Splunk. I also exported all the RAW events from Splunk and tried to manually search for those events but they are not part in splunk.

I wonder how can that happen?

0 Karma

valiquet
Contributor

index=_internal "sourcetypename" log_level!=INFO

SPlunk might thinks your files are the same because or the CRC check (the first 256bytes). You can increase that or add the file names as the salt in props.

0 Karma

anjambha
Communicator

Hi zealvora,

Your are not seeing that record could be because of records are merging. Are you using props.conf for this if not then you can try this.

props.conf:
[your sourcetype]
BREAK_ONLY_BEFORE_DATE = true

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...