Security

Why is Splunk not indexing all the data from the log file?

zealvora
Engager

Hi

We have a script that stores all the user commands to a log file named commands.log and entire file is monitored by splunk. Initially for few minutes it seems to be working well however there are certain lines within commands.log which do not show up in splunk.

Sample commands.log file from the server :

Feb 28 12:01:41  ec2-user: ec2-user [12024]: date [0]
Feb 28 12:04:52  ec2-user: ec2-user [12024]: cat butter.txt [1]
Feb 28 12:05:02  ec2-user: ec2-user [12024]: cat pease.txt [1]
Feb 28 12:05:44  ec2-user: ec2-user [12024]: touch butter01.txt [0]
Feb 28 12:05:49  ec2-user: ec2-user [12024]: touch pease01.txt [0]
Feb 28 12:06:40  ec2-user: ec2-user [12024]: nslookup pingdom.com [0]
Feb 28 12:06:44   ec2-user: ec2-user [12024]: ping c1 motorola.com [2]

Splunk Events:

Feb 28 12:06:44  ec2-user: ec2-user [12024]: ping c1 motorola.com [2]
Feb 28 12:06:40  ec2-user: ec2-user [12024]: nslookup pingdom.com [0]
Feb 28 12:01:41  ec2-user: ec2-user [12024]: date [0]

So basically what has happened is four command.log file lines (cat butter.txt, cat pease.txt, touch butter01.txt, touch pease01.txt) are not indexed in Splunk. I also exported all the RAW events from Splunk and tried to manually search for those events but they are not part in splunk.

I wonder how can that happen?

0 Karma

valiquet
Contributor

index=_internal "sourcetypename" log_level!=INFO

SPlunk might thinks your files are the same because or the CRC check (the first 256bytes). You can increase that or add the file names as the salt in props.

0 Karma

anjambha
Communicator

Hi zealvora,

Your are not seeing that record could be because of records are merging. Are you using props.conf for this if not then you can try this.

props.conf:
[your sourcetype]
BREAK_ONLY_BEFORE_DATE = true

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...