Why is Splunk not indexing all the data from the log file?

[zealvora]

Hi

We have a script that stores all the user commands to a log file named commands.log and entire file is monitored by splunk. Initially for few minutes it seems to be working well however there are certain lines within commands.log which do not show up in splunk.

Sample commands.log file from the server :

Feb 28 12:01:41  ec2-user: ec2-user [12024]: date [0]
Feb 28 12:04:52  ec2-user: ec2-user [12024]: cat butter.txt [1]
Feb 28 12:05:02  ec2-user: ec2-user [12024]: cat pease.txt [1]
Feb 28 12:05:44  ec2-user: ec2-user [12024]: touch butter01.txt [0]
Feb 28 12:05:49  ec2-user: ec2-user [12024]: touch pease01.txt [0]
Feb 28 12:06:40  ec2-user: ec2-user [12024]: nslookup pingdom.com [0]
Feb 28 12:06:44   ec2-user: ec2-user [12024]: ping c1 motorola.com [2]

Splunk Events:

Feb 28 12:06:44  ec2-user: ec2-user [12024]: ping c1 motorola.com [2]
Feb 28 12:06:40  ec2-user: ec2-user [12024]: nslookup pingdom.com [0]
Feb 28 12:01:41  ec2-user: ec2-user [12024]: date [0]

So basically what has happened is four command.log file lines (cat butter.txt, cat pease.txt, touch butter01.txt, touch pease01.txt) are not indexed in Splunk. I also exported all the RAW events from Splunk and tried to manually search for those events but they are not part in splunk.

I wonder how can that happen?

[valiquet]

index=_internal "sourcetypename" log_level!=INFO

SPlunk might thinks your files are the same because or the CRC check (the first 256bytes). You can increase that or add the file names as the salt in props.

[anjambha]

Hi zealvora,

Your are not seeing that record could be because of records are merging. Are you using props.conf for this if not then you can try this.

props.conf:
[your sourcetype]
BREAK_ONLY_BEFORE_DATE = true