Splunk Search

How come Splunk is not picking up the first few lines (3-5 line) of our log files?

aknsun
Path Finder

Hi,

I have an issue where Splunk is not picking up the first few lines (3-5 line) of log files when doing a search. There is no customization done via the props and transforms.

I have also checked and didn't find any messages in $SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder that pointed to any issue of these lines being skipped.

Any suggestions?

Regards,

AKN.

Tags (1)
0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Hi @aknsun

I can't see anything obviously wrong with your log that would cause events to go missing.

The following things could be happening:

  • The automatic datetime detection is not working properly for your timestamps and Splunk thinks the events are either in the future or very far in the past. Try running this search to identify if this is the cause: index ="whatever" source="path of the log file" earliest=0 latest=+10d
  • You might be using a source or sourcetype that is discarding your events. Splunk out-of-the-box does come with some special configurations for some sourcetypes. You should run btool on the server to try and identify if this is the case. Example /opt/splunk/bin/splunk btool props list <sourcetype> --debug

Hope this helps.

0 Karma

aknsun
Path Finder

Hi @chrisyoungerjds

  1. I checked the first option and the result seems to be the same. Some events are missing.
  2. the sourcetype is log4j. So I believe that should be ok.

Regards,
AKN

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Hi aknsun, Are you able to share an example of the log file lines that are not displaying along with the search you are running?

0 Karma

aknsun
Path Finder

Search

Index = "index name" source = "path of the log file"

Search only returns the 3rd line in this case. The first 2 lines are not returned.

Log details (Masked here)
2019-01-23 04:18:04,537 INFO [pool-1-thread-1] Create ******** success.
2019-01-23 11:03:01,994 INFO [pool-1-thread-2] Create ******** success.
2019-01-23 11:37:14,436 INFO [pool-1-thread-3] Create ******** success.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...