Splunk Search

How come Splunk is not picking up the first few lines (3-5 line) of our log files?

aknsun
Path Finder

Hi,

I have an issue where Splunk is not picking up the first few lines (3-5 line) of log files when doing a search. There is no customization done via the props and transforms.

I have also checked and didn't find any messages in $SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder that pointed to any issue of these lines being skipped.

Any suggestions?

Regards,

AKN.

Tags (1)
0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Hi @aknsun

I can't see anything obviously wrong with your log that would cause events to go missing.

The following things could be happening:

  • The automatic datetime detection is not working properly for your timestamps and Splunk thinks the events are either in the future or very far in the past. Try running this search to identify if this is the cause: index ="whatever" source="path of the log file" earliest=0 latest=+10d
  • You might be using a source or sourcetype that is discarding your events. Splunk out-of-the-box does come with some special configurations for some sourcetypes. You should run btool on the server to try and identify if this is the case. Example /opt/splunk/bin/splunk btool props list <sourcetype> --debug

Hope this helps.

0 Karma

aknsun
Path Finder

Hi @chrisyoungerjds

  1. I checked the first option and the result seems to be the same. Some events are missing.
  2. the sourcetype is log4j. So I believe that should be ok.

Regards,
AKN

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Hi aknsun, Are you able to share an example of the log file lines that are not displaying along with the search you are running?

0 Karma

aknsun
Path Finder

Search

Index = "index name" source = "path of the log file"

Search only returns the 3rd line in this case. The first 2 lines are not returned.

Log details (Masked here)
2019-01-23 04:18:04,537 INFO [pool-1-thread-1] Create ******** success.
2019-01-23 11:03:01,994 INFO [pool-1-thread-2] Create ******** success.
2019-01-23 11:37:14,436 INFO [pool-1-thread-3] Create ******** success.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...