Splunk Search

I'm missing some events from the search index

erwinpastor
Explorer

I have set up log index in Splunk to monitor Airwatch logs on several servers. However it seems that not all entries for some of the servers are coming into Splunk. For example, I have search string source="D:\AirWatch\Logs\MobileAccessGateway\MAGService.log" "Status [4]" which should give me a total of 96 events but Splunk is only showing 92 for server XXX.

My inputs.conf is as follows:

[monitor://D:\Airwatch\Logs\MobileAccessGateway\MAGService.log]
index = index_name
sourcetype = sourcetype_name

I have not applied any custom logging in props.conf and transforms.conf.

Anyone who can please help as to what probably went wrong and how this kind of scenario can be mitigated. Also, is there a way that I can recover the missing events? Thanks!

0 Karma

kml_uvce
Builder

check if your events are broken properly when indexed in splunk , possibility that some events from files are merged and became single event

0 Karma

erwinpastor
Explorer

@kml_uvce, it's been awhile. Haven't got the chance to get back to you asap.

The source is just coming from one file. The log looks like the following:

10/02/2015 19:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 19:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 19:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 19:45:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:45:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:45:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:45:42 - UpdateServiceStatus: Status [4]
.
.
.

Comparing the results from Splunk and from the actual logs, Splunk is missing some events. Any parameter I need to add to the inputs.conf?

[monitor://D:\Airwatch\Logs\MobileAccessGateway\MAGService.log]
index = index_name
sourcetype = sourcetype_name
0 Karma

satishsdange
Builder

Hi -

I indexed sample data provided by you but I did not see any loss. If its test environment, could you please clear the index & re-index your logs.

0 Karma

erwinpastor
Explorer

Thanks @satishsdange It's a PROD environment though so I can't do it. The sample I provided is just a snapshot, there are hundreds of lines (of the same format) in the actual logs. Any other advise you can recommend?

0 Karma

MuS
SplunkTrust
SplunkTrust

check your splunkd.log for any messages from TailingProcessor related to that log file in question

0 Karma

satishsdange
Builder

Could you please share indexes.conf (mask imp information)

0 Karma

erwinpastor
Explorer

@satishsdange, please see indexes.conf setup below:

[index_name]
thawedPath = /opt2/splunk/data/ols_cold/index_name/thaweddb
maxDataSize = 100
maxHotSpanSecs = 172800
homePath.maxDataSizeMB = 1000
coldPath = /opt2/splunk/data/ols_cold/index_name/db
maxTotalDataSizeMB = 5000
coldPath.maxDataSizeMB = 4000
maxWarmDBCount = 30
maxHotBuckets = 3
homePath = /opt2/splunk/data/ols_hot/index_name/db
frozenTimePeriodInSecs = 32000000
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...