I'm missing some events from the search index

[erwinpastor]

I have set up log index in Splunk to monitor Airwatch logs on several servers. However it seems that not all entries for some of the servers are coming into Splunk. For example, I have search string source="D:\AirWatch\Logs\MobileAccessGateway\MAGService.log" "Status [4]" which should give me a total of 96 events but Splunk is only showing 92 for server XXX.

My inputs.conf is as follows:

[monitor://D:\Airwatch\Logs\MobileAccessGateway\MAGService.log]
index = index_name
sourcetype = sourcetype_name

I have not applied any custom logging in props.conf and transforms.conf.

Anyone who can please help as to what probably went wrong and how this kind of scenario can be mitigated. Also, is there a way that I can recover the missing events? Thanks!

[kml_uvce]

check if your events are broken properly when indexed in splunk , possibility that some events from files are merged and became single event

[erwinpastor]

@kml_uvce, it's been awhile. Haven't got the chance to get back to you asap.

The source is just coming from one file. The log looks like the following:

10/02/2015 19:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 19:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 19:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 19:45:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 20:45:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 21:45:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:00:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:15:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:30:42 - UpdateServiceStatus: Status [4]
10/02/2015 22:45:42 - UpdateServiceStatus: Status [4]
.
.
.

Comparing the results from Splunk and from the actual logs, Splunk is missing some events. Any parameter I need to add to the inputs.conf?

[monitor://D:\Airwatch\Logs\MobileAccessGateway\MAGService.log]
index = index_name
sourcetype = sourcetype_name

[satishsdange]

Hi -

I indexed sample data provided by you but I did not see any loss. If its test environment, could you please clear the index & re-index your logs.

[erwinpastor]

Thanks @satishsdange It's a PROD environment though so I can't do it. The sample I provided is just a snapshot, there are hundreds of lines (of the same format) in the actual logs. Any other advise you can recommend?

[MuS]

check your splunkd.log for any messages from TailingProcessor related to that log file in question

[satishsdange]

Could you please share indexes.conf (mask imp information)

[erwinpastor]

@satishsdange, please see indexes.conf setup below:

[index_name]
thawedPath = /opt2/splunk/data/ols_cold/index_name/thaweddb
maxDataSize = 100
maxHotSpanSecs = 172800
homePath.maxDataSizeMB = 1000
coldPath = /opt2/splunk/data/ols_cold/index_name/db
maxTotalDataSizeMB = 5000
coldPath.maxDataSizeMB = 4000
maxWarmDBCount = 30
maxHotBuckets = 3
homePath = /opt2/splunk/data/ols_hot/index_name/db
frozenTimePeriodInSecs = 32000000