How to trouble shoot Security Logs with Low Event Count logs are not parsing at realtime?

[Hemnaath]

Hi All,

We have been notified by the security team as they are seeing low security events counts and the logs are not parsing at realtime, based on the alert set to trigger in the splunk.

Query details

| inputlookup LOW-EVENT-COUNT-SECURITY.csv
| join type=outer sourcetype [metadata type=sourcetypes]
| eval recent=strftime(recentTime,"%m/%d/%y %H:%M:%S")
| eval last=strftime(lastTime,"%m/%d/%y %H:%M:%S")
| eval nowTime=now()
| eval now=strftime(nowTime,"%m/%d/%y %H:%M:%S")
| eval diff=(nowTime-recentTime)
| table data sourcetype recentTime recent lastTime last now nowTime diff 

data          sourcetype    recentTime  recent                lastTime              last                     now          nowTime         diff
F5 BIG-IP   f5:bigip:syslog 1531492099  07/13/18 10:28:19   1531506498  07/13/18 14:28:18   07/13/18 10:28:22   1531492102  3

When checked for error logs in splunkd.log found below error details from the search heads

07-13-2018 11:01:30.655 -0400 ERROR DistributedBundleReplicationManager - HTTP response code 400 (HTTP/1.1 400 Bad Request). Error applying delta=/opt/splunk/var/run/searchpeers/C090FDA2-105E-4875-A110-3F13FF986151-1531493889-1531494056.delta, searchHead=C090FDA2-105E-4875-A110-3F13FF986151, prevTime=1531493889, prevChksum=4389239684162954908, curTime=1531494056, curChksum=16674274590358345014: Error copying /opt/splunk/v

Error logs in splunkd.log from Indexer instances

07-13-2018 10:32:35.344 -0400 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/C090FDA2-105E-4875-A110-3F13FF986151-1531492185.bundle

Error logs in splunkd.log from Heavy forwarder instances

a) 07-13-2018 07:01:03.262 -0400 ERROR DistributedBundleReplicationManager - Reading reply to upload: rv=-2, Receive from=https://splunk03:8089 timed out; exceeded 60sec, as per=distsearch.conf/[replicationSettings]/sendRcvTimeout


b) 07-13-2018 07:01:39.014 -0400 ERROR DistributedBundleReplicationManager - got non-200 response from peer. uri=https://splunk02:8089, reply="HTTP/1.1 204 No Content" response_code=204

c) Unable to upload bundle to peer named splunk02 with uri=https://splunk02:8089.

Questions:

1) We had recently upgraded the search head/indexer instance to 7.0.4 from 6.6.1 and but we have not upgraded the HF instances where all the data are parsing before getting indexed. whether this might be causing the issue ?

2) How/where to start the investigation and trouble this issue to find out the root cause for the low event count.

Kindly guide me to fix this issue.

[CarsonZa]

Sounds like you have a lookup(s) that may be way to big to replicate see if this applies.

https://answers.splunk.com/answers/139192/error-distributedbundlereplicationmanager-got-non-200-resp...