Hi All,
We have been notified by the security team as they are seeing low security events counts and the logs are not parsing at realtime, based on the alert set to trigger in the splunk.
Query details
| inputlookup LOW-EVENT-COUNT-SECURITY.csv
| join type=outer sourcetype [metadata type=sourcetypes]
| eval recent=strftime(recentTime,"%m/%d/%y %H:%M:%S")
| eval last=strftime(lastTime,"%m/%d/%y %H:%M:%S")
| eval nowTime=now()
| eval now=strftime(nowTime,"%m/%d/%y %H:%M:%S")
| eval diff=(nowTime-recentTime)
| table data sourcetype recentTime recent lastTime last now nowTime diff
data sourcetype recentTime recent lastTime last now nowTime diff
F5 BIG-IP f5:bigip:syslog 1531492099 07/13/18 10:28:19 1531506498 07/13/18 14:28:18 07/13/18 10:28:22 1531492102 3
When checked for error logs in splunkd.log found below error details from the search heads
07-13-2018 11:01:30.655 -0400 ERROR DistributedBundleReplicationManager - HTTP response code 400 (HTTP/1.1 400 Bad Request). Error applying delta=/opt/splunk/var/run/searchpeers/C090FDA2-105E-4875-A110-3F13FF986151-1531493889-1531494056.delta, searchHead=C090FDA2-105E-4875-A110-3F13FF986151, prevTime=1531493889, prevChksum=4389239684162954908, curTime=1531494056, curChksum=16674274590358345014: Error copying /opt/splunk/v
Error logs in splunkd.log from Indexer instances
07-13-2018 10:32:35.344 -0400 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/C090FDA2-105E-4875-A110-3F13FF986151-1531492185.bundle
Error logs in splunkd.log from Heavy forwarder instances
a) 07-13-2018 07:01:03.262 -0400 ERROR DistributedBundleReplicationManager - Reading reply to upload: rv=-2, Receive from=https://splunk03:8089 timed out; exceeded 60sec, as per=distsearch.conf/[replicationSettings]/sendRcvTimeout
b) 07-13-2018 07:01:39.014 -0400 ERROR DistributedBundleReplicationManager - got non-200 response from peer. uri=https://splunk02:8089, reply="HTTP/1.1 204 No Content" response_code=204
c) Unable to upload bundle to peer named splunk02 with uri=https://splunk02:8089.
Questions:
1) We had recently upgraded the search head/indexer instance to 7.0.4 from 6.6.1 and but we have not upgraded the HF instances where all the data are parsing before getting indexed. whether this might be causing the issue ?
2) How/where to start the investigation and trouble this issue to find out the root cause for the low event count.
Kindly guide me to fix this issue.
