When I see number of events in Forwarder server it shows me total line count 24130
cat /opt/xxt/xx/*gz | zgrep ST-xxxx | grep identity | wc -l
When I do the same search from Splunk , it shows the event count 24018
Index= main sourcetype=xx ST-xxxx identity
I got following result:
24018 events(15/02/2015 00:00:00:000 to 16/02/2015 00:00:00:000)
Check @woodcock's answer first, if everything is okay there then we need to see your source of input props.conf and transforms.conf related to that. As with props.conf and transforms.conf events can be ignored conditionally.
Some events may have expired and been removed but more likely you are not running your search for All Time and some events are mis-timestamped so that they are in the future (which most versions of Splunk include inside All Time but I have heard that some versions of All Time use latest=now). These future events should never happen (but trust me: they do) and will only show up if you do a search for All Time (or use latest=999999999). If this is the problem, you most likely have a TIMEZONE issue.