Forwarder data not being indexed, but is shown in metrics as processed

[Kindred]

Hey,

I'm trying to debug a really strange issue with a forwarder on one of our VMs. Basically we cloned a working VM (that does get indexed), but the new VM data doesn't get indexed, yet metrics shows it's processing data from it:

08-20-2013 16:02:15.229 +0000 INFO Metrics - group=tcpin_connections, 10.55.57.68:46601:9997, connectionType=cooked, sourcePort=46601, sourceHost=10.55.57.68, sourceIp=10.55.57.68, destPort=9997, _tcp_Bps=14.58, _tcp_KBps=0.01, _tcp_avg_thruput=0.03, kb=0.42, _tcp_Kprocessed=72.13, _tcp_eps=0.03, build=163460, version=5.0.3, os=Linux, arch=x86_64, hostname=xx-xx-mgmt, guid=A9B4ADDB-3751-41ED-9109-64F6060552D5, fwdType=uf, ssl=false, lastIndexer=50.xx.xx.xx:9997, ack=false

I've gone through warnings and errors in S.o.S and nothing shows anything related to connection issues.

Are there any other places to look for information? The config is the same as the working machine (same inputs etc) so I don't believe it's a config issue.

[yannK · Accepted solution]

Maybe not related but : If you clone a forwarder, it's required to give him a unique identity.
Look if the metrics logs of the indexer is you do not have multiple forwarders with the exact same guid=A9B4ADDB-3751-41ED-9109-64F6060552D5

Then go on the forwarder and delete the $SPLUNK_HOME/etc/instance.cfg and restart to recreate a new one.

For detail about cloning an instance
http://docs.splunk.com/Documentation/Splunk/5.0.4/Deploy/Makeadfpartofasystemimage

[Kindred]

Thanks, I found the file and it made a new one on restart, and I can see the data now so looks like it was that.