Indexer not indexing forwarded data

[gauravnj1]

I have an forwarder that's set up to monitor a log file at the location: /var/log/mhn/mhn-splunk.log.

inputs.conf on forwarder:

[monitor:///var/log/mhn/mhn-splunk.log]
sourcetype = mhn
index = mhn
disabled = false

outputs.conf on forwarder:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = Dest IP:9997

[tcpout-server://Dest IP:9997]

On the forwarder

• I have verified connection using netstat
  tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 5600/splunkd*
  tcp 0 0 Source IP:48652 Dest IP:9997 ESTABLISHED 5600/splunkd

• Checked splunkd.log
  02-22-2018 02:04:04.790 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
  02-22-2018 02:27:07.846 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
  02-22-2018 02:29:03.860 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.

On the indexer:

• I have verified the index, mhn, exists and is enabled.

• Listener is setup on the right port
  tcp 0 0 0.0.0.0:9997 0.0.0.0: LISTEN 31490/splunkd
  tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 31490/splunkd*
  tcp 0 0 0.0.0.0:8000 0.0.0.0: LISTEN 31490/splunkd*
  tcp 0 0 Dest IP:9997 Source IP:48652 ESTABLISHED 31490/splunkd

• metrics.log is showing as receiving the events from the forwarder
  02-21-2018 23:40:19.593 -0800 INFO Metrics - group=tcpin_connections, Source IP:48652:9997, connectionType=cooked, sourcePort=48652, sourceHost=Source IP, sourceIp=Source IP, destPort=9997, kb=7.95, _tcp_Bps=262.59, _tcp_KBps=0.26, _tcp_avg_thruput=0.52, _tcp_Kprocessed=346.17, _tcp_eps=0.19, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.23, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=03bbabbd5c0f, version=7.0.2, os=Linux, arch=x86_64, hostname=ubuntu, guid=BEB9358D-17D6-4C65-B408-99DF4C038DFA, fwdType=uf, ssl=false, lastIndexer=Dest IP:9997, ack=false

Can't quite figure out why I'm not seeing the events in index=mhn. I was hoping the Splunk community might be able to tell me if there was anything I was missing.

[skoelpin]

You should check the logs and see if your forwarder is sending over data. You can also check the forwarder logs

Run this search

index=_internal sourcetype=splunkd

[richgalloway · Accepted solution]

Just in case timestamps are not being parsed correctly, try searching index=mhn over All Time.

[gauravnj1]

@richgalloway, you were right. There's something messed up with the timestamps. I'll write another question on how to untangle that mess. Thank you for pointing me in the right direction.