Indexer not indexing forwarded data

gauravnj1 — Tue, 29 Sep 2020 18:09:17 GMT

I have an forwarder that's set up to monitor a log file at the location: /var/log/mhn/mhn-splunk.log.

inputs.conf on forwarder:

[monitor:///var/log/mhn/mhn-splunk.log]
sourcetype = mhn
index = mhn
disabled = false

outputs.conf on forwarder:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = Dest IP:9997

[tcpout-server://Dest IP:9997]

On the forwarder

I have verified connection using netstat
tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 5600/splunkd*
tcp 0 0 Source IP:48652 Dest IP:9997 ESTABLISHED 5600/splunkd
Checked splunkd.log
02-22-2018 02:04:04.790 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
02-22-2018 02:27:07.846 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.
02-22-2018 02:29:03.860 -0500 INFO TcpOutputProc - Connected to idx=Dest IP:9997, pset=0, reuse=0.

On the indexer:

I have verified the index, mhn, exists and is enabled.
Listener is setup on the right port
tcp 0 0 0.0.0.0:9997 0.0.0.0: LISTEN 31490/splunkd
tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 31490/splunkd*
tcp 0 0 0.0.0.0:8000 0.0.0.0: LISTEN 31490/splunkd*
tcp 0 0 Dest IP:9997 Source IP:48652 ESTABLISHED 31490/splunkd
metrics.log is showing as receiving the events from the forwarder
02-21-2018 23:40:19.593 -0800 INFO Metrics - group=tcpin_connections, Source IP:48652:9997, connectionType=cooked, sourcePort=48652, sourceHost=Source IP, sourceIp=Source IP, destPort=9997, kb=7.95, _tcp_Bps=262.59, _tcp_KBps=0.26, _tcp_avg_thruput=0.52, _tcp_Kprocessed=346.17, _tcp_eps=0.19, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.23, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=03bbabbd5c0f, version=7.0.2, os=Linux, arch=x86_64, hostname=ubuntu, guid=BEB9358D-17D6-4C65-B408-99DF4C038DFA, fwdType=uf, ssl=false, lastIndexer=Dest IP:9997, ack=false

Can't quite figure out why I'm not seeing the events in index=mhn. I was hoping the Splunk community might be able to tell me if there was anything I was missing.

Re: Indexer not indexing forwarded data

richgalloway — Thu, 22 Feb 2018 14:12:56 GMT

Just in case timestamps are not being parsed correctly, try searching index=mhn over All Time.

Re: Indexer not indexing forwarded data

skoelpin — Thu, 22 Feb 2018 14:36:56 GMT

You should check the logs and see if your forwarder is sending over data. You can also check the forwarder logs

Run this search

index=_internal sourcetype=splunkd

Re: Indexer not indexing forwarded data

gauravnj1 — Sat, 24 Feb 2018 21:33:37 GMT

@richgalloway, you were right. There's something messed up with the timestamps. I'll write another question on how to untangle that mess. Thank you for pointing me in the right direction.

topic Indexer not indexing forwarded data in Getting Data In

Indexer not indexing forwarded data

Re: Indexer not indexing forwarded data

Re: Indexer not indexing forwarded data

Re: Indexer not indexing forwarded data