Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

[sanjayjp99]

Hi,

I am new to Splunk and needs to take care of existing Splunk setup. I am trying to forward large CSV file from Universal Forwarder(UF) to Heavy Forwarder(HF) and then indexer.
Our existing setup is 1 MA , 2 HF, 2 SH and 2 INX. right now I am forwarding live logs from Arcsight server (non splunk forwarder) to HF which divide into 10 different indexes based on data type.

Without disturbing the above setup I want to forward large CSV files (5-10MB) from Linux server using UF.

See below the config settings that I did so far, I am receiving events on HF but not sure how to redirect them to a specific index.

on Universal forwarder

**input.conf**

[default]
host = server1.mydomain.com
[monitor://opt/client/reports/archive/Splunk/]
sourcetype = csv
index = main

**Output.conf**

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = hfserver2.mydomain.com:6514
[tcpout-server://hfserver2.mydomain.com:6514]

On Heavy Forwarder

**input.conf**

[default]
host = hfserver2.mydomain.com
# receive events from the ArcSight Forwarding connector
[tcp:4514]
disabled = 0
sourcetype = cefevents

#For Reports (to receive events from Splunk universal forwarder ) 
[splunktcp:6514]                                                                 
disabled = 0
sourcetype = csv


**Output.conf** (I haven't made any changes here ) 

# use indexer discovery to identify the Indexers dynamically
[indexer_discovery:splunk_master]
pass4SymmKey = ********************************
master_uri = https://ma1server.mydomain.com:8089

# don't index any events on the Heavy Forwarder, just forward events to the Indexers
[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers

[tcpout:SplunkIndexers]
# identify which Indexers to send events to by querying the master node
indexerDiscovery = splunk_master

With this setup, I am receiving events on HF (verified via tcpdump) but I can't see it on the indexer. I think its getting discarded.
I am doing all the changes in config files since I don't know how to do it on Master's web UI.

Please help me.
Thanks in advance

[sanjayjp99 · Accepted solution]

I am searching data on search head and indexer, I have also tried capturing packet in HF and indexer.
No, master is not indexing data.

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false

[sanjayjp99]

I am searching data on Search head and indexer.
No, master is not indexing data.
In above config I am using master's url for indexer discovery and as I said its working for other indexes. (port 4514)

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false

[abhishekkoli]

HI sanjayjp99,

As per given config files data is moving like UF-->HF-->MA.

You need to make conf file in order to send data from UF-->HF-->INDEXER.

Make sure you are sending data to indexers from the cluster.

[abhishekkoli]

HI
Where you are searching for the data ? (MA or SH)
IS MA is used to index data ?
WHat is output cong for MA ?

[sanjayjp99]

As per given config files data is moving like UF-->HF-->MA

I think that's how you send data in cluster environment, sending data straight to indexer will be limited to that specific indexer will not be cluster. correct me if I am wrong.
above setting is working for indexes that are coming to HF via non Splunk forwarder.

Thanks

[abhishekkoli]

Hi sanjayjp99,
Data should point out all the indexers in the cluster not Indexer master.
Indexer Master will decide the which indexer has to receive data .

1. UF-->HF-->MA this is not recommendation.
2. UF-->HF-->IDX1,IDX2- Point the data to all indexers in cluster through output file of heavy Forward.(IDM) will take care of Replication and search factor.

[sanjayjp99]

If I send direct to my indexer it will not be cluster right?

[sanjayjp99]

still waiting on possible solution

[woodcock]

You should not be indexing this way unless you have a reason to. You should be sending directly from your UF to your Indexer tier. Why are you not doing this?