topic Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer? in Getting Data In

Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99 — Fri, 20 Apr 2018 20:50:46 GMT

Hi,

I am new to Splunk and needs to take care of existing Splunk setup. I am trying to forward large CSV file from Universal Forwarder(UF) to Heavy Forwarder(HF) and then indexer.
Our existing setup is 1 MA , 2 HF, 2 SH and 2 INX. right now I am forwarding live logs from Arcsight server (non splunk forwarder) to HF which divide into 10 different indexes based on data type.

Without disturbing the above setup I want to forward large CSV files (5-10MB) from Linux server using UF.

See below the config settings that I did so far, I am receiving events on HF but not sure how to redirect them to a specific index.

on Universal forwarder

**input.conf**

[default]
host = server1.mydomain.com
[monitor://opt/client/reports/archive/Splunk/]
sourcetype = csv
index = main

**Output.conf**

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = hfserver2.mydomain.com:6514
[tcpout-server://hfserver2.mydomain.com:6514]

On Heavy Forwarder

**input.conf**

[default]
host = hfserver2.mydomain.com
# receive events from the ArcSight Forwarding connector
[tcp:4514]
disabled = 0
sourcetype = cefevents

#For Reports (to receive events from Splunk universal forwarder ) 
[splunktcp:6514]                                                                 
disabled = 0
sourcetype = csv


**Output.conf** (I haven't made any changes here ) 

# use indexer discovery to identify the Indexers dynamically
[indexer_discovery:splunk_master]
pass4SymmKey = ********************************
master_uri = https://ma1server.mydomain.com:8089

# don't index any events on the Heavy Forwarder, just forward events to the Indexers
[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers

[tcpout:SplunkIndexers]
# identify which Indexers to send events to by querying the master node
indexerDiscovery = splunk_master

With this setup, I am receiving events on HF (verified via tcpdump) but I can't see it on the indexer. I think its getting discarded.
I am doing all the changes in config files since I don't know how to do it on Master's web UI.

Please help me.
Thanks in advance

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

woodcock — Fri, 20 Apr 2018 23:16:42 GMT

You should not be indexing this way unless you have a reason to. You should be sending directly from your UF to your Indexer tier. Why are you not doing this?

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99 — Mon, 23 Apr 2018 13:25:07 GMT

If I send direct to my indexer it will not be cluster right?

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99 — Wed, 13 Jun 2018 15:57:53 GMT

still waiting on possible solution

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

abhishekkoli — Wed, 18 Jul 2018 16:05:29 GMT

HI sanjayjp99,

As per given config files data is moving like UF-->HF-->MA.

You need to make conf file in order to send data from UF-->HF-->INDEXER.

Make sure you are sending data to indexers from the cluster.

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99 — Mon, 30 Jul 2018 20:15:57 GMT

As per given config files data is moving like UF-->HF-->MA

I think that's how you send data in cluster environment, sending data straight to indexer will be limited to that specific indexer will not be cluster. correct me if I am wrong.
above setting is working for indexes that are coming to HF via non Splunk forwarder.

Thanks

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

abhishekkoli — Tue, 31 Jul 2018 18:30:13 GMT

Hi sanjayjp99,
Data should point out all the indexers in the cluster not Indexer master.
Indexer Master will decide the which indexer has to receive data .

UF-->HF-->MA this is not recommendation.
UF-->HF-->IDX1,IDX2- Point the data to all indexers in cluster through output file of heavy Forward.(IDM) will take care of Replication and search factor.

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

abhishekkoli — Mon, 06 Aug 2018 14:16:45 GMT

HI
Where you are searching for the data ? (MA or SH)
IS MA is used to index data ?
WHat is output cong for MA ?

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99 — Fri, 07 Sep 2018 14:33:53 GMT

I am searching data on Search head and indexer.
No, master is not indexing data.
In above config I am using master's url for indexer discovery and as I said its working for other indexes. (port 4514)

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false

Re: Why am I unable to see the events on the indexer when forwarding a large CSV file from UF to HF to indexer?

sanjayjp99 — Fri, 07 Sep 2018 14:37:54 GMT

I am searching data on search head and indexer, I have also tried capturing packet in HF and indexer.
No, master is not indexing data.

master_uri = https://ma1server.mydomain.com:8089

[indexAndForward]
index = false

[tcpout]
defaultGroup = SplunkIndexers
forwardedindex.filter.disable = true

[tcpout:SplunkIndexers]
# discover the Indexers from the master node
indexerDiscovery = splunk_master

# heartbeats between forwarder and indexer, default is 30 seconds
heartbeatFrequency = 30

# increase the maximum output queue size
# default setting is auto
# default size is 500KB when useAck is disabled and 21MB when enabled
maxQueueSize = auto

# ensure reliable delivery by confirming with the Indexers that they have received each event
useACK = true

# disable SSL compression
useClientSSLCompression = false