topic Re: How to index data which is forwarded to DNS? in Getting Data In

How to index data which is forwarded to DNS?

saifuddin9122 — Thu, 21 Jul 2016 20:55:06 GMT

Hello

I am using DNS lists for load balancing. I am pointing my forwarders to send data to my DNS, but I was wondering how can an indexer listen for data which is being forwarded to DNS?

I searched for document, but I could not find it, so can anyone please let me know how I can solve it?

Is it simply by enabling the listening port on my indexers?
or do I need any connection between my DNS and indexers?

thanks in advance

Re: How to index data which is forwarded to DNS?

somesoni2 — Thu, 21 Jul 2016 21:14:01 GMT

You need to setup Indexer to receive data onto a port (e.g. 9997). Then you need to configure your DNS LB to forward oncoming traffic to that port (9997) on the Indexers, So here is how it'll look like

                To DNS LB                       Forward to 
Forwarders------------------> DNS LB --------------------> Indexers (receiving on 9997)
               on some port                Indexers on 
                say 9997                          port 9997

Re: How to index data which is forwarded to DNS?

ShaneNewman — Thu, 21 Jul 2016 21:28:28 GMT

This is not recommended. This was being done at a previous company I consulted for and resulted in terrible performance, upon talking to a few at Splunk we discovered that the load balancers were breaking up the stream from the UF. It is much better to allow the SUFs to load balance for you - as they were designed.

Re: How to index data which is forwarded to DNS?

saifuddin9122 — Fri, 22 Jul 2016 13:44:59 GMT

SUFs do you mean static list load balancing??

can you please provide me some clear point what your are talking about LBS.

Re: How to index data which is forwarded to DNS?

ShaneNewman — Fri, 22 Jul 2016 16:20:30 GMT

Yes, the static list found in the outputs.conf.

Load balancing

Re: How to index data which is forwarded to DNS?

saifuddin9122 — Fri, 22 Jul 2016 16:22:49 GMT

ok thanks for clarification !!!

working on both as a part of testing

Re: How to index data which is forwarded to DNS?

Richfez — Sun, 24 Jul 2016 01:29:47 GMT

It may be possible that what you are looking for is the Splunk App for Stream. Stream uses the underlying packet capturing mechanisms on the various platforms to capture data off the wire and send it in via the Forwarder.

The topic is a bit lengthy to get into the solution here, I'd recommend reading and following the extensive documentation in the app and its areas themselves.

The Splunk App for Stream.
How to Install the Splunk App for Stream.
One of the many pages on how to Easily Set Up the Splunk App for Stream. "Easily" being relative, you know.

Re: How to index data which is forwarded to DNS?

Richfez — Sun, 24 Jul 2016 01:31:11 GMT

Huh. Somehow I didn't see any of the other comments against the original question. This may not be your solution. I'll leave it for now just in case, but no worries if it isn't.

Re: How to index data which is forwarded to DNS?

ddrillic — Sun, 24 Jul 2016 17:47:07 GMT

Please also keep in mind that Splunk app for Stream is a Splunk-supported free app - supported and free!

Re: How to index data which is forwarded to DNS?

saifuddin9122 — Mon, 25 Jul 2016 20:20:54 GMT

Hello

i have followed the way u specified but i could not see any events in indexer.
on my forwarder i am getting a message as Forwarding to indexer group default-autolb-group blocked for 100 seconds.

here is my outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk-dns-test.XXXX:9997

[tcpout-server://splunk-dns-test.test-XXXX:9997]

and on my indexer i enabled listening

can you please help me

thanks

Re: How to index data which is forwarded to DNS?

Richfez — Mon, 25 Jul 2016 20:46:26 GMT

Can you confirm with tcpdump (or wireshark depending on your OS) that the packets are being seen on your interface on the Splunk server?