All Apps and Add-ons

Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

HangGlider
New Member

I've run through the installation process and quadruple-checked my work, but nothing is showing up in Splunk. We have 3 indexers and 1 search head. One thing that isn't clear is whether port 9997 (referenced in the install doc) is UDP or TCP. Our search head isn't using "Forwarding and Receiving", so I just configured UDP 9997 and TCP 9997 in Settings->Data Inputs->UDP (and TCP respectively). The Bit9 server is writing trace files to my export directory as expected. I'm a Splunk newbie, and I've obviously screwed up something, but I'm at a loss to know where else to look.

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee
  • you need your forwarder to send the data to the indexer (=search peer), not search head
  • are you looking at the forwarder logs?

HangGlider
New Member

Thanks for your answer!! Where can I double-check whether I'm sending to an indexer?

There are only a few log files that have today's date. None of them are giving me obvious hints... The closest thing I see to an error is in the splunkd log file:

02-26-2016 13:27:19.348 -0800 WARN  TailReader - Enqueing a very large file=E:\Bit9_export\EventTrace-20160226.bt9 in the batch reader, with bytes_to_read=524288774, reading of other large files could be delayed
02-26-2016 13:27:22.679 -0800 INFO  WatchedFile - Will begin reading at offset=31809902 for file='E:\Bit9_export\EventTrace-20160226-2.bt9'.
02-26-2016 13:27:27.692 -0800 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
0 Karma

bit9
Path Finder

In the directory where your Universal Forwarder is installed, go into etc\system\local. If you look at outputs.conf you should see some stanzas that reference where it is forwarding by default. If it's not your indexer, you should change it so that it is. Once you've done that, restart the forwarder.

0 Karma

HangGlider
New Member

Thanks! Besides a 'README', there are only 3 files in etc\system\local: deploymentclient.conf, inputs.conf, and server.conf. None appear to have anything about where I'm forwarding too. 😞

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

try this:
- go to splunk/bin directory
- ./splunk cmd btool outputs list --debug

0 Karma

handlin2014
New Member

So, how is this resolved? Do we need to create an outputs.conf file in the etc\system\default file directory?
Have this same issue. Please advise.

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee
0 Karma

HangGlider
New Member

Well, something happens when I execute that command - it flashes another command prompt window - filled with text - then it disappears. I've tried piping it to a file, but that doesn't work either. I appreciate your help, but I'm not a Windows guy - so I'll track down someone around here to help me capture the output of the command you've provided - and then I'll be back, Thanks again for all your help!!

0 Karma

haliakbar_splun
Splunk Employee
Splunk Employee

Hi From what I am reading you are opening this within Windows. Please open a command prompt as administrator. Then run the command and you will get the results.

0 Karma

bit9
Path Finder

Does it say anything in etc\system\default\outputs.conf about where it's forwarding?

0 Karma

HangGlider
New Member

Unfortunately, no.

0 Karma

bit9
Path Finder

@mreynov_splunk is correct, you need to send the data to an indexer. The configuration (Forwarding and Receiving) part should also happen on an indexer. (There is no choice of protocol there - it's automatically TCP, and it's usually already configured.)

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!