All Apps and Add-ons
Highlighted

Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

New Member

I've run through the installation process and quadruple-checked my work, but nothing is showing up in Splunk. We have 3 indexers and 1 search head. One thing that isn't clear is whether port 9997 (referenced in the install doc) is UDP or TCP. Our search head isn't using "Forwarding and Receiving", so I just configured UDP 9997 and TCP 9997 in Settings->Data Inputs->UDP (and TCP respectively). The Bit9 server is writing trace files to my export directory as expected. I'm a Splunk newbie, and I've obviously screwed up something, but I'm at a loss to know where else to look.

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

Splunk Employee
Splunk Employee
  • you need your forwarder to send the data to the indexer (=search peer), not search head
  • are you looking at the forwarder logs?
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

Path Finder

@mreynov_splunk is correct, you need to send the data to an indexer. The configuration (Forwarding and Receiving) part should also happen on an indexer. (There is no choice of protocol there - it's automatically TCP, and it's usually already configured.)

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

New Member

Thanks for your answer!! Where can I double-check whether I'm sending to an indexer?

There are only a few log files that have today's date. None of them are giving me obvious hints... The closest thing I see to an error is in the splunkd log file:

02-26-2016 13:27:19.348 -0800 WARN  TailReader - Enqueing a very large file=E:\Bit9_export\EventTrace-20160226.bt9 in the batch reader, with bytes_to_read=524288774, reading of other large files could be delayed
02-26-2016 13:27:22.679 -0800 INFO  WatchedFile - Will begin reading at offset=31809902 for file='E:\Bit9_export\EventTrace-20160226-2.bt9'.
02-26-2016 13:27:27.692 -0800 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

Path Finder

In the directory where your Universal Forwarder is installed, go into etc\system\local. If you look at outputs.conf you should see some stanzas that reference where it is forwarding by default. If it's not your indexer, you should change it so that it is. Once you've done that, restart the forwarder.

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

New Member

Thanks! Besides a 'README', there are only 3 files in etc\system\local: deploymentclient.conf, inputs.conf, and server.conf. None appear to have anything about where I'm forwarding too. 😞

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

Path Finder

Does it say anything in etc\system\default\outputs.conf about where it's forwarding?

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

New Member

Unfortunately, no.

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

Splunk Employee
Splunk Employee

try this:
- go to splunk/bin directory
- ./splunk cmd btool outputs list --debug

0 Karma
Highlighted

Re: Bit9 Security Platform: How to troubleshoot why no data is getting indexed?

New Member

Well, something happens when I execute that command - it flashes another command prompt window - filled with text - then it disappears. I've tried piping it to a file, but that doesn't work either. I appreciate your help, but I'm not a Windows guy - so I'll track down someone around here to help me capture the output of the command you've provided - and then I'll be back, Thanks again for all your help!!

0 Karma