Getting Data In

Splunk 6.2 - "Add Data Source" with Local Server Data Itself Can't See Past 2 Tiers of Folders From / (root)

teward001
Path Finder

I'm getting an odd error. When using Splunk 6.2.3, and trying to access files in a folder that is read/write/execute to the world, containing files which I want Splunk to ingest (the files contain JSON), the system says "Not Found" when trying to traverse into directories beyond the second tier (so, for example, Splunk can't access anything inside of /opt/splunk, its own home directory, as it says "Not Found", or similar directories which have world-readable settings).

I've NEVER seen this issue before, so I need some advice on what to do.

Details of the setup: Splunk listens locally on port 8000 on a box, and we have an nginx forwarder in front of it to listen on port 80 and proxy_pass the data between the clients and Splunk transparently. Connecting direct to 8000 is not an option on the network where the clients are. Everything runs on an Ubuntu 12.04 VM on a VMware hypervisor.


WEB CLIENTS/BROWSERS TRAVERSAL PATHWAY:
Browser <--> NGINX Listener on 80/tcp so web gateway at Browser won't fail to access the interface <--> Splunk Web Interface on 8000/tcp.

DATA LOCATION:
VM Name: Splunk
Data Location: /opt/splunk-data/{foo OR bar OR baz}/ on the root filesystem of Splunk (VM).
Issue: Splunk says "Not Found" when through the Web Interface defining the files inside the folders to ingest. /opt can be clicked on and seen fine. Click on /opt/splunk-data and it says "Not Found". Splunk user via CLI can cd into the directory and read data as normal as it has rights.

THERE ARE NO TCP LISTENERS OR FORWARDERS PROVIDING DATA TO SPLUNK - THE DATA FILES EXIST ON THE SAME FILESYSTEM AS THE SPLUNK SYSTEM.

0 Karma
1 Solution

teward001
Path Finder

Apparently something in 6.2.x doesn't want to behave with Ubuntu 12.04. To that end, I grabbed the 6.1.x .deb installer from the archived copy we had on the network here, and installed it, and it seems to work fine.

Something is nasty with the 6.2.x copy we have...

View solution in original post

0 Karma

teward001
Path Finder

Apparently something in 6.2.x doesn't want to behave with Ubuntu 12.04. To that end, I grabbed the 6.1.x .deb installer from the archived copy we had on the network here, and installed it, and it seems to work fine.

Something is nasty with the 6.2.x copy we have...

0 Karma

teward001
Path Finder

Also, to the people running the Answers site here, FIX YOUR INPUT PARSER! It's parsing the left and right arrow chars as HTML chars in a code block and it fails.

0 Karma

woodcock
Esteemed Legend

Are all directories on the same filesystem or are you cross-linking? These are the 2 settings that immediately jumped to mind:

recursive = [true|false]
* If false, Splunk will not monitor subdirectories found within a monitored directory.
* Defaults to true.

followSymlink = [true|false]
* Tells Splunk whether or not to follow any symbolic links within a directory it is monitoring.
* If set to false, Splunk will ignore symbolic links found within a monitored directory.
* If set to true, Splunk will follow symbolic links and monitor files at the symbolic link's destination.
* Additionally, any whitelists or blacklists defined for the stanza also apply to files at the symbolic link's destination.
* Defaults to true.

0 Karma

teward001
Path Finder

All directories are on the same file system. If you're specifying settings, tell me where they are, as I've jumped between Splunk configurations, Apache, Python stuff, etc. the past two weeks so I've forgotten where most settings exist xD.

We haven't even gotten so far as to set up the data source ingestion - it quite literally can't see anything beyond two directories. We have a test directory at /splunk-test-data/, with foo bar and baz as directories, with JSON files inside there containing test1.json test2.json test3.json files. When we try and add the datasource in Splunk and navigate to /splunk-test-data/foo it says "Error: Cannot find directory, do we have permissions?". Recursive chown to splunk:splunk (user/group which the debian package installs) and chmod 777 recursively on that directory also doesn't help, so I'm not sure what else to do. We've also confirmed Splunk is running as the splunk system user too.

This is also a fresh Splunk installation without any additional configuration yet, too, so all settings are 'default' for now.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...