Splunk Forwarder - Specific Directory to Send Data to in Splunk Enterprise

[ZellNorman]

Hello all,

Let me preface that I may be misunderstanding what the Splunk Universal Forwarder does, but I'll jump in because I wasn't able to find the answers I was looking for in the docs nor in other answers.

• On Splunk Enterprise I configured this instance's inputs.conf file to receive data on port 9997
• On the Spluk Forwarder I configured it to send data to my Splunk Enterprise instance on port 9997
• BUT On my Splunk Enterprise instance, it is currently only monitoring one directory (/home/[user])
• Additionally I configured my Forwarder to monitor the directory /var/[user]/test and filled it with 2 basic .json files, but those are (yes, obviously) not being reflected in Splunk Enterprise's Frontend (continue below please))

Most of the posts I've seen seem to make the Forwarder look like it sends only streams of files (I doubt that conclusion is right), but either way is there a default directory that Splunk Enterprise saves results from the Forwarder in (I'm trying to avoid doing an rsync or scp)? Secondly is there a way to specify a directory Splunk Forwarder results will be saved in or that Splunk Forwarder will send results to? Third, I wasn't clear on this but is the log that would show activity related to Splunkd transferring files, splunkd.log?

Thanks guys.

ANSWER may involve unforseen firewall rules (Running CentOS 7 sometimes firewalld will takeover your iptables service
Alright, so I have good news and frustrating news. Running CentOS 7, the problem was initially for some reason my iptables rules to open up port 9997 were not setting even though I edited my sysconfig/iptables and restarted the networking service on the system hosting my Splunk Enterprise.

However afterward when I updated and restarted the Splunk Enterprise System using the "reboot" command, the web interface was no longer accessible. Turns out that when you restart all of a sudden "firewalld" takes over the port rules (which was frustrating because lsof -i :9997) said splunkd was listening on that port which can really throw people off. So what I had to do was reopen the ports using firewalld. When I opened ports 8000, 8089, and 9997 using firewalld, waited a minute, and created some new json files on the forwarder directory I was monitoring, The results successfully showed up in Splunk

[niketn]

@ZellNorman good to know that you were able to solve your issue yourself. Please convert your Comment with solution as answer and accept the same to mark this question as answered.

Up vote the other answers that have helped you (unless you have already done so).

[sudosplunk]

Hi ZellNorman,

1. Yes. Splunk saves all logs in a directory based structure called 'index' (in splunk's language). If you did not define any index while configuring your inputs, then splunk will store everything into 'main' index by default.
2. Yes. You can specify index name in your inputs.conf file. Locate appropriate inputs.conf file usually located under $SPLUNK_HOME/etc/system/local or $SPLUNK_HOME/etc/apps/local directories and define as below:

Under inputs.conf:
[monitor:///custom_path_to_log_files]
index = your_custom_index_name
sourcetype =
your_custom_sourcetype_name ##
Sourcetype is used to differentiate
data types in a single index. Highly
recommended by splunk to use assign
the right sourcetype while configuring
your inputs. More info @
http://docs.splunk.com/Documentation/Splunk/7.1.1/Data/Listofpretrainedsourcetypes

And finally, Yes. splunkd.log it is. Also, you can check metrics.log file for information about data thruput and dest-source connectivity and more.

Hope this will get you started.

[ZellNorman]

And nittala_surya as well, your answer was extraordinarily helpful also and I really appreciate the extra help you provided with the config.

Just to be sure, you're saying change inputs.conf on the SplunkForwarder or in my Splunk Enterprise instance? Additionally, just for onlookers, the path to inputs.conf on my system are more akin to $SPLUNK_HOME/etc/apps/search/local/inputs.conf

And, alrighty, I added in an index name and sourcetype, ran "lsof -i :9997" to verify the my Splunk Enterprise instance is listening on port 9997, but interestingly enough I'm not seeing data populated in Splunk Enterprise related to the Forwarder. Hmmmm.... I'm gonna do a little experimentation and get right back to you. One second.

I definitely appreciate the help and pointers you've given me.

[ZellNorman]

Hi nittala_surya

Alright, so I have good news and frustrating news. Running CentOS 7, the problem was initially for some reason my iptables rules to open up port 9997 were not setting even though I edited my sysconfig/iptables and restarted the networking service on the system hosting my Splunk Enterprise.

However afterward when I updated and restarted the Splunk Enterprise System using the "reboot" command, the web interface was no longer accessible. Turns out that when you restart all of a sudden "firewalld" takes over the port rules (which was frustrating because lsof -i :9997) said splunkd was listening on that port which can really throw people off. So what I had to do was reopen the ports using firewalld. When I opened ports 8000, 8089, and 9997 using firewalld, waited a minute, and created some new json files on the forwarder directory I was monitoring, The results successfully showed up in Splunk

This was quite a journey. Thanks so much for all of the help. I'm off to happy Splunking

[FrankVl]

Forwarders don't send data to folders, they send data to an index in Splunk. Depending on the index you configured you should be able to search for the data from your Splunk Enterprise instance by querying:

index=yourindex

or if you didn't configure the input to go to a specific index:

index=main

Use "all time" in the time picker, to make sure you don't miss anything for which timestamping didn't work.

Also: you should be able to see events from the universal forwarder in index=_internal.

But you might want to go through some basic documentation / training to get to grips with the key concepts first 🙂

[ZellNorman]

Thanks a ton FrankVl. This was extremely helpful. Alrighty, I'm gonna have to do configure indexing a bit more appropriately from here in order to get the data in at this point. Wish me luck and really, thanks a ton!