Splunk 6.2 universal forwarder on Mac OSX 10.10.2? no data

[kcarroll]

Hey All, I am new to Splunk and trying to gain some insight. I have an all mac home and I am trying to gain some insight to what's taking place in my network and whats leaving it.

1. Mac Mini OS X 10.10.2 with Splunk 6.2.2 (indexer\search)
2. MacBookPro with 10.10.2 OS X
3. Universal Forwarder 6.2.2 on MacBookPro

I have installed the server successfully and have logged in and changed the password.

I have DL'd the .DMG from splunk and ran the installer, I have launched the UF with the short cut on my desktop. (so far so good)

This is what it all goes pair shaped so to say. I have drilled down via the terminal app to the Applications\SplunkForwarder\etc\apps\SplunkUniversalForwarder
when I am in here I can only see default and meta

I select default and see lots of files, like outputs.conf, limits.conf, inputs.conf and so on. I believe that I am in the right space based on what I have read. I see in some of the docs that this location over writes or over rules the other outputs.conf in other locations. So this is the one I need to setup the server to send the data to from what I can gather.

I edit them and add the lines for the following:
outputs.conf

Version 6.2.2

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection)
forwardedindex.filter.disable = false

[tcpout:my_indexer]
server=NN.NN.NN.NN:9997 <--- this is what I added

inputs.conf
[monitor:///var/log]
sourcetype=syslog
host=mymachinename

I stop the Splunk service and start it again with the desktop icon.

Now I go to the serverwebpage:8000 and I am all excited and yep, nothing at all. Back to reading more loads of doc's that don't seem to related really to MAC OS X (aka unix, i get it) . I am not a UNIX admin nor have I ever been. So its little clumsy to fumble around but i get there sooner or later.

It dawns on me that maybe I need to make sure the server is actually listening on that port. I got to "settings/forwarding and receiving" and select add new under the "receiving data" header. I add the port 9997. I restart splunk on the laptop and I wait about 10 mins......still nothing.

Troubleshooting

• I can ping the server and vice versa
• I can ssh to the server
• firewall is off on the macbook\server
• I can telnet to port 9997 on the server from the laptop
• I can telnet to port 8089 on the laptop from server

Logic is I got good connectivity via ip or dns. So this has to be some config logic I am missing.

Can anyone offer some direction on what load of doc's I must be not finding? Its can't be this freakin hard to make a client to talk to the indexer with a UF? Right? (stumped)

[s2_splunk · Accepted solution]

Best practice: Never edit files in default.
Create a folder called local in the same directory that has default/meta and make all your configurations there.

Your outputs.conf for a single indexer should look like this:

[tcpout-server://nn.nn.nn.nn:9997]

This is documented pretty well here

Then you do the same for your inputs.conf, i.e. create a new file in the local directory and add your settings.
I'd recommend reading this until you understand how Splunk processes .conf files.

You don't need to enable receiving on your forwarder system, but you do need to enable it on the indexer for the port you are using (Settings->Forwarding And Receiving->Receive Data).

[kcarroll]

Next Challenge is going to be finding out why I have 53 hosts all my laptop with different names or variations of names.

I was going to try and show that but something do with Karma point's keeping me from making this useful. o well.

[kcarroll]

Hello and thanks for the answers. .

I need to adjust the stanza on the client it would seem. The doc you linked is the one I was reading and I am using the first style in that doc as to where you're suggesting the third style. Which I will be giving a go tonight and see where it takes me.

As far as editing the defaults, yes I know better, but i got lazy and frustrated. I should have just mkdir local and then vi inputs.conf and outputs.conf. Then I could have dealt with them, doh! something that is easy to fix tonight also.

again thanks for the links and answers. let ya know how it works out tonight.

[kcarroll]

So freaking Awesome..... it's working, it's working (in my best Anikin Skywalker voice)

Thanks much