Getting Data In
Highlighted

How to forward specific log files to specific indexes?

Explorer

Hello, I'm trying to figure out the following setup:

At the moment we have one rotating log file that should be forwarded to one specific Splunk index / source type. But this will most likely not stay the only log we want to have in Splunk, but the other logs we will be forwarding later, should end up in another index / source type.

From my understanding, I would have to set up a monitor in the inputs.conf for each file or folder I'd like to monitor. What I don't get is, how to set the target index/source type for that monitor in the outputs.conf, or if I'm on the right way at all with my assumption.

So basically, what I'm trying to accomplish is to set up the Universal Forwarder to do the following:
Forward:
ourfirst.log to index 1 / source type 1
our
second.log to index 2 / source type 2
and so on

Is there a finished example anywhere how to get this done? I can't figure out the connection of the inputs.conf and outputs.conf from the documentation.

Thanks a lot

0 Karma
Highlighted

Re: How to forward specific log files to specific indexes?

Super Champion

In your inputs.conf, (eg let's say the files are in /var/log/myapp/)
Both sourcetype and index is set at inputs.conf

[monitor:///var/log/myapp/our_first.log]
sourcetype=sourcetype1
index = myindex1

[monitor:///var/log/myapp/our_second.log]
sourcetype=sourcetype2
index = myindex2

View solution in original post

Highlighted

Re: How to forward specific log files to specific indexes?

Explorer

Thanks,

I configured one monitor like the example now and have an outputs.conf like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = xxx.xxx.xxx.xxx:9997

[tcpout-server://xxx.xxx.xxx.xxx:9997]

With:

list forward-server

I can see the following result:

Active forwards:
        xxx.xxx.xxx.xxx:9997
Configured but inactive forwards:
        None

And with:

list monitor

I can see my monitored file

But there's no new events in Splunk. I have port 9997 active in Splunk under settings -> Forwarding and receiving -> Receive data

Any ideas where to start searching for errors? Is there an error log I could check? Or any possibility to see if the universal forwarder is even trying to forward events?

Thanks again

0 Karma
Highlighted

Re: How to forward specific log files to specific indexes?

Super Champion

you can check lots of places. Do you have iptables blocking port 9997?
- check internal index on your master Splunk to see if handshake is made . Handshake normally happens on management port 8089
- login to the Universal forwarder and check if any errors are present in "splunkd.log"
- Try logging to the UF and do a ssh -v -p 9997 {ip
of_indexer} to see if it contact the 9997 port
- More advanced issues like ulimits etc (but these won't happen unless its a huge prod system with lot of files open in tandem)

0 Karma
Highlighted

Re: How to forward specific log files to specific indexes?

Explorer

OK, thanks, I'll keep that in mind for future problems. But I just logged in this morning and the events were there. I suppose they were initially not fast enough to show up. But now we get the events in near realtime. Which is what we were looking for.

0 Karma