How to configure a universal forwarder to forward 2 syslogs from 1 IP address to 2 different indexes?

[rene847]

Hi,
I have a Linux server and the syslog is functional. Here is UF config (inputs.conf) :

[udp://xx.xx.xx.140:514]
        host = MyHostName
        acceptFrom = xx.xx.xx.140
        index = TI-LNXEVENTS
        disabled = false

Today, new project…. They request to add "same server" and the logs from specific software with index "TI-lav". But I already send logs (events) to Splunk with Index "TI-LNXEVENTS"

I make test with this (in inputs.conf):

[udp://xx.xx.xx.140:514]
        source = lav_syslog
        sourcetype = lav:prod
        host = MyHostName
        connection_host = none
        acceptFrom = xx.xx.xx.140
        disabled = false
        index = TI-lav

But obviously it sends to index TI-LNXEVENTS.

My question is: I would like to have 2 logs and 2 indexes from this server. What are the good practices?

Best Regards

[esix_splunk]

It depends on the purpose and reason behind this. As mentioned, you shouldnt double index data unless there is a specific requirement for segregation of data where RBAC (role based access control) wont be sufficient for purposes. (Double data = double license usage )

While not best practices, the quickest way to achieve what you are wanting to do specifically would be to install an additional UF instance on the machine, and configure and inputs.conf on that UF to monitor and index that to the TI-lav index.

There is a trade off here of course, the additional client to have to manage and maintain on the host.

[esix_splunk]

This is simple for Splunk.

First you need to identify how and where your application logs to on the file system. After you do this, create a new monitor statement in inputs.conf.
In that monitor stanza, you can define the index and the source type for the Application log files.

[rene847]

Thank you for your reply, I appreciate.

However, Linux events are one thing (TI-LNXEVENTS). These are specific events to the operating systems only.

The other index (TI-lav) is a specific application, an this application generate connections and events specific to this application, never recorded by Linux events.

That's my problem, 1 ip and 2 index from the same server.
Do you have any ideas?

[ekost · Accepted solution]

The best practice for ingesting the same log/stream/event twice is: don’t do it. Unless the second destination index is completely isolated (ex. limited Role access) from being searched by most users, any search across both indexes would turn up duplicate events. This causes more havoc later with any search requiring a count on those events. Use search time knowledge to stitch the data together, as long as there are little to no security restrictions on who can search in either index.
Example: use a macro that searches TI-LNXEVENTS for the syslog events (by host, source, or sourcetype, whichever is easier); and teach the other team how to integrate the macro into their searches. I’m sure there are other cool ways to do this as well.

[ekost]

Hello, You'll have to be very clear about your use-case and details to get a good response. I am guessing that:
1. All system and application events coming from an unnamed Linux host are being sent to Splunk via syslog.
2. You want to parse the syslog stream and separate application events from system events by index.
3. If possible tag system and application events with a unique sourcetype.

If that's the case, you would:
1. Get the events to Splunk the same way you already are, but add a sourcetype in the inputs.conf
2. On the main Splunk instance (or indexer) write a props.conf/transforms.conf combination to filter the destination index based upon the sourcetype and a regex match. The setting you want is the Metadata:Index to change the destination index. There are partial examples several places in Answers, such as: here.

[rene847]

Thank you for your reply, I appreciate.

However, Linux events are one thing (TI-LNXEVENTS). These are specific events to the operating systems only.

The other index (TI-lav) is a specific application, an this application generate connections and events specific to this application, never recorded by Linux events.

That's my problem, 1 ip and 2 index from the same server.
Do you have any ideas?

[jeffland]

If you want to distinguish the events from the linux system from those of the application in a search, you could go by sourcetype. If that is what you were trying to accomplish.