topic Re: How to seperate different index and sourcetype? in Getting Data In

How to seperate different index and sourcetype?

johnwyane — Mon, 28 Sep 2020 14:53:43 GMT

HI,
I have one Splunk server. I would like to receive data from some servers and network devices.
1 I would send F5 (10.10.10.10 and 10.10.10.11) log and cisco router (172.16.16.10 etc.) and firewall Fortigate (10.10.10.20 & 10.10.10.21)log to my Splunk server by UDP 514.
2 Also, I would use light universal forwarder to send data of windows & Unix servers to my splunk server.
However, I would like to use different sourcetype and index to identify the different devices.
For example, sourcetype=win_AD index=win_AD , sourcetype=sun index=sun , sourcetype=asm index=asm , sourcetype=cisco index=cisco , sourcetype=forti index=forti

After I study some manuals and some of splunkbase answers. I know I have to modified props.conf and transforms.conf .
1 How to configure above two files?
2 How to setup unix and window AD with universal forwarder configured file?
Any one could give me a complete method to set it up?
Thank you very much!

Anthony

Re: How to seperate different index and sourcetype?

Ayn — Fri, 04 Oct 2013 09:12:04 GMT

What information are you missing from the docs/splunkbase answers you found? We can help, but we can't do all the work for you - you need to understand the mechanisms before you can implement this.

Re: How to seperate different index and sourcetype?

johnwyane — Mon, 28 Sep 2020 14:54:17 GMT

I know how to set index using Splunk web interface.
Also, I know how to receive the data from UDF514.

But I don't know about the meaning of "Format" and "$0" and "metadata" "_metadata" even I check the transforms.conf.sample file.
1 So anyone can explain it ?
2 Also, if possible , correct my configuration if anything is wrong!!
3 Should I assign different sourcetype for F5, cisco, fortigate? How? Which file?
Thank you very much!

Here is my configuration
props.conf
[F5_syslog]
TRANSFORMS-F5=asm_log

[cisco-router_syslog]
TRANSFORM-cisco=cisco-router

[fortigate_syslog]
TRANSFORM-forti=fortigate

transforms.conf
[asm_log]
REGEX = 10.10.10.10|10.10.10.11
DEST_KEY = _MetaData:Index
FORMAT = F5_ASM

[cisco-router]
REGEX = 172.16.16.*
DEST_KEY = _MetaData:Index
FORMAT = cisco_log

[Fortigate]
REGEX = 10.10.10.20|10.10.10.21
DEST_KEY = _MetaData:Index
FORMAT = Forti_log

For my 2nd question
at windows server (with forwarder):
I do nothing just using windows-ta-app.

at unix server(with forwarder):
[monitor:///var/log]
sourcetype=unix
index=unix

Re: How to seperate different index and sourcetype?

kristian_kolb — Sat, 05 Oct 2013 10:32:30 GMT

You'll need to ensure that the incoming data on port 514 will be separated into different sourcetypes before this configuration will work (since you are making transforms based on sourcetypes, right).

Also, quite separately, your REGEXes might be a bit too unspecific, since they will match on the existence of a string (in your case ip-addresses) regardless of where they appear in the message. Perhaps the addition of a SOURCE_KEY attribute in your transforms can help you with that (i.e. ensure that the REGEX is matching on the host and not on the whole _raw event.