Getting Data In

How to seperate different index and sourcetype?

johnwyane
New Member

HI,
I have one Splunk server. I would like to receive data from some servers and network devices.
1 I would send F5 (10.10.10.10 and 10.10.10.11) log and cisco router (172.16.16.10 etc.) and firewall Fortigate (10.10.10.20 & 10.10.10.21)log to my Splunk server by UDP 514.
2 Also, I would use light universal forwarder to send data of windows & Unix servers to my splunk server.
However, I would like to use different sourcetype and index to identify the different devices.
For example, sourcetype=win_AD index=win_AD , sourcetype=sun index=sun , sourcetype=asm index=asm , sourcetype=cisco index=cisco , sourcetype=forti index=forti

After I study some manuals and some of splunkbase answers. I know I have to modified props.conf and transforms.conf .
1 How to configure above two files?
2 How to setup unix and window AD with universal forwarder configured file?
Any one could give me a complete method to set it up?
Thank you very much!

Anthony

Tags (2)
0 Karma

johnwyane
New Member

I know how to set index using Splunk web interface.
Also, I know how to receive the data from UDF514.

But I don't know about the meaning of "Format" and "$0" and "metadata" "_metadata" even I check the transforms.conf.sample file.
1 So anyone can explain it ?
2 Also, if possible , correct my configuration if anything is wrong!!
3 Should I assign different sourcetype for F5, cisco, fortigate? How? Which file?
Thank you very much!

Here is my configuration
props.conf
[F5_syslog]
TRANSFORMS-F5=asm_log

[cisco-router_syslog]
TRANSFORM-cisco=cisco-router

[fortigate_syslog]
TRANSFORM-forti=fortigate

transforms.conf
[asm_log]
REGEX = 10.10.10.10|10.10.10.11
DEST_KEY = _MetaData:Index
FORMAT = F5_ASM

[cisco-router]
REGEX = 172.16.16.*
DEST_KEY = _MetaData:Index
FORMAT = cisco_log

[Fortigate]
REGEX = 10.10.10.20|10.10.10.21
DEST_KEY = _MetaData:Index
FORMAT = Forti_log

For my 2nd question
at windows server (with forwarder):
I do nothing just using windows-ta-app.

at unix server(with forwarder):
[monitor:///var/log]
sourcetype=unix
index=unix

0 Karma

kristian_kolb
Ultra Champion

You'll need to ensure that the incoming data on port 514 will be separated into different sourcetypes before this configuration will work (since you are making transforms based on sourcetypes, right).

Also, quite separately, your REGEXes might be a bit too unspecific, since they will match on the existence of a string (in your case ip-addresses) regardless of where they appear in the message. Perhaps the addition of a SOURCE_KEY attribute in your transforms can help you with that (i.e. ensure that the REGEX is matching on the host and not on the whole _raw event.

0 Karma

Ayn
Legend

What information are you missing from the docs/splunkbase answers you found? We can help, but we can't do all the work for you - you need to understand the mechanisms before you can implement this.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Thanks for the Memories: .conf26 Took Learning to New Heights

Thank you, Splunk Community, for making .conf26 in Denver one for the books. From packed Splunk University ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...