topic Re: Can data have more then one source type? in Getting Data In

Can data have more then one source type?

stefanlasiewski — Mon, 19 Mar 2012 23:30:59 GMT

Our Splunk server receives data through syslog, and all data is tagged with 'sourcetype=syslog'.

I am interested in using the Splunk App for Web Intelligence to parse data from several dozen different Apache webservers. My weblogs are sent to Splunk, but they are sent via syslog and thus Splunk automatically assigns the 'syslog' sourcetype to this data. The Splunk App for Web Intelligence doesn't work with 'sourcetype=syslog'. I need to assign a new kind of sourcetype to this data.

Can data belong to multiple kinds of sourcetypes? If I define the Sourcetype like 'access_combined' for my Apache HTTP logs, will it still belong to the 'syslog' Sourcetype?

Re: Can data have more then one source type?

Damien_Dallimor — Mon, 19 Mar 2012 23:45:31 GMT

You can perform a sourcetype transform on the data coming in to your syslog input.

So you can identify particular messages by host or a regex pattern on the message contents etc.. and dynamically update the sourcetype field and any other meta fields.

inputs.conf

[udp://514]
connection_host = dns
sourcetype = syslog
disabled = false

props.conf

[syslog]
TRANSFORMS-web = set_web_sourcetype

transforms.conf

[set_web_sourcetype]
REGEX = (.+Some Regex Pattern.+)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::my_new_sourcetype

Alternatively ,you could just set up a dedicated syslog input for your webserver data.

inputs.conf

[udp://5678]
connection_host = dns
sourcetype = my_custom_sourcetype
disabled = false
index = my_index

Re: Can data have more then one source type?

gkanapathy — Tue, 20 Mar 2012 00:19:39 GMT

Data may only have a single sourcetype. However, if you define a new sourcetype, you can set it to perform multiple transforms and field extractions that would apply to, say, both syslog and to access_combined, but looking at and duplicating the configurations of each in props.conf.