Getting Data In

Changing sourcetype of logs coming in as syslog

slee75
New Member

Hello, we currently have Apache logs coming into our Splunk instance via rsyslog. As such, the sourcetype of the apache logs is "syslog". Is there a way to change the sourcetype of just the apache logs to access_common and/or access_combined?

0 Karma
1 Solution

skoelpin
SplunkTrust
SplunkTrust

I'm assuming your logging the logs and forwarding them with a UF on the syslog server? If so then you need to change the sourcetype in inputs.conf on the remote forwarder and restart it.. Keep in mind, you will lose all fields you created under the current sourcetype since fields are relative to sourcetype..

Whats the reason for wanting to change sourcetypes?

View solution in original post

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I'm assuming your logging the logs and forwarding them with a UF on the syslog server? If so then you need to change the sourcetype in inputs.conf on the remote forwarder and restart it.. Keep in mind, you will lose all fields you created under the current sourcetype since fields are relative to sourcetype..

Whats the reason for wanting to change sourcetypes?

0 Karma

slee75
New Member

Thank you! We want to change sourcetypes so we can extract the httpd access log fields, ie source IP, etc. Is it possible to configure props.conf to change sourcetype based on process? I'm assuming that's created by the syslog field extraction, so I don't know if props.conf applies before or after the logs are indexed? I know the apache logs get the field "process" set as httpd, otherwise I'm not sure how to differentiate these logs from other linux syslog files (like from /var/log/messages)

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Sorry about that, I had a typo.. The sourcetype is defined in inputs.conf on the forwarder.. Are you using the deployment server? If so then it will most likely be under $SPLUNK_HOME/etc/deployment-apps/<APP-NAME>/local/ on the DS.. It could also be under $SPLUNK_HOME/etc/system/local on the UF which is against Splunks best practices..

How long have you been indexing this data for? You could delete all the data from that previous sourcetype and re-index on the new sourcetype if it hasn't been a while.. If you have a ton of data, then you should probably not re-index and wait for the previous sourcetype to drop off.. You could also create custom field extractions under that sourcetype which can capture the source IP.. How many fields do you want to capture under your current sourcetype?

0 Karma

slee75
New Member

We're actually using rsyslogd to forward the logs on port 514, which is what makes it a bit tougher. I suppose we can create custom field extractions, those would apply to every log wouldn't they? Can we make it conditional? I come from an ELK background, and you could use logstash to parse based on conditions (if programname=httpd { mapping => mapping.json } for example)

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Ah I see.. I would suggest you log the data and forward it via Universal Forwarder to Splunk. You're going to run into issues down the road if you forward the traffic directly to Splunk (You will lose data if need to restart Splunk)

You probably can assign the sourcetype conditionally, but it will require some tricky conf changes.. But I would suggest assigning sourcetype based on the "shape" of the data.. If both sourcetypes have the same "shape" then your only adding more admin work on your end to maintain it.. Also remember that line breaking and timestamp identification depends on sourcetype so you will need to account for this along with extracting fields.

In your case, I would suggest keeping your current sourcetype and apply extractions on your sourcetype

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Did this answer solve your issue? If so can you accept it to close it out?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...