Mulitple versions of logs creating separate sourcetypes?

[onegreydot]

I have a log file that I created a transforms.conf and props.conf for specifying the log source in the props with [source::.../name.log]. The application generating the log was recently updated and the log file now has the same name.log but different header fields. Righ now I am extracting the fields using the REPORT statement and having DELIMS and FIELDS. But since the log file changed the FIELDS are now different. What would be the best way to maintaining both versions of logs inside of my splunk instance without having to rename the actual log?

[emiller42]

Do the existing transforms still function on the new format of data, just with incorrect fields? Or does it not apply at all?

If it doesn't apply at all, then you can simply add another transforms.conf stanza for the new format, and have the props.conf stanza use both. Splunk will try both and apply whichever works.

However, if your extraction still technically works, just is incorrect because the data changed, that complicates things. Can you provide sample data and your existing configuration?

[emiller42]

You may have to get more explicit and use regex to parse out the fields then. I would try to make regex extractions that only match one of the two formats. That way when both are applied, there's no way the fields can be swapped around like that.

[onegreydot]

Tried adding that in the props.conf and it does extract both sets now.

However it still doesn't distinguish between logs, because the headers have changed. I should have put that in the example above.

So instead of "src_ip", "src_port", "dest_ip" ...
It may be "src_ip", "src_port", "host", "dest_ip" ...

The above makes it so that host could have the dest_ip and vice versa when searching the logs. If there were just additional headers to the log the above solution would work great!

Thanks!

[emiller42]

Try just this in your props.conf settings:

REPORT_name_log = old_log,new_log

[onegreydot]

The format of the log files in the transforms.conf looks something like:
old log:

[old_log] FIELDS="timestamp", "src_ip", "src_port", "dest_ip", "dest_port"
DELIMS=","

new log:
[new_log]
FIELDS="timestamp", "src_ip", "src_port", "dest_ip", "dest_port", "url", "level", "message"
DELIMS=","

In my props.conf I have:
[source::.../name.log]
sourcetype = name_log

[name_log]
REPORT-old_log = old_log
...
...

As you can see the new version of the log has a few more fields. I will actually try putting both REPORT statments in the props.conf and see how that goes.

• Thanks for the help.