All Apps and Add-ons

Multiple Log format in a single sourcetype?

rjyetter
Path Finder

I have a problem where we had an older revision of Bluecoat and then one by one they upgraded the bluecoats and the field mappings to a newer revision.

Problem - mixed fields in the same sourcetype
- Older format is parsed correctly with correct field mappings
- Newer format is not parsed correctly.

All of our bluecoats are now updated to the latest revision now.

I fixed the field mappings to the new BC fields so current data is being parsed and reported on correctly.

Problem - Historical data does not get reported on with the updated fields.

Question:

Is it possible to have multiple log formats within a single sourcetype and be able to parse the data out correctly even through field names may be the same but in a different position?

grantsales
Engager

I think splunk needs to add the ability to perform regex restrictions or filtering prior to field extractions. That would help me in my current set up.

I have 3 different logging formats that change based on a specific field within the same source type.

Basically I need

Case 1: " TEXTA "
rex fileds

Case2: " TEXTB "
rex fields

Case3: " TEXTC "
rex fields

Case4:
default unparsed, no rex fields.

0 Karma

rjyetter
Path Finder

This will probably get ugly, I kind of figure this was the case.. This is what happens when proper change control is not followed!

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

maybe. the neatest way would be to either version or to have different sourcetypes (e.g., bluecoat, bluecoat_v2) but that takes foreknowledge that you might not have, and it's too late now.

you can simply put multiple field extractions to a sourcetype, and both will run, but you might have conflicts. you might be able to deal with this with regex extractions, provided you write the regexes so each one could only match one of the formats, and fail on the other(s).

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...