Splunk Search

How to extract the same fields from multiple log formats?

SridharS
Path Finder

Hi,

I have a 3 different log files and there are 8 different formats in them. All formats have the same fields in them (cpu,memory etc.) and regex is similar. Hence, based on the system name, I should get its values (level, OS, primary_drive). Below is the sample log:

Nov 08 12:55:07 servername: {"cpu_cores":"4","ram_size":"6","system_name":"NAME1","level":"info","OS":"WINDOWS","primary_drive":"C"}
Nov 08 12:54:07 servername: {"cpu_cores":"2","ram_size":"6","BIOS_Version":"A12","system_model":"Opti","system_name":"NAME2","level":"info","OS":"WINDOWS","primary_drive":"D"}
Nov 08 12:52:07 servername: {"cpu_cores":"4","ram_size":"4","system_name":"NAME3","level":"info","OS":"WINDOWS","primary_drive":""}

What I did here is I gave the 2 different regex for 2 log formats in props.conf. Hence I get both these log formats in and other 6 formats are ignored

props.conf

[source::/source/file name]
TRANSFORMS-set = outside,inside

transforms.conf

[outside]
REGEX =.
DEST_KEY = queue
FORMAT = nullQueue

[inside]
REGEX = <{\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)>

REGEX =  {\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:BIOS_Version\")\:\"(?<BIOS_Version>.+)\"\,\"(?:system_model\")\:\"(?<system_model>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w) 

DEST_KEY = queue
FORMAT = indexQueue

Also for field extraction, I made the change in props.conf as below, but I get the field extraction for only regex2. The field extraction for regex 1 is not working, hence it's not getting extracted. I am not sure whether same field getting extracted for different patterns might be a problem. Can someone help me, so that I should get field extractions for both regex types?

props.conf

[my_sourcetype]
EXTRACT-fields = <{\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)>

EXTRACT-fields =  {\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:BIOS_Version\")\:\"(?<BIOS_Version>.+)\"\,\"(?:system_model\")\:\"(?<system_model>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w) 
1 Solution

SridharS
Path Finder

Hi, finally I found it. Instead of giving EXTRACT-fields = regex, I gave the all the field names .

EXTRACT-field1, field2, field3....... = regex 1
EXTRACT-field1, field2, field3....... = regex 2

View solution in original post

0 Karma

SridharS
Path Finder

Hi, finally I found it. Instead of giving EXTRACT-fields = regex, I gave the all the field names .

EXTRACT-field1, field2, field3....... = regex 1
EXTRACT-field1, field2, field3....... = regex 2

0 Karma

bkumarm
Contributor

Sridhar,
I had come across a similar problem. I am explaining the approach I have taken, may be useful for you.
1. Combine all the three log files into an eventtype.
Example: eventtype myevent="index=* host=hostname source="source1.log" OR source="source2.log"
2. in each log file define field extracts for the fields as you are currently doing. Use same field name across files (or use field aliases)
3. search for the common fields using eventtype.

example: eventtype="myevent" ID=*
this query will give you values for ID from all three files.
-Bharath

0 Karma

sundareshr
Legend

How about trying something like this

REGEX ="([^"]+)":"([^"]*),?
FORMAT = $1::$2
0 Karma

SridharS
Path Finder

This works. But in my case I need to provide 2 different regex patterns for a single log file. When I give the 2 patterns in props.conf it extracts only one in result(where the fields for both patterns are same).

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...