Re: Field Extractions in Search Head GUI

anandhalagarasa · ‎01-04-2018

Hi Team,

I have an event which is getting segregated with pipe (|) symbol and i want to separate those events with a regex expression how to proceed further.

Sample Event are below:

So I have tried to segregate the same in search head GUI by clicking the Field extractions and by using delimit option i choose pipe (|) symbol to split it.

The fields are getting extracted and I have renamed the field names too. But once I saved it and clicked the extracted fields which is in left hand column its getting with the field name and with the key value. But actually i want the key value alone.

For Example:

If i have delimit the field using (|) symbol and rename the field to "Name" and saved it.

Post saving it when i click the "Name" in the extracted field it should show as "Men" as mentioned in sample event.

But instead when i click the "Name" field it shows as "Name=Men"

So kindly let me know the regex to extract only the key value alone since i need to create multiple field extractions for the same.

micahkemp · ‎01-09-2018

To implement my previous answer via the UI, follow these steps:

Settings -> Fields -> Field Transformations -> New

alt text

Set sharting to global

Settings -> Field Extractions -> New

alt text

Set sharing to global

Settings -> Source Types -> Select source type (may have to uncheck "Show only popular")

Expand Advanced settings -> New setting -> Name = KV_MODE Value = none (I can't attach another screenshot, sorry) -> Save

This extracts the fields as you've specified for the line:

including:

FileId: xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx

micahkemp · ‎01-08-2018

transforms.conf:

[pipe_key_equals_value]
REGEX = (?<_KEY_1>[^|=]+)=(?<_VAL_1>[^|]+)

props.conf:

[<sourcetype>]
REPORT-pipe_key_equals_value = pipe_key_equals_value

To see this in action: https://regex101.com/r/otQuZ6/1

Note: When _KEY_1 and _VAL_1 are used, splunk will use the value of _KEY_1 as the field name, and the value of _VAL_1 as the value of that field.

abhijeet01 · ‎01-04-2018

Hey @anandhalagarasan

Ignore my previous answer.
You can try below regex.

I have checked this on regex101.com and its working fine.You can also use this regex in transforms.conf file.

anandhalagarasa · ‎01-08-2018

Hi abhijeet01,

Thanks for your response. But still it didn't worked.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

micahkemp · ‎01-04-2018

Splunk appears to automatically extract these fields, as they are Key=Value. Does your sourcetype set KV_MODE = none?

mayurr98 · ‎01-04-2018

Hey

Go to settings>fields>field extractions>new and put below regex

Level=(?P<Level>[^|]*)|Name=(?P<Name>[^|]*)|Id=\((?P<Id>[^\)]*)\)|Job=\((?P<Job>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|Cycle=\((?P<Cycle>[^\)]*)\)|Value=\((?P<Value>[^\)]*)\)|Field=\((?P<Field>[^\)]*)\)|Process\s(?P<Process>.*)

Also you can try in search query and look for field value using below query.

index=your_index | rex field=_raw “Level=(?P<Level>[^|]*)|Name=(?P<Name>[^|]*)|Id=\((?P<Id>[^\)]*)\)|Job=\((?P<Job>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|Cycle=\((?P<Cycle>[^\)]*)\)|Value=\((?P<Value>[^\)]*)\)|Field=\((?P<Field>[^\)]*)\)|Process\s(?P<Process>.*)”

After that you can see all the field on the left side. With field value pair as you wanted

Let me know if this helps you !

mayurr98 · ‎01-04-2018

hey @anandhalagarasan

have you tried above regex?
you will be able to extract process with a value xxxxas well with all the desired results !

let me know if this works!

anandhalagarasa · ‎01-08-2018

Hi mayurr98,

Thanks for your response. But still it didn't worked.

Let me show the actual raw event from this example can you help to retrieve those 8 fields.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

mayurr98 · ‎01-08-2018

see for this particular event you can use

ServiceName=(?P<Name>[^|]*)|DropId=(?P<Id>[^\|]*)|Job=\((?P<Job>[^\)]*)\)|JobNumber=\((?P<JobNumber>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|StampCycle=(?P<StampCycle>[^\|]*)|TotalFiles\=(?P<Field>[^\|]*)|FileId=(?P<FileId>[^\|]*)

Let me know if this helps !

anandhalagarasa · ‎01-09-2018

Thanks for your response.
I have applied the regex as provided but still the last field alone (FileId) is fetching the information along with the message that might be whatever even-though we have a pipe symbol in between..

Once the field is extracted and when i click the FileId it shows the result as below:

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

Can you kindly help on this.

Elsurion · ‎01-04-2018

One way is to take the name as the beginning pattern for the regex.

| rex field=_raw "\|Name=(?<name>\S+)|ID=\((?<id>\S+)\)\|"

another way is to put these regex into a transforms.conf on the searchhead.

/local/props.conf

[<your_sourcetype>]
REPORT_extract_b1  = extr_pats

/local/transforms.conf

[extr_pats]
REGEX = \|Name=(?<name>\S+)|ID=\((?<id>\S+)\)\|

Another way is with DELIMS

[extr_pats]
DELIMS = "=|"
FIELDS = "temp1","name",temp2","id"

anandhalagarasa · ‎01-04-2018

Thanks for your suggestion. All fields are working fine except the last one that is "Field".

Sample Event :

Here in this case the "Field" is getting extracted along with Process information.

When I click "Field" in extracted fields the results are as below:

xxxxxxx
xxxxxxx|Processs xxxxxx

But i need to exclude the Process one and get only the key value xxxxxxx.

So kindly help to provide regex for the same.

Elsurion · ‎01-05-2018

Did you use the inline or the props/transforms one?

For inline it would be

| rex field=Field "\|\w+\s+(?<process>.+)"

for props/transforms with DELIMS it would be this example
tranforms.conf rewrite

 [<your_sourcetype>]
 REPORT_extract_b1  = extr_pats
 REPORT_newfield = new_extr

props.conf add

[new_extr]
REGEX = "\|\w+\s+(?<process>.+)"
SOURCE_KEY = Field

anandhalagarasa · ‎01-08-2018

Hi,

Consider this exact sample event and from here i want to extract those fields.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

Elsurion · ‎01-09-2018

The full regex for this string would be this one:

\S+\s\S+\|(?<Status>\w+)\|\w+=(?<ServiceName>\w+)\|\w+=(?<DropId>(\w+-){4}\w+)\|\w+=\(?<JobNumber>(\w+)\)\|\w+=\(?<DropNumber>(\w+)\)|\|\w+=(?<StampCycle>\w+)\|\w+=(?<TotalFiles>\w+)\|\w+=(?<FileId>(\w+-){4}\w+)\|\w+:\s(?<QueueName>\w+),\s\w+:\s(?<GUID>(\w+-){4}\w+)

It does not include now special characters like äöü.
You could also expand the placeholder \w+ to it's real name like ServiceName, etc. But this is only a solution when you getting faults with the extraction.

When you encounter some mismatch, you can test the regex also here:
https://regexr.com/
but you have to remove the fielddefintions, since this page does not recognize it.

anandhalagarasa · ‎01-08-2018

Hi Elsurion,

Thanks for your response. But still it didn't worked.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

Field Extractions in Search Head GUI

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL

Buttercup Games: Further Dashboarding Techniques (Part 5)

Customers Increasingly Choose Splunk for Observability