Re: Field Extractions in Search Head GUI

anandhalagarasa · ‎01-04-2018

Hi Team,

I have an event which is getting segregated with pipe (|) symbol and i want to separate those events with a regex expression how to proceed further.

Sample Event are below:

So I have tried to segregate the same in search head GUI by clicking the Field extractions and by using delimit option i choose pipe (|) symbol to split it.

The fields are getting extracted and I have renamed the field names too. But once I saved it and clicked the extracted fields which is in left hand column its getting with the field name and with the key value. But actually i want the key value alone.

For Example:

If i have delimit the field using (|) symbol and rename the field to "Name" and saved it.

Post saving it when i click the "Name" in the extracted field it should show as "Men" as mentioned in sample event.

But instead when i click the "Name" field it shows as "Name=Men"

So kindly let me know the regex to extract only the key value alone since i need to create multiple field extractions for the same.

micahkemp · ‎01-09-2018

To implement my previous answer via the UI, follow these steps:

Settings -> Fields -> Field Transformations -> New

Set sharting to global

Settings -> Field Extractions -> New

Set sharing to global

Settings -> Source Types -> Select source type (may have to uncheck "Show only popular")

Expand Advanced settings -> New setting -> Name = KV_MODE Value = none (I can't attach another screenshot, sorry) -> Save

This extracts the fields as you've specified for the line:

including:

FileId: xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx

micahkemp · ‎01-08-2018

transforms.conf:

[pipe_key_equals_value]
REGEX = (?<_KEY_1>[^|=]+)=(?<_VAL_1>[^|]+)

props.conf:

[<sourcetype>]
REPORT-pipe_key_equals_value = pipe_key_equals_value

To see this in action: https://regex101.com/r/otQuZ6/1

Note: When _KEY_1 and _VAL_1 are used, splunk will use the value of _KEY_1 as the field name, and the value of _VAL_1 as the value of that field.

abhijeet01 · ‎01-04-2018

Hey @anandhalagarasan

Ignore my previous answer.
You can try below regex.

I have checked this on regex101.com and its working fine.You can also use this regex in transforms.conf file.

anandhalagarasa · ‎01-08-2018

Hi abhijeet01,

Thanks for your response. But still it didn't worked.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

micahkemp · ‎01-04-2018

Splunk appears to automatically extract these fields, as they are Key=Value. Does your sourcetype set KV_MODE = none?

mayurr98 · ‎01-04-2018

Hey

Go to settings>fields>field extractions>new and put below regex

Level=(?P<Level>[^|]*)|Name=(?P<Name>[^|]*)|Id=\((?P<Id>[^\)]*)\)|Job=\((?P<Job>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|Cycle=\((?P<Cycle>[^\)]*)\)|Value=\((?P<Value>[^\)]*)\)|Field=\((?P<Field>[^\)]*)\)|Process\s(?P<Process>.*)

Also you can try in search query and look for field value using below query.

index=your_index | rex field=_raw “Level=(?P<Level>[^|]*)|Name=(?P<Name>[^|]*)|Id=\((?P<Id>[^\)]*)\)|Job=\((?P<Job>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|Cycle=\((?P<Cycle>[^\)]*)\)|Value=\((?P<Value>[^\)]*)\)|Field=\((?P<Field>[^\)]*)\)|Process\s(?P<Process>.*)”

After that you can see all the field on the left side. With field value pair as you wanted

Let me know if this helps you !

mayurr98 · ‎01-04-2018

hey @anandhalagarasan

have you tried above regex?
you will be able to extract process with a value xxxxas well with all the desired results !

let me know if this works!

anandhalagarasa · ‎01-08-2018

Hi mayurr98,

Thanks for your response. But still it didn't worked.

Let me show the actual raw event from this example can you help to retrieve those 8 fields.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

mayurr98 · ‎01-08-2018

see for this particular event you can use

ServiceName=(?P<Name>[^|]*)|DropId=(?P<Id>[^\|]*)|Job=\((?P<Job>[^\)]*)\)|JobNumber=\((?P<JobNumber>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|StampCycle=(?P<StampCycle>[^\|]*)|TotalFiles\=(?P<Field>[^\|]*)|FileId=(?P<FileId>[^\|]*)

Let me know if this helps !

anandhalagarasa · ‎01-09-2018

Thanks for your response.
I have applied the regex as provided but still the last field alone (FileId) is fetching the information along with the message that might be whatever even-though we have a pipe symbol in between..

Once the field is extracted and when i click the FileId it shows the result as below:

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

Can you kindly help on this.

Elsurion · ‎01-04-2018

One way is to take the name as the beginning pattern for the regex.

| rex field=_raw "\|Name=(?<name>\S+)|ID=\((?<id>\S+)\)\|"

another way is to put these regex into a transforms.conf on the searchhead.

/local/props.conf

[<your_sourcetype>]
REPORT_extract_b1  = extr_pats

/local/transforms.conf

[extr_pats]
REGEX = \|Name=(?<name>\S+)|ID=\((?<id>\S+)\)\|

Another way is with DELIMS

[extr_pats]
DELIMS = "=|"
FIELDS = "temp1","name",temp2","id"

anandhalagarasa · ‎01-04-2018

Thanks for your suggestion. All fields are working fine except the last one that is "Field".

Sample Event :

Here in this case the "Field" is getting extracted along with Process information.

When I click "Field" in extracted fields the results are as below:

xxxxxxx
xxxxxxx|Processs xxxxxx

But i need to exclude the Process one and get only the key value xxxxxxx.

So kindly help to provide regex for the same.

Elsurion · ‎01-05-2018

Did you use the inline or the props/transforms one?

For inline it would be

| rex field=Field "\|\w+\s+(?<process>.+)"

for props/transforms with DELIMS it would be this example
tranforms.conf rewrite

 [<your_sourcetype>]
 REPORT_extract_b1  = extr_pats
 REPORT_newfield = new_extr

props.conf add

[new_extr]
REGEX = "\|\w+\s+(?<process>.+)"
SOURCE_KEY = Field

anandhalagarasa · ‎01-08-2018

Hi,

Consider this exact sample event and from here i want to extract those fields.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

Elsurion · ‎01-09-2018

The full regex for this string would be this one:

\S+\s\S+\|(?<Status>\w+)\|\w+=(?<ServiceName>\w+)\|\w+=(?<DropId>(\w+-){4}\w+)\|\w+=\(?<JobNumber>(\w+)\)\|\w+=\(?<DropNumber>(\w+)\)|\|\w+=(?<StampCycle>\w+)\|\w+=(?<TotalFiles>\w+)\|\w+=(?<FileId>(\w+-){4}\w+)\|\w+:\s(?<QueueName>\w+),\s\w+:\s(?<GUID>(\w+-){4}\w+)

It does not include now special characters like äöü.
You could also expand the placeholder \w+ to it's real name like ServiceName, etc. But this is only a solution when you getting faults with the extraction.

When you encounter some mismatch, you can test the regex also here:
https://regexr.com/
but you have to remove the fielddefintions, since this page does not recognize it.

anandhalagarasa · ‎01-08-2018

Hi Elsurion,

Thanks for your response. But still it didn't worked.

I just want to extract the following fields without any issues.

ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId

And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.

When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.

xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx

But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.

Is there any way to get it achieved.

NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.

Field Extractions in Search Head GUI

Introducing the 2024 SplunkTrust!

Introducing the 2024 Splunk MVPs!

Splunk Custom Visualizations App End of Life