Hi Team,
I have an event which is getting segregated with pipe (|) symbol and i want to separate those events with a regex expression how to proceed further.
Sample Event are below:
2017-06-04 03:51|Level=***|Name=Men|Id=(xxxxx)|Job=(xxxx)|DropNumber=(xxxx)|Cycle=(xxx)|Value=(xxx)|Field=(xxxx)|Process xxxxxxxx
So I have tried to segregate the same in search head GUI by clicking the Field extractions and by using delimit option i choose pipe (|) symbol to split it.
The fields are getting extracted and I have renamed the field names too. But once I saved it and clicked the extracted fields which is in left hand column its getting with the field name and with the key value. But actually i want the key value alone.
For Example:
If i have delimit the field using (|) symbol and rename the field to "Name" and saved it.
Post saving it when i click the "Name" in the extracted field it should show as "Men" as mentioned in sample event.
But instead when i click the "Name" field it shows as "Name=Men"
So kindly let me know the regex to extract only the key value alone since i need to create multiple field extractions for the same.
To implement my previous answer via the UI, follow these steps:
Settings -> Fields -> Field Transformations -> New
Set sharting to global
Settings -> Field Extractions -> New
Set sharing to global
Settings -> Source Types -> Select source type (may have to uncheck "Show only popular")
Expand Advanced settings -> New setting -> Name = KV_MODE Value = none (I can't attach another screenshot, sorry) -> Save
This extracts the fields as you've specified for the line:
2018-01-07 01:50:39,296|INFO|ServiceName=xxxx|DropId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx|JobNumber=(xxx)|DropNumber=(xxx)|StampCycle=xxxx|TotalFiles=xx|FileId=xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx, GUID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
including:
FileId: xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx
transforms.conf:
[pipe_key_equals_value]
REGEX = (?<_KEY_1>[^|=]+)=(?<_VAL_1>[^|]+)
props.conf:
[<sourcetype>]
REPORT-pipe_key_equals_value = pipe_key_equals_value
To see this in action: https://regex101.com/r/otQuZ6/1
Note: When _KEY_1
and _VAL_1
are used, splunk will use the value of _KEY_1
as the field name, and the value of _VAL_1
as the value of that field.
Hey @anandhalagarasan
Ignore my previous answer.
You can try below regex.
rex field=_raw "^(?P.+)|Level=(?P[^|]+)|Name=(?P.+)|Id=((?P.+))|Job=((?P.+))|DropNumber=((?P.+))|Cycle=((?P.+))|Value=((?P.+))|Field=((?P.+))|Process\s(?P.+)"
I have checked this on regex101.com and its working fine.You can also use this regex in transforms.conf file.
Hi abhijeet01,
Thanks for your response. But still it didn't worked.
Let me show the actual raw event from this example can you help to retrieve those 8 fields.
2018-01-07 01:50:39,296|INFO|ServiceName=xxxx|DropId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx|JobNumber=(xxx)|DropNumber=(xxx)|StampCycle=xxxx|TotalFiles=xx|FileId=xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx, GUID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
I just want to extract the following fields without any issues.
ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId
And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.
When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.
xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx
But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.
Is there any way to get it achieved.
NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.
Splunk appears to automatically extract these fields, as they are Key=Value. Does your sourcetype set KV_MODE = none
?
Hey
Go to settings>fields>field extractions>new and put below regex
Level=(?P<Level>[^|]*)|Name=(?P<Name>[^|]*)|Id=\((?P<Id>[^\)]*)\)|Job=\((?P<Job>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|Cycle=\((?P<Cycle>[^\)]*)\)|Value=\((?P<Value>[^\)]*)\)|Field=\((?P<Field>[^\)]*)\)|Process\s(?P<Process>.*)
Also you can try in search query and look for field value using below query.
index=your_index | rex field=_raw “Level=(?P<Level>[^|]*)|Name=(?P<Name>[^|]*)|Id=\((?P<Id>[^\)]*)\)|Job=\((?P<Job>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|Cycle=\((?P<Cycle>[^\)]*)\)|Value=\((?P<Value>[^\)]*)\)|Field=\((?P<Field>[^\)]*)\)|Process\s(?P<Process>.*)”
After that you can see all the field on the left side. With field value pair as you wanted
Let me know if this helps you !
hey @anandhalagarasan
have you tried above regex?
you will be able to extract process
with a value xxxx
as well with all the desired results !
let me know if this works!
Hi mayurr98,
Thanks for your response. But still it didn't worked.
Let me show the actual raw event from this example can you help to retrieve those 8 fields.
2018-01-07 01:50:39,296|INFO|ServiceName=xxxx|DropId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx|JobNumber=(xxx)|DropNumber=(xxx)|StampCycle=xxxx|TotalFiles=xx|FileId=xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx, GUID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
I just want to extract the following fields without any issues.
ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId
And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.
When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.
xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx
But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.
Is there any way to get it achieved.
NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.
see for this particular event you can use
ServiceName=(?P<Name>[^|]*)|DropId=(?P<Id>[^\|]*)|Job=\((?P<Job>[^\)]*)\)|JobNumber=\((?P<JobNumber>[^\)]*)\)|DropNumber=\((?P<DropNumber>[^\)]*)\)|StampCycle=(?P<StampCycle>[^\|]*)|TotalFiles\=(?P<Field>[^\|]*)|FileId=(?P<FileId>[^\|]*)
Let me know if this helps !
Thanks for your response.
I have applied the regex as provided but still the last field alone (FileId) is fetching the information along with the message that might be whatever even-though we have a pipe symbol in between..
Once the field is extracted and when i click the FileId it shows the result as below:
xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx
Can you kindly help on this.
One way is to take the name as the beginning pattern for the regex.
| rex field=_raw "\|Name=(?<name>\S+)|ID=\((?<id>\S+)\)\|"
another way is to put these regex into a transforms.conf on the searchhead.
/local/props.conf
[<your_sourcetype>]
REPORT_extract_b1 = extr_pats
/local/transforms.conf
[extr_pats]
REGEX = \|Name=(?<name>\S+)|ID=\((?<id>\S+)\)\|
Another way is with DELIMS
[extr_pats]
DELIMS = "=|"
FIELDS = "temp1","name",temp2","id"
Thanks for your suggestion. All fields are working fine except the last one that is "Field".
Sample Event :
2017-06-04 03:51|Level=***|Name=Men|Id=(xxxxx)|Job=(xxxx)|DropNumber=(xxxx)|Cycle=(xxx)|Value=(xxx)|Field=(xxxx)|Process xxxxxxxx
Here in this case the "Field" is getting extracted along with Process information.
When I click "Field" in extracted fields the results are as below:
xxxxxxx
xxxxxxx|Processs xxxxxx
But i need to exclude the Process one and get only the key value xxxxxxx.
So kindly help to provide regex for the same.
Did you use the inline or the props/transforms one?
For inline it would be
| rex field=Field "\|\w+\s+(?<process>.+)"
for props/transforms with DELIMS it would be this example
tranforms.conf rewrite
[<your_sourcetype>]
REPORT_extract_b1 = extr_pats
REPORT_newfield = new_extr
props.conf add
[new_extr]
REGEX = "\|\w+\s+(?<process>.+)"
SOURCE_KEY = Field
Hi,
Consider this exact sample event and from here i want to extract those fields.
2018-01-07 01:50:39,296|INFO|ServiceName=xxxx|DropId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx|JobNumber=(xxx)|DropNumber=(xxx)|StampCycle=xxxx|TotalFiles=xx|FileId=xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx, GUID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
I just want to extract the following fields without any issues.
ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId
And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.
When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.
xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx
But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.
Is there any way to get it achieved.
NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.
The full regex for this string would be this one:
\S+\s\S+\|(?<Status>\w+)\|\w+=(?<ServiceName>\w+)\|\w+=(?<DropId>(\w+-){4}\w+)\|\w+=\(?<JobNumber>(\w+)\)\|\w+=\(?<DropNumber>(\w+)\)|\|\w+=(?<StampCycle>\w+)\|\w+=(?<TotalFiles>\w+)\|\w+=(?<FileId>(\w+-){4}\w+)\|\w+:\s(?<QueueName>\w+),\s\w+:\s(?<GUID>(\w+-){4}\w+)
It does not include now special characters like äöü.
You could also expand the placeholder \w+ to it's real name like ServiceName, etc. But this is only a solution when you getting faults with the extraction.
When you encounter some mismatch, you can test the regex also here:
https://regexr.com/
but you have to remove the fielddefintions, since this page does not recognize it.
Hi Elsurion,
Thanks for your response. But still it didn't worked.
Let me show the actual raw event from this example can you help to retrieve those 8 fields.
2018-01-07 01:50:39,296|INFO|ServiceName=xxxx|DropId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx|JobNumber=(xxx)|DropNumber=(xxx)|StampCycle=xxxx|TotalFiles=xx|FileId=xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx, GUID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
I just want to extract the following fields without any issues.
ServiceName
DropId
JobNumber
DropNumber
StampCycle
TotalFiles
FileId
And as mentioned in your regex expression i can able to extract all fields perfectly except "FileId" alone.
When i extract FileId and when i click the FileId its getting extracted along with (|) pipe symbol and QueueName information.
xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxxxxx|QueueName: xxx_xxxxx_xxxxxx
But actually i need only the FileId value alone and it should not be extracting any messages post FileId Since I need to extract only till FileId.
Is there any way to get it achieved.
NOTE:
For few instances alone its getting extracted with | information followed by FileId and the remaining FileId values are getting extracted perfectly.