Splunk Search

How do I configure Splunk to recognize my custom delimiter for proper field extraction?

emamedov
Explorer

I currently have a log statement which has a custom delimiter: {|}

Where an example log statement would look like:

Oct-13 12:17:13 | INFO| [Logger:152] Message{|}Activity1{|}userDeletedProfile{|}John Smith{|}Smith Securities{|}Test1{|}5512{|}324166{|}552341{|}260

However, when I try to conduct a field extraction where DELIMS = "{|}", the fields aren't being extracted properly. However, testing the above log statement in another application that is capable of delimiting yields successful results.

0 Karma

gokadroid
Motivator

Hi @emamedov

I tried to extract fields using the Field Extractor and it did extract the fields with {|} however for some odd reason it considered pipe in between as a field too. So all of below in " " are each field as they appear on my field extractor.

"Oct-13 12:18:13 | INFO| [Logger:152] Message"  "|"  "Activity1" "|" "userDeletedProfile" "|" "John" "|" "KMI SECUR" "|" "Test1" "|" "5512" "|" "324166" "|" "552341" "|" "260"

If you are not fine with above then alternatively you can use below regex in "write my own regular expression" in FieldExtractor or during search time:


^(?< dateTime>[^|]+)|\s+?(?< loggerLevel>[^|]+)|\s+?(?< message>[^{]+){|}(?< activity>[^{]+){|}(?< profile>[^{]+){|}(?< userName>[^{]+){|}(?< securities>[^{]+){|}(?< test>[^{]+){|}(?< id1>[^{]+){|}(?< id2>[^{]+){|}(?< id3>[^{]+){|}(?< id4>[^{]+)

Note Please remove space in each of the tags above < dateTime> < loggerLevel> and so on

0 Karma

gcusello
SplunkTrust
SplunkTrust

You have to extract fields using multiple regex.
I cannot use my pc now, Tomorrow morning I'll send you an example.
Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

in props. conf
REPORT-myfields = myfields_Fields

in transforms.conf
[myfields_Fields]
REGEX = ](.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.)
FORMAT = Field1::"$1" Field2::"$2" Field3::"$3" Field4::"$4" Field5::"$5" Field6::"$6" Field7::"$7" Field8::"$8" Field9::"$9" Field10::"$10"
WRITE_META = true

Bye.
Giuseppe

0 Karma

rjthibod
Champion

I think you cannot use the DELIMS setting to use ordered groups of characters as delimeters. Reading http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf seems to say that you can only specify a single character delimeter.

Instead of DELIM you will probably have to specify the fields/delimeters with REGEX-based extractions.

rjthibod
Champion

The community probably needs more info to help. Can you share more details about the settings in props.conf that are associated with this sourcetype?

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...