REGEX command in props/transforms conf files for field extraction

[SplunkDash]

Hello,

I was trying to use REGEX command within props/transforms conf files to extraction fields, but field extraction is not working. Two sample events and my props/transforms conf files are given below. Any recommendations will be highly appreciated. Thank you so much.

props.conf

[mysourcetype]

SHOULD_LINEMERGE=false

LINE_BREAKER = ([\r\n]+)

TIME_PREFIX=^

TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N

MAX_TIMESTAMP_LOOKAHEAD=24

TRUNCATE = 9999

REPORT-fieldEtraction = fieldEtraction

transforms.conf

[fieldEtraction]

REGEX =

\{\"UserID\":\"(?P<UserID>\w+)\","UserType":\"(?P<UserType>\w+)\","System":\"(?P<System>\w+)\","UAT":\"(?P<UAT>.*)\","EventType":\"(?P<EventType>.*)\","EventID":"(?P<EventID>.*)\","Subject":"(?P<Subject>.*)\","EventStatus":"(?P<EventStatus>.*)\","TimeStamp":\"(?P<TimeStamp>.*)\","Device":"(?P<Device>.*)\","MsG":"(?P<Message>.*)\"}

Samples Events

 2023-10-03T18:56:31.099Z OTESTN097MA4513020 TEST[20248] {"UserID":"8901A","UserType":"EMP","System":"TEST","UAT":"UTA-True","EventType":"TEST","EventID":"Lookup","Subject":"A516617222","EventStatus":"00","TimeStamp":"2023-10-03T18:56:31.099Z","Device":" OTESTN097MA4513020","Msg":"lookup ok"}

2023-10-03T18:56:32.086Z OTESTN097MA4513020 TEST[20248] {"UserID":"8901A","UserType":"EMP","System":"TEST","UAT":"UTA-True","EventType":"TEST","EventID":"Lookup","Subject":"A516617222","EventStatus":"00","TimeStamp":"2023-10-03T18:56:32.086Z","Device":" OTESTN097MA4513020","Msg":"lookup ok"}

[yeahnah · Accepted solution]

Hi @SplunkDash 

I can see a few things...

1. This is a search time extraction so need to be specified it on the search head (maybe you have)
     - this can be done via the SH UI too
2.  You've specified the REGEX like an inline field extraction (see props.conf.EXTRACT) so you would not have to use transforms configuration for this (though I believe it does work in transforms).
3. The regex is not quite right

Try this instead (inline or transforms) 

{"UserID":"(?P<UserID>\w+)","UserType":"(?P<UserType>\w+)","System":"(?P<System>\w+)","UAT":"(?P<UAT>.+?)","EventType":"(?P<EventType>.+?)","EventID":"(?P<EventID>.+?)","Subject":"(?P<Subject>.+?)\","EventStatus":"(?P<EventStatus>.+?)","TimeStamp":"(?P<TimeStamp>.+?)","Device":"(?P<Device>.+?)","Msg":"(?P<Message>.+?)"}

Having said that, the example events are a good example of using the power of transforms to match reoccurring patterns and extract them as key value pairs.   In that case the transforms.conf settings would be something like this.

[fieldEtraction]
REGEX = "([^"]+?)":"([^"]+?)"
FORMAT = $1::$2

The REGEX specifies two capture groups and the FORMAT extracts each group as field = value, repeatedly (up to 10000 times, by default).

Some doc links as reference
https://docs.splunk.com/Documentation/Splunk/8.2.7/Admin/Transformsconf#transforms.conf.example

https://docs.splunk.com/Documentation/Splunk/8.2.7/Admin/Propsconf#Field_extraction_configuration

Hope that helps

[inventsekar]

Hi @SplunkDash ... on the indexer, pls try to do "Add data" (Settings---->Add Data)... upload the sample log file, and at the 2nd or 3rd step, there is a field extraction task.. 

in which, Splunk will help/guide you on the rex command for the field extraction. pls try that idea. (dont follow all steps in that "Add Data" and upload the data.. if you do, you will get duplicate logs then.)

once you are good with the field extraction, you can copy that rex command use it inside transforms.conf. 

Splunk newbie learning videos, for absolute beginners:
https://www.youtube.com/@SiemNewbies101/playlists

i have added 24 small videos of rex... completely for splunk newbies and beginners. thanks.