How to extract the fields using regex and props.conf at indexing time?

[kiran331]

Hello,

How to use Regex in props.conf to extract the fields in the below sample event with source type "syslog".

08/11/17 13:30:34 abckdefrg44 openfep[1123]: [log.c][411]: CPM ALSLLER (ID 5): pin_alarm-DOFRI 22.33.22.5.0 -c 453 1 0 -r 449 0 -v "ONLINE" "FEP" "level Group 33. status" "Status now online" &

Group: level Group 33. status
Status: Status now online

[cpetterborg]

Please read this article. It is insightful about the use of index time field extractions:

https://answers.splunk.com/answers/396276/should-i-use-an-index-time-field-extraction.html

Splunk best practice is to do search time field extractions. There are those that are questioning this as you can see by reading the whole answer and comment on this in this article:

https://answers.splunk.com/answers/2535/search-time-vs-index-time-field-extraction.html

If you are still intent on doing index-time extraction, read this article, as it might give you some help in doing it:

https://answers.splunk.com/answers/234011/index-time-field-extraction-regexp-issue.html

And for the official documentation, see this page:

http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Data/Configureindex-timefieldextraction

And finally, this is the REGEX I would use:

REGEX="(?<Group>[^"]*)"\s+"(?<Status>[^"]*)"\s*&$

But it should go in the transforms.conf, not props.conf.

[santiagoaloi]

Assuming that there's some level of consistency in your syslog event, and "FEP" will always be there, you could do the following props.conf extract, using "FEP" as a starting point for your regex.

[syslog]
EXTRACT-Group = \"FEP\"\s\"(.*?)\"
EXTRACT-Status = \"FEP\"\s\"(?:.*?)\"\s\"(.*?)\"

Your could use regex101 to practice your regex extractions, make sure to anonymize the data before doing so, nobody knows what that site does to your sample events.

Hope it helps, shout if you need more help.

[lfedak_splunk]

Hey @kiran331, this documentation might help http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Advancedsourcetypeoverrides