topic Re: Erex example in Splunk Search

Erex example

amarish_vlabs — Thu, 23 Nov 2017 15:07:50 GMT

Hi, I have the below log and values for "days" field are 4, 10 , 15, 30. Could you please extract the "days" field using the "erex" command.

Log :

2017-11-21 04:55:34,060 tn="[14347886-7337]" ll=INFO cn="s.c.m" - id="57ef4442-aa9f-444d-bd80-e4fd3018f82", action="execute", operation="put", collection="Messages", "days" : "4", "Code" : "491833", useSecondary="false", retries="0"

Re: Erex example

kamlesh_vaghela — Thu, 23 Nov 2017 16:55:09 GMT

Can you please try this configuration?

props.conf

[my_sourcetype]
REPORT-mydays = mydays

transforms.conf

[mydays]
REGEX = \"days\"\s:\s\"(?<days>.+)\",\s\"
FORMAT = $1

Thanks

Re: Erex example

niketn — Thu, 23 Nov 2017 17:01:09 GMT

@amarish_vlabs, any reason why you want to rely on erex and not rex or regular expression?

Re: Erex example

niketn — Thu, 23 Nov 2017 17:14:33 GMT

Following is a run anywhere search example for erex command to extract the field you are interested in (based on the sample provided). Please try out and confirm.

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  erex days examples="4,13"

Ideally you should use rex command and once you have tested the same save your regular expression as Field Extraction for reusability and maintenance. Following is the regular expression:

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

While the above examples use makeresults and append to mock some sample events as per question. You can try out the final pipe with erex or rex in your base search returning data as per your question:

Using rex command

<YourBaseSearch>
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

Using erex command

<YourBaseSearch>
|  erex days examples="4,13"

PS: erex might not be robust for field extraction in production data or else you will have to use a lot of sample examples and counterexamples.

Re: Erex example

amarish_vlabs — Wed, 29 Nov 2017 13:42:58 GMT

@niketnilay, I extracted the days field by using "rex" command. I just want to know how to extract with "erex". If this is possible with "erex", Please let me know. Otherwise just ignore. Thank you sir for your reply.

Re: Erex example

amarish_vlabs — Wed, 29 Nov 2017 13:44:18 GMT

Thanks for your clear explanation. It is very useful.

Re: Erex example

amarish_vlabs — Wed, 29 Nov 2017 13:44:48 GMT

Thank you so much for your reply.

Re: Erex example

niketn — Wed, 29 Nov 2017 14:37:25 GMT

@amarish_vlabs, glad you found it useful. Please accept/up vote the answer if it helped 🙂

Re: Erex example

saurabh_tek11 — Wed, 06 Dec 2017 06:12:15 GMT

@amarish_vlabs, Here is your sample Event :

2017-11-21 04:55:34,060 tn="[14347886-7337]" ll=INFO cn="s.c.m" - id="57ef4442-aa9f-444d-bd80-e4fd3018f82", action="execute", operation="put", collection="Messages", "days" : "4", "Code" : "491833", useSecondary="false", retries="0"

using erex:

index=* source="amarish_vlabs.txt"  | erex newfield_days examples=4

Job says : Successfully learned regex. Consider using: | rex "(?i)\-e(?P<newfield_days>\d+)"

Now we can see new Search-time field "newfield_days" has been extracted on-the-fly (not persistent field) and moving forward better we use the rex command as that's efficient & faster.
Using rex :

index=* source="amarish_vlabs.txt"  | rex "(?i)\-e(?P<newfield_days>\d+)"

This answer is to start rolling the ball based on one given event. To generalize this on larger set of data and generate (possibly) precise regular expression using erex command, use the optional arguments like counterexamples, fromfield & maxtrainers.

here is syntax of erex :

erex [<field>] examples=<string> [counterexamples=<string>] [fromfield=<field>] [maxtrainers=<int>]

You may see more examples here.

Re: Erex example

saurabh_tek11 — Wed, 20 Dec 2017 12:21:36 GMT

Hi @amarish_vlabs, does this answers your question or you have some query? Please feel free to ask.
If no query, please accept the answer so as to close this open question. 🙂 Thank you - Saurabh