Erex example

[amarish_vlabs]

Hi, I have the below log and values for "days" field are 4, 10 , 15, 30. Could you please extract the "days" field using the "erex" command.

Log :

2017-11-21 04:55:34,060 tn="[14347886-7337]" ll=INFO cn="s.c.m" - id="57ef4442-aa9f-444d-bd80-e4fd3018f82", action="execute", operation="put", collection="Messages", "days" : "4", "Code" : "491833", useSecondary="false", retries="0"

[saurabh_tek11]

@amarish_vlabs, Here is your sample Event :

2017-11-21 04:55:34,060 tn="[14347886-7337]" ll=INFO cn="s.c.m" - id="57ef4442-aa9f-444d-bd80-e4fd3018f82", action="execute", operation="put", collection="Messages", "days" : "4", "Code" : "491833", useSecondary="false", retries="0"

using erex:

index=* source="amarish_vlabs.txt"  | erex newfield_days examples=4

Job says : Successfully learned regex. Consider using: | rex "(?i)\-e(?P<newfield_days>\d+)"

Now we can see new Search-time field "newfield_days" has been extracted on-the-fly (not persistent field) and moving forward better we use the rex command as that's efficient & faster.
Using rex :

index=* source="amarish_vlabs.txt"  | rex "(?i)\-e(?P<newfield_days>\d+)"

This answer is to start rolling the ball based on one given event. To generalize this on larger set of data and generate (possibly) precise regular expression using erex command, use the optional arguments like counterexamples, fromfield & maxtrainers.

here is syntax of erex :

erex [<field>] examples=<string> [counterexamples=<string>] [fromfield=<field>] [maxtrainers=<int>]

You may see more examples here.

[saurabh_tek11]

Hi @amarish_vlabs, does this answers your question or you have some query? Please feel free to ask.
If no query, please accept the answer so as to close this open question. 🙂 Thank you - Saurabh

[niketn · Accepted solution]

Following is a run anywhere search example for erex command to extract the field you are interested in (based on the sample provided). Please try out and confirm.

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  erex days examples="4,13"

Ideally you should use rex command and once you have tested the same save your regular expression as Field Extraction for reusability and maintenance. Following is the regular expression:

|  makeresults
|  eval _raw="2017-11-21 04:55:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f82\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"4\", \"Code\" : \"491833\", useSecondary=\"false\", retries=\"0\""
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:56:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f83\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"13\", \"Code\" : \"491834\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:57:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f84\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"24\", \"Code\" : \"491835\", useSecondary=\"false\", retries=\"0\""]
|  append
    [|  makeresults
    |  eval _raw="2017-11-21 04:58:34,060 tn=\"[14347886-7337]\" ll=INFO cn=\"s.c.m\" - id=\"57ef4442-aa9f-444d-bd80-e4fd3018f85\", action=\"execute\", operation=\"put\", collection=\"Messages\", \"days\" : \"14\", \"Code\" : \"491836\", useSecondary=\"false\", retries=\"0\""]
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

While the above examples use makeresults and append to mock some sample events as per question. You can try out the final pipe with erex or rex in your base search returning data as per your question:

Using rex command

<YourBaseSearch>
|  rex "\"days\"\s+:\s+\"(?<days>[^\"]+)\""

Using erex command

<YourBaseSearch>
|  erex days examples="4,13"

PS: erex might not be robust for field extraction in production data or else you will have to use a lot of sample examples and counterexamples.

[amarish_vlabs]

Thanks for your clear explanation. It is very useful.

[niketn]

@amarish_vlabs, glad you found it useful. Please accept/up vote the answer if it helped 🙂

[niketn]

@amarish_vlabs, any reason why you want to rely on erex and not rex or regular expression?

[amarish_vlabs]

@niketnilay, I extracted the days field by using "rex" command. I just want to know how to extract with "erex". If this is possible with "erex", Please let me know. Otherwise just ignore. Thank you sir for your reply.

[kamlesh_vaghela]

Hi

Can you please try this configuration?

props.conf

[my_sourcetype]
REPORT-mydays = mydays

transforms.conf

[mydays]
REGEX = \"days\"\s:\s\"(?<days>.+)\",\s\"
FORMAT = $1

Thanks

[amarish_vlabs]

Thank you so much for your reply.