topic Re: use props.conf and or transforms.conf to parse log file in Getting Data In

How do I use props.conf and or transforms.conf to parse log file?

eholz1 — Tue, 27 Sep 2022 20:53:27 GMT

I would like to use props.conf and/or transforms.conf to parse data coming from a generic

single line log file using regex to search for "Error" or "Notice" I did a test for my regex in regex 101, and the regex seems ok. regex = (?<=Error)(.*$)

I do have a sourcetype for the incoming data - what should I be looking for and what files should I edit to

allow this?

Thanks,

eholz1

Re: use props.conf and or transforms.conf to parse log file

richgalloway — Tue, 27 Sep 2022 19:39:42 GMT

To parse that one field at search-time (the best option, IMO), you can use an EXTRACT setting in props.conf. No transform is needed.

[mysourcetype] EXTRACT-error = Error(?<Error>.*)$

Note the required named capture group in the regex. Also, avoid using lookbehind in regexes as they're expensive and usually unnecessary.

Re: use props.conf and or transforms.conf to parse log file

eholz1 — Tue, 27 Sep 2022 20:27:53 GMT

hello richgallowy,

I will give this a shot, and see what happens.

and come back if I have any questions. could I add an "OR" to this like:

EXTRACT-error = Error(?<Error>.*)$

OR Notice(?<Notice>.*)$

thanks for the support,

eholz1

Re: use props.conf and or transforms.conf to parse log file

richgalloway — Tue, 27 Sep 2022 21:09:55 GMT

There is nothing to stop you from doing that, but it may not do what you expect.

If you need to extract both Error and Notice fields then use two EXTRACT settings.

EXTRACT-error = Error(?<Error>.*)$ EXTRACT-notice = Notice(?<Notice>.*)$

Re: use props.conf and or transforms.conf to parse log file

eholz1 — Tue, 27 Sep 2022 22:01:55 GMT

Thanks again for the tips.

Now one last dump question - how do I verify that this works?

Do I need any other settings in the props.conf file to allow the data to be viewed in a search?

When I configure the props.conf file with the source "my_source", I still see all the data in the log file that i do not need.

Thanks again,

eholz1

Re: use props.conf and or transforms.conf to parse log file

richgalloway — Wed, 28 Sep 2022 00:12:05 GMT

You'll know the extractions work when you see the new fields on the left side of the results or when you see values from | table Error Notice.

Yes, you will still see field "you do not need". Nothing in the OP asked about getting rid of anything. You probably don't want to get rid of anything at this points, because you may need it for another use case. You can, however, only show what you need by using the table command as above.

Re: use props.conf and or transforms.conf to parse log file

eholz1 — Wed, 28 Sep 2022 14:05:38 GMT

Thanks again for the clarification - I am still learning!