topic Re: Map fields to sourcetype in Getting Data In

Map fields to sourcetype

DanAlexander — Wed, 18 Oct 2023 14:06:08 GMT

Hi All,

I need help building a SPL that would return all available fields mapped to their sourcetypes/source

Looking across all Indexers crawling through all indexes index=*

I currently use to strip off all the fields and their extracted fields but I have no idea where they are coming from, what is their sourcetype and source:

index=* fieldsummary | search values!="[]" | rex field=values max_match=0 "\{\"value\":\"(?<extracted_values>[^\"]+)\"" | fields field extracted_values

Thank you!

Re: Map fields to sourcetype

ITWhisperer — Wed, 18 Oct 2023 08:22:44 GMT

You could start with something like this

It may fail due to the amount of data being brought back, so you might want to break it up by index. Also, it works by looking at the fields returned in the events, so if some fields are not used in the time period covered, they will not show up, so you might want to run it a different times of the day, rather than for longer periods.

Re: Map fields to sourcetype

DanAlexander — Wed, 18 Oct 2023 11:36:22 GMT

Thanks for the reply @ITWhisperer

Would it be possible to help me out creating something similar which can include both source and sourcetype, please?

Thank you!

Re: Map fields to sourcetype

ITWhisperer — Wed, 18 Oct 2023 12:09:56 GMT

Try something like this

index=* | fields - _time _raw | foreach * [| eval <<FIELD>>=if("<<FIELD>>"=="index",index,if("<<FIELD>>"=="source",source,sourcetype))] | table * | fillnull value="N/A" | foreach * [eval sourcetype=if("<<FIELD>>"!="sourcetype" AND "<<FIELD>>"!="source" AND "<<FIELD>>"!="index",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)] | dedup sourcetype | table index source sourcetype

Re: Map fields to sourcetype

DanAlexander — Wed, 18 Oct 2023 14:49:15 GMT

Thank you.

I tried it and it looks like there are lots of duplicates and for example SysMon fields were as same as to XmlWinEvtLogs and there was no difference between fields within Application and Security for example.

Would you be able to have a look into the SPL and see if it can be optimized, please?

Kind regards,

Dan