Map fields to sourcetype

[DanAlexander]

Hi All,

I need help building a SPL that would return all available fields mapped to their sourcetypes/source 

Looking across all Indexers crawling through all indexes index=*

I currently use to strip off all the fields and their extracted fields but I have no idea where they are coming from, what is their sourcetype and source:

index=*

fieldsummary

| search values!="[]"

| rex field=values max_match=0 "\{\"value\":\"(?<extracted_values>[^\"]+)\""

| fields field extracted_values

Thank you!

[ITWhisperer · Accepted solution]

You could start with something like this

index=*
| fields - _time _raw
| foreach *
    [| eval <<FIELD>>=if("<<FIELD>>"=="index",index,sourcetype)]
| table *
| fillnull value="N/A"
| foreach *
    [eval sourcetype=if("<<FIELD>>"!="sourcetype",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype
| table sourcetype

It may fail due to the amount of data being brought back, so you might want to break it up by index. Also, it works by looking at the fields returned in the events, so if some fields are not used in the time period covered, they will not show up, so you might want to run it a different times of the day, rather than for longer periods.

[DanAlexander]

Thanks for the reply @ITWhisperer 

Would it be possible to help me out creating something similar which can include both source and sourcetype, please?

Thank you!

[ITWhisperer]

Try something like this

index=*
| fields - _time _raw
| foreach *
    [| eval <<FIELD>>=if("<<FIELD>>"=="index",index,if("<<FIELD>>"=="source",source,sourcetype))]
| table *
| fillnull value="N/A"
| foreach *
    [eval sourcetype=if("<<FIELD>>"!="sourcetype" AND "<<FIELD>>"!="source" AND "<<FIELD>>"!="index",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype 
| table index source sourcetype

[DanAlexander]

Thank you.

I tried it and it looks like there are lots of duplicates and for example SysMon fields were as same as to XmlWinEvtLogs and there was no difference between fields within Application and Security for example.

Would you be able to have a look into the SPL and see if it can be optimized, please?

Kind regards,

Dan