Hi All,
I need help building a SPL that would return all available fields mapped to their sourcetypes/source
Looking across all Indexers crawling through all indexes index=*
I currently use to strip off all the fields and their extracted fields but I have no idea where they are coming from, what is their sourcetype and source:
index=*
fieldsummary
| search values!="[]"
| rex field=values max_match=0 "\{\"value\":\"(?<extracted_values>[^\"]+)\""
| fields field extracted_values
Thank you!
You could start with something like this
index=*
| fields - _time _raw
| foreach *
[| eval <<FIELD>>=if("<<FIELD>>"=="index",index,sourcetype)]
| table *
| fillnull value="N/A"
| foreach *
[eval sourcetype=if("<<FIELD>>"!="sourcetype",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype
| table sourcetype
It may fail due to the amount of data being brought back, so you might want to break it up by index. Also, it works by looking at the fields returned in the events, so if some fields are not used in the time period covered, they will not show up, so you might want to run it a different times of the day, rather than for longer periods.
You could start with something like this
index=*
| fields - _time _raw
| foreach *
[| eval <<FIELD>>=if("<<FIELD>>"=="index",index,sourcetype)]
| table *
| fillnull value="N/A"
| foreach *
[eval sourcetype=if("<<FIELD>>"!="sourcetype",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype
| table sourcetype
It may fail due to the amount of data being brought back, so you might want to break it up by index. Also, it works by looking at the fields returned in the events, so if some fields are not used in the time period covered, they will not show up, so you might want to run it a different times of the day, rather than for longer periods.
Thanks for the reply @ITWhisperer
Would it be possible to help me out creating something similar which can include both source and sourcetype, please?
Thank you!
Try something like this
index=*
| fields - _time _raw
| foreach *
[| eval <<FIELD>>=if("<<FIELD>>"=="index",index,if("<<FIELD>>"=="source",source,sourcetype))]
| table *
| fillnull value="N/A"
| foreach *
[eval sourcetype=if("<<FIELD>>"!="sourcetype" AND "<<FIELD>>"!="source" AND "<<FIELD>>"!="index",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype
| table index source sourcetype
Thank you.
I tried it and it looks like there are lots of duplicates and for example SysMon fields were as same as to XmlWinEvtLogs and there was no difference between fields within Application and Security for example.
Would you be able to have a look into the SPL and see if it can be optimized, please?
Kind regards,
Dan