Getting Data In

Map fields to sourcetype

DanAlexander
Communicator

Hi All,

I need help building a SPL that would return all available fields mapped to their sourcetypes/source 

Looking across all Indexers crawling through all indexes index=*

I currently use to strip off all the fields and their extracted fields but I have no idea where they are coming from, what is their sourcetype and source:

index=*

fieldsummary

| search values!="[]"

| rex field=values max_match=0 "\{\"value\":\"(?<extracted_values>[^\"]+)\""

| fields field extracted_values

Thank you!

0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

You could start with something like this

index=*
| fields - _time _raw
| foreach *
    [| eval <<FIELD>>=if("<<FIELD>>"=="index",index,sourcetype)]
| table *
| fillnull value="N/A"
| foreach *
    [eval sourcetype=if("<<FIELD>>"!="sourcetype",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype
| table sourcetype

It may fail due to the amount of data being brought back, so you might want to break it up by index. Also, it works by looking at the fields returned in the events, so if some fields are not used in the time period covered, they will not show up, so you might want to run it a different times of the day, rather than for longer periods.

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

You could start with something like this

index=*
| fields - _time _raw
| foreach *
    [| eval <<FIELD>>=if("<<FIELD>>"=="index",index,sourcetype)]
| table *
| fillnull value="N/A"
| foreach *
    [eval sourcetype=if("<<FIELD>>"!="sourcetype",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype
| table sourcetype

It may fail due to the amount of data being brought back, so you might want to break it up by index. Also, it works by looking at the fields returned in the events, so if some fields are not used in the time period covered, they will not show up, so you might want to run it a different times of the day, rather than for longer periods.

DanAlexander
Communicator

Thanks for the reply @ITWhisperer 

Would it be possible to help me out creating something similar which can include both source and sourcetype, please?

Thank you!

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try something like this

index=*
| fields - _time _raw
| foreach *
    [| eval <<FIELD>>=if("<<FIELD>>"=="index",index,if("<<FIELD>>"=="source",source,sourcetype))]
| table *
| fillnull value="N/A"
| foreach *
    [eval sourcetype=if("<<FIELD>>"!="sourcetype" AND "<<FIELD>>"!="source" AND "<<FIELD>>"!="index",if('<<FIELD>>'!="N/A",mvappend(sourcetype,"<<FIELD>>"),sourcetype),sourcetype)]
| dedup sourcetype 
| table index source sourcetype
0 Karma

DanAlexander
Communicator

Thank you.

I tried it and it looks like there are lots of duplicates and for example SysMon fields were as same as to XmlWinEvtLogs and there was no difference between fields within Application and Security for example.

Would you be able to have a look into the SPL and see if it can be optimized, please?

Kind regards,

Dan

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...