topic Re: Optimizing query designed to retrieve source and field names only in Getting Data In

Optimizing query designed to retrieve source and field names only

jsam019 — Sat, 14 Mar 2020 18:23:01 GMT

Using the REST api, I am currently retrieving a set of events from Splunk and extracting all of the field names and log sources, simultaneously building a map of log sources and fields belonging to them. Is there any way that I can retrieve this data with a minimal payload? For example, if I pull back 1 record that is from LogSource1 and has Property1 equal to [some really long string], I really don't want that whole string back. I just need to consume LogSource1 and Property1. I'm open to any ideas.

Re: Optimizing query designed to retrieve source and field names only

woodcock — Sat, 14 Mar 2020 19:54:35 GMT

Where is Splunk in this? The source of the data? The destination of the data? You have told us almost nothing. You need to try again and give ALL the details.

Re: Optimizing query designed to retrieve source and field names only

jsam019 — Sat, 14 Mar 2020 19:56:40 GMT

...it is the splunk REST api, sir. That is where the events are located.

Re: Optimizing query designed to retrieve source and field names only

woodcock — Sat, 14 Mar 2020 20:15:33 GMT

That is not all the details; that is just one.

Re: Optimizing query designed to retrieve source and field names only

jsam019 — Sat, 14 Mar 2020 21:15:40 GMT

I'm looking for suggestions to optimally retrieve event data via splunk's API aside from loading the entire event. I currently send basic SPL queries with a time range and pull out the fields and sources I see.. that results in gigantic payloads which I extract only those 2 pieces of data. I'm not sure what else needs to be clarified. I know about the field summary option, but that doesn't give me the log sources used for each field.

Re: Optimizing query designed to retrieve source and field names only

woodcock — Sun, 15 Mar 2020 01:23:25 GMT

You can always end your SPL with | table Just the fields I need.

Re: Optimizing query designed to retrieve source and field names only

jsam019 — Sun, 15 Mar 2020 21:53:53 GMT

The issue is that I don't know what fields are available since we have several log sources.

Re: Optimizing query designed to retrieve source and field names only

to4kawa — Sun, 15 Mar 2020 22:05:14 GMT

your search
| fieldsummary

try this and check your fields.

Re: Optimizing query designed to retrieve source and field names only

jsam019 — Sun, 15 Mar 2020 22:15:30 GMT

@to4kawa I've alluded to that already. The issue is that it won't indicate which sources contained the fields.

Re: Optimizing query designed to retrieve source and field names only

to4kawa — Sun, 15 Mar 2020 22:22:57 GMT

after searching, select source from left side extract fields
and then, check your fields again.

Re: Optimizing query designed to retrieve source and field names only

woodcock — Sun, 15 Mar 2020 22:46:27 GMT

You are making this impossible. You need to back all the way and explain the problem FULLY and clearly.

Re: Optimizing query designed to retrieve source and field names only

jsam019 — Sun, 15 Mar 2020 22:56:50 GMT

Thanks @to4kawa - I'm not sure what the SPL looks like for this but I'll try to play around with this. In the end, I want to be able to tell senior mgmt "here are the 10 fields we have, and these 2 are from source 1 while these 2 come from source 2" for today" so this seems to be closer to what I'm looking for.

Re: Optimizing query designed to retrieve source and field names only

to4kawa — Mon, 16 Mar 2020 21:08:27 GMT

source="A" |table * | foreach * [ eval <<FIELD>>="sourceA" ]  |append [ search source="B" | table * | foreach * [ eval <<FIELD>>="sourceB" ]  ] |stats values(*) as * | transpose 0 | where mvcount('row 1')=1

This query shows the fields from only one source. How about this?