Python SDK: StreamingCommand only returns data in fields where fields are in the first record.

[harrison_tamu]

I'm writing a search command using the Splunk Python SDK to pull in data from an external API into search results. The goal is to add fields to each record based on the data returned from the API. Example: search ... | CUSTOM_COMMAND source_ip outputs the search results with enriched data from the API.

The external API returns different fields based on the query; for example, one query could return fields A, B, and C, but another query could only return fields A and B. Due to this, different records could have different fields. I make the Splunk field name whatever the key of the API data is. For example, if the API returns {'keyA': 'valueA', 'keyC': 'valueC'}, then new fields called keyA and keyC will be added to the Splunk record and returned to the search.

Here is the issue... it appears that if Splunk doesn't see a key in the first result, it won't show that key for any of the later results even if a value was added to that key. If the first record is returned where fields keyA and keyC added from the external API call, then I'll be able to see any other records below that have values for keyA and keyC. However, if there is a record later down the search results where a value is added to a field named keyB, the value will not be displayed in the results; keyB will be blank for all results unless there is some value for keyB in the first record. If I manually add some junk value to keyB in the first record, all records below that are supposed to have a value for keyB will display that value.

I've been operating under the assumption that Splunk doesn't really care about records having different fields, but I'm not too sure what to think of this... Am I misunderstanding something about how Splunk operates? Please let me know what I can clarify.

[douglasmsouza]

Hi @harrison_tamu, did you solve this problem?

I'm, having the same issue here.

[harrison_tamu]

@douglasmsouza I have not found any great solution. What I'm currently doing is just making sure that every record returned has every possible field even if the field is blank. It's not really a fix and it feels wrong, but that's the most reasonable thing I came up with.

[douglasmsouza]

@harrison_tamu I ended up doing it in a similar way. Adding every possible fields just on the first record worked for me.