Getting Data In

How to efficiently push data to Splunk

Valentin
Engager

Dear Splunk community,

I have a Python application that pushes data to Splunk every time is executed. Multiple events are pushed using JSON format. Only a subset of the data being sent, namely two fields are changing during job execution, the rest are constant per job execution (think of them as some sort of job metadata). I would like to have that metadata in splunk so I can filter it, but I do not like also pushing lots of identical data for each event.  I guess what I am looking for is some sort of bulk tagging after each import where each job metadata field would be a label.  

I appreciate any thoughts/suggestions how to do this usinng splunk BKMs.

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You probably could associate some INGEST_EVAL settings with the appropriate sourcetype or source so Splunk will automatically add fields to the events, but it's far easier to have the Python app continue to do it.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You probably could associate some INGEST_EVAL settings with the appropriate sourcetype or source so Splunk will automatically add fields to the events, but it's far easier to have the Python app continue to do it.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

Valentin
Engager

If I use INGEST_EVAL, is Splunk going to literally add those fields to each event, or is it going to do some internal JOIN? What I want to avoid is the fact that some metadata fields like `a-quite-long-string-that-really-does-not-change` is actually copied to each event. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, INGEST_EVAL adds the result as a field to each event.

Bear in mind that any field that is NOT in an event cannot be used as a filter on that event.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Valentin
Engager

thank you. it seems I have no other choice than adding the metadata in the events.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!