Other Usage

How to collect (with | collect) the output of a search in a new sourcetype created dynamically within the search itself?

Gian89
Explorer

Hello community,

on my desk, I have a pretty edgy request that is giving me quite a headache.

I would need to collect (with | collect) the output of a search in a new sourcetype created dynamically within the search itself.

Here you can find a simple ad hoc example:

| makeresults 
| eval letter1="A", letter2="B", letter3="C"
| eval variabile="NewSourcetype"
| eval _raw=_time + ": " + _raw
| collect index=garbage sourcetype=variabile

Problem is that the event is stored under  sourcetype=variabile instead of sourcetype=NewSourcetype.
Any idea how to manage such a situation?

Thanks in advance for your kind support.

Labels (1)
0 Karma
1 Solution

Gian89
Explorer

Not the best approach but it is working:

| makeresults
| eval letter1="A", letter2="B", letter3="C"
| append
[| makeresults
| eval letter1="D", letter2="E", letter3="F"]
| eval _raw=_time + ": " + _raw
| appendpipe
[| where letter1="A"
| collect index="trash" sourcetype=testA
| where false() ]
| appendpipe
[| where letter1="D"
| collect index="trash" sourcetype=testD
| where false()

Originally posted in this thread:
https://community.splunk.com/t5/Knowledge-Management/collect-index-quot-based-on-values-quot/m-p/473...

 

View solution in original post

0 Karma

mbjerkeland_spl
Splunk Employee
Splunk Employee

In case @PickleRick 's suggestion wasn't clear, you can do this:

| makeresults count=5
| eval n=(random() % 10)
| eval sourcetype="something" . n
| fields - n
| collect index=your_summary_index output_format=hec

 It will respect the sourcetype set, in this case a value between something0 to something9

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK, if you read the docs for the collect command, you come across this passage:

[...]

output_format

Syntax: output_format=[raw | hec]

Description: Specifies the output format for the summary indexing. If set to raw, uses the traditional non-structured log style summary indexing stash output format.If set to hec, it generates HTTP Event Collector (HEC) JSON formatted output:

  • All fields are automatically indexed when the stash file is indexed.
  • The file that is written to the var/spool/splunk path ends in .stash_hec instead of .stash.
  • Allows the source, sourcetype, and host from the original data to be used directly in the summary index. Does not re-map these fields to the extract_host/extracted_sourcetype/... path.
  • The index and splunk_server fields in the original data are ignored.
  • You cannot use the addtime, host, marker, source, or the sourcetype options when output_format=hec.

[...]

So you should format the event before calling the collect, setting up proper host/source/sourcetype values (which in this case can be evaluated dynamically as any other field). And then just collect with output_format=hec.

 

Gian89
Explorer

Not the best approach but it is working:

| makeresults
| eval letter1="A", letter2="B", letter3="C"
| append
[| makeresults
| eval letter1="D", letter2="E", letter3="F"]
| eval _raw=_time + ": " + _raw
| appendpipe
[| where letter1="A"
| collect index="trash" sourcetype=testA
| where false() ]
| appendpipe
[| where letter1="D"
| collect index="trash" sourcetype=testD
| where false()

Originally posted in this thread:
https://community.splunk.com/t5/Knowledge-Management/collect-index-quot-based-on-values-quot/m-p/473...

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...