Other Usage

How to collect (with | collect) the output of a search in a new sourcetype created dynamically within the search itself?

Gian89
Explorer

Hello community,

on my desk, I have a pretty edgy request that is giving me quite a headache.

I would need to collect (with | collect) the output of a search in a new sourcetype created dynamically within the search itself.

Here you can find a simple ad hoc example:

| makeresults 
| eval letter1="A", letter2="B", letter3="C"
| eval variabile="NewSourcetype"
| eval _raw=_time + ": " + _raw
| collect index=garbage sourcetype=variabile

Problem is that the event is stored under  sourcetype=variabile instead of sourcetype=NewSourcetype.
Any idea how to manage such a situation?

Thanks in advance for your kind support.

Labels (1)
0 Karma
1 Solution

Gian89
Explorer

Not the best approach but it is working:

| makeresults
| eval letter1="A", letter2="B", letter3="C"
| append
[| makeresults
| eval letter1="D", letter2="E", letter3="F"]
| eval _raw=_time + ": " + _raw
| appendpipe
[| where letter1="A"
| collect index="trash" sourcetype=testA
| where false() ]
| appendpipe
[| where letter1="D"
| collect index="trash" sourcetype=testD
| where false()

Originally posted in this thread:
https://community.splunk.com/t5/Knowledge-Management/collect-index-quot-based-on-values-quot/m-p/473...

 

View solution in original post

0 Karma

mbjerkeland_spl
Splunk Employee
Splunk Employee

In case @PickleRick 's suggestion wasn't clear, you can do this:

| makeresults count=5
| eval n=(random() % 10)
| eval sourcetype="something" . n
| fields - n
| collect index=your_summary_index output_format=hec

 It will respect the sourcetype set, in this case a value between something0 to something9

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK, if you read the docs for the collect command, you come across this passage:

[...]

output_format

Syntax: output_format=[raw | hec]

Description: Specifies the output format for the summary indexing. If set to raw, uses the traditional non-structured log style summary indexing stash output format.If set to hec, it generates HTTP Event Collector (HEC) JSON formatted output:

  • All fields are automatically indexed when the stash file is indexed.
  • The file that is written to the var/spool/splunk path ends in .stash_hec instead of .stash.
  • Allows the source, sourcetype, and host from the original data to be used directly in the summary index. Does not re-map these fields to the extract_host/extracted_sourcetype/... path.
  • The index and splunk_server fields in the original data are ignored.
  • You cannot use the addtime, host, marker, source, or the sourcetype options when output_format=hec.

[...]

So you should format the event before calling the collect, setting up proper host/source/sourcetype values (which in this case can be evaluated dynamically as any other field). And then just collect with output_format=hec.

 

Gian89
Explorer

Not the best approach but it is working:

| makeresults
| eval letter1="A", letter2="B", letter3="C"
| append
[| makeresults
| eval letter1="D", letter2="E", letter3="F"]
| eval _raw=_time + ": " + _raw
| appendpipe
[| where letter1="A"
| collect index="trash" sourcetype=testA
| where false() ]
| appendpipe
[| where letter1="D"
| collect index="trash" sourcetype=testD
| where false()

Originally posted in this thread:
https://community.splunk.com/t5/Knowledge-Management/collect-index-quot-based-on-values-quot/m-p/473...

 

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...