topic Re: How to set the source in a collect command to a variable? in Getting Data In

How to set the source in a collect command to a variable?

creiglow — Sun, 03 Nov 2019 15:12:41 GMT

I am working with the collect command an want to set the source to a variable, not a string.

| eval myDynamicSource=index."-".sourcetype
| collect index=garbage source=myDynamicSource

However, the source in the collect is always just the name of the variable even if I use no quotes, double quotes or single quotes.

Any idea what I am missing?

Thanks,
--Keith

Re: How to set the source in a collect command to a variable?

arjunpkishore5 — Sun, 03 Nov 2019 23:53:44 GMT

To my knowledge, collect operates on the entire Input, as in, it's not a streaming command. This is why if you cancel the search(with a collect statement at the end) before it has completed, none of the records are saved. So you cannot dynamically change the metadata directly on the collect, but you can do a workaround using props and transforms. here is how.

your collect remains as is | collect index=garbage source=myDynamicSource

In transforms.conf, you have this. In this, you're extracting the value of myDynamicSource and assigning it to the metadata field source

[setDynamicSource]
REGEX = (myDynamicSource)=(.+)
FORMAT = source::$2
DEST_KEY= MetaData:Source

In props.conf, you just apply the above transform to the source myDynamicSource

[source::myDynamicSource]
TRANSFORMS-set_source = setDynamicSource

Hope this helps.

Cheers

Re: How to set the source in a collect command to a variable?

frankfowler — Thu, 10 Sep 2020 22:53:56 GMT

I dont see this as working :

I have transforms.conf and props.conf set as you stated above in my etc/system/local area.

my search is as follows :

index=xxx | eval myDynamicSource=source | collect index=xxx source=myDynamicSource

i see the source still set to myDynamicSource.

what am i missing ?