topic changing `host` and persisting the result in Splunk Search

changing `host` and persisting the result

mushkevych — Tue, 11 Sep 2018 23:22:47 GMT

I am trying to make this query work:
index="main" | eval host=asset_id | collect index="scanned_app"
where asset_id is a field, not a static value.

Two observations regarding the query:
- without | collect ..., the search shows data as i expect it - with the meta-field host changed
- with | collect ..., the resulting index carries host unchanged from the main index

Q: how do i change the host, so that it can be persisted in another index ?
index="main" | eval *magic_here* | collect index="scanned_app"

Re: changing `host` and persisting the result

adonio — Tue, 11 Sep 2018 23:57:40 GMT

can you elaborate? is asset_id a field or a static value?
also, what is it that you are trying to accomplish? i sense lookup will serve you better here

Re: changing `host` and persisting the result

mushkevych — Wed, 12 Sep 2018 15:53:32 GMT

asset_id is a field.
my goal is to transform the data set by changing host value and persist it in another index.
P.S. Updated the question to reflect your comment

Re: changing `host` and persisting the result

pwild_splunk — Tue, 29 Sep 2020 21:41:32 GMT

Try this

index="main" | eval _raw=_raw.",host=".asset_id | collect index="scanned_app"

Re: changing `host` and persisting the result

pwild_splunk — Tue, 16 Oct 2018 04:36:40 GMT

This may not do what you want. The events in the summary index will contain a host field that is multi-valued, containing the indexed host field as well as the auto-extracted host value. If the purpose of this is to create a dashboard or graph, you may be able to work with the data by removing the first value with something like this.

| eval host=mvindex(host,1)

Re: changing `host` and persisting the result

jbjerke_splunk — Tue, 29 Sep 2020 21:37:44 GMT

You can try and wrap you search in the map command that dynamically let's you generate another search.

This generates an event in the summary index with host=hello set from the outer search.

|makeresults count=1
| eval asset_id="hello"
| map search="
search
index=\"main\"
| collect index=scanned_app host=$asset_id$
"

Re: changing `host` and persisting the result

pwild_splunk — Tue, 29 Sep 2020 21:41:40 GMT

This will work. Remember that the map command, by default, is limited to only 10 sub-search iterations. Use the option maxsearches=10000 or something more appropriate for your data set.

Converting the above to your actual search, see below. You probably don't need "_time=$orig_time$," in the eval.

Re: changing `host` and persisting the result

mushkevych — Tue, 29 Sep 2020 21:37:50 GMT

@jbjerke_splunk and @pwild_splunk thank you for comments. could you perhaps help me understand why the SPL index="main" | eval host=asset_id | collect index="scanned_app" works without |collect... and does not work with |collect... ? What is happening during |collect...?

Re: changing `host` and persisting the result

Vijeta — Tue, 16 Oct 2018 16:31:14 GMT

@mushkevych -Since host is a default field , and collect command will look for default fields for source sourcetype host unless you override it in collect command

Re: changing `host` and persisting the result

mushkevych — Tue, 16 Oct 2018 16:35:15 GMT

@Vijeta thank you for reply. Perhaps you can advise how to override default field host in collect command?

Re: changing `host` and persisting the result

Vijeta — Tue, 29 Sep 2020 21:37:53 GMT

Since you want host value to be assigned to a variable asset_id , you will have to use map command as mentioned by @pwild_splunk

Re: changing `host` and persisting the result

pwild_splunk — Wed, 30 Sep 2020 03:17:46 GMT

To understand why this doesn't work as you're expecting you have to understand how the collect command works. When you pipe a search result into collect, it dumps the output of the command into a text file on your splunk server, which is then picked up by a monitor input for indexing in the same way as any other input. Just like when configuring a monitor input, you can specify the host field once for the input, you can't set it on an event by event basis. With collect, when you define fields like index=A, sourcetype=B, host=C you are defining them in the same way you would in an inputs.conf. Those fields are applied to the output for processing by Splunk's data pipeline.