Knowledge Management

Getting Markers to show up as fields for summary index

usethedata
Path Finder

The collect command has a marker option which can be "A string, usually of key-value pairs, to append to each event written out. Optional, default is empty.". However, I can't find any good examples of where markers are used and parseable. I've tried comma and space delimited key value pairs, but I can't get those parsed in the index search results. As an example,

source="/log/sources/myvpn.log" | geoip REMOTE_IP | eval country_source=if(REMOTE_IP_country_code="US","domestic","international") | bucket _time span=1h | stats count by _time,vpn_action,country_source | collect index=mysummary marker="summary_type=vpn summary_span=3600 summary_method=bucket"

is an attempt to collect hourly stats on VPN connects and disconnects by country. I've not been able to find an incantation that would let me use things like search index=mysummary summary_type=vpn to pull out VPN-related summary index stats. I see the key-value pairs in the result, but they're not parsed as fields. What am I missing here in terms of how to construct the marker argument to collect?

Tags (2)
0 Karma

usethedata
Path Finder

After a moderate amount of experimentation, I found what works, which is that there has to be a comma and a space between the key=value pairs. And if the value contains spaces or commas, it needs to be escape quoted. And the stream should have an | addinfo included for the manual population of the index (backfill):

... | stats count by _time,vpn_action,country_source | addinfo | collect index=mysummary marker="summary_type=vpn, summary_span=3600, summary_method=bucket, search_name=\"vpn starts and stops\""

Using a search of this form gives me fields I can use when I do a search (e.g. search index=mysummary). The | addinfo ensures that the search results contain fields that specify when the search was run to populate these particular index values. I think that gets added automatically for scheduled searches. But if you're doing a manual search to backfill the index, throw the | addinfo into the stream before the collect.

In the category of "error between keyboard and monitor", I thought I'd tried this before posting a question, but I hadn't (quite). The space after the comma is necessary. I do wonder why space-separated doesn't work, since space separated key=value pairs are automatically parsed at search time for _raw fields otherwise.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...