Knowledge Management
Highlighted

Getting Markers to show up as fields for summary index

Path Finder

The collect command has a marker option which can be "A string, usually of key-value pairs, to append to each event written out. Optional, default is empty.". However, I can't find any good examples of where markers are used and parseable. I've tried comma and space delimited key value pairs, but I can't get those parsed in the index search results. As an example,

source="/log/sources/myvpn.log" | geoip REMOTE_IP | eval country_source=if(REMOTE_IP_country_code="US","domestic","international") | bucket _time span=1h | stats count by _time,vpn_action,country_source | collect index=mysummary marker="summary_type=vpn summary_span=3600 summary_method=bucket"

is an attempt to collect hourly stats on VPN connects and disconnects by country. I've not been able to find an incantation that would let me use things like search index=mysummary summary_type=vpn to pull out VPN-related summary index stats. I see the key-value pairs in the result, but they're not parsed as fields. What am I missing here in terms of how to construct the marker argument to collect?

Tags (2)
0 Karma
Highlighted

Re: Getting Markers to show up as fields for summary index

Path Finder

After a moderate amount of experimentation, I found what works, which is that there has to be a comma and a space between the key=value pairs. And if the value contains spaces or commas, it needs to be escape quoted. And the stream should have an | addinfo included for the manual population of the index (backfill):

... | stats count by _time,vpn_action,country_source | addinfo | collect index=mysummary marker="summary_type=vpn, summary_span=3600, summary_method=bucket, search_name=\"vpn starts and stops\""

Using a search of this form gives me fields I can use when I do a search (e.g. search index=mysummary). The | addinfo ensures that the search results contain fields that specify when the search was run to populate these particular index values. I think that gets added automatically for scheduled searches. But if you're doing a manual search to backfill the index, throw the | addinfo into the stream before the collect.

In the category of "error between keyboard and monitor", I thought I'd tried this before posting a question, but I hadn't (quite). The space after the comma is necessary. I do wonder why space-separated doesn't work, since space separated key=value pairs are automatically parsed at search time for _raw fields otherwise.