topic Does each Splunk event have a unique identifier? in Splunk Search

Does each Splunk event have a unique identifier?

araitz — Fri, 15 Jan 2010 02:52:14 GMT

I would like to tag some specific events to group them together for incident response and forensics purposes. Is this possible with Splunk?

Re: Does each Splunk event have a unique identifier?

Stephen_Sorkin — Fri, 15 Jan 2010 02:58:59 GMT

No, it is not yet.

Each event does have a unique id, the tuple (splunk_server, index, _cd), but "_cd" is not searchable (only filterable). You could use lookup tables to map this to a tag or key.

When we make _cd searchable, that will allow searching on the tags or groups.

Re: Does each Splunk event have a unique identifier?

deloach — Wed, 10 Aug 2011 17:01:58 GMT

Any updates on being able to tag specific events or time frame when this functionality might be available?

Re: Does each Splunk event have a unique identifier?

pembleton — Thu, 11 Jul 2013 16:22:18 GMT

any update?

Re: Does each Splunk event have a unique identifier?

mikaelbje — Tue, 21 Jul 2015 12:34:32 GMT

Update for Splunk 6.2.1.

_cd is still not searchable after 5 years. I suggest using the following method which calculates a hash based on the raw event:

your search here | eval id=md5(_raw) | id="VALUE_YOU_ARE_LOOKING_FOR"

Re: Does each Splunk event have a unique identifier?

thisissplunk — Tue, 29 Sep 2020 09:30:38 GMT

I've found that you can at least access _cd for a stats if you do a rename first:

| rename _cd as unique_id
| stats count by unique_id

If you want to search on that data, you can do this:

index=awesome sourcetype=woah
rename _cd as unique_id
| search unique_id=9320:49207386

Re: Does each Splunk event have a unique identifier?

christianhuber — Wed, 14 Jun 2017 13:23:07 GMT

I downvoted this post because there is a newer and better answer

Re: Does each Splunk event have a unique identifier?

koshyk — Fri, 23 Jun 2017 09:49:35 GMT

I feel it is unfair to downvote an answer after 7+ years. (may be at that time, it was the only solution.).

Re: Does each Splunk event have a unique identifier?

Lowell — Tue, 29 Sep 2020 14:34:37 GMT

So there's been some movement on this question in the recent weeks, so I'll drop in and give some commentary. Hopefully this helps someone. Bottom line, there still isn't a really good answer to this, as far as I know. Most of when I hear this question asked, it's because of a misunderstanding of what Splunk does and how it works. If you really need a feature like this, open a splunk support ticket and explain your use case (that's how new features like this get introduced.) Otherwise, enjoy the following weeds:

Before going much further let me point out that Sorkin's answer was from before Splunk indexer clustering, and therefore won't work reliably in on clustered environments today. The answer with the next highest points (mikaelbje) shows an id based on a hash of the event's raw text. But there's no fundamental guarantee of uniqueness for _raw (e.g., the same message could repeat multiple time per second, or data could be ingested twice, ...), and more importantly, there's no fast way to search on it. If you're trying to pick out one event out of a few thousand, then this is probably acceptable, but it does not scale. (This is because Splunk has to pull back every event from disk, then calculate the checksum, and then compare it. There's no fast index operations involved, therefore it will be slow.)

So there's a couple issues that index clustering bring to the mix that changes things dramatically. First, with bucket replication, there's no guarantee that the same indexer will always be returning the same event in the future, it could easily be handled off to another peer who has a replicated copy. (Also note that splunk_server represents the current serverName of that indexer, which may be different from index time.) So Splunk introduced the _bkt field (around the same time as indexer clustering, if I remember correctly.) And it always returns the same bucket name, even if the hostname changes (because it uses a GUID not a hostname). This works even if the original host is decommissioned. (Yes, that still breaks if the index name changes.)

So the modern equivalent of (splunk_server, index, _cd) tuple, is now (_bkt, _cd).

And fortunately, _btk is always available and consistent, even if you're not on a cluster. (Although in that case the GUID just represents the current GUID of the server, not the GUID embedded in the bucket's name.) The _btk field is composition of (1) index name, (2) simple bucket id ( or "local id", a simple incrementing integer), and (3) GUID of the initial bucket creator. And, as always, _cd is a combination of (1) bucket id (integer only), and (2) the internal event number for that bucket. (If you look under the covers, this is the id that Splunk uses to "delete", aka hide, events. Notr: It's unclear to me if the event id (stored in_cd) is perstient across a bucket rebuild (if thawed after being frozen, or if passed through exporttool/importtool for some other reasons.) Again, it's probably not best to rely on this mechanism.)

But as of Splunk 6.5, you can't search on "_cd" in the base search, and while searching for "_bkt" in the base search works, according to LISPY, it tries to find it as a raw string, which sure doesn't look efficient. (Also makes me wonder if fields.conf is setup wrong.... another research problem for another day...)

Bottom line, if this is a feature that you actually need, file an enhancement request. If you just need something in Splunk with a unique and consistent key, take a look at the KV store--they say it's magic!

Re: Does each Splunk event have a unique identifier?

tcgerhard — Wed, 16 Aug 2017 02:16:56 GMT

My use case is that I want to select an event from a list and display more details about the event in another panel via drilldown. Some sort of ID (and ideally, indexed) would make it straightforward.

Re: Does each Splunk event have a unique identifier?

dangsecurity — Tue, 02 Jan 2018 16:20:30 GMT

Why didn't you link to the "newer and better answer" so others can benefit as well?