Splunk Search

After transaction: "Field '_raw' does not exist in the data"

acdevlin
Communicator

I'm using a transaction to group events within 30 minutes of one another. What I want to do after that is "undo" one of the large events created by the transaction command, so I can then view the data in more detail.

With some searching, I found this thread, which talks about how to split a transaction. Unfortunately, this method is not working for me; when I try nick's suggested approach, I get the error "Field '_raw' does not exist in the data" and the transaction event is not split up.

The full search is long and complicated. The section relevant to my question looks something like this:

... | transaction user maxpause=30m mvlist=t maxevents=-1 | search start_time="06/15/2011 17:21:52"| eval _raw = split(_raw, "\n") | mvexpand _raw | stats values(req_time)

Any ideas? Am I using the _raw field incorrectly?

1 Solution

Lowell
Super Champion

Your "_raw" field wouldn't exist anymore after your | stats values(req_time). Not sure if that's relevant to your entire search or not.

Have you tried using a field name other than _raw (since it has a special purpose) while debugging your search? Sometimes renaming the field and then naming it back lets you get around some internal assumptions built in to some of the search commands.

For example, you could try:

... | eval raw=split(_raw, "\n") | mvexpand raw | rename raw as _raw | ...

View solution in original post

hexx
Splunk Employee
Splunk Employee

Here's an alternative method using rex.

For testing purposes let's associate events from metrics.log in clumps of 4 with :

index=_internal source=*metrics.log | transaction maxevents=4 source

This yields transaction meta-events that look like this :


12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=queue, name=typingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=58, smallest_size=0
12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=realtime_search_data, system total, drop_count=0
12-29-2011 08:51:52.940 -0800 INFO  Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0

To get the original _raw field back (and therefore, display the original events) I use rex with a "delimiter" regex matching the date at the beginning of my event, and then all characters until a CLRF. This creates one multi-value field per transaction containing the pre-transaction values of _raw, which we can then expand back to a single-value field with mvexpand:

index=_internal source=*metrics.log | transaction maxevents=4 source | eval transaction_raw=_raw | rex max_match=1000 "(?msi)^(?[01]\d-[0-3]\d-2011\s[^\r\n]*?)$" | mvexpand raw | eval _raw=raw

Remarks :

  • For the delimiter regex to work, you need to specify max_match with a value equal to or higher than the maximum number of events you expect in your transaction.
  • The regex will need to be reworked if your original events span multiple lines.
  • A transition field (here raw) is necessary. You cannot restore _raw directly unto itself.
  • We conserved the transaction's own _raw in transaction_raw which allows to still report on the transaction results. The transaction special fields are also conserved.

Lowell
Super Champion

Your "_raw" field wouldn't exist anymore after your | stats values(req_time). Not sure if that's relevant to your entire search or not.

Have you tried using a field name other than _raw (since it has a special purpose) while debugging your search? Sometimes renaming the field and then naming it back lets you get around some internal assumptions built in to some of the search commands.

For example, you could try:

... | eval raw=split(_raw, "\n") | mvexpand raw | rename raw as _raw | ...

carasso
Splunk Employee
Splunk Employee

in 4.3, there's mvraw=true, which keeps _raw as a multivalued field

0 Karma

Krishna_R
Path Finder

@acdevlin,

I'm trying the same here (to split a transaction back to original events after filtering on transaction) and I hit the same snag 'split(_raw, "\n")' isn't splitting... Could you please tell which character worked for you?

0 Karma

acdevlin
Communicator

Suddenly realized this morning that the problem was the \n delimiter; changing it to a different character helped break up events from _raw. Kind of embarrassing.

Thanks again Lowell for helping me out!

0 Karma

acdevlin
Communicator

Thanks for the speedy reply Lowell.

Your idea DOES get rid of the error about "_raw"...but it seems the transaction event is not being split up. I still get a single event at the end of the query. Do you have any further suggestions on solving this?

Also, the "| stats values(req_time)" is what I'm trying to get out of the search as a whole: the specific "req_time" values associated with this particular series of events in the transaction. If that at all makes sense.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...