concatenating fields at search time in props.conf and/or transforms.conf

[bcavagnolo]

I have a bunch of existing regexs that operate on an HTTP URI (E.g., "/foobar?x=1&y=2"). I have logs of two different source types, one in which the URI is reported as a single field, and another in which the query portion of the URI is elswhere in the event (E.g., "/foobar HTTP/1.1 200 x=1&y=2"). I see that I can define an index-time field that concatenates the base URI and its query string. But is there a search-time way to do this?

Edit: Okay. It's starting to sound like this cannot be achieved at searchtime. But because these URLs represent most of my data, I'm really not excited about the bloating this will cause by indexing this field. What's worse, my interpretation of the documentation on how to do this is that I would have to index the field for both of my sourcetypes. That is, I could not maintain an index-time field for one sourcetype and a search-time field for the other sourcetype with the same name.

But Before I give up, let me (re)suggest these three ideas in case anyone in splunkland can see a way to make one of them work:

Idea #1: Is there is a succinct way to apply a transform.conf stanza to multiple SOURCE_KEYs without duplicating the REGEX?

Idea #2: Can two transform.conf stanzas refer to a single REGEX somehow?

Idea #3: Can I define a search-time field in transforms.conf using something like SEDCMD to replace the text intervening between the two field components with an empty string?

[Wilcooley]

Assuming that you are asking about concatenating fields of an individual event, I think you can use FORMAT in transforms.conf:

[my_sourcetype]
REGEX = (\S+)\s+HTTP/\d\.\d\s+\d+\s+(\S+)
FORMAT = myurl::$1/$2

At least, there are examples in both the README and default directories of using extracted bits in the name and value of a single field, such as the auto-kv extraction and this from README/transforms.conf.example:

[session-anonymizer]
REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
FORMAT = $1SessionId=########$2
DEST_KEY = _raw

Oh, but RTFMing, I see in README/transforms.conf.spec:121 for 5.0.3:

• FORMAT for search-time extractions:
  • ...
  • NOTE: You cannot create concatenated fields with FORMAT at search time. That functionality is only available at index time.

So I think I am going to have to change my answer to, "You can't have that wish, my Little Bear."

[bcavagnolo]

@Wilcooley: Yeah that's right.

[Wilcooley]

Just for clarification: You're asking for concatenating the fields of an individual event of the 2nd sourcetype, not concatenating fields between events of different sourcetypes, correct?

[dart]

You can do this with a calculated field, using eval syntax:

### props.conf
[my_sourcetype]
EVAL-concatenated = fielda + fieldb

[bcavagnolo]

The problem is that I need the concatenated field to be available for REPORT-xyz statements that invoke transforms from transforms.conf. The EVAL documentation suggests that only lookups can use the results of EVALs. So I'm afraid this won't work. Please advise if I am missing something or if there is another way. I think I may have to use an index-time field.