Indexing the results of a search

[beano501]

All,

I'm ingesting data from Azure that contains (as part of it) a syslog message, I have the vendor specific application for this syslog message format.

Simplified structure below.

{
   CollectorHostName: xxxxx
   Computer: xxxxx
   EventTime: 2025-06-27T07:19:45Z
   Facility: local7
   HostIP: xx.xx.xx.xx
   SyslogMessage: logver=704072731 timestamp=1750983585 devname="xxx" devid="xxx" vd="root" date=2025-06-27 time=00:19:45 eventtime=1751008785866135964 tz="-0700" logid="0100032002" type="event" subtype="system" level="alert" ........."
}

 What I would ideally like to be able to do, is extract (via a search) the "SyslogMessage" field and then re-index this into a new index (and appropriate sourcetype) so that the syslog message can be processed in "the normal way" by the vender specific application. 

Does anyone know how I can achieve this? Many thanks in advance.

[gcusello]

Hi @beano501 ,

as also @ITWhisperer hinted, the best solution is scheduling a search that extracts only the fields you need and storage results in a summary index using the collect command.

Only one attention: you cannot use a sourcetype different than "stash" because, otherwise you have to pay twice the license.

Ciao.

Giuseppe

[beano501]

Thanks for the responses, I had only really considered using "summary indexes" as part of the usual summary index commands sitimechart etc.

What I have got working is

index=xxxxx sourcetype="mscs:kql"
| eval _raw = SyslogMessage
| fields _raw
| collect index=main sourcetype=fortigate_event run_in_preview=true

Which achieves what I am after. I appreciate this approach would impact licensing, but it will be low volume. 

Thanks again

[gcusello]

Hi @beano501 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[ITWhisperer · Accepted solution]

As @gcusello says, with summary indexing, you have to bear in mind the licensing costs, and it partly depends on the sourcetype (stash vs non-stash), but it also depends on ingest- or workload- -based licensing model - for more details see here 

[ITWhisperer]

Have you considered using a summary index to hold the extracted field(s)?