topic Re: Search-time Mask in Getting Data In

Search-time Mask

sdwilkerson — Thu, 27 May 2010 10:17:24 GMT

Some events flow into the Splunk instance via syslog sockets.

For a brief period of time, the sourcetypes that came in via syslog were prepended with the standard syslog code like <142> (so the line would read <142>May 6 ...). This was only for a few days.

Is there a way I can do a built-in search-time extraction to mask the <\d+> prepended to these sourcetypes so that 1. It is not seen and 2. All of the rest of the props/transforms extractions will work as if it wasn't there?

Note: I do not mean to use something like rex in the search-command, but rather, make the change in the splunk configs to mask these characters from any search results.

Sean

Re: Search-time Mask

gkanapathy — Thu, 27 May 2010 11:27:37 GMT

I recommend against it, but you could do:

[mysourcetype]
EXTRACT-out = (?s)^(?:\<\d+\>)?(?<altraw>.*)
FIELDALIAS-raw = altraw as _raw

Update: Oh, but FIELDALIAS runs after all other extractions, so no the other extractions won't work as if it wasn't there. And as you've no doubt found, you can't overwrite _raw with an EXTRACT or REPORT.

Re: Search-time Mask

sdwilkerson — Thu, 27 May 2010 21:00:05 GMT

If sensitive info were accidentally indexed (e.g. PII like SSN#) as part of other logdata that you didn't want any Splunk users to see, would the only solution be to delete, edit(raw) and then reIndex?

Thanks,
Sean

Re: Search-time Mask

gkanapathy — Thu, 27 May 2010 21:33:29 GMT

You can use the | delete command to prevent specific events from being returned in search. (Note it does not reclaim disk space or remove the data from the disk, but simply makes it unretrieveable via Splunk.) Yes, then you can edit the raw data (or set up a TRANSFORM or SEDCMD to edit the data as is comes in and reindex.