Getting Data In

Search-time Mask

sdwilkerson
Contributor

Some events flow into the Splunk instance via syslog sockets.

For a brief period of time, the sourcetypes that came in via syslog were prepended with the standard syslog code like <142> (so the line would read <142>May 6 ...). This was only for a few days.

Is there a way I can do a built-in search-time extraction to mask the <\d+> prepended to these sourcetypes so that 1. It is not seen and 2. All of the rest of the props/transforms extractions will work as if it wasn't there?

Note: I do not mean to use something like rex in the search-command, but rather, make the change in the splunk configs to mask these characters from any search results.

Sean

Tags (2)

gkanapathy
Splunk Employee
Splunk Employee

I recommend against it, but you could do:

[mysourcetype]
EXTRACT-out = (?s)^(?:\<\d+\>)?(?<altraw>.*)
FIELDALIAS-raw = altraw as _raw

Update: Oh, but FIELDALIAS runs after all other extractions, so no the other extractions won't work as if it wasn't there. And as you've no doubt found, you can't overwrite _raw with an EXTRACT or REPORT.

gkanapathy
Splunk Employee
Splunk Employee

You can use the | delete command to prevent specific events from being returned in search. (Note it does not reclaim disk space or remove the data from the disk, but simply makes it unretrieveable via Splunk.) Yes, then you can edit the raw data (or set up a TRANSFORM or SEDCMD to edit the data as is comes in and reindex.

0 Karma

sdwilkerson
Contributor

If sensitive info were accidentally indexed (e.g. PII like SSN#) as part of other logdata that you didn't want any Splunk users to see, would the only solution be to delete, edit(raw) and then reIndex?

Thanks,
Sean

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!