Splunk Search

Use rex to strip certain characters from fields

bowesmana
SplunkTrust
SplunkTrust

Unicode punctuation characters U+2000 to U+206f seem to make Splunk want to put the requirement for Simplified Chinese fonts in exported PDFs, so I want to convert these characters to ASCII equivalents.

I can add the following to the search command

rex field=Course mode=sed "s/(‘|’)/'/g"

where the replacement chars above are U+2018 and U+2019 and they are replaced with 0x27, but I want to put something in props.conf to force it to happen always.

How would I do this?

Tags (2)
0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Yes, but keep in mind this is an index time function, so it will change indexed data on the way in... permanently.

[yoursourcetype]

sedcmd-course = s/(‘|’)/'/g

You can read about it HERE
and I have excerpted below:
SEDCMD- = 
* Only used at index time.
* Commonly used to anonymize incoming data at index time, such as credit card or social
  security numbers. For more information, search the online documentation for "anonymize
  data."
* Used to specify a sed script which Splunk applies to the _raw field.
* A sed script is a space-separated list of sed commands. Currently the following subset of
  sed commands is supported:
        * replace (s) and character substitution (y).
* Syntax:
        * replace - s/regex/replacement/flags
                * regex is a perl regular expression (optionally containing capturing groups).
                * replacement is a string to replace the regex match. Use \n for backreferences,
                  where "n" is a single digit.
                * flags can be either: g to replace all matches, or a number to replace a specified
                  match.
        * substitute - y/string1/string2/
                * substitutes the string1[i] with string2[i]
With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...

Skip the Awkward Silence: Have a .conf-ersation at .conf26

Picture this. You arrive at .conf26 already having your socializing and networking plans mapped out. No ...

Rethinking Zero Trust: From Product Purchases to Logical Control Evidence

Implementing Zero Trust (ZT) across complex environments often falters at the very beginning due to a ...