SEDCMD not working as designed

[jslamcle]

We have a SEDCMD masking a field that correctly masks data as shown in the event however in the expanded info on the event it is not masked.  Anyone seen this before?  Working with Proofpoint logs. 

[PickleRick]

E_NOT_ENOUGH_INFO

We don't know your setup. We don't know how you're ingesting data, we don't know where you have your SEDCMD defined. We don't know what your effective config is.

Did you verify your sed syntax? Did you do the btool? Did you put the props.conf in the proper place?

[tscroggins]

Hi @jslamcle,

If the input includes indexed extractions, either in a monitor stanza or in a modular input script, then SEDCMD wouldn't mask the extracted field values; it would only mask the _raw value.

E.g., given /tmp/foo.json:

{"foo":"bar"}
{"foo":"baz"}

and:

# inputs.conf
[monitor:///tmp/foo.json]
sourcetype = foo_json
INDEXED_EXTRACTIONS = json

# props.conf
[foo_json]
SEDCMD-foo = s/"foo":"[^"]+"/"foo":""/

_raw will be indexed as:

{"foo":""}
{"foo":""}

but the events will have indexed values of foo=bar and foo=baz, respectively:

[jslamcle]

Does it have to do with the highlighted parameter INDEXED_EXTRACTIONS, 

Example:Isolation:Web doesn’t have any SEDCMDs

[Example:Isolation:Web]

EVAL-vendor_region = lower('region'."-".'zone')

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_1 = userName AS user

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_2 = disposition AS action

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_4 = categories{} AS category

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_5 = fileName AS file_name

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_6 = fileSize AS file_size

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_7 = fileMimeType AS http_content_type

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_8 = parentPageURL AS http_referrer

FIELDALIAS-aob_gen_Example_Isolation_Web_alias_9 = classification AS type

INDEXED_EXTRACTIONS = json

AUTO_KV_JSON = 0

KV_MODE = none

SHOULD_LINEMERGE = 0

TIMESTAMP_FIELDS = date

category = Example Web Isolation

pulldown_type = 1

local/props.onf
[Example:Isolation:Url]

SEDCMD-sanitize_jsessionid = s/jsessionid=[0-9A-Za z]+/jsessionid=masked_by_splunk/g

SEDCMD-sanitize_url_parameter = s/([#&])(access_token|id_token)=[^\s&",]+/\1\2=masked_by_splunk/g

SEDCMD-sanitize_url_parameters_password = s/([Pp][Aa][Ss][Ss][Ww][Oo][Rr][Dd])=[^\s"&']+/\1=masked_by_splunk/g

[tscroggins]

Hypothetically, Example:Isolation:Url would have some other configuration extracting jsessionid, access_token, id_token, or password, possibly through another props stanza, e.g. [host::...] or [source::...], matching the input.

[tscroggins]

You may be able to use a TRANSFORM or INGEST_EVAL to edit the indexed field as well, although INDEXED_EXTRACTIONS from a forwarder would need to be routed back through pasrsingQueue. Since SEDCMD is already working, the re-route probably isn't necessary. Note the use of := to edit the existing field, if present:

[remove-foo]
INGEST_EVAL = foo:=null()

# or

[mask-and-replace-foo]
INGEST_EVAL = foo:=""