topic Re: does Splunk UF has capability to mask data as it send to indexer? in Getting Data In

does Splunk UF has capability to mask data as it send to indexer?

abi2023 — Wed, 27 Mar 2024 19:36:27 GMT

Is it possible in Splunk to have one props.conf file on one server's Universal Forwarder (UF) for a specific app, and another props.conf file on a different server for the same app, but with one file masking a certain field and the other not?

Re: does Splunk UF has capability to mask data as it send to indexer?

richgalloway — Wed, 27 Mar 2024 21:34:33 GMT

UFs are independent so it is possible to have different configurations on each.

If the UFs are managed by a Deployment Server, however, you cannot have different props.conf files in the same app. You would have to create separate apps and put them in different server classes for the UFs to have different props for the same sourcetype.

To answer the second part of the question, you *should* be able to put force_local_processing = true in the props.conf file to have the UF perform masking. Of course, you would also need SEDCMD settings to define the maskings themselves. I say "should" because I don't have experience with this and the documentation isn't clear about what the UF will do locally.

Re: does Splunk UF has capability to mask data as it send to indexer?

marnall — Wed, 27 Mar 2024 22:07:09 GMT

Would it fit your use case to set inputs.conf and outputs.conf such that the UF forwards the same logs to two different indexer servers, then those indexer servers have different props.conf which can mask and not mask the fields?

It seems like props.conf on the UF won't solve your problem.

Re: does Splunk UF has capability to mask data as it send to indexer?

gcusello — Thu, 28 Mar 2024 06:35:47 GMT

Hi @abi2023 ,

as @marnall said, you can create different apps and deploy to the UFs using different serverclasses.

About data mascking, for my knowledge UFs enter only in the input phase, but the other phases (merge and parsing) are in the first full Splunk instance that data are passing through.

In other words, in the Indexers or (when present) in the first Heavy Forwarder, but not in the UFs.

If your doubt is that data are sent in clear mode, you can encrypt them between the UFs and the Indexers (or HFs), and then mask them on these other systems.

Ciao.

Giuseppe

Re: does Splunk UF has capability to mask data as it send to indexer?

abi2023 — Fri, 29 Mar 2024 13:29:01 GMT

Can I do this buy source? and does source can take different props conf.

Re: does Splunk UF has capability to mask data as it send to indexer?

PickleRick — Sat, 30 Mar 2024 08:24:04 GMT

Yep. While the local processing on UF part is supposed to work _somehow_ it's indeed not very well docummented and not recommended.

If you want only some sources masked, you could use source-based or host-based stanzas in props.conf on your indexer(s) to selectively apply your SEDCMD or transform only to specific part of your data.

Re: does Splunk UF has capability to mask data as it send to indexer?

tscroggins — Sun, 31 Mar 2024 03:58:35 GMT

Hi @abi2023,

If you need to mask data pre-transit for whatever reason and the force_local_processing setting doesn't meet your requirements, you can use the unarchive_cmd props.conf setting to stream inputs through Perl, sed, or any command or script that reads input from stdin and writes output to stdout.

For example, to mask strings that might be IPv4 addresses in a log file using Perl:

/tmp/foo.log

This is 1.2.3.4. Uh oh, 5.6.7.8 here. Definitely not an IP address: a.1.b.4. 512.0.1.2 isn't an IP address. Oops.

inputs.conf

[monitor:///tmp/foo.log] sourcetype = foo

props.conf

[source::/tmp/foo.log] unarchive_cmd = perl -pe 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/*.*.*.*/' sourcetype = preprocess-foo NO_BINARY_CHECK = tru [preprocess-foo] invalid_cause = archive is_valid = False LEARN_MODEL = false [foo] DATETIME_CONFIG = NONE SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) EVENT_BREAKER_ENABLE = true EVENT_BREAKER = ([\r\n]+)

The transmitted events will be masked before they're sent to the receiver:

Regular expressions that work with SEDCMD should work with Perl without modifications.

The unarchive_cmd setting is a flexible alternative to scripted and modular inputs. The sources do not have to be archive files.

As others have noted, you can deploy different props.conf configurations to different forwarders. Your props.conf settings for line breaks, timestamp extraction, etc., should be deployed to the next downstream instance of Splunk Enterprise (heavy forwarder or indexer) or Splunk Cloud.

Re: does Splunk UF has capability to mask data as it send to indexer?

tscroggins — Sun, 31 Mar 2024 03:59:57 GMT

My previous answer used a single source example, but you can modify unarchive_cmd settings per-source as needed.

Re: does Splunk UF has capability to mask data as it send to indexer?

tscroggins — Sun, 31 Mar 2024 04:05:16 GMT

Small correction:

NO_BINARY_CHECK = true

(If I edit my original answer, the formatting will be mangled.)