Getting Data In

does Splunk UF has capability to mask data as it send to indexer?

abi2023
Path Finder

Is it possible in Splunk to have one props.conf file on one server's Universal Forwarder (UF) for a specific app, and another props.conf file on a different server for the same app, but with one file masking a certain field and the other not?

0 Karma

tscroggins
Influencer

Hi @abi2023,

If you need to mask data pre-transit for whatever reason and the force_local_processing setting doesn't meet your requirements, you can use the unarchive_cmd props.conf setting to stream inputs through Perl, sed, or any command or script that reads input from stdin and writes output to stdout.

For example, to mask strings that might be IPv4 addresses in a log file using Perl:

/tmp/foo.log

This is 1.2.3.4.
Uh oh, 5.6.7.8 here.
Definitely not an IP address: a.1.b.4.
512.0.1.2 isn't an IP address. Oops.

inputs.conf

[monitor:///tmp/foo.log]
sourcetype = foo

props.conf

[source::/tmp/foo.log]
unarchive_cmd = perl -pe 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/*.*.*.*/'
sourcetype = preprocess-foo
NO_BINARY_CHECK = tru

[preprocess-foo]
invalid_cause = archive
is_valid = False
LEARN_MODEL = false

[foo]
DATETIME_CONFIG = NONE
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)

The transmitted events will be masked before they're sent to the receiver:

tscroggins_0-1711857184100.png

Regular expressions that work with SEDCMD should work with Perl without modifications.

The unarchive_cmd setting is a flexible alternative to scripted and modular inputs. The sources do not have to be archive files.

As others have noted, you can deploy different props.conf configurations to different forwarders. Your props.conf settings for line breaks, timestamp extraction, etc., should be deployed to the next downstream instance of Splunk Enterprise (heavy forwarder or indexer) or Splunk Cloud.

0 Karma

tscroggins
Influencer

Small correction:

NO_BINARY_CHECK = true

(If I edit my original answer, the formatting will be mangled.)

0 Karma

abi2023
Path Finder

Can I do this buy source?  and does  source can take different props conf.

0 Karma

tscroggins
Influencer

My previous answer used a single source example, but you can modify unarchive_cmd settings per-source as needed.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @abi2023 ,

as @marnall said, you can create different apps and deploy to the UFs using different serverclasses.

About data mascking, for my knowledge UFs enter only in the input phase, but the other phases (merge and parsing) are in the first full Splunk instance that data are passing through.

In other words, in the Indexers or (when present) in the first Heavy Forwarder, but not in the UFs.

If your doubt is that data are sent in clear mode, you can encrypt them between the UFs and the Indexers (or HFs), and then mask them on these other systems.

Ciao.

Giuseppe

marnall
Builder

Would it fit your use case to set inputs.conf and outputs.conf such that the UF forwards the same logs to two different indexer servers, then those indexer servers have different props.conf which can mask and not mask the fields? 

It seems like props.conf on the UF won't solve your problem.

richgalloway
SplunkTrust
SplunkTrust

UFs are independent so it is possible to have different configurations on each. 

If the UFs are managed by a Deployment Server, however, you cannot have different props.conf files in the same app.  You would have to create separate apps and put them in different server classes for the UFs to have different props for the same sourcetype.

To answer the second part of the question, you *should* be able to put force_local_processing = true in the props.conf file to have the UF perform masking.  Of course, you would also need SEDCMD settings to define the maskings themselves.  I say "should" because I don't have experience with this and the documentation isn't clear about what the UF will do locally.

---
If this reply helps you, Karma would be appreciated.

PickleRick
SplunkTrust
SplunkTrust

Yep. While the local processing on UF part is supposed to work _somehow_ it's indeed not very well docummented and not recommended.

If you want only some sources masked, you could use source-based or host-based stanzas in props.conf on your indexer(s) to selectively apply your SEDCMD or transform only to specific part of your data.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...