Splunk Enterprise

Filtering out logs based on sourcetype

jessieb_83
Path Finder

I'm at a loss and hoping for an assist. 

Running a distributed Splunk instance, I used the Deployment Server to push props.conf and transforms.conf to my heavy forwarders to drop specific external customer logs logs at the HF. 

 

We're receiving logs from several external customers, each with their own index. I'm in progress of dividing each customer into sub indexes like {customer}-network, {customer}-authentication and {customer}-syslog

Yes, I'm trying to dump all Linux syslog. This a temporary move while their syslog are flooding millions of errors before we're able to finish moving them to their new {customer}-syslog index. I did inform them and they're working it, with no ETA.

I've been over a dozen posts on the boards, I've asked two different AI's how to do this backwards and forwards, I've triple checked spelling, placement & permissions.
I tried pushing the configs to the indexers from the cluster manager and that didn't work either.
I created copies of the configs in ~/etc/system/local/ and no dice. 
I've done similar in my lab with success. 

I verified the customer inputs.conf is declaring the sourcetype as linux_messages_syslog

I'm at a total loss of why this isn't working.

 

props.conf:

[linux_messages_syslog]
TRANSFORMS-dropLog = dropLog

transforms.conf:

[dropLog]
REGEX = (?s)^.*$
DEST_KEY = queue
FORMAT = nullQueue

 

Anyone have any idea what got'cha I'm getting got by?

Labels (1)
0 Karma
1 Solution

livehybrid
Ultra Champion

Hi @jessieb_83 

Are they sending from a UF or a HF? If you arent having much look with this props/transform combo then it sounds like the data might be arriving to you already parsed, thus what you're doing here wont have an affect.

If the data has already been through a HF then you could try this instead:

# props.conf
[linux_messages_syslog]
RULESET-dropSyslog = dropLog

# transforms
[dropLog]
INGEST_EVAL = queue="nullQueue"

If you are sure its coming from a UF then you could try setting "REGEX = ." on your existing config, however I think what you had should have worked.

:glowing_star: Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

View solution in original post

livehybrid
Ultra Champion

Hi @jessieb_83 

Are they sending from a UF or a HF? If you arent having much look with this props/transform combo then it sounds like the data might be arriving to you already parsed, thus what you're doing here wont have an affect.

If the data has already been through a HF then you could try this instead:

# props.conf
[linux_messages_syslog]
RULESET-dropSyslog = dropLog

# transforms
[dropLog]
INGEST_EVAL = queue="nullQueue"

If you are sure its coming from a UF then you could try setting "REGEX = ." on your existing config, however I think what you had should have worked.

:glowing_star: Did this answer help you? If so, please consider:

  • Adding karma to show it was useful
  • Marking it as the solution if it resolved your issue
  • Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

jessieb_83
Path Finder

THATS what I forgot to mention. They are pushing everything to a HF on their side, that is pushing to my HF's. I will try that out. Thanks!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Then @livehybrid 's solution should work. When you're getting data from a HF (or any other "full" Splunk instance) you're getting it as already parsed and it completely bypasses most of the props/transforms mechanics, except for RULESETs.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...