Props.Conf - By-Pass

[RAVISHANKAR]

Hello ,

i have a common log file which same name in both production and stage with different name for sourcetype.

As i don't want that logs to be ingested from Production i have added below entry in props.conf.

[source::<Log file path>]

Transforms-null= setnull

transforms.conf

[setnull]
REGEX = BODY
DEST_KEY = queue
FORMAT = nullQueue

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

But i want same log file from stage and not from production - in props.conf adding the sourctype of prod will restrict the logs from production and ingest the logs from stage where sourcetype name is different??

[source::<Log file path>]

[sourcetype = <Prod Sourcetype>]

Transforms-null= setnull

in Addition - Prod Source Type i have other two logs and i don't want that get stopped because of this configuration changes.

Thanks

[RAVISHANKAR]

Hi matty,

Thanks for your quick response.

the lab and prod file paths are the same - yes , but sourcetype name is different for prod and stage

i can't pass sourcetype in props because three log files are part of one sourcetype and among that i am restricting one log file - but i want all three logs in file stage.

Also are you on-prem or cloud? - on-Prem

What does your inputs.conf stanza look like?

[monitor:<path>]
sourcetype= <sourcetype name>

Thanks

[mattymo]

So all 3 files are picked up by this one monitor stanza?

Are the files all truly the same format aka "sourcetype"?

Can you explain a bit more about why we omitting just one file? 

What can we use to uniquely identify this particular source? the host? Sounds like it has to be source + something else to make it unique. 

If you cant differentiate them at the source, then perhaps something like ingest_eval or a "sourcetype rename" is needed. 

Seems to me you might just be overloading the config...I mean maybe just dont deploy an input that picks up this file in prod? thats why i asked if they truly are all the same sourcetype/format....

[mattymo]

Hi! 

If I am following your question, you are concerned because the lab and prod file paths are the same?

You are not required to set the source path to the file in props.conf to get the desired outcome. If your sourcetype is being set in your inputs that pick up this file, you can simply configure the props to match on the sourcetype to do the processing.  Also I don't think you want to duplicate the stanza names in transforms.conf ie. [setnull] is named twice. could lead to unintended consequences. 

What does your inputs.conf stanza look like? How are you sending this file (UF to indexers? Uf to HF to idx?)

Also are you on-prem or cloud? I ask because "Ingest Actions" (and other solutions like ingest or edge processor) provides a UI for you to do this to help validate and avoid config mistakes. 

Regardless, please always test your configs in a local lab environment to avoid having a bad day 🙂

[RAVISHANKAR]

Adding sourcetype additionally in props.conf fulfilled my requirement.

Thanks

[mattymo]

can you post what you ended up with and accept an answer that helped as the solution? even if it's your own ( i believe you can do that)

Glad to hear you got where you needed to go!