topic Re: Multiple Inputs on the same File in Getting Data In

Multiple Inputs on the same File

TheEggi98 — Thu, 29 Aug 2024 06:59:14 GMT

Hi there,

i have a file monitoring stanza on a universal forwarder where i filter using transforms.conf to only get logentries i need, because the server writes logentries of multiple business processes into the same logfile.

Now i need entries of another process with different ACL in a different index from that logfile but in our QS cluster while the first datainput still ingests into our PROD cluster

So i have my inputs.conf

[monitor://<path_to_logfile>] disabled = 0 index = <dataspecific index 1> sourcetype = <dataspecific sourcetype 1>

a props.conf

[<dataspecific sourcetype 1>] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE_DATE = true TRUNCATE = 1500 TIME_PREFIX = ^ MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = [%y/%m/%d %H:%M:%S] TRANSFORMS-set = setnull, setparsing

and a transforms.conf

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = (<specific regex>) DEST_KEY = queue FORMAT = indexQueue

As standalone Stanza i would need the new input like this, with its own setparsing transforms

[monitor://<path_to_logfile>] disabled = 0 index = <dataspecific index 2> sourcetype = <dataspecific sourcetype 2> _TCP_ROUTING = qs_cluster

to be honest i could just create a second stanza thats a little different and still reads the same file, but i dont want two tailreader on the same file.

What possibilities do i have?
Thanks in advance

Re: Multiple Inputs on the same File

gcusello — Thu, 29 Aug 2024 07:09:06 GMT

Hi @TheEggi98 ,

you cannot read the same files in two input stanzas, ony one (by precedence rules) will be used.

If in the same path, you have to read different files for each input, you can specify in the stanzas the correct file to read.

If instead data are in the same file, the only solution is to read it with one input stanza and then override index and eventually sourcetype values on the Indexers or (if present) on Heavy Forwarders, following the instructions at

for sourcetype https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/Advancedsourcetypeoverrides?_gl=1*4u2o7n*_gcl_au*NTk4MTY5Ny4xNzI0ODM2ODI0*FPAU*NTk4MTY5Ny4xNzI0ODM2ODI0*_ga*MTY2Mzg1NDI2Mi4xNzI0ODM2ODI0*_ga_5EPM2P39FV*MTcyNDkxNDM3OC41LjEuMTcyNDkxNTE2NS4wLjAuMTM4NTMxMDQ2NA..*_fplc*SVZreWQzalBQTTVYVjFvczZ3Sm45a3lBd1REUGtiV3c4bktjeDdzejliWm9NbEYlMkJ2Z2VGb2E4JTJCYzdsNld4QSUyQmJ0NnAwVTNKaU93OWJGbk1uSmVBa0R3M3l4ZWcyNElnZFZISldBS0VlOSUyRmxycm00UUp5NXdDd2xXb3clMkJXQSUzRCUzRA..

and for index https://community.splunk.com/t5/Getting-Data-In/Route-data-to-index-based-on-host/td-p/10887?_gl=1*1079w7n*_gcl_au*NTk4MTY5Ny4xNzI0ODM2ODI0*FPAU*NTk4MTY5Ny4xNzI0ODM2ODI0*_ga*MTY2Mzg1NDI2Mi4xNzI0ODM2ODI0*_ga_5EPM2P39FV*MTcyNDkxNDM3OC41LjEuMTcyNDkxNTIxNS4wLjAuMTM4NTMxMDQ2NA..*_fplc*M29uUHdZbnRsT3VuRlgxaktXenFld01sdDJ1dXVqcFExYTRDSFF5OTZRd2pTRmFzRGE4OU11YUZaS0dtdG5iSWNuckRxTGRFT2l4cDVrZDlQTnNLUTFEOVVIemRxQyUyQjhyTmpvJTJGeUZ5bUs1Vng2eWtkMUxWcEpSdDFQWEtZQSUzRCUzRA..

Ciao.

Giuseppe

Re: Multiple Inputs on the same File

TheEggi98 — Thu, 29 Aug 2024 07:17:03 GMT

Hi @gcusello
thanks for the fast response.

if im not wrong i theoretically could bypass the precedence by doing this (at least btool dont complain) but i will not do that

[monitor://<path to logfile>.log] ... [monitor://<path to same logfile>.lo*] ...

When overriding sourcetype and index on the indexer, am i able to route data of the second sourcetype to our qs cluster to build dashboards?

Re: Multiple Inputs on the same File

gcusello — Thu, 29 Aug 2024 07:22:52 GMT

Hi @TheEggi98 ,

if the file to read is always the same in both inputs, Splunk doesn't read twice a file and the solution is the second one I described (overriding).

If instead you have different files in the same path to read in the two inputs, you can specify in the input stanza the different file name to read also using the same path.

Ciao.

Giuseppe

Re: Multiple Inputs on the same File

TheEggi98 — Thu, 29 Aug 2024 07:36:47 GMT

Alright Thank you

i will use sourcetype and index overriding and then make the data of the newly added available for our qs cluster to build dashboards

Re: Multiple Inputs on the same File

gcusello — Thu, 29 Aug 2024 07:40:17 GMT

Hi @TheEggi98 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉