Multiple Inputs on the same File

[TheEggi98]

Hi there,

i have a file monitoring stanza on a universal forwarder where i filter using transforms.conf to only get logentries i need, because the server writes logentries of multiple business processes into the same logfile.

Now i need entries of another process with different ACL in a different index from that logfile but in our QS cluster while the first datainput still ingests into our PROD cluster

So i have my inputs.conf

[monitor://<path_to_logfile>]
disabled = 0
index = <dataspecific index 1>
sourcetype = <dataspecific sourcetype 1>

a props.conf

[<dataspecific sourcetype 1>]
SHOULD_LINEMERGE        = true
BREAK_ONLY_BEFORE_DATE  = true
TRUNCATE                = 1500
TIME_PREFIX             = ^
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT             = [%y/%m/%d %H:%M:%S]
TRANSFORMS-set 			= setnull, setparsing

and a transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (<specific regex>)
DEST_KEY = queue
FORMAT = indexQueue

As standalone Stanza i would need the new input like this, with its own setparsing transforms

[monitor://<path_to_logfile>]
disabled = 0
index = <dataspecific index 2>
sourcetype = <dataspecific sourcetype 2>
_TCP_ROUTING = qs_cluster

to be honest i could just create a second stanza thats a little different and still reads the same file, but i dont want two tailreader on the same file.

What possibilities do i have?
Thanks in advance

[TheEggi98]

Hi @gcusello 
thanks for the fast response.

if im not wrong i theoretically could bypass the precedence by doing this (at least btool dont complain) but i will not do that

[monitor://<path to logfile>.log]
...

[monitor://<path to same logfile>.lo*]
...

 
When overriding sourcetype and index on the indexer, am i able to route data of the second sourcetype to our qs cluster to build dashboards?

[gcusello]

Hi @TheEggi98 ,

if the file to read is always the same in both inputs, Splunk doesn't read twice a file and the solution is the second one I described (overriding).

If instead you have different files in the same path to read in the two inputs, you can specify in the input stanza the different file name to read also using the same path.

Ciao.

Giuseppe

[TheEggi98]

Alright Thank you

i will use sourcetype and index overriding and then make the data of the newly added available for our qs cluster to build dashboards

[gcusello]

Hi @TheEggi98 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello · Accepted solution]

Hi @TheEggi98 ,

you cannot read the same files in two input stanzas, ony one (by precedence rules) will be used.

If in the same path, you have to read different files for each input, you can specify in the stanzas the correct file to read.

If instead data are in the same file, the only solution is to read it with one input stanza and then override index and eventually sourcetype values on the Indexers or (if present) on Heavy Forwarders, following the instructions at 

for sourcetype https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/Advancedsourcetypeoverrides?_gl=1*4u....

and for index https://community.splunk.com/t5/Getting-Data-In/Route-data-to-index-based-on-host/td-p/10887?_gl=1*1....

Ciao.

Giuseppe