Getting Data In

Multiple Inputs on the same File

TheEggi98
Path Finder

Hi there,

i have a file monitoring stanza on a universal forwarder where i filter using transforms.conf to only get logentries i need, because the server writes logentries of multiple business processes into the same logfile.

Now i need entries of another process with different ACL in a different index from that logfile but in our QS cluster while the first datainput still ingests into our PROD cluster

So i have my inputs.conf

[monitor://<path_to_logfile>]
disabled = 0
index = <dataspecific index 1>
sourcetype = <dataspecific sourcetype 1>

a props.conf

[<dataspecific sourcetype 1>]
SHOULD_LINEMERGE        = true
BREAK_ONLY_BEFORE_DATE  = true
TRUNCATE                = 1500
TIME_PREFIX             = ^
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT             = [%y/%m/%d %H:%M:%S]
TRANSFORMS-set 			= setnull, setparsing

and a transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (<specific regex>)
DEST_KEY = queue
FORMAT = indexQueue


As standalone Stanza i would need the new input like this, with its own setparsing transforms

[monitor://<path_to_logfile>]
disabled = 0
index = <dataspecific index 2>
sourcetype = <dataspecific sourcetype 2>
_TCP_ROUTING = qs_cluster

 

to be honest i could just create a second stanza thats a little different and still reads the same file, but i dont want two tailreader on the same file.

What possibilities do i have?
Thanks in advance

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @TheEggi98 ,

you cannot read the same files in two input stanzas, ony one (by precedence rules) will be used.

If in the same path, you have to read different files for each input, you can specify in the stanzas the correct file to read.

If instead data are in the same file, the only solution is to read it with one input stanza and then override index and eventually sourcetype values on the Indexers or (if present) on Heavy Forwarders, following the instructions at 

for sourcetype https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/Advancedsourcetypeoverrides?_gl=1*4u....

and for index https://community.splunk.com/t5/Getting-Data-In/Route-data-to-index-based-on-host/td-p/10887?_gl=1*1....

Ciao.

Giuseppe

View solution in original post

TheEggi98
Path Finder

Hi @gcusello 
thanks for the fast response.

if im not wrong i theoretically could bypass the precedence by doing this (at least btool dont complain) but i will not do that

[monitor://<path to logfile>.log]
...

[monitor://<path to same logfile>.lo*]
...

 
When overriding sourcetype and index on the indexer, am i able to route data of the second sourcetype to our qs cluster to build dashboards?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TheEggi98 ,

if the file to read is always the same in both inputs, Splunk doesn't read twice a file and the solution is the second one I described (overriding).

If instead you have different files in the same path to read in the two inputs, you can specify in the input stanza the different file name to read also using the same path.

Ciao.

Giuseppe

0 Karma

TheEggi98
Path Finder

Alright Thank you

i will use sourcetype and index overriding and then make the data of the newly added available for our qs cluster to build dashboards

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TheEggi98 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @TheEggi98 ,

you cannot read the same files in two input stanzas, ony one (by precedence rules) will be used.

If in the same path, you have to read different files for each input, you can specify in the stanzas the correct file to read.

If instead data are in the same file, the only solution is to read it with one input stanza and then override index and eventually sourcetype values on the Indexers or (if present) on Heavy Forwarders, following the instructions at 

for sourcetype https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/Advancedsourcetypeoverrides?_gl=1*4u....

and for index https://community.splunk.com/t5/Getting-Data-In/Route-data-to-index-based-on-host/td-p/10887?_gl=1*1....

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...