Getting Data In

sourcetypes best practice

Branden
Builder

I'm thinking about adding certain application server logs to our Splunk environment. At first, it seemed simple: I would make a file named "appsrv.log" be indexed as sourcetype "appsrv" (or something like that).

The problem is our test host contains several codelines/environments, which means it may have several different, but similarly named appsrv.log files. In a Splunk query, it may be difficult to distinguish which appsrv.log file the results come from unless the user looks at the path of the file itself in the search results. Given how long the path can be, this can be rather annoying.

I'm wondering if it would be better to give a different sourcetype to each codeline. For example: qa-appsrv, test-appsrv, dmo-appsrv.

Does this go against best practice? I'm wondering if anyone has any thoughts on this approach.

Thanks!

Tags (1)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

you may wish instead to create lookup tables against the path of the file, or add a custom indexed field indicating the environment. the lookup table approach is more flexible. basically, you create a CSV that maps the name/path of the file to the "codeline", using wildcard matches if necessary. The file would be something like:

source,codeline
/var/log/blah/*,blah
/var/log/qa/*,qa
/log/some/other/path/appsrv.log,test

the props

[appsrv]
LOOKUP-codeline = sourcecodeline source OUTPUT codeline

and the transforms

[sourcecodeline]
filename = mysourcetocodelinefile.csv
match_type = WILDCARD(source)

if set as an automatic lookup, users will be able to search on the codeline, e.g., using sourcetype=appsrv codeline=qa or similar.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

you may wish instead to create lookup tables against the path of the file, or add a custom indexed field indicating the environment. the lookup table approach is more flexible. basically, you create a CSV that maps the name/path of the file to the "codeline", using wildcard matches if necessary. The file would be something like:

source,codeline
/var/log/blah/*,blah
/var/log/qa/*,qa
/log/some/other/path/appsrv.log,test

the props

[appsrv]
LOOKUP-codeline = sourcecodeline source OUTPUT codeline

and the transforms

[sourcecodeline]
filename = mysourcetocodelinefile.csv
match_type = WILDCARD(source)

if set as an automatic lookup, users will be able to search on the codeline, e.g., using sourcetype=appsrv codeline=qa or similar.

Branden
Builder

Just tried it. Works great, thanks again! Out of curiosity, where do you typically store your CSVs?

0 Karma

Branden
Builder

Aaah! Good idea, thanks! I like the CSV approach. I think I will give that a try.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

And you could also set "codeline" in the inputs.conf as an indexed field, by adding: _meta = codeline::qa (for example) to the input stanza for the qa codeline files.

0 Karma
Get Updates on the Splunk Community!

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...

Splunk App Developers | .conf25 Recap & What’s Next

If you stopped by the Builder Bar at .conf25 this year, thank you! The retro tech beer garden vibes were ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...