Getting Data In

sourcetypes best practice

Branden
Builder

I'm thinking about adding certain application server logs to our Splunk environment. At first, it seemed simple: I would make a file named "appsrv.log" be indexed as sourcetype "appsrv" (or something like that).

The problem is our test host contains several codelines/environments, which means it may have several different, but similarly named appsrv.log files. In a Splunk query, it may be difficult to distinguish which appsrv.log file the results come from unless the user looks at the path of the file itself in the search results. Given how long the path can be, this can be rather annoying.

I'm wondering if it would be better to give a different sourcetype to each codeline. For example: qa-appsrv, test-appsrv, dmo-appsrv.

Does this go against best practice? I'm wondering if anyone has any thoughts on this approach.

Thanks!

Tags (1)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

you may wish instead to create lookup tables against the path of the file, or add a custom indexed field indicating the environment. the lookup table approach is more flexible. basically, you create a CSV that maps the name/path of the file to the "codeline", using wildcard matches if necessary. The file would be something like:

source,codeline
/var/log/blah/*,blah
/var/log/qa/*,qa
/log/some/other/path/appsrv.log,test

the props

[appsrv]
LOOKUP-codeline = sourcecodeline source OUTPUT codeline

and the transforms

[sourcecodeline]
filename = mysourcetocodelinefile.csv
match_type = WILDCARD(source)

if set as an automatic lookup, users will be able to search on the codeline, e.g., using sourcetype=appsrv codeline=qa or similar.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

you may wish instead to create lookup tables against the path of the file, or add a custom indexed field indicating the environment. the lookup table approach is more flexible. basically, you create a CSV that maps the name/path of the file to the "codeline", using wildcard matches if necessary. The file would be something like:

source,codeline
/var/log/blah/*,blah
/var/log/qa/*,qa
/log/some/other/path/appsrv.log,test

the props

[appsrv]
LOOKUP-codeline = sourcecodeline source OUTPUT codeline

and the transforms

[sourcecodeline]
filename = mysourcetocodelinefile.csv
match_type = WILDCARD(source)

if set as an automatic lookup, users will be able to search on the codeline, e.g., using sourcetype=appsrv codeline=qa or similar.

Branden
Builder

Just tried it. Works great, thanks again! Out of curiosity, where do you typically store your CSVs?

0 Karma

Branden
Builder

Aaah! Good idea, thanks! I like the CSV approach. I think I will give that a try.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

And you could also set "codeline" in the inputs.conf as an indexed field, by adding: _meta = codeline::qa (for example) to the input stanza for the qa codeline files.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...

Manual Instrumentation with Splunk Observability Cloud: How to Instrument Frontend ...

Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward ...