topic Re: Dynamic Sourcetype Extraction based in stanza in Splunk Search

Dynamic Sourcetype Extraction based in stanza

gauravmsharma — Mon, 31 Aug 2020 05:52:06 GMT

Need some suggestions related to dynamic sourcetype extraction:

Does splunk supports sourctype extraction from the stanza which i am using in monitor of inputs.conf file. For ex:

/var/log/test-function_name.log

In this log i want to extract the function-name and use it as a source type. This also means there will be multiple log files under /var/log folder based on function_name.

The reason why i am using this is because my log events does not include this fuction name is each and evert event .

The events are more specifially comming in like:

Log forwarding initializing for function =test/function-name

job_id: XX created

----------------Logs-----------------

jo_id: XX killed

So what is the best way to extract the function_name.

Re: Dynamic Sourcetype Extraction based in stanza

yeahnah — Mon, 31 Aug 2020 07:35:39 GMT

Hi @gauravmsharma

I'm not exactly sure what you're asking for here but I'll give an answer a go. At least it might lead to some better understanding.

I don't believe "dynamic sourcetype extraction based on the stanza" entry is something that can be done, or even should be done, if it was even possible in Splunk.

The sourcetype determines how Splunk software processes the incoming data stream into individual events according to the nature of the data (ref: https://docs.splunk.com/Documentation/Splunk/latest/Data/Aboutdefaultfields#Source_vs_sourcetype). It a useful way of grouping similar data sources based on their data format/syntax/structure. If you could do this then you would also, potentially, have to configure lots of sourcetype entries to tell Splunk how to process the incoming data into events .

It does not really matter anyway as you also have the source metadata field, which already contains the function-name (as you call it) in the monitor stanza. You can extract/manipulate it for every event when you search it.

For example

# inputs.conf [monitor:///var/log/test-function_name.log] sourcetype = some_sourcetype

Events found in this file would all be assigned with source="/var/log/test-function_name.log"

Now you can use regex or eval functions in the Splunk search language to extract the function_name from the source field's value, e.g.

... my base search ... | rex field=source "\/(?<func_name>)[^\.]+)" ...

Resulting in all events in the inputs file having a new key value pair of func_name="test-function_name" . This could also be automated in Splunk's field extractions so it just happens automatically without specifying it in the search query,

Note, I'm just using examples above to show what could be done and you will need to adjust for your own needs.

Hope this helps or gives you some ideas.

Re: Dynamic Sourcetype Extraction based in stanza

gauravmsharma — Mon, 31 Aug 2020 10:24:00 GMT

I guess that is the problem with the events which i am trying to parse. Not every event include this function_name which makes it difficult to dynamically asign the source type. In my case the the events starts which a message

2020-08-31T05:27:58 : Log forwarding initializing for job=job::/test/function-name

It ends with

2020-08-31T05:28: job.id: XX Killed

In between there are logs which has no reference for this so called function-name

So the query remains can be use this initializing line for creating a sourcetype and assign logs to the same sourcetype till it recived a message Killed in the events.

Or will process each and every event based on the conf files props.conf and tranform.conf.

Since in this case i am using func_name as a variable and it can comeup with multiple values.

Re: Dynamic Sourcetype Extraction based in stanza

yeahnah — Mon, 31 Aug 2020 23:00:02 GMT

Hi @gauravmsharma

I think you need to read up on what a sourcetype is in Splunk and how it is used and should be used. It will not help you here and is confusing when used in the context of this question.

So for your question

So the query remains can be use this initializing line for creating a sourcetype and assign logs to the same sourcetype till it recived a message Killed in the events.

The answer is no.

From what I can tell you are trying to tie individual events in a log file together, but these individual events have no unique key value to tie then together.

Depending on the log content and the order of events in the log, you may be able to tie the individual events together using the Splunk processing language (SPL) with commands like transaction and stats. Without a good look at the source log data it hard to know though. If you attach some to this question then may be able to help.

Re: Dynamic Sourcetype Extraction based in stanza

gauravmsharma — Thu, 03 Sep 2020 08:52:30 GMT

I guess i found what i was looking for:

https://community.splunk.com/t5/Getting-Data-In/dynamically-assign-sourcetype-on-folder/td-p/281104

So this is what i tested and it's working like a charm till now 🙂