topic How to combine and chart data from multiple files in the same folder, but only if a certain type of file exists? in Splunk Search

How to combine and chart data from multiple files in the same folder, but only if a certain type of file exists?

avinashreddy539 — Mon, 28 Sep 2020 17:57:11 GMT

Hi,

I am new to splunk and need help with my use case below.
Whenever a request is made to my application, it will create a new unique folder and start writing the information to three files called 1.err, 2.xml, 3.xml. 1.err file will be created only whenever any error is identified., otherwise this file will not be created.

My 1.err file contains

FAIL_ERROR_MESSAGE = Error due to function: function name.

If the file 1.err exists in the folder then i need to get all the FAIL_ERROR_MESSAGE count and from 2.xml file i need to get
SENDID
attribute value
from that folder and from 3.xml file i need to get GFM value as shown in below.

          <name>GFM</name>
          <value>Test</value>
          <dataType>String</dataType>
        </attribute>

I need to prepare a chart where it can show error messages by SENDID and by GFM value. The data should be collected from all the folders where the folder contains 1.err file.

Thanks in advance.

Re: How to combine and chart data from multiple files in the same folder, but only if a certain type of file exists?

jeremiahc4 — Tue, 21 Oct 2014 14:51:10 GMT

This is an interesting use case. I'm not aware of any way to tell splunk only to index the files if it sees a .err file. You can index all of them, then alert based on the presence of the error message.

You would need some kind of transaction ID that is shared between the 2 xml & 1 err file in order to get the specifics you need. Perhaps the unique directory name could be used as that transaction ID. You would then use the xmlkv command to extract the SENDID and GFM values extracted into your report also.

Re: How to combine and chart data from multiple files in the same folder, but only if a certain type of file exists?

avinashreddy539 — Tue, 21 Oct 2014 18:32:25 GMT

we can index all files. But while generating the report we are interested in only .err files. if we have .err file in folder then only we need to fetch the fields from other files in same folder.

Re: How to combine and chart data from multiple files in the same folder, but only if a certain type of file exists?

jeremiahc4 — Tue, 21 Oct 2014 19:01:29 GMT

I'd say you'd want to setup your inputs.conf on the server where you are getting these .err files such that you index all the files using a regex "*" in place of the unique directory.

[monitor:///path/to/files/*/*.xml]
...
[monitor:///path/to/files/*/*.err]
...

Then you can extract that unique directory using rex expression. You can then put those together in a transaction and search for the failure string from the .err file and extract the values you need

base search |  rex "\/path\/to\/files\/(?P<transID>[\w\d]*\/.*" | transaction transID | xmlkv | table field1 field2 SENDID GFM