Getting Data In

A very singular use case (I think) in indexing XML files.

tcmarquesi
Explorer

I think I have a very particular scenario using XML files. At least I did not find somebody having the same issue here. There we go!

I would like to index this XMLs where:
1. Each file is an entire event, there isn't event break inside the file.
2. Inside the file there isn't timestamp, only in the file name.

So:
1. Which sourcetype may I choose in this scenario? Should I create a custom one?
2. How can I make splunk identify the timestamp properly?

I'll thank very much if somebody can help me, very much.

Regards,

Tiago

Tags (3)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

You should create your own custom sourcetype to handle all the custom things that you want to do. Add a impossible to find Line breaker for your events so that all data from the file will appear as one event. It will be very inefficient to have Splunk read timestamp from the filename as there is no direct way to do that. Also, only the date can be extracted from file name, not the time. See this for more details on how to configure timestamp recognition in Splunk and what's the order Splunk follows.
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/HowSplunkextractstimestamps#How_Splunk_softwa...

I would suggest you to use the current time (time when Splunk sees the file and reads it) as the timestamp. If everything is working fine and your application is generating files with latest timestamp, then using current time will be the closest anyways. My suggestion would be like this

[YourSourceTypeName]
SHOULD_LINEMERGE = false
LINE_BREAKER =([\r\n]+)(?=YouWillNeverFindThisPattern)
DATETIME_CONFIG = current
TRUNCATE = 25000        #default to 10000 character, adjust based on your file content).
KV_MODE = xml

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You should create your own custom sourcetype to handle all the custom things that you want to do. Add a impossible to find Line breaker for your events so that all data from the file will appear as one event. It will be very inefficient to have Splunk read timestamp from the filename as there is no direct way to do that. Also, only the date can be extracted from file name, not the time. See this for more details on how to configure timestamp recognition in Splunk and what's the order Splunk follows.
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/HowSplunkextractstimestamps#How_Splunk_softwa...

I would suggest you to use the current time (time when Splunk sees the file and reads it) as the timestamp. If everything is working fine and your application is generating files with latest timestamp, then using current time will be the closest anyways. My suggestion would be like this

[YourSourceTypeName]
SHOULD_LINEMERGE = false
LINE_BREAKER =([\r\n]+)(?=YouWillNeverFindThisPattern)
DATETIME_CONFIG = current
TRUNCATE = 25000        #default to 10000 character, adjust based on your file content).
KV_MODE = xml
0 Karma

tcmarquesi
Explorer

Thank you @somesoni2 🙂

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...