Getting Data In

How do I assign a single timestamp to multiple events in one XML document at index-time?

himynamesdave
Contributor

Hi all,

I have an XML feed that returns data like this:

<feed lastUpdate="1438698061185" version="2.0">

<doc>
<num>1</num>
<num2>13</num2>
</doc>

<doc>
<num>1</num>
<num2>13</num2>
</doc>

</feed> 

As you can see, the timestamp (lastUpdate=) is at the top of the document, which contains 2 events (in this example). The actual doc is much larger (1000+ events), so trying to avoid splitting events at search time (using spath).

Is there any way I can assign this time to each event at index time? Or do I need to pre-process the XML doc?

Thanks!

1 Solution

woodcock
Esteemed Legend

You have 2 options:

1: If the dumping of the data is in near-realtime, you can use DATETIME_CONFIG = CURRENT which will cause Splunk to timestamp your event with the Indexer's current system time (so that _time = _indextime).

2: Configure linebreaking and timestamping in the normal way and, believe it or not, it will actually work as you would like but the downside is you will get a huge number of logs like this:

2014 22:22:16.138 +0000 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Oct 22 22:22:14 2014). Context: source::XXX|host::YYY|ZZZ|3549

Because it defaults to the timestamp of the previous event: WIN!

View solution in original post

woodcock
Esteemed Legend

You have 2 options:

1: If the dumping of the data is in near-realtime, you can use DATETIME_CONFIG = CURRENT which will cause Splunk to timestamp your event with the Indexer's current system time (so that _time = _indextime).

2: Configure linebreaking and timestamping in the normal way and, believe it or not, it will actually work as you would like but the downside is you will get a huge number of logs like this:

2014 22:22:16.138 +0000 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Wed Oct 22 22:22:14 2014). Context: source::XXX|host::YYY|ZZZ|3549

Because it defaults to the timestamp of the previous event: WIN!

himynamesdave
Contributor

Oh! This is really interesting, I did not know of that behaviour.

I could use index time for timestamp for this use case, however, was just checking if I had missed something. This is super useful to know. Thanks!

0 Karma

somesoni2
Revered Legend

Agreed, you would have to pre-process the xml file before dropping it to monitored folder OR you can setup a scripted input which will do this processing and send events to Splunk.

How much delay you see between the value in header-lastUpdate versus time when files is dropped to monitored folder?

himynamesdave
Contributor

Thought so. I could use index time for timestamp for this use case, however, was just checking if I had missed something Splunk could have done.

0 Karma

woodcock
Esteemed Legend

I don't know of any way to do this outside of pre-processing it.

Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...