Getting Data In

When indexing don't create an event every time EPOCH date is shown, but still index the data

j666gak
Communicator

Hi,

I'm having a bit of a headache. I am trying to index an XML file however I want the event date to be the date the file is updated, and not each EPOCH date listed in the file. Although I do still want to index this data.

The second thing that I want to do is is convert the EPOCH date in to a human readable format.

Example of log file

domain created="1364516883">url.com /domain>

domain created="1364516882">url.net /domain>

domain created="1364516882">url.info /domain>




I would appreciate any help anybody can give me.


Thanks

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You can do this force this by creating a datetime.xml file containing only:

<timePatterns />
<dataPatterns />

and then indexing with

DATETIME_CONFIG = /etc/apps/myapp/local/mydatetime.xml

or whatever the appropriate relative path to your file is. This will cause date parsing from the data and from the file name to fail (because no patterns exist that can be matched), until the remaining option is to use the modification date of the input file. Instead of trying to modify the epoch date when the data is indexed, you should just convert it at search time when viewing the data, e.g., using the | convert search command or strptime() eval function.

Ayn
Legend

Use DATETIME_CONFIG = CURRENT in props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

As for converting epoch to human readable in the actual raw event, that's not possible to do via an index-time transform at least. If it's really important to you, you could consider switching to a script-based input and put that kind of transforming logic into the script instead.

0 Karma

Ayn
Legend

Yes. The assumption would be that Splunk will pick events up right as the file is updated, so index time will be roughly the same as file modification time.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

This will set the timestamp to the time the file is indexed, rather than the modification time of the file.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...