topic Re: Incorrect Event Date Issue in Splunk Search

Incorrect Event Date Issue

usdreamz — Thu, 12 Dec 2013 23:31:48 GMT

We have Splunk free version protected by IBM Tivoli Access Manager. SPlunk indexes the access logs from access manager.
There are no logs in the system before Sep 2013 since system is just implemented.
Whenever I run a search in Splunk for events e.g. from Feb 2013 onwards the my access gets logged in access manager log with following string

splunk/en-US/app/search/flashtimeline?q=search%20*&earliest=1360573200&latest=1384074000

Splunk indexes this as event occurred in Feb 2013 (as per my example above) and show this under Feb 2013 events while the actual timestamp in the log is todays date . Why Splunk is treating the above as Feb 2013 event and how to fix this issue?

Re: Incorrect Event Date Issue

lukejadamec — Fri, 13 Dec 2013 00:54:21 GMT

Which index holds this event?

Re: Incorrect Event Date Issue

usdreamz — Fri, 13 Dec 2013 02:09:34 GMT

index="main" hold these events.

Re: Incorrect Event Date Issue

lukejadamec — Fri, 13 Dec 2013 02:31:33 GMT

That is odd. The event you posted looks like an event that is recorded in the _internal index.

Does your Splunk Access Role include _internal as one of the default search indexes?

Re: Incorrect Event Date Issue

Lucas_K — Fri, 13 Dec 2013 02:56:55 GMT

What do your raw full events look like from access manager?

If its plain text like you provided without timestamps, then reason is due to splunk guessing that the url epoch time is a timestamp (which it is not!).

Do you have a props configured for this source type?

I suggest you revisit how those logs are being parsed. Easiest way is to take a sample and put it through the gui data input (manager/settings -> data inputs -> add data -> choose your sample file). Play with your data inside that until you get the timestamp extraction right.

I'm guessing if what you provided is the actual raw event you'll need to use something like DATETIME_CONFIG=current to add the time at which the event was seen.

Re: Incorrect Event Date Issue

usdreamz — Fri, 13 Dec 2013 05:41:37 GMT

Thanks for quick response. Please see more information below.

The raw log in the access manager is mentioned below. Every event / log in the access manager starts with and ends with

The event log in access manager for the search performed in Splunk

 <event rev="1.2">
    <date>2013-12-11-21:16:04.828-09:00I-----</date>
    <outcome status="0">0</outcome>
    <component rev="1.2">http</component>
    <event_id>xxx</event_id>
    <action>xxx</action>
    <location>accessmanagerserver</location>
    </originator>
    <accessor name="">
    <user_location>xxxxx</user_location>
    <user_location_type>xxxx</user_location_type>
    </accessor>
    <target resource="5">
    <object>/splunk/en-US/api/search/jobs/1386828913.220/summary?min_freq=0.5&#x0026;earliest_time=1233478800&#x0026;latest_time=1235898000&#x0026;output_time_format=%Y-%m-%dT%H:%M:%S.%Q%z&#x0026;_=1386828964438</object>
    <object_nameinapp>/splunk/en-US/api/search/jobs/1386828913.220/summary?min_freq=0.5&amp;earliest_time=1233478800&amp;latest_time=1235898000&amp;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&amp;_=1386828964438</object_nameinapp>
    </target>
    <resource_access>
    <action>httpRequest</action>
    search/jobs/1386828913.220/summary?min_freq=0.5&amp;earliest_time=1233478800&amp;latest_time=1235898000&amp;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&amp;_=1386828964438
    <method>xxxxx</method>
    <response>xxxx</response></resource_access>
    <data>
    GET ?min_freq=0.5&#x0026;earliest_time=1233478800&#x0026;latest_time=1235898000&#x0026;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&#x0026;_=1386828964438 
    search/flashtimeline?auto_pause=true&#x0026;q=search%20host%3D%22webseal2%22
    </data>
    </event>

Please see the event parsed and indexed by Splunk. I am not sure why only part of the event is parsed here. This behavior is only observed for the searches performed in Splunk and logged in access manager logs and indexed by Splunk. The access logs for other applications in access manager are indexed by splunk as well and it works well in the above format (i.e. complete event with start and end with event tag). Why Splunk is parsing / filtering only some part of the complete event?

GET min_freq=0.5&#x0026;earliest_time=1233478800&#x0026;latest_time=1235898000&#x0026;output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%25z&#x0026;_=1386828964438

search/flashtimeline?auto_pause=true&q=search%20host%3D%22webseal2%22

Prop file has following properties

BREAK_ONLY_BEFORE = <event rev="1.2">
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER = </event>

Re: Incorrect Event Date Issue

Lucas_K — Mon, 28 Sep 2020 15:29:19 GMT

Your events are not breaking correctly and as such the timestamp is not being extracted properly either.

Try something like this in your props.conf

BREAK_ONLY_BEFORE=
MAX_TIMESTAMP_LOOKAHEAD=35
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d-%H:%M:%S.%3N
TIME_PREFIX=