Hi all,
I have a problem with a field call "timestamp".
I have created a custom python script and added as "Data input". The script is executed every 5 minutes and makes an API call, parse the json response and send it to the indexer.
This is a sample raw event:
{"rev_pingdeath_count": 0, "fwd_tiny_count": 0, "dst_address": "X.X.X.X", "timestamp": "2017-09-15T16:05:00.000Z", "start_timestamp": "1505491512000000", "fwd_cwr_count": 0, "user_src_location": null, "rev_synrst_count": 0, "fwd_xmas_count": 0, "server_app_latency_usec": 0, "rev_rst_count": 0, "dst_is_internal": "true", "user_dst_businessUnit": null, "fwd_bytes": 1022, "fwd_synfin_count": 0, "rev_ack_count": 8, "user_dst_pod": null, "total_perceived_latency_usec": 0, "bandwidth_bytes_per_second": "0", "user_src_department": null, "user_src_businessUnit": null, "rev_psh_count": 4, "dst_hostname": "appServerXXXX", "user_src_lifecycle": null, "rev_pkts": 8, "fwd_nc_count": 0, "src_address": "Y.Y.Y.Y", "rev_finnoack_count": 0, "user_src_pod": null, "dst_enforcement_epg_name": [], "rev_nc_count": 0, "rev_cwr_count": 0, "user_src_datacenter": null, "fwd_synrst_count": 0, "fwd_ack_count": 7, "srtt_available": "SRTT_NONE", "rev_allzero_count": 0}
There is only one timestamp field on each event, as far I have been able to see, but when I do a ** index=main source=mysource | head 1 | table timestamp** I get the following data:
Where is the none value coming from ??. This none is present in every single event.
Splunk version 6.5.2, single instance.
Thanks and regards,
Hi mgarciar,
I think that you used "timestamp" field as timestamp in log ingestion
so try to use index=main source=mysource | head 1 | table _time
Bye.
Giuseppe
Splunk is using the current time to set the _time field value and not timestamp.
It should be better to configure your input to use timestamp field as timestamp,
Anyway try
index=main source=mysource | head 1 | rename timestamp AS _time | table _time
Bye.
Giuseppe
just tried to use your sample in a text input, it worked fine, below is how the event looks like when i ingest it through a text file...see and compare against your event
{ [-]
bandwidth_bytes_per_second: 0
dst_address: X.X.X.X
dst_enforcement_epg_name: [ [+]
]
dst_hostname: appServerXXXX
dst_is_internal: true
fwd_ack_count: 7
fwd_bytes: 1022
fwd_cwr_count: 0
fwd_nc_count: 0
fwd_synfin_count: 0
fwd_synrst_count: 0
fwd_tiny_count: 0
fwd_xmas_count: 0
rev_ack_count: 8
rev_allzero_count: 0
rev_cwr_count: 0
rev_finnoack_count: 0
rev_nc_count: 0
rev_pingdeath_count: 0
rev_pkts: 8
rev_psh_count: 4
rev_rst_count: 0
rev_synrst_count: 0
server_app_latency_usec: 0
src_address: Y.Y.Y.Y
srtt_available: SRTT_NONE
start_timestamp: 1505491512000000
timestamp: 2017-09-15T16:05:00.000Z
total_perceived_latency_usec: 0
user_dst_businessUnit: null
user_dst_pod: null
user_src_businessUnit: null
user_src_datacenter: null
user_src_department: null
user_src_lifecycle: null
user_src_location: null
user_src_pod: null
}
And when i ran the query | table timestamp,_time , this is what i get:
timestamp _time
2017-09-15T16:05:00.000Z 2017-09-15 21:35:00
so mgarciar _timee is not really the current time , is it?
I think if you do the event compare between my text ingestion and your event you will get some clues
Hi mgarciar,,
could you explain:
you ingested this event from a text file in index=main using as TimeStamp the current time;
you extracted the "timestamp" field using a regex (like this timestamp: (?<timestamp>.*)\s+total
);
after you run my search:
index=main source=mysource | head 1 | rename timestamp AS _time | table timestamp _time
and you have different values between timestamp and _time?
I performed the same test and I had the same values, could you repeat this full test?
Bye.
Giuseppe
Hi,
I'm ingesting events from a Python script that is calling an API. This script writes to splunk the events as shown on my original post. When I do a search, like this:
index=main source=mysource | head 1
I can see Splunk returns a single event, everything looking good. But when add:
| table timestamp
where timestamp is a field from the json formatted event it returns two values, none and the actual date time value (as shown on the original picture). The none is not visible anywhere.
Hi mgarciar,
in your python script try to use this search:
index=main source=mysource
| head 1
| rex "timestamp: (?<timestamp>.*)\s+total"
| table timestamp
Bye.
Giuseppe
Hi Giuseppe,
Thank you for your suggestions. That will probably allow me to extract the correct value at search time.
But the problem persists right?, the question is more: why I'm getting that none value inserted on the field? Is it that timestamp is a reserved field maybe?
Hi mgarciar,
probably you didn't correctly extracted the timestamp field, try to re configure field extraction enabled for the role that you use in python script and probably you'll not have the problem.
if you're satisfied by this answer, please accept or upvote it.
Bye.
Giuseppe
There is not field extraction at all. The value is "as is". That's the value coming from the Api (with the exception of the none). The _time field is just set to be the system date/time (for now).
There is not field extraction, props, transform, anything related with the "timestamp" field.
try to add a field extraction for timestamp field using my regex, in this way you should use this field also in script.
Bye.
Giuseppe
If this answer satisfies your question, please accept or upvote it.
Bye.
Giuseppe
Cannot reproduce, please share your props.conf for this sourcetype
props.conf
[mysourcetype]
DATETIME_CONFIG = CURRENT
MAX_EVENTS = 1000
TRUNCATE = 100000
But I did find something interesting ... I did a export raw events and this is what I got:
Sorry actually shows both values: none and the actual field value