OTX data not importing

[jpolcari22]

Just installed the app and i've configured my api and subscribed to some sources in OTX. However, no data is coming in. I'm currently seeing these messages:

02-07-2019 12:43:15.653 -0500 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/TA-otx/bin/otx.py" Completed polling. Logged 3358 pulses and 76409 indicators.

02-07-2019 12:40:56.893 -0500 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/TA-otx/bin/otx.py" Retrieving subscribed pulses since: 2018-11-09 12:40:56.893778

02-07-2019 12:40:57.863 -0500 WARN DateParserVerbose - A possible timestamp match (Fri Jul 31 16:07:04 2020) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=otx://otx_data|host=xxxxxxxxxxxxxx|otx:indicator|\n 24 similar messages suppressed. First occurred at: Thu Feb 7 12:18:48 2019

It looks like maybe the timestamping is incorrect? Any ideas?

[luke_monahan]

That certainly seems like a mis-parsed timestamp.

Can you do a search into the future so I can see the raw format the timestamps are being returned from OTX for you?

e.g.

index=otx earliest=now latest=+5y

[jpolcari22]

Just gave that a shot but it returned no results. When I view the list of indexes it shows otx as having 0 events.

[luke_monahan]

I think the easiest way to get some debugging output will be to relax the time-window restriction for the sourcetype and restart the OTX ingest so we can see what's coming back from the API.

1. Add "max_days_hence=10950" in props.conf for the [otx:pulse] and [otx:indicator] sourcetypes
2. Remove the checkpoint file for OTX input ($SPLUNK_HOME/var/lib/modinputs/otx/*.json by default)

The next run of the input should attempt the backlog again, and index it in the future rather than dumping it. Once that's done we should have the raw data, which will show us what the timestamp format is coming back as. If this fails, I'll send you a version of the input which will log the timestamps in splunkd.log so we can see them.

The input expects %Y-%m-%dT%H:%M:%S.xxxxxx (without milliseconds for indicators) for what it's worth. This is fairly ISO standard, so I'm not sure why it would ever change. I just quickly tested and got this format.