All Apps and Add-ons

OTX data not importing

jpolcari22
New Member

Just installed the app and i've configured my api and subscribed to some sources in OTX. However, no data is coming in. I'm currently seeing these messages:

02-07-2019 12:43:15.653 -0500 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/TA-otx/bin/otx.py" Completed polling. Logged 3358 pulses and 76409 indicators.

02-07-2019 12:40:56.893 -0500 INFO ExecProcessor - message from "python /opt/splunk/etc/apps/TA-otx/bin/otx.py" Retrieving subscribed pulses since: 2018-11-09 12:40:56.893778

02-07-2019 12:40:57.863 -0500 WARN DateParserVerbose - A possible timestamp match (Fri Jul 31 16:07:04 2020) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=otx://otx_data|host=xxxxxxxxxxxxxx|otx:indicator|\n 24 similar messages suppressed. First occurred at: Thu Feb 7 12:18:48 2019

It looks like maybe the timestamping is incorrect? Any ideas?

0 Karma

luke_monahan
Path Finder

That certainly seems like a mis-parsed timestamp.

Can you do a search into the future so I can see the raw format the timestamps are being returned from OTX for you?

e.g.

index=otx earliest=now latest=+5y
0 Karma

jpolcari22
New Member

Just gave that a shot but it returned no results. When I view the list of indexes it shows otx as having 0 events.

0 Karma

luke_monahan
Path Finder

I think the easiest way to get some debugging output will be to relax the time-window restriction for the sourcetype and restart the OTX ingest so we can see what's coming back from the API.

  1. Add "max_days_hence=10950" in props.conf for the [otx:pulse] and [otx:indicator] sourcetypes
  2. Remove the checkpoint file for OTX input ($SPLUNK_HOME/var/lib/modinputs/otx/*.json by default)

The next run of the input should attempt the backlog again, and index it in the future rather than dumping it. Once that's done we should have the raw data, which will show us what the timestamp format is coming back as. If this fails, I'll send you a version of the input which will log the timestamps in splunkd.log so we can see them.

The input expects %Y-%m-%dT%H:%M:%S.xxxxxx (without milliseconds for indicators) for what it's worth. This is fairly ISO standard, so I'm not sure why it would ever change. I just quickly tested and got this format.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!