Hi, I am testing this Add-on to retrieve data from REST API. Connection is good and it pulls the data but every event gets truncated around the same line. Also the response doesn't seem end properly(a complete page where next can be requested). It seems as if a limit has been reached.   I changed the Add-on logging level to DEBUG but do not see any error messages. Is there anything else I can check? Splunk Version - 7.2.4.2 Add-on Version - 1.9.8 Thanks, ~ Abhi ... View more

Hi, We have a use-case where responses(host_addr) returned from DNS queries are passed through AbuseIPDB API to check for any potential matches.  Since the API has a set limit we dont want to query an IP more than once. To achieve this, stats is used to get distinct values and then it is passed through the API. It works well but due to the use of "stats", we lose all the other crucial fields from the original data, e.g. src_ip, query etc. Here's a sample query: <Base Search> | stats count by host_addr | table host_addr | abuseip ipfield=host_addr | sort - AbuseConfidence Could eventstats come to the rescue here? If so, what could be a potential syntax of that search? From the other examples I saw, eventstats sees to be more useful when performing a actual stats function like sum etc. End goal is to create something like | table src_ip, query, host_addr, LastReportedAt, AbuseConfidence  but also keeping API limits in check(Using only unique values of host_addr). Any pointers on this will be appreciated. Thanks, ~ Abhi   ... View more

Hi, We are using Splunk Stream to pull logs from DNS Servers. All the target Servers have similar naming convention and do show up under preview based on the Regex Rule for the group. But one of them never becomes part of the group. This Server(003) ends up under defaultgroup. Preview lists all matched assets   003 is not added to the group Description for defaultgroup reads "Used when there is no matching group found for a given stream forwarder ID", but in this case 003 clearly matches a group along with others. Are there any other parameters apart from the name which might be playing a role here? Thanks, ~ Abhi ... View more

Hi, We are using Splunk Stream to get DNS logs into Splunk and it maps seamlessly with the Network Resolution Data model as well. To get cleaner data, I created another DNs stream but this time with some aggregation as shown below. Intent is to get concise summaries of DNS events without overwhelming the Indexers.   Now instead of field "query",  data is  coming  as field values(query) which no longer maps to Network Resolution Data Model. DNS.query now reads unknown. Has anyone come across this issue? Would renaming the field resolve the issue or is there a better way to fix the mapping? Thanks, ~ Abhi     ... View more

After Upgrading to SSE Version 3.2.2, we are getting the following error while loading the MITRE Map. Error in 'lookup' command: Could not find all of the specified lookup fields in the lookup table.       Has anyone come across this while using the App? Thanks, ~ Abhi ... View more

Splunk Version - 7.2.4.2 Splunk ES Version - 5.3.0 Hi, I am trying to add a custom lookup within ES to define Category/Priority for certain assets. Followed this article to the letter to create lookup Table & Definitions with correct permissions. https://docs.splunk.com/Documentation/ES/5.3.0/Admin/Createlookups Lookup was also formatted as required. I was able to add the Lookup definition under Configure > Data Enrichment > Identity Management but still the new Categories do not show up under any search for the asset nor are they being used by ES for Correlations. I do see another location Configure > Content > Content Management > Create New Content > Managed Lookup but when I try to add a new Managed Lookup, this new lookup definition is not listed in the drop-down. Could this be causing ES not to read/merge the data from this new custom lookup? What is the difference between adding lookup under these two locations? Note: As a test, I added the same data in the built-in assets.csv lookup and now at least ES Asset Center can see the updated Categories for those assets but it still doesn't get added when running Searches/Data Model correlations etc. Thanks, ~ Abhi ... View more

Hi, I am trying to add Snort data into Splunk by monitoring barnyard2.alert file using Universal Forwarders.   [monitor:///var/log/barnyard2/barnyard2.alert] sourcetype=snort_unified2 index=snort   As explained here https://community.splunk.com/t5/Getting-Data-In/Why-is-Splunk-line-breaking-a-single-IDS-Alert-event-into-two/m-p/287641#M54947 , I added same settings to props.conf(Indexer and SH) but Splunk still ends up breaking each line as a separate event, as shown below: props.conf   [snort_unified2] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)\[\*\*\] TIME_PREFIX = ^([^\r\n]+[\r\n]+){2} MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %m/%d-%H:%M:%S.%6N category = Network & Security   But when I try these settings by manually adding same input it works just fine. Any ideas on what could be going wrong with props.conf? Thanks, ~Abhi   ... View more

Hi, We are using the latest version for TA-dmarc add-on for Splunk (3.2.1). Connection seems to be successful and debug logs show that add-on is able to read the messages: 2020-05-21 09:00:54,162 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | get_dmarc_messages: successfully connected to x.y.com 2020-05-21 09:00:54,769 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX 2020-05-21 09:00:54,916 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX match subject "Report domain:" For new messages I can also see Add-on writing the data. 2020-05-21 09:30:54,469 DEBUG pid=131084 tid=MainThread file=base_modinput.py:log_debug:286 | write_part_to_file: saved file /tmp/tmpEH0vFq/test.com!xyz.com!1589932800!1590019200!star.xml from uid 48 But I do not see anything showing up under index=dmarc (This is the index set in the config). I do see the following repeated: 2020-05-21 09:00:54,938 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 (body: {}) 2020-05-21 09:00:54,942 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 HTTP/1.1" 200 389 2020-05-21 09:00:54,943 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.005052 2020-05-21 09:00:50,982 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:51,953 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:52,712 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:54,026 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | Success creating temporary directory /tmp/tmptGt5ME 2020-05-21 09:00:54,026 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | Start processing imap server x.y.com with use_ssl True 2020-05-21 09:00:54,162 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | get_dmarc_messages: successfully connected to x.y.com 2020-05-21 09:00:54,769 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX 2020-05-21 09:00:54,916 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX match subject "Report domain:" 2020-05-21 09:00:54,917 INFO pid=165192 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling 2020-05-21 09:00:54,917 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/config/TA_dmarc_checkpointer (body: {}) 2020-05-21 09:00:54,918 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:54,927 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/config/TA_dmarc_checkpointer HTTP/1.1" 200 5241 2020-05-21 09:00:54,928 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.010589 2020-05-21 09:00:54,928 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/config/ (body: {'offset': 0, 'count': -1, 'search': 'TA_dmarc_checkpointer'}) 2020-05-21 09:00:54,931 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/config/?offset=0&count=-1&search=TA_dmarc_checkpointer HTTP/1.1" 200 4439 2020-05-21 09:00:54,932 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003590 2020-05-21 09:00:54,934 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_25 (body: {}) 2020-05-21 09:00:54,937 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_25 HTTP/1.1" 200 388 2020-05-21 09:00:54,938 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003830 2020-05-21 09:00:54,938 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 (body: {}) 2020-05-21 09:00:54,942 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 HTTP/1.1" 200 389 2020-05-21 09:00:54,943 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.005052 2020-05-21 09:00:54,943 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_31 (body: {}) 2020-05-21 09:00:54,947 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_31 HTTP/1.1" 200 340 2020-05-21 09:00:54,948 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.004498 2020-05-21 09:00:54,948 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_38 (body: {}) 2020-05-21 09:00:54,952 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_38 HTTP/1.1" 200 388 2020-05-21 09:00:54,953 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.004457 2020-05-21 09:00:54,953 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_42 (body: {}) 2020-05-21 09:00:54,956 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_42 HTTP/1.1" 200 340 2020-05-21 09:00:54,957 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.004431 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids on imap set([25, 42, 28, 38, 31]) 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids on imap set([25, 42, 28, 38, 31]) 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids in checkp set([25, 42, 28, 38, 31]) 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids new set([]) 2020-05-21 09:00:54,958 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | Ended processing imap server x.y.com 2020-05-21 09:00:54,959 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | Success deleting temporary directory /tmp/tmptGt5ME How can I troubleshoot this further? Thanks, ~ Abhi ... View more

All, We are working on setting up a Input for SQL Server. DBX Version is 3.1.4. Table currently contains roughly 2M rows and there is a Auto-increment column(int) that we are using for Rising column. Under "Fetch Size" for both Connection as well as Input I've set 5000 rows. Max Rows is left at 0. When the Input is saved, first query pulls around 21K rows. Next time when it runs(Every 5 minutes), DBX pulls only 344 Rows each time. I am not able to figure out how to have it retrieve more than that. We are also noticing that after 2nd run time, the Rising column isin't rising anymore and DBConnect keeps pulling same 344 rows over and over at each interval. I read that Fetch Size might not be considered depending on the driver but if that is the case here can anything else be done to fetch all the rows from DB? We are using MS-SQL Server Using jTDS Driver . At this point our main goal is to get all the rows from DB into Splunk to Analyze the data. [UPDATE] Screenshot from modinputs. Rising Column stops after few entries. Thanks in Advance, ~ Abhi ... View more

Hi, I am using DB Connect V2(Configured on Indexer) to pull data from MSSQL Database with timestamp as the Rising Column. Data in the preview looks good but once the input is saved, each query pulls everything from the checkpoint timestamp onward instead of pulling only the rows that were added after the last updated checkpoint. SELECT COLUMN1, COLUMN2, COLUMN3, COLUMN4, TIMESTAMP FROM <Table_Name> WHERE TIMESTAMP > ? ORDER BY TIMESTAMP DESC Is there anything else I am missing in the configuration? ~ Abhi ... View more

Hi, Has anyone come across the following issue where Splunk web loads but gets stuck at Loading prompt? This was noticed after we upgraded ES app via deployer(5.0.0 -> 5.2.2). But we are also having some SHC related issue so not sure if this is anything related to a setting in Splunk ES app or completely unrelated. Splunk version is on 7.2.4.2 Any help is appretiated.. Thanks, ~ Abhi ... View more

Hi, We have a Search Head Cluster with 3 members. Originally when the cluster was setup there was a 4th member that was eventually removed and turned into a Stand-alone SH. If we check the shcluster-status, only the 3 current members are listed and command run on 4th clearly says that clustering is not enabled. But for some reason, Monitoring Console still thinks the 4th member is part of the cluster although it doesn't have any clustering data on it. Not sure where this information is coming from but is there a way to clean this up? Thanks in advance, ~ Abhi ... View more