Hi, We are using the latest version for TA-dmarc add-on for Splunk (3.2.1). Connection seems to be successful and debug logs show that add-on is able to read the messages: 2020-05-21 09:00:54,162 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | get_dmarc_messages: successfully connected to x.y.com 2020-05-21 09:00:54,769 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX 2020-05-21 09:00:54,916 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX match subject "Report domain:" For new messages I can also see Add-on writing the data. 2020-05-21 09:30:54,469 DEBUG pid=131084 tid=MainThread file=base_modinput.py:log_debug:286 | write_part_to_file: saved file /tmp/tmpEH0vFq/test.com!xyz.com!1589932800!1590019200!star.xml from uid 48 But I do not see anything showing up under index=dmarc (This is the index set in the config). I do see the following repeated: 2020-05-21 09:00:54,938 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 (body: {}) 2020-05-21 09:00:54,942 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 HTTP/1.1" 200 389 2020-05-21 09:00:54,943 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.005052 2020-05-21 09:00:50,982 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:51,953 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:52,712 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:54,026 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | Success creating temporary directory /tmp/tmptGt5ME 2020-05-21 09:00:54,026 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | Start processing imap server x.y.com with use_ssl True 2020-05-21 09:00:54,162 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | get_dmarc_messages: successfully connected to x.y.com 2020-05-21 09:00:54,769 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX 2020-05-21 09:00:54,916 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | get_dmarc_messages: 5 messages in folder INBOX match subject "Report domain:" 2020-05-21 09:00:54,917 INFO pid=165192 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling 2020-05-21 09:00:54,917 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/config/TA_dmarc_checkpointer (body: {}) 2020-05-21 09:00:54,918 INFO pid=165192 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1 2020-05-21 09:00:54,927 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/config/TA_dmarc_checkpointer HTTP/1.1" 200 5241 2020-05-21 09:00:54,928 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.010589 2020-05-21 09:00:54,928 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/config/ (body: {'offset': 0, 'count': -1, 'search': 'TA_dmarc_checkpointer'}) 2020-05-21 09:00:54,931 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/config/?offset=0&count=-1&search=TA_dmarc_checkpointer HTTP/1.1" 200 4439 2020-05-21 09:00:54,932 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003590 2020-05-21 09:00:54,934 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_25 (body: {}) 2020-05-21 09:00:54,937 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_25 HTTP/1.1" 200 388 2020-05-21 09:00:54,938 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003830 2020-05-21 09:00:54,938 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 (body: {}) 2020-05-21 09:00:54,942 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_28 HTTP/1.1" 200 389 2020-05-21 09:00:54,943 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.005052 2020-05-21 09:00:54,943 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_31 (body: {}) 2020-05-21 09:00:54,947 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_31 HTTP/1.1" 200 340 2020-05-21 09:00:54,948 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.004498 2020-05-21 09:00:54,948 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_38 (body: {}) 2020-05-21 09:00:54,952 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_38 HTTP/1.1" 200 388 2020-05-21 09:00:54,953 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.004457 2020-05-21 09:00:54,953 DEBUG pid=165192 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_42 (body: {}) 2020-05-21 09:00:54,956 DEBUG pid=165192 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-dmarc/storage/collections/data/TA_dmarc_checkpointer/x.y.com_dmarc%40xyz.com_42 HTTP/1.1" 200 340 2020-05-21 09:00:54,957 DEBUG pid=165192 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.004431 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids on imap set([25, 42, 28, 38, 31]) 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids on imap set([25, 42, 28, 38, 31]) 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids in checkp set([25, 42, 28, 38, 31]) 2020-05-21 09:00:54,958 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | filter_seen_messages: uids new set([]) 2020-05-21 09:00:54,958 INFO pid=165192 tid=MainThread file=base_modinput.py:log_info:293 | Ended processing imap server x.y.com 2020-05-21 09:00:54,959 DEBUG pid=165192 tid=MainThread file=base_modinput.py:log_debug:286 | Success deleting temporary directory /tmp/tmptGt5ME How can I troubleshoot this further? Thanks, ~ Abhi ... View more

All, We are working on setting up a Input for SQL Server. DBX Version is 3.1.4. Table currently contains roughly 2M rows and there is a Auto-increment column(int) that we are using for Rising column. Under "Fetch Size" for both Connection as well as Input I've set 5000 rows. Max Rows is left at 0. When the Input is saved, first query pulls around 21K rows. Next time when it runs(Every 5 minutes), DBX pulls only 344 Rows each time. I am not able to figure out how to have it retrieve more than that. We are also noticing that after 2nd run time, the Rising column isin't rising anymore and DBConnect keeps pulling same 344 rows over and over at each interval. I read that Fetch Size might not be considered depending on the driver but if that is the case here can anything else be done to fetch all the rows from DB? We are using MS-SQL Server Using jTDS Driver . At this point our main goal is to get all the rows from DB into Splunk to Analyze the data. [UPDATE] Screenshot from modinputs. Rising Column stops after few entries. Thanks in Advance, ~ Abhi ... View more

Hi, I am using DB Connect V2(Configured on Indexer) to pull data from MSSQL Database with timestamp as the Rising Column. Data in the preview looks good but once the input is saved, each query pulls everything from the checkpoint timestamp onward instead of pulling only the rows that were added after the last updated checkpoint. SELECT COLUMN1, COLUMN2, COLUMN3, COLUMN4, TIMESTAMP FROM <Table_Name> WHERE TIMESTAMP > ? ORDER BY TIMESTAMP DESC Is there anything else I am missing in the configuration? ~ Abhi ... View more

Hi, Has anyone come across the following issue where Splunk web loads but gets stuck at Loading prompt? This was noticed after we upgraded ES app via deployer(5.0.0 -> 5.2.2). But we are also having some SHC related issue so not sure if this is anything related to a setting in Splunk ES app or completely unrelated. Splunk version is on 7.2.4.2 Any help is appretiated.. Thanks, ~ Abhi ... View more

Hi, We have a Search Head Cluster with 3 members. Originally when the cluster was setup there was a 4th member that was eventually removed and turned into a Stand-alone SH. If we check the shcluster-status, only the 3 current members are listed and command run on 4th clearly says that clustering is not enabled. But for some reason, Monitoring Console still thinks the 4th member is part of the cluster although it doesn't have any clustering data on it. Not sure where this information is coming from but is there a way to clean this up? Thanks in advance, ~ Abhi ... View more

Hi, We have an alert set to sent email each time a Firewall failover occurs. Alert condition is pretty straightforward. sourcetype=<Source Type> m=144 OR m=145 m=144 is "Primary firewall has transitioned to Active" where m=145 is "Secondary firewall has transitioned to Active". "host" field lets us know the firewall in question. Alert is working as expected. We get one alert when the secondary becomes active and then another when Primary takes over. Is there a way to create an condition such that once it detects Secondary is Active(i.e. m=145), Splunk will send one alert and then continue to do so at a set frequency(Say once every 2 hours etc) until it sees a m=144 event type, i.e. Primary is now Active , coming from the same firewall(host field). Looks like I'll have to use some correlation command like transaction to achieve this. Any ideas? Thanks, ~ Abhi ... View more

Hi, I am trying to setup a new input on Splunk DBConnect (V 3.1.4) to connect and retrieve data from MSSQL Server using MS-SQL Server Using jTDS Driver . Identity and Connection were setup successfully and a simple select * from <View Name> gets the data into the Preview Data pane. But as soon as I select any Rising Column and execute the query, following error is show. Tried different types for rising column(int, datetime etc) but they all end up with the error show above. In another thread it was mentioned that permissions on /opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect might be a reason but those seem fine to me, although the folder itself is empty. There is no checkpoint value stored. What could be causing this issue? Anything wrong with the version I am using? Thanks, ~ Abhi ... View more

Hi, I am trying to determine total license usage in GB by a certain group of assets where hostname starts with "xyz". There are a total of 24 such hosts that are currently sending data in Splunk, but I tried two different searches to get license count and both reported a different number of hosts. Following query gave results for 10 hosts. index=_internal host=<License Master> source=*license_usage.log* type="Usage" h=xyz* | eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | bin _time span=1d | stats sum(b) as b by _time, pool, s, st, h, idx | eval b=b/(1024*1024*1024) | timechart span=1d sum(b) AS volumeB by h fixedrange=false useother=f Whereas the following gave data only for 7 of them. index=_internal source=*metrics.log group="tcpin_connections" hostname=xyz* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | stats sum(kb) as KB by sourceHost | eval KB = round(KB) We have just one license master and both queries above were run for a 24 Hour window. How can I get the total sum of data sent by these hosts(xyz*) in the last 24 hours? Thanks, ~ Abhi ... View more

Hi, While tuning Splunk ES whenever there is a need to see if a datamodel can see required fields from a specific sourcetype, we use the following search | datamodel Malware search | search sourcetype=<sourcetype> sourcetype=* works but we still need to specify a datamodel. I was wondering if it is possible to search across all the Datamodels & All sourcetypes at once in a single query? If it is then maybe we can stats by datamodel, sourcetype to get a full picture. Thanks, ~ Abhi ... View more