Hello All I have a lookup table that I created that only has ip address and hostnames. I want to run the following search against the lookup table but I am not getting the results I expect. index=_internal sourcetype=splunkd [inputlookup dmzhosts.csv | table ip | rename ip as search | format] group=tcpin_connections NOT eventType=* | stats max(_time) as last_connected, sum(kb) as sum_kb by guid, hostname | addinfo | eval "Source Host" = hostname | eval ttnow = now() | eval Current = strftime(ttnow,"%m-%d-%Y %H:%M:%S") | eval Status = if(isnull(sum_kb) or (sum_kb <= 0) or (last_connected < (info_max_time - 60)), "Not Reachable", "active") | eval "Last Connected" = strftime(last_connected,"%m-%d-%Y %H:%M:%S") | where Status = "Not Reachable" | table "Source Host" "Last Connected" Current Status The search seems to run but I know it isn't really working as the lookup table has 160 IP addresses and the events only show 46 sourceIp's. What I really need is the is a for loop it seems so that the search will set the sourceIp to the ip from the lookup table and then provide a list of all the ones that are missing at the end of the search. Ideas? thanks ed ... View more

Hello All DB connect is failing for some weird reason. I get the following in my logs. It connects successfully but it fails to return any data. When I setup the inputs I can see data for the query but that is the only time. I have another input setup for the same exact database and it works just fine. Ideas as the logs really don't give me a direction to go. thanks ed [eventdata] connection = svr-sql-lnl-14 description = Eventdata disabled = 0 host = svr-sql-lnl-14 index = lenel index_time_mode = dbColumn input_timestamp_column_number = 13 input_timestamp_format = "yyyy-MM-dd HH:mm:ss.SSZ" interval = 30 mode = rising query = SELECT *\ FROM "events"."dbo"."EventLog"\ WHERE UNQ_KEY > ?\ ORDER BY UNQ_KEY ASC source = dbo.eventviews sourcetype = event_data tail_rising_column_number = 1 fetch_size = 300 4/30/18 9:34:35.905 AM 2018-04-30 09:34:35.905 -0700 [QuartzScheduler_Worker-17] INFO org.easybatch.core.job.BatchJob - Job 'eventdata' finished with status: FAILED host = splk-srch-01 source = opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:35.905 AM 2018-04-30 09:34:35.905 -0700 INFO c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=svr-sql-lnl-14 jdbc_url=null status=FAILED input_name=eventdata batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2018-04-30_09:34:35 end_time=2018-04-30_09:34:35 duration=9 read_count=0 write_count=0 filtered_count=0 error_count=0 host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_job_metrics.log sourcetype = dbx_job_metrics 4/30/18 9:34:35.898 AM 2018-04-30 09:34:35.898 -0700 [QuartzScheduler_Worker-17] INFO com.splunk.dbx.connector.logger.AuditLogger - operation=dbinput connection_name=svr-sql-lnl-14 stanza_name=eventdata state=success sql='SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC' host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_audit_server.log sourcetype = dbx_audit 4/30/18 9:34:35.896 AM 2018-04-30 09:34:35.896 -0700 [QuartzScheduler_Worker-17] INFO c.s.d.s.dbinput.recordreader.DbInputRecordReader - action=db_input_record_reader_is_opened input_task="eventdata" query=SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:35.896 AM 2018-04-30 09:34:35.896 -0700 [QuartzScheduler_Worker-17] INFO org.easybatch.core.job.BatchJob - Job 'eventdata' starting host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:05.904 AM 2018-04-30 09:34:05.904 -0700 [QuartzScheduler_Worker-14] INFO org.easybatch.core.job.BatchJob - Job 'eventdata' finished with status: FAILED host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:05.904 AM 2018-04-30 09:34:05.904 -0700 INFO c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=svr-sql-lnl-14 jdbc_url=null status=FAILED input_name=eventdata batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2018-04-30_09:34:05 end_time=2018-04-30_09:34:05 duration=9 read_count=0 write_count=0 filtered_count=0 error_count=0 host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_job_metrics.log sourcetype = dbx_job_metrics 4/30/18 9:34:05.898 AM 2018-04-30 09:34:05.898 -0700 [QuartzScheduler_Worker-14] INFO com.splunk.dbx.connector.logger.AuditLogger - operation=dbinput connection_name=svr-sql-lnl-14 stanza_name=eventdata state=success sql='SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC' host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_audit_server.log sourcetype = dbx_audit 4/30/18 9:34:05.895 AM 2018-04-30 09:34:05.895 -0700 [QuartzScheduler_Worker-14] INFO c.s.d.s.dbinput.recordreader.DbInputRecordReader - action=db_input_record_reader_is_opened input_task="eventdata" query=SELECT * FROM "events"."dbo"."EventLog" WHERE UNQ_KEY > ? ORDER BY UNQ_KEY ASC host = splk-srch-01 source = /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log sourcetype = dbx_server 4/30/18 9:34:05.895 AM 2018-04-30 09:34:05.895 -0700 [QuartzScheduler_Worker-14] INFO org.easybatch.core.job.BatchJob ... View more

Hello All, I am trying to figure out how to create drop down menu that dynamically feeds another drop down menu. I am trying to create a dashboard for management to select a person who owns data in Splunk and based on that select have the next drop down menu auto fill with the roles that are assigned to said data owner. Example: Bob owns two roles bobs_users and bobs_powers Tom owns three roles toms_users, toms_powers and toms_admin Cat owns two roles cats_user and cats_admin Then based on the two drop down menus have it auto populate the roles in the following simple search: | rest /services/authentication/users splunk_server=local | fields title roles capabilities | rename title as User | search roles=$role$ I am not having much luck as the owners are not index any where in Splunk. I mean I could create a one time lookup file and use that to try and auto populate the roles drop down, but trying to do everything from drop down menus/xml without having to do extra work if possible. Any ideas? thanks ed ... View more

Hello All, I am a little confused as to what the heck is going wrong with my time stamps. We have the following raw logs: 2018-02-19 11:13:00 - INFO - ENTITLEMENT - UsersDaoImpl:124 - 036cc5fa-a0b3-4a54-978f-9f34747fd126 - Inside UsersDaoImpl- getUserByUserId method 2018-02-19 11:13:00 - INFO - ENTITLEMENT - EMSJobOrderServiceImpl:38 - 036cc5fa-a0b3-4a54-978f-9f34747fd126 - SalesOrderDTO object type received. 2018-02-19 11:13:00 - WARN - ENTITLEMENT - EMSJobOrderServiceImpl:54 - 036cc5fa-a0b3-4a54-978f-9f34747fd126 - Returning the Job Params... 2018-02-19 11:13:00 - INFO - ENTITLEMENT - UsersDaoImpl:124 - 036cc5fa-a0b3-4a54-978f-9f34747fd126 - Inside UsersDaoImpl- getUserByUserId method 2018-02-19 11:13:00 - INFO - ENTITLEMENT - UsersDaoImpl:124 - 036cc5fa-a0b3-4a54-978f-9f34747fd126 - Inside UsersDaoImpl- getUserByUserId method 2018-02-19 11:13:00 - INFO - ENTITLEMENT - UsersDaoImpl:124 - 036cc5fa-a0b3-4a54-978f-9f34747fd126 - Inside UsersDaoImpl- getUserByUserId method The timezone for the logs/server is PST, but when the logs get ingested they are coming in with a timestamp as follows: The props.conf for said data is as follows: [ems_catalina] SHOULD_LINEMERGE = false TIME_PREFIX = <6> MAX_TIMESTAMP_LOOKAHEAD = 24 TIME_FORMAT = %Y-%m-%dT%H%M%SZ [ems_applogs] TIME_FORMAT = %Y-%m-%d %H:%M:%S TZ = US/Pacific #[source::/apps/tomcat/logs/ems_entitlement_services.log] #TZ = America/Los_Angeles The ems_applogs is the sourcetype which I am having issues with. Any ideas/help. thanks ed ... View more

Hello All, I really need to get good at regex and learn to do this myself but alas there are so many other things that seem to be a priority right now. I have the following log file names. log_SVR-IES-PAN-RAMA-01-20170806 log_SVR-ORW-PAN-RAMA-01-20170806 log_SVR-IES-PAN-RAMA-01-20170813 log_SVR-ORW-PAN-RAMA-01-20170813 log_SVR-IES-PAN-RAMA-01-20170820 log_SVR-ORW-PAN-RAMA-01-20170820 log_SVR-IES-PAN-RAMA-01-20170827 log_SVR-ORW-PAN-RAMA-01-20170827 log_SVR-IES-PAN-RAMA-01-20170903 log_SVR-ORW-PAN-RAMA-01-20170903 log_SVR-IES-PAN-RAMA-01-20170910 log_SVR-ORW-PAN-RAMA-01-20170910 log_SVR-IES-PAN-RAMA-01 log_SVR-ORW-PAN-RAMA-01 I am monitoring the log files with the following stanza: [monitor:///var/log2/gns/palo/log_*] index = panlog host_regex = (?<=log_).+-01 sourcetype = pan:log no_appending_timestamp = true So the question is will the host_regex just give the host name svr-orw|ies-pan-rama-01? According to the regexr.com/v1 site it should but I want to make sure it is correct before I implement it. THanks ed ... View more

Hello All, Basically, I am confused as to what is actually happening in our environment. VMware shows that we are roughly writing at 6MB/s and Splunk Distributed Management Console shows indexing rate of 219KB/s. So what is actually happening to magnify the writes so much? ... View more

Hello All I am working with our CheckPoint FW admin to figure out why their tool shows 17 million events for the past 8 hrs, and Splunk is only showing roughly 5500 events. I have looked at the errors and this is the only error I could find. 7/26/16 11:38:46.179 AM 2016-07-26 18:38:46,179 +0000 log_level=ERROR, pid=31312, tid=Thread-1, file=event_writer.py, func_name=_do_write_events, code_line_no=79 | EventWriter encounter exception which maycause data loss, queue leftsize=2 Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_checkpoint-opseclea/bin/splunk_ta_checkpoint_opseclea/splunktalib/event_writer.py", line 63, in _do_write_events write(evt) IOError: [Errno 32] Broken pipe I have all 5 sourcetypes being logged as well, Firewall Events, Firewall Audit, Firewall Non-Audit, Firewall VPN and Firewall SmartDefense. Again a search for errors in the TA for checkpoint only shows this one error. We are using the latest version of Splunk Add-on for Checkpoint LEA. -ed ... View more