I am testing out the Splunk Operator Helm chart to deploy a C3 architecture Splunk instance. At the moment everything deploys without errors, My cluster manager will pull and install apps via the AppFramework config, and SmartStore is receiving data from the indexer cluster. However, after creating ingress objects for each Splunk instance in the deployment (LM, CM, MC, SHC, IDXC) I have been able to successfully log into every WebGUI except for the indexer cluster. The behavior I am experience is basically like getting kicked out of the GUI the second I type the username and password then hit enter. The web page refreshes and I am back at the log in screen. I double checked that the Kubernetes secret containing the admin password is the same for all of the Splunk instances, and also intentionally typed in a bad password and got a login failed message instead of the screen refresh I get when entering the correct password. I am not really sure how to go about troubleshooting this. I searched through the _internal index but didn't come up with a smoking gun. ... View more

The Cisco Networks Add-on for Splunk Enterprise is licensed under Creative Commons. This license does not allow for commercial use...I have been unable to track down a way to "purchase" a license that would allow me to utilize this Add-on legally. Is there any chance someone can point me in the right direction? ... View more

I am working with linux auditd data The first search is below which pulls together all of the applications executed by a user during the duration of their session index=os sourcetype=auditd NOT exe=/usr/sbin/crond | transaction ses startswith=USER_START endswith=USER_END | rename hostname AS src | eval in_time=_time | eval login_time=strftime(in_time,"%d-%b-%Y %H:%M:%S.%3N") | eval out_time=_time + duration | eval logout_time=strftime(out_time,"%d-%b-%Y %H:%M:%S.%3N") | search src=$field2$ auid=$field3$ host=$field4$ | table login_time,logout_time,duration,src,host,uid,auid,exe,key The drilldown looks like this, which take the host, & originating user name from the first search and finds all command line executions that user performed. index=os sourcetype=auditd host=$field4$ | `find_commands` | transaction timestamp | search auid=$field2$ type=EXECVE | table timestamp,host,ppid,pid,auid,uid,command,proc_command,success | sort timestamp Where I am struggling is to get the timestamp from the login_time and logout_time fields from the first search to populate the timestamp picker of the drill down. Dashboard Source <form> <label>Linux Auditd</label> <description>User session monitoring and the applications they ran</description> <fieldset submitButton="true"> <input type="time" token="field1"> <label></label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> <input type="text" token="field2"> <label>Source System</label> <default>*</default> </input> <input type="text" token="field4"> <label>Target System</label> <default>*</default> </input> <input type="text" token="field3"> <label>Source User</label> <default>*</default> </input> </fieldset> <row> <panel> <title>Session Monitoring</title> <table> <search> <query>index=os sourcetype=auditd NOT exe=/usr/sbin/crond | transaction ses startswith=USER_START endswith=USER_END | rename hostname AS src | search src=$field2$ | eval in_time=_time | eval login_time=strftime(in_time,"%d-%b-%Y %H:%M:%S.%3N") | eval out_time=_time + duration | eval logout_time=strftime(out_time,"%d-%b-%Y %H:%M:%S.%3N") | search auid=$field3$ host=$field4$ | table login_time,logout_time,duration,src,host,uid,auid,exe,key</query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">20</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="refresh.display">progressbar</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <drilldown> <link target="_blank">search?q=index%3Dos%20sourcetype%3Dauditd%20host%3D$field4$%20%7C%20%60find_commands%60%20%7C%20transaction%20timestamp%20%7C%20search%20auid%3D$field2$%20type%3DEXECVE%20%7C%20table%20timestamp%2Chost%2Cppid%2Cpid%2Cauid%2Cuid%2Ccommand%2Cproc_command%2Csuccess%20%7C%20sort%20timestamp&amp;earliest=$row.login_time$&amp;latest=$row.logout_time$</link> </drilldown> </table> </panel> </row> </form>   ... View more

Hey all, I am really struggling to create a parser for a specific section of the Windows-TerminalServices-Gateway/Operational Event log. Everything I have added to props & transforms appears to be correct, and I can get the sections I want to parse via SPL but whenever I add it to the sourcetype and refresh nothing changes. inputs.conf on the Windows 2016 Server [WinEventLog://Microsoft-Windows-TerminalServices-Gateway/Operational] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 renderXml = true index = windows [WinEventLog://Microsoft-Windows-TerminalServices-Gateway/Admin] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 renderXml = true index = windows props.conf on search head [XmlWinEventLog:Microsoft-Windows-TerminalServices-Gateway/Operational] rename = XmlWinEventLog [XmlWinEventLog:Microsoft-Windows-TerminalServices-Gateway/Admin] rename = XmlWinEventLog [xmlwineventlog] REPORT-0xml_kv_extract = system_props_xml_kv,system_props_xml_attributes,eventdata_xml_data,rendering_info_xml_data,userdata_xml_kv,userdata_xml_attributes [XmlWinEventLog] REPORT-0xml_kv_extract = system_props_xml_kv,system_props_xml_attributes,eventdata_xml_data,rendering_info_xml_data,userdata_xml_kv,userdata_xml_attributes transforms.conf on search head [userdata_xml_kv] # Extracts anything in the form of <tag>value</tag> as tag::value SOURCE_KEY = UserData_Xml REGEX = (?ms)<(\w*)>([^<]*)<\/\1> FORMAT = $1::$2 MV_ADD = 1 [userdata_xml_attributes] # Extracts values from following fields: # EventInfo: xmlns SOURCE_KEY = UserData_Xml REGEX = (?ms)([^\s=]+)\s*=\s*(\'[^<\']*\'|"[^<"]*") FORMAT = $1::$2 MV_ADD = 1 ... View more

I am having trouble wrapping my head around how to configure a HF to forward the sourcetypes of syslog and auditd to a 3rd party syslog host as well as to an indexer, without sending other sourcetypes as well. I am trying to use a combination of these to docs to help but I have not been successful yet. Route and filter data Forward data to third-party systems My configs look like this right now. props.conf [syslog] TRANSFORMS-routing = routeAll, send_to_syslog [auditd] TRANSFORMS-routing = routeAll, send_to_syslog [cpu] TRANSFORMS-routing = routeAll [ps] TRANSFORMS-routing = routeAll transforms.conf [routeAll] REGEX = (.) DEST_KEY = _TCP_ROUTING FORMAT = default-autolb-group [send_to_syslog] REGEX = (.) DEST_KEY = _SYSLOG_ROUTING FORMAT = syslogGroup outputs.conf [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = Y.Y.Y.Y:9997 [tcpout-server://Y.Y.Y.Y:9997] [indexAndForward] index = false [syslog] defaultGroup=syslogGroup [syslog:syslogGroup] server = X.X.X.X:514 sendCookedData = false type = tcp ... View more

After updating our Splunk environment from Splunk 7.0.3 & ES 5.0 to Splunk 7.2.0 & Enterprise Security 5.1.1, many of the ES dashboards are not functioning. This appears to be a problem with all ES dashboards that use a dropdown input or Field Picker that includes an ALL option. For example: the Access Center dashboard should have 4 panels of data, however all 4 panels show the error “Search is waiting for input”. I have tried adjusting all of the input fields at the top of the dashboard and hitting search again but nothing changes. ... View more

I have a Splunk instance in a Development & Test lab that uses what we call "repeatable time" to test software updates against a known good checkpoint of data. During a test all of the server times are rolled back to March 16 2011. I have been struggling to figure out how to get Splunk to ignore the Date in the logged events and only care about the "time". The logs are being monitored in a directory structure as follows /var/adm/splunk/2017_"today's_julian_date"/"server_name"/*.log Example Log: header,123,321,2011-03-16 17:35:36.035 +00:00,subject,............. Effectively I am trying to get these events indexed with today's date but the Time from the log. so it would have: _time=2017-12-11 17:35:36.035 I have tried playing with the TIME_PREFIX & MAX_TIMESTAMP_LOOKAHEAD settings in the sourcetype but have not been successful. TIME_PREFIX = ^[^\s]+\s MAX_TIMESTAMP_LOOKAHEAD = 25 This is a critical issue for me to sort out so any help would be greatly appreciated. ... View more

Hello all, I collect all of my *nix logs into a central server that I has a UF installed on it. I have the splunk_ta_nix installed on my single instance indexer/sh as well as installed at the UF. inputs.conf on the UF only has the [monitor:///var/log] stanza enabled Everything from the centralized location for /var/log/messages is getting the sourcetype of "syslog" and the host field is populating properly based off of the contents of the event rather then with the hostname of the central log server. Everything from /var/log/secure is getting the sourcetype of linux_secure but every event is populated with the hostname of the central log server in the host field regardless of contents of the event. I added the following to Splunk_TA_nix/local/transforms.conf [linux_secure_host] REGEX = ^\w+\s\d{2}\s\d{2}:\d{2}:\d{2}\s(\S+) FORMAT = host::$1 DEST_KEY = MetaData:Host And this to Splunk_TA_nix/local/props.conf [linux_secure] TRANSFORMS-linux_secure_host = linux_secure_host And everything from the centralized /var/log/secure now has the correct host field value. Hoo-ray! Lastly, I attempted to tackle all of the auditd logs that live in /var/log/audit/audit.log These events get the sourcetype of linux_audit and show the same behaviour as the previous example I was able to fix, so I edited transforms.conf like so [linux_audit_host] REGEX = \snode=(\S+) FORMAT = host::$1 DEST_KEY = MetaData:Host and props.conf like this [linux_audit] TRANSFORMS-linux_audit_host = linux_audit_host but i have had no luck populating the correct value into the host field for the events that go into this sourcetype Here is an example of a log from /var/log/audit/audit.log node=ipa01.test.linux type=USER_END msg=audit(1505793661.317:6773): pid=13781 uid=0 auid=0 ses=917 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' Any help with this issue would be amazing. ... View more