I have events with the following format - [Thread-2505_GOOGLE_INT_20170424155901301f9e61-1493049600619-NSRLM_2_1_RTL_39088504_2_R_PCLN,PCLN] 2017-04-24 12:00:02 : T:0.047 secs S:XXXX-SSSS-SSSSSS A:Availability M:INT_20170424155901301f9e61-1493049600619-NSRLM_2_1_RTL_39088504_2_R_1 CMD: [<?xml version="1.0" ?><AvailabilityRequestV2 xmlns="http://xml.google.com" siteid="1470249" apikey="SFGSDGSDFSDFGSFG" async="false" waittime="5"><Type>4</Type><Id>460573</Id><Radius>0</Radius><Latitude>0.0</Latitude><Longitude>0.0</Longitude><CheckIn>2017-10-01</CheckIn><CheckOut>2017-10-17</CheckOut><Rooms>1</Rooms><Adults>2</Adults><Children>0</Children><Language>en-us</Language><Currency>000</Currency></AvailabilityRequestV2>] Notice couple of things - event starts with a thread name in square brackets, followed by date/time, followed by an unwanted ":", followed by several key/value pairs separated by a ":", and finally at the end you have content in XML inside square brackets. I want to extract/transform this into the following so it is easy to search and run spath to extract fields from xml - Thread = Thread-2505_GOOGLE_INT_20170424155901301f9e61-1493049600619-NSRLM_2_1_RTL_39088504_2_R_PCLN,PCLN DateTime = 2017-04-24 12:00:02 ResponseTime = 0.047 secs S = XXXX-SSSS-SSSSSS A = Availability M = INT_20170424155901301f9e61-1493049600619-NSRLM_2_1_RTL_39088504_2_R_1 Action = CMD XML = <?xml version="1.0" ?><AvailabilityRequestV2 xmlns="http://xml.google.com" siteid="1470249" apikey="SFGSDGSDFSDFGSFG" async="false" waittime="5"><Type>4</Type><Id>460573</Id><Radius>0</Radius><Latitude>0.0</Latitude><Longitude>0.0</Longitude><CheckIn>2017-10-01</CheckIn><CheckOut>2017-10-17</CheckOut><Rooms>1</Rooms><Adults>2</Adults><Children>0</Children><Language>en-us</Language><Currency>000</Currency></AvailabilityRequestV2> Notice that the last key CMD is put into a new key "Action" and value is put inside XML. I was able to do some clean up like removing square brackets, removing extra ":" etc with the following SED Command - SEDCMD-remove-xml-header = s/\<\?xml[^\>]*\>//g SEDCMD-remove-square-brackets = s/\[|\]//g SEDCMD-remove-colon = s/ : / /g SECCMD-reame-threadname = s/Thread-/Thread:/1 How do I achieve the rest? Is a key value transformer? ... View more

cisco_ironport_web.log has the following events - Event - 1 1489714117.601 56 27.1.11.11 TCP_REFRESH_HIT/200 54491 GET http://www.flatbed-scanner-review.org/inter-banner_flatbed.jpg bhussain@buttercupgames.com DIRECT/www.flatbed-scanner-review.org image/jpeg DEFAULT_CASE-DefaultGroup-Demo_Clients-NONE-NONE-DefaultRouting <nc,ns,0,-,-,-,-,0,-,-,-,-,-,-,-,nc,-> - http://www.flatbed-scanner-review.org/ Event - 2 1489713615.376 809 211.166.11.101 TCP_MISS/200 147639 GET http://www.vindy.com/ myuan@buttercupgames.com DIRECT/www.vindy.com text/html DEFAULT_CASE-DefaultGroup-Demo_Clients-NONE-NONE-DefaultRouting <IW_news,3.4,0,-,-,-,-,0,-,-,-,-,-,-,-,IW_news,-> - - I use the following reg-ex to extract user, url and domain "field1","field2","field3","field4","field5","field6","url","user","field9","field10","field11","field12","field13","domain" It doesn't work for second event, because domain fields has '-'. How do I fix it? ... View more

Wondering if there a default sorucetype that can be used to extract source_ip and user from secure.log files? source_ip is easy because of the format user is a little bit tricky because of different format of log lines Example events - Thu Mar 16 2017 05:42:48 www1 sshd[3061]: Failed password for invalid user ubuntu from 78.111.167.117 port 3346 ssh2 Thu Mar 16 2017 05:31:31 www1 sshd[9216]: Accepted password for djohnson from 10.3.10.46 port 2750 ssh2 Thu Mar 16 2017 04:42:27 www1 sshd[56680]: pam_unix(sshd:session): session opened for user djohnson by (uid=0) In this case, should we run the extraction several times and save the reg-ex pattern for each type of event separately? ... View more

After browsing through Splunk Answers, the closest I could get is the following SPL to list all Indexes and Sourcetypes in a single table - | eventcount summarize=false index=* index!=_* | dedup index | fields index | map maxsearches=100 search="| metadata type=sourcetypes index=\"$index$\" | eval retention=tostring(abs(lastTime-firstTime), \"duration\") | convert ctime(firstTime) ctime(lastTime) | sort lastTime | rename totalCount AS \"TotalEvents\" firstTime AS \"FirstEvent\" lastTime AS \"LastEvent\" | eval index=\"$index$\"" | rename index as "Index" "sourcetype" as "SourceType" | fields Index SourceType TotalEvents FirstEvent LastEvent I want to provide the users with the ability to filter by indexes and sourcetypes. Here is what I have so far - <form> <label>Splunk Indexes and SourceTypes</label> <fieldset submitButton="false"> <input type="multiselect" token="index" searchWhenChanged="true"> <label>Select Index</label> <search> <query>| rest /servicesNS/-/-/data/indexes|rename "title" as index | eval dy = (frozenTimePeriodInSecs/86400) % 365 | eval retention = dy . " days" | dedup index | stats count by index</query> </search> <fieldForLabel>index</fieldForLabel> <fieldForValue>index</fieldForValue> <choice value="\&quot;$index$\&quot;">ALL</choice> <default>"\""$index$\"""</default> <initialValue>\"$index$\"</initialValue> </input> <input type="multiselect" token="source_type" searchWhenChanged="true"> <label>Select SourceType(s)</label> <search> <query>| metadata type=sourcetypes index=* | stats count by sourcetype</query> </search> <fieldForLabel>sourcetype</fieldForLabel> <fieldForValue>sourcetype</fieldForValue> <prefix>(</prefix> <suffix>)</suffix> <valuePrefix>SourceType=</valuePrefix> <delimiter> OR </delimiter> <choice value="*">ALL</choice> <default>*</default> <initialValue>*</initialValue> </input> </fieldset> <row> <panel> <table> <search> <query>| eventcount summarize=false index=* index!=_* | dedup index | fields index | map maxsearches=100 search="| metadata type=sourcetypes index=\"$index$\" | eval retention=tostring(abs(lastTime-firstTime), \"duration\") | convert ctime(firstTime) ctime(lastTime) | sort lastTime | rename totalCount AS \"TotalEvents\" firstTime AS \"FirstEvent\" lastTime AS \"LastEvent\" | eval index=\"$index$\"" | rename index as "Index" "sourcetype" as "SourceType" | fields Index SourceType TotalEvents FirstEvent LastEvent | search $source_type$</query> <earliest>-3d@d</earliest> <latest>now</latest> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> </row> </form> I am unable to achieve 2 things here - When I filter indexes, I want the respective sourcetypes to be filtered in the sourctypes dropdown Display the table with selected indexes and sourcetypes only (should be able to select multiple in both case) The query seems to be slow, but it gives the expected output. Any advice? Thanks! ... View more