You can use the Unix app and a light weight forwarder / Universal forwarder on the remote machines if they unix based, or you can use the Windows app and a light weight forwarder / Universal forwarder for windows machines. You can setup these forwarders to collect the statistics and send them back to your main splunk instance. Here's a great blog link discussing this topic: Choosing a Forwarder, or not And here's the documentation on the setup: Set up forwarding and receiving ... View more

Breaks need to be placed in the props.conf. I assume in your inputs.conf your forcing the sourcetype to iss? If so, inputs.conf [script://blah.exe] sourcetype = iis props.conf [iis] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE = Date BREAK_ONLY_BEFORE = ###_End_Of_Mail_Message_### you can also do this by the source as well, props.conf [source::Email] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE = Date BREAK_ONLY_BEFORE = ###_End_Of_Mail_Message_### That being said, you can only use 1 break_only_before block. If you want to do "or" type clauses, use the "|" delimiter (Date)|(###_End_Of_Mail_Message_###) or did you mean to type: BREAK_ONLY_AFTER = ###_End_Of_Mail_Message_### ... View more

What is the error in the splunkd.log that says why it wasn't indexed? ... View more

limit the search going into the transaction command, it'll be faster, |rex field=_raw "(?<url>http://[A-z0-9./]+)" |search url="http://na.api.inrix.com/V3/Traffic/Inrix.ashx" |transaction url startswith="POST request" endswith="request complete" maxevents=2 |stats count as Total_Requests avg(duration) as Average_Response_Time ... View more

The issue you're going to run into is you'll have to keep pushing the summarized data forward. Also when your searching the summarized data also is based on time, so you'll have to either run a constant summary or run a subsearch to gather the data your after. In the use case you described above, I'd personally use a lookup table. You can populate the lookup table with a list of fields, then use |inputlookup append=t to the end of a stats command and append the list of fields your expecting. Since lookups aren't time sensitive you won't have to use a subsearch to gather your data. ... View more

and then if you only care about the 200's do a search afterwards | search "status 200 OK" so your final search would be: |rex field=_raw "(?<url>http://[A-z0-9./]+)" |transaction url startswith="POST request" endswith="request complete" maxevents=2 |stats count as Total_Requests avg(duration) as Average_Response_Time ... View more

to extract the text into a field, you either need to use a regex expression in props.conf / transforms.conf or you can use the "rex" command. In your case, assuming the url is only printed 1x on each line, use: |rex field=_raw "(?<url>http://[A-z0-9./]+)" then use the transaction command exactly as I have above: |transaction url startswith="POST request" endswith="request complete" maxevents=2 | top url by duration | sort - duration As a side note, your endswith will only close a transaction if the response code was a 200, you may want to capture all responses as I have in mine. ... View more

It really just depends on what the ultimate goal is, if you're just trying to get notified if a field is empty, you may be better off doing a real-time alert. The summary index really in the case your using, only tells you if you had a field and it disappeared, since new fields probably don't just "appear" you may be better off building a static list. ... View more

if you extract the url from both lines, you could use the transaction command, |transaction url startswith="POST request" endswith="request complete" maxevents=2 | stats count as Total_Requests avg(duration) as Average_Response_Time This will show you the total requests and the average duration of each request. By using the transaction command you get the "duration" field automatically. Without doing the stats command you'll be able to see the duration of each event and search on duration. For instance you could do things like: |transaction url startswith="POST request" endswith="request complete" maxevents=2 | top url by duration | sort - duration This would list the top durations and their corresponding URL's so you can see which pages take the longest to load. or |stats max(duration) by url to see what was the longest amount of time taken to load any url. ... View more

can you elaborate a little more on what you're looking for? You are wanting a count of events where httpResponse = 200 or 404? or you're looking for how many fields have 200 and 404 in them? Or are you looking for a list of values that each field has at the end of the day, basically the "values" or "list" function of timechart/stats/chart? ... View more

I think you're hitting the hard limit in limits.conf for a join. I don't think there is a way around this without modifying the conf files. [join] subsearch_maxout = 50000 subsearch_maxtime = 60 subsearch_timeout = 120 ... View more

I tried changing the rest_handler.py to info and debug and can't get any of my custom rest calls to output anything to python.log. I'm very confused on why. I'm hoping someone else can post why in here, or this may be a bug. I know changing the base log.cfg as discussed below used to record everything python called. ... View more

I'm not sure why python.log isn't being updated. Might be worth importing logger into your script and use a try clause to catch your own exception. Then you could also log the updates / returns from when the script is ran. ... View more

but the calls to the lookup are being recorded in the python.log? ... View more

Modify etc/log.cfg for the following blocks: # # define splunk python logging properties # # logging classes are defined by a logging declaration at the log of each # file. # # splunk # splunk.appserver # splunk.search # [python] splunk = INFO splunk.appserver = INFO splunk.appserver.controllers = INFO splunk.appserver.lib = WARN Setting them to DEBUG should echo everything from each script inside the python.log file. ... View more

I just noticed you're using the search app, but you modified the unix app file, or did you copy these settings to the search app? Incase you missed, can you also post what the event looks like in splunk. If you modified the transforms only in the unix app, they may be app specific and you may need to globalize your configuration files. ... View more

Since the copied text is a subsearch, can you check and see if the full search is listed on the jobs page? If it's not, can you post the full search from the audit.log ([$SPLUNK_HOME]/var/log/splunk/) ... View more

if the values are marked as multivalue fields, you can pass them through the transaction command using the mvlist flag. After they are both listed on the transaction, you can then use streamstats to add them even-though they are the same name. index=iis sourcetype=iis-w3c (cs_method=POST cs_uri_stem=/page1) OR (cs_uri_stem=/page2) | transaction c_ip maxspan=12s startswith="cs_method=POST cs_uri_stem=/pagea" endswith="cs_method=GET cs_uri_stem=/pageb" mvlist=time_taken | streamstats sum(time_taken) as Total_Time_Taken window=1 this would mean if page1 has time_taken=1 and page2 has time_taken=2, the resulting transaction will have 2 time_taken fields, one marked 1 and one marked 2. Streamstats will then create a new field "Total_Time_Taken" and add all occurrences of "time_taken", resulting in a value of 3. ... View more

Can you post an example of the data set? also you don't need the |search *, index=myIndex| transaction myId maxspan=30m startswith="MsgNo=0" endswith="Hang up" should produce the same result with less overhead. ... View more