My use case is to find out how many transactions went out to a customer for a particular day. The results will include a tie out of counts and dollars from end to end. In some cases receipts go out to both a supplier and customer, so for that transaction I want to be able to see that it went out to both. TPCode is a field that we use in Splunk to differentiate all customers, and I have a lookup that lists the supplier TPCodes along with their customer TPCodes. These particular TPCodes that I want to differentiate are also a specific domain of customers, so simply grouping by TPCode will include other domains. So, my lookup consists of the the following headers (among others): supplier_TPCode, customer_TPCode Initially, my first thought is to do something like this in my search: | lookup ctg_supplier_customer_tpcode_lookup.csv supplier_TPCode AS TPCode OUTPUT customer_TPCode My issue is tying customer_TPCode in with the results to show that the transaction went out to supplier, customer,. or both. At an individual transaction level of results, my search looks like this: (index=gentran sourcetype=oms_invoice EDI=1 earliest=-19d@d latest=-12d@d eDiversity=Y) OR (index=contract_gateway sourcetype=esb_audit bp_bp_name=Invoice*) | lookup ctg_supplier_customer_tpcode_lookup.csv supplier_TPCode AS TPCode OUTPUT customer_TPCode,processing_type,split_flag,format | eval SourceSystem=case(isnotnull(SourceSystem), SourceSystem, sourcetype=="oms_invoice","OMS" ) | eval Amount=coalesce(Amount, InvoiceAmount) | eval endpoint_name=coalesce(Endpoint,svc_context_name) | eval transaction_format=coalesce(TransactionFormat,transaction_format) | rex field=msg_stuff "[^|]*\|(?<transaction_format>[^|]*)\|.*" | stats count as event_count, earliest(_time) AS start_time, latest(_time) as end_time, latest(bp_bp_name) as bp_bp_name, latest(Amount) as Amount, latest(TN_RECEIVERID) as customer_name values(TPCode) as TPCode, values(transaction_format) as transaction_format, latest(MasterAccountNumber) as MasterAccountNumber, latest(endpoint_name) as endpoint_name, latest(status) as exit_status, latest(other) as exit_message, values(SourceSystem) as SourceSystemValues, latest(bp_context_id) as bp_context_id, by InvoiceNumber | where like(SourceSystemValues, "%OMS%") From here I rollup the transactions to get a high level count of total invoices sent/not sent for these particular customers: | eval duration=tostring( (end_time - start_time) , "duration" ) | eval exit_status=upper(exit_status) | lookup temp_ctg_status_rollup_lookup.csv ctg_status AS exit_status OUTPUT ctg_rollup_status | eval failure_dollars=if(ctg_rollup_status=="FAILED", Amount, 0) | eval success_dollars=if(ctg_rollup_status=="SUCCESS", Amount, 0) | eval processing_dollars=if(ctg_rollup_status=="PROCESSING", Amount, 0) | stats count as total_invoice_count, sum(Amount) as sum_total, sum(failure_dollars) as sum_failure_rollup_dollars, count(eval(ctg_rollup_status=="FAILED")) as not_sent_count, sum(success_dollars) as sum_success_rollup_dollars, count(eval(ctg_rollup_status=="SUCCESS")) as sent_count, count(eval(ctg_rollup_status=="PROCESSING")) as should_have_been_processed_by_now, sum(processing_dollars) as sum_processing_rollup_dollars, by TPCode | table TPCode, total_invoice_count,sum_total,sum_failure_rollup_dollars,not_sent_count,sum_success_rollup_dollars,sent_count,should_have_been_processed_by_now,sum_processing_rollup_dollars How can I expand this search to add the customer_TPCode to determine for a particular day how many transactions went out to either supplier_tpcode, customer_TPCode, or both? If it helps I also have a split_type column in the lookup that tells us whether these customers are configured for receipts to go out to supplier, customer, or both, but by grouping by InvoiceNumber we should be able to tell from the results. Would it be best to just add all of the TPCodes in one column and search only on those? Any insight is greatly appreciated. Thank you! ... View more

I have a lookup that lists x number of values. I would like to be able to discover how many of those aren't actually logged in Splunk and table them. Initially, I tried something like this: inputlookup lookup_values.csv NOT [search index=contract_gateway sourcetype=esb_audit bp_bp_name=* | fields *] | table values | dedup values I am not getting any results, but I know I am missing results by the count of results I see when I pull all data that I can find in Splunk for that list of values. Any insights on query enhancements would be greatly appreciated. Thanks in Advance ... View more

I'm trying to find the best way to join the results of one search, and essentially feed that result set to match with another. Overall I want to correlate a list of invoices for a particular day. I am mostly able to do it with one search based on the logic of the systems and how things are logged. Another good-to-know is one file is being indexed manually while logging is being created (sourcetype=oms_invoice below). I can just do something like this: (index=gentran sourcetype=gentran_audit InvoiceNumber=* NOT InvoiceNumber=00 source="/log/file/path/wholeLog.20150610") OR (index=gentran sourcetype=oms_invoice InvoiceNumber=*) earliest=-9d@d latest=-7d@d Then I use stats to group stuff which allows us to bypass the 10,000 threshold limit: |stats count as event_count, earliest(_time) AS entered, latest(_time) as exited, values(SourceSystem) as SourceSystemValues latest(status) as last_seen_status latest(other) as last_seen_status_message by InvoiceNumber For this result I get 45,000 invoices went from one system to the next. Now I want to take this output and try to match with what we have in another environment. The problem here is matching by date will not work, as the time an invoice made it to that environment. may be different IE the following day. Without any unique identifier being logged in the first search above, I'm finding it difficult to tie out the invoice number list to show the complete correlation between the first two hops and the last. The main goal is to take the first result set and macth the same set of invoice numbers with what is in the last hop. Is there anyway I can take the first result set and match that invoice list with what is in that last hop? I'm thinking similarly on the lines of using inputlookup. The problem with that is the 10,000 limit of sub search, so I'd like to find a way to get around that. Any ideas and insight are much appreciated! ... View more

I'm working on sending requests to Splunk's REST API for the first time and have a few questions after reading through the tutorial/manual: I can't locate the /services/ directory under /splunk/home/ . Is this created out of the box, or something we need to add in /etc/apps ? Our dev environment is currently http - will sending requests to http://localhost:8080/... be sufficient for testing? Any insight is greatly appreciated. Thanks! ... View more

I have a CSV file that I would like to index one time only. There are two fields (Date, Time) that I want to be able to use as _time so that I can create a correlation of avg/median response times with data being forwarded in to a different index down the line. Are there attributes I can add to props.conf to help with this? Here is what my current props configurations look like: [oms_invoice_data] HEADER_MODE = firstline SHOULD_LINEMERGE = false TIME_PREFIX = ^ TIME_FORMAT = %Y%m%d,%H%M%S MAX_TIMESTAMP_LOOKAHEAD = 15 EXTRACT-oms_invoice_data=(?<Date>[^,]*),\s*(?<Time>[^,]*),\s*(?<Division>[^,]*),\s*(?<Customer>[^,]*),\s*(?<BillTo>[^,]*),\s*(?<DiversityCustomer>[^,]*),\s*(?<InvoiceTypes>[^,]*),\s*(?<EDI>[^,]*),\s*(?<PDV>[^,]*),\s*(?<PAPER>[^,]*),\s*(?<InvoiceNumber>[^,]*),\s*(?<InvoiceAmount>[^,]*),\s*(?<LegacyEDI>.*) Alternatively, is there a way I could manipulate the search to concatenate these values, and then later coalesce _time with time (custom time field) to be able to calculate earliest/latest _time and find durations that way? I tried doing something like this but things didn't work out as I had hoped (knowing I can't reassign _time respectively: index=main sourcetype=oms_invoice_data source="/tmp/oms_invoice_data_0504_0510.csv" EDI=1 OR LegacyEDI=1 InvoiceTypes=EDI* Date=20150504 | table Date,Time, InvoiceNumber InvoiceAmount, | mvexpand InvoiceNumber | eval SourceSystem="OMS" | eval Date = strftime(Date, "%Y/%m/%d") | eval Time = strftime(Time,"%H:%M:%S") | eval time=Date. " " .Time The weird thing here is that time is displayed as 1970/08/22 09:46:53 rather than 2015/05/04 09:46:53 Essentially what I am looking for is being able to calculate earliest(_time) and latest(_time) so that for one day, 05/04 for example), I can calculate the end-to-end avg and median response time for each transaction. Any thoughts on this would be greatly appreciated. Thanks in Advance! ... View more

I would like to graph by month/day of the week how many times we have restarted two servers in particular. Rather than using _internal, I want to use the index=os data coming in for these two hosts so that the search/ graph can be shared across a large distribution and not just the VIP Splunkers with admin access (only two of us :-)) The search I have will let us know when the hosts aren't forwarding data: index=_internal host=host01* OR host=host02* earliest=-15m | eval Age = (now() - _time ) | stats first(Age) as Age, first(_time) as LastTime by host | convert ctime(LastTime) as "Last Active On" | eval Status= case(Age < 180,"Running",Age > 180,"DOWN") | table host, Age, "Last Active On", Status I want to be able to bucket the time and visualize it via a timechart to show how many times the server was "restarted" over time. Any recommendations on a search that could help with this would be greatly appreciated. Thanks in advance! ... View more