About mpdude

mpdude · ‎11-27-2019

Dear fellow Splunkers, I have a use case where I believe Splunk could provide great insight, alerts and dashboards, but I do not know if the way data has to be acquired makes it the right tool for the job. The data in question is timesheet reporting, with the additional challenge that timesheets might be updated (data entry errors fixed) later on. For example, I could run a script every day that would import records consisting of: ID/Name of the user Current timestamp = the time that data was read from the underlying operational system Timesheet period: Date, begin and end time Project being worked on Maybe additional categories So, it might happen that I import some of these tuples, but then – say the next day – re-run the import and one of the following happens: A particular period is no longer present, maybe because it has been deleted (time recorded by mistake) A particular period has changed in duration (e. g. forgot to stop timer) New periods are added (forgot to start timer) Would it be feasible to work with this data in Splunk at all? I guess the problem is that Splunk is not a (relational) database but an append-only index, right? I mean, how could I easily add to all relevant searches that for a particular day, only those events (imported records) are to be considered that have been imported at the time where data for that day has last been updated? Does that problem description make sense?

mpdude · ‎11-25-2017

Are you sure about | dedup count by tx_id, extra ? That seems to yield an empty result. What exactly should the dedup remove? I would like to keep the duplicate "error" event, maybe I got this wrong...?

mpdude · ‎11-25-2017

That almost did it 😉 but finally helped me to solve it – thanks! First, I found that the "base" result can be obtained much more efficiently by doing * [ search sourcetype="A" event="error" | fields tx_id ] I suspect that's because I have only few "A" type errors, but lots of "B" type events and so Splunk can avoid retrieving the unneeded events in the first place? Then, | stats values(*) AS * BY tx_id seems like a feat to me. Am I right that this is somewhat like using |transaction , in that it "groups" together all events with the same tx_id , but maintains the different field values? Does that create multi-valued fields per tx_id ? Last, in fact I had not only one extra field but two ( extra and extra2 ). I omitted those in the initial question for clarity. Thus, | stats values(url) AS url BY extra did not work out, but | stats values(extra), values(extra2) by url did. Final search: * [ search sourcetype="A" event="error" | fields tx_id ] | stats values(*) AS * BY tx_id | stats values(extra), values(extra2) by url Thanks!

mpdude · ‎11-25-2017

I have two sourcetypes with data as follows: First sourcetype: tx_id=1, event=error, extra=foo tx_id=1, event=error, extra=bar tx_id=2, event=info tx_id=2, event=error, extra=baz tx_id=3, event=info Second sourcetype: tx_id=1, url=/A tx_id=2, url=/B tx_id=3, url=/C What I would like to get is a table with all error events from the first log, the particular extra info and the corresponding url from the second log. That is, (ideally) the result would be extra=foo, url=/A extra=bar, url=/A extra=baz, url=/B I know SQL fairly well and this would be a plain, simple "join". But I just cannot get this to work in Splunk with a | transaction (which will not give me the two rows in the tx_id=1 case), a | join or | stats . As this must be really straightforward, I am probably missing the obvious and would appreciate any help or pointers. Thanks a lot!

mpdude · ‎04-22-2016

Dear fellow Splunkers, I have seen the docs on index-time field extractions and a few related answers here, there or there with the general guidance that an index-time extraction is rarely ever needed or beneficial. However, I have a dedicated index that holds Apache logfiles for a lot of different virtual hosts. I have set up search-time field extractions to get the apache_virtualhost and HTTP status code. Now following search index=web apache_virtualhost=some.virtual.host | timechart count by status is very slow and does not complete even after a few minutes, keeping the CPU 100% busy. However, index=web source=/path/to/logs/for/this/vhost-only.log | timechart count by status returns a result in an acceptable amount of time. Being used to relational DBs, I immediately thought "sure, in the second case Splunk can retrieve the small subset of matching rows from the index, whereas the first case needs to push all rows through the regexp first". But that is probably not the way Splunk works. So, is there an simple explanation for this? Have I found one of the rare cases where index-time field extraction would make sense? Thanks for sharing your insights!

Posts	5
Solutions	0
Karma Given	1
Karma Received	2
Member Since	‎03-01-2016

Online Status	Offline
Date Last Visited	‎06-18-2020 12:39 PM

Can Splunk process data that is "updated" over tim...

"Join" information from two log sources

Should I use an index-time field extraction?

Can Splunk process data that is "updated" over tim...

Re: "Join" information from two log sources

Re: "Join" information from two log sources

"Join" information from two log sources

Should I use an index-time field extraction?