How to do a real-time search with inputcsv?

[nick405060]

Per the real-time search documentation, you cannot use inputcsv in a real-time search. I'm looking to display real-time deviation from normality, where dynamically-generated CSVs are used to determine the normality baseline.

1) How would I go about doing this if the dynamically-generated CSVs are generated by daily Splunk reports?
2) How would I go about doing this, assuming the dynamically generated CSVs are externally generated? (This second question only needs to be answered if the first question does not answer it)

[DalJeanis · Accepted solution]

Two things to consider -

1) Do you need realtime, or near-real-time? Many people think they want realtime, when the actual SLA is several minutes or even hours. A scheduled search that runs every 2 minutes provides the same functionality as real-time, with less stress on the machine.

Here are two places where we discuss that ...
https://answers.splunk.com/answers/681681/in-real-time-alert-if-i-use-lookup-command-too-man.html
https://answers.splunk.com/answers/678548/are-real-time-dashboards-for-noc-putting-too-much.html

2) Is there a reason to use a CSV rather than a lookup? If you load the CSV into a lookup table, and use inputlookup with append=t some of your limitations may go away. but only do that if you really need realtime.

https://answers.splunk.com/answers/111220/real-time-search-url-loader-and-inputlookup-issue.html
https://answers.splunk.com/answers/132490/how-to-reload-inputlookup-csv-file-used-in-a-saved-realtim...

[nick405060]

Thanks for your answer! I partially took your suggestion and used a lookup (lookup, not inputlookup) and that worked. Initially I tried to fix the problem using a post-process search in my XML code, where the base search used inputcsv and then the post-process search was in real-time and joined with the normality baseline that had been inputted. Turns out that didn't work either, because you can neither inputcsv nor join in a real-time search.

So, I just used a lookup as you said, edited transforms.conf, and made sure my dynamic csvs were being put in the etc/apps/my_app/local/lookups directory. If this helps anyone else, I wrote the following Python script to use in a Splunk search to move files around (this specific code only takes an item in var/run/splunk, and moves it to var/run/splunk/... rest of path here, but can be ../../../ ...):

# movecsv file directory
# ... where directory is in Splunk/var/run/splunk

import shutil
import sys

file = sys.argv[1]
dir = sys.argv[2]

shutil.move('C:\\Program Files\\Splunk\\var\\run\\splunk\\'+file, 'C:\\Program Files\\Splunk\\var\\run\\splunk\\'+dir+'\\'+file)

[nick405060]

It needs to be real-time - we're looking to have a display that shows all incoming emails in real-time that deviate from our normality metric

[DalJeanis]

@nick405060 - Please reread those first two references. Will a human or automated system need to react to whatever is pictured on the dashboard in less than 90 seconds? 120 seconds? 150 seconds?

If that fast real-time reaction by humans is required as an SLA, then you may have a real time use case. If not, then what you want is standard near-real-time... a search that runs every 1m or 2m or 3m and refreshes the dash.

Do not let a user or a manager fall in love with the term "real time". Nothing is actually real time, and the thing that is called a "real time" search is actually a constantly updating stream of not-yet-complete information. This is because almost every installation has a minute or so delay before transmission and ingestion of any kind of data is complete, so those last "real-time" data points are wrong. Always.

The key is to always ask your management not "do you want real time" but "what is the SLA to respond to information on the dash?" If the answer is more than 5 minutes before a human must DO something, then it is clearly not a real time requirement.

"So, when you say you want real time, you mean you want the most recent complete data that is available, correct?"

"Yes"

"Okay, our typical transmission and ingestion delay is 45 seconds, with occasional spikes to 65 seconds, so we will refresh the dash every 1m with a 1m delay. That way, the most recent data point will be 1 minute past, and the last data point will be on at least 97 percent complete."

That's how to handle the politics... with the technical truth.

If multiple real time dashboards proliferate, you will degrade your entire Splunk installation. Unless that ONE particular dashboard is the most critical dashboard in your environment, you should protect your environment by using a standard high-frequency update instead, tuned to exploit the actual ingestion time for that data.

[nick405060]

Our use case is real-time, without any delay on top of a sunk 45-65s ingestion delay. I understand that delay is very small, especially since the 60 second refresh is going to cover most of that sunk ingestion delay anyways, but real-time DOES get you a faster display time on average, and there is benefit to seeing each data point as it comes in for human readability and situational awareness purposes. Coming from the DoD, I don't think it's exactly rare or a foreign concept to need a true real-time display. Unless I'm missing something, the harm from using real-time displays is minimal - if your installation routinely gets overloaded, turn your less important real-time displays into a refresh or disable them temporarily.

It is a bit disappointing that Splunk as a SIEM can't handle real-time well. I imagine that's a major weakness they want to patch in the near future.