topic Re: Updating data added to Splunk in Splunk Enterprise

Updating data added to Splunk

GirolamoBo — Fri, 04 Dec 2015 21:21:21 GMT

I used ./splunk add oneshot “/your/log/file/myfile.log” –sourcetype myfile to add data to my instance of Splunk Light successfully. I followed steps from this blog post http://blogs.splunk.com/2014/03/21/search-command-coalesce/ to do it. Here is the content of the file:

Thu Mar 6 11:33:49 EST 2014 src_ip=1.1.1.1
Thu Mar 6 11:33:45 EST 2014 sourceip=8.1.2.3
Thu Mar 6 11:33:48 EST 2014 source_ip=1.1.1.0
Thu Mar 6 11:33:47 EST 2014 sip=1.1.1.199
Thu Mar 6 11:33:46 EST 2014 ip=
Thu Mar 6 11:33:46 EST 2014 ip=22.22.22.22

However when I made change to that file (added another line to it)

Thu Mar 7 11:33:46 EST 2014 ip=22.22.22.22

Splunk did not reflect the additional event. The same happens when I add a source using the UI, in Add Data>Upload files from your computer. Consequent changes to the file are not reflected in Splunk. Only the data that existed at the time of import is available.
What is the way to make Splunk to include later additions of content to the local file? I understand that this is a contrived example (adding an extra line manually is not what happens in real practice). Thank you.

Re: Updating data added to Splunk

ChrisG — Fri, 04 Dec 2015 21:24:16 GMT

oneshot is exactly that: a one-shot upload. You want to use monitor. See Monitor files and directories in the Splunk Enterprise Getting Data In manual.

Re: Updating data added to Splunk

GirolamoBo — Fri, 04 Dec 2015 21:42:33 GMT

Thank you @ChrisG When I try this command:

./splunk add monitor source "/Users/myuser/path/mylogs/firewall.log" -index newindex

I get this error:

Parameters must be in the form '-parameter value'

Re: Updating data added to Splunk

ChrisG — Fri, 04 Dec 2015 21:49:05 GMT

Try it without quotes:

./splunk add monitor source /Users/myuser/path/mylogs/firewall.log -index newindex

Re: Updating data added to Splunk

GirolamoBo — Fri, 04 Dec 2015 21:56:47 GMT

Unfortunately I tried it without quotes, with double quotes and with single quotes with the same outcome:
Parameters must be in the form '-parameter value'

Re: Updating data added to Splunk

ChrisG — Sat, 05 Dec 2015 00:01:47 GMT

Sorry!

./splunk add monitor -source /Users/myuser/path/mylogs/firewall.log -index newindex

./splunk add monitor "/Users/myuser/path/mylogs/firewall.log" -index newindex

I mashed up the two, my apologies.

Re: Updating data added to Splunk

malmoore — Sat, 05 Dec 2015 00:16:33 GMT

Hi,

In our discussion we found a conflation in the doc page. It doesn't say that you can use either

./splunk monitor -source <source>

./splunk monitor <source>

I've updated the page to say that you can use either of those, but I've changed the examples to remove the -source argument since you will hardly ever use it unless you specifically want to. Apologies for any confusion.

Re: Updating data added to Splunk

GirolamoBo — Sat, 05 Dec 2015 04:58:28 GMT

Thank you. I tried this: ./splunk add index -name "newindex"

which returned this: Index "newindex" added

then added monitor:

./splunk add monitor "/Users/myuser/Desktop/path/mylogs/firewall.log" -index newindex

which returned this, which seems to indicate success:

Added monitor of /Users/myuser/Desktop/path/mylogs/firewall.log

But the new sourcetype is not created. When I used add oneshot a new sourcetype was added. But add monitor command did not. Would you please let me know what else I am missing?

Re: Updating data added to Splunk

ChrisG — Sat, 05 Dec 2015 05:31:56 GMT

It should be there. Check the time range of your search to make sure that it includes the interval when you updated the file.

Re: Updating data added to Splunk

GirolamoBo — Sat, 05 Dec 2015 08:27:06 GMT

I used the content of the original file that is in my post. It has time intervals. I don't even see it as a sourcetype. I have not even tested updating it since it is not showing in Spunk. I also used
-sourcetype firewall with the add monitor command but still no sourcetype is created.