Getting Data In

Split views, adding date to incomplete timestamp, and more

New Member

Hello,

I just started evaluating Splunk. So please apologize if I should ask for the obvious.

My test case is a post-mortem analysis of three interacting Linux applications. Each writes its own log files. I have just installed Splunk on a Windows machine, created a test index, and indexed excerpts from those three files that cover the time interval I'm interested in (about half an hour).

  • Does Splunk offer a split view of the events from the three different sources? I would like to see events that occurred at the "same" time, e.g. within a specific second, side by side.

  • One application's timestamps do not contain the date. I can't modify the application. Can I "suggest" a date to Splunk?

  • Can I delete all data from just one of those three sources?

  • How do I delete all the test data I've indexed?

  • Looking at Manager > Indexes, there are event in _audit and _internal. Where do these come from?

Thanks in advance, Malte

Tags (3)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

Taking your questions one by one:

Does Splunk offer a split view of the events from the three different sources? I would like to see events 
that occurred at the "same" time, e.g. within a specific second, side by side.

You can create your own view, that would offer something similar to that (though less robust than I imagine you're thinking -- e.g., two by two, or stacked), but the general theory with Splunk would be that you look at the event stream, with the events intermingled. If your timestamps are accurate, that will show you everything in one stream.

One application's timestamps do not contain the date. I can't modify the application. 
Can I "suggest" a date to Splunk?

I think this should get you up and running, but if you have your logs going in on a regular basis, it should work just fine. I'm not exactly sure how you'd have to set it up if you want to batch import past logs, so if that's your use case, you might need to open a different Splunk answers specificically for that issue. http://answers.splunk.com/questions/9531/extract-timestamp-without-date

Can I delete all data from just one of those three sources?

The easiest way is to grant yourself the can_delete role and then you can do a search for source="xyz" | delete. Note that this will make it not reported by Splunk, but won't actually remove it from the filesystem. For that, you would need to put each source in a different index and then...

How do I delete all the test data I've indexed?

From the command line, you can run splunk clean eventdata -index myindex (or without the index, to do the entire system). See this for more: http://www.splunk.com/base/Documentation/latest/Admin/RemovedatafromSplunk#Remove_data_from_indexes_...

Looking at Manager > Indexes, there are event in _audit and _internal. Where do these come from?

Those are internal splunk indexes. One contains audit information (searches run, etc.) and the other contains statistics about throughput, license information, etc. You can look at the contents of them by running a search for index=_audit or index=_internal or even index=_audit OR index=_internal

http://www.splunk.com/base/Documentation/latest/Admin/AuditSplunkactivity

View solution in original post

Splunk Employee
Splunk Employee

Taking your questions one by one:

Does Splunk offer a split view of the events from the three different sources? I would like to see events 
that occurred at the "same" time, e.g. within a specific second, side by side.

You can create your own view, that would offer something similar to that (though less robust than I imagine you're thinking -- e.g., two by two, or stacked), but the general theory with Splunk would be that you look at the event stream, with the events intermingled. If your timestamps are accurate, that will show you everything in one stream.

One application's timestamps do not contain the date. I can't modify the application. 
Can I "suggest" a date to Splunk?

I think this should get you up and running, but if you have your logs going in on a regular basis, it should work just fine. I'm not exactly sure how you'd have to set it up if you want to batch import past logs, so if that's your use case, you might need to open a different Splunk answers specificically for that issue. http://answers.splunk.com/questions/9531/extract-timestamp-without-date

Can I delete all data from just one of those three sources?

The easiest way is to grant yourself the can_delete role and then you can do a search for source="xyz" | delete. Note that this will make it not reported by Splunk, but won't actually remove it from the filesystem. For that, you would need to put each source in a different index and then...

How do I delete all the test data I've indexed?

From the command line, you can run splunk clean eventdata -index myindex (or without the index, to do the entire system). See this for more: http://www.splunk.com/base/Documentation/latest/Admin/RemovedatafromSplunk#Remove_data_from_indexes_...

Looking at Manager > Indexes, there are event in _audit and _internal. Where do these come from?

Those are internal splunk indexes. One contains audit information (searches run, etc.) and the other contains statistics about throughput, license information, etc. You can look at the contents of them by running a search for index=_audit or index=_internal or even index=_audit OR index=_internal

http://www.splunk.com/base/Documentation/latest/Admin/AuditSplunkactivity

View solution in original post

Splunk Employee
Splunk Employee

I don't think there is any way, out of the box, to have them scroll in a synchronized fashion. That said, you can access any of the javascript that makes splunk work, so you could certainly code that process yourself. That's getting well above my head, though, and there may well be a much simpler way to get it done. If you're going through a sales channel for your eval, that might be a good question to have them relay. Alternatively, if you don't get another reply to this, you might want to open another question specifically for that issue, to see if anyone else can think of a better way.

0 Karma

New Member

Thanks for your help!

I think, splicing events from different sources into one stream is a good concept. Unfortenatly, it probably does not work in my case because the resolution of the timestamps varies between the sources and it is not fine enough for the interdependencies between the applications to be properly represented in the result stream.

I assume I might manage to build a custom view that shows data from multiple sources side by side. But would it be possible to scroll those views in a synchronized fashion, i.e. based on their timestamps?

0 Karma