topic Document Flow tracking over multiple sources. Linking unique field in Splunk Search

Document Flow tracking over multiple sources. Linking unique field

MorphiusX — Fri, 22 Jul 2022 04:44:20 GMT

Beginner user here.

PART 1
Wanting to track documents over multiple sources to ensure they reach their destination

Source 1 - Source 2 or 3 - Source 4
Start Point (Sent) - Middle Points (Accepted or Rejected) - End Point (Received)

Each document has the following
ID = Unique to each document
DATE \ TIME STAMP = Says what time the document arrived to that point
DESCRIPTION = like a subject what the document contains

All documents have a unique ID that is tracked on each source.
I want to track this ID and ensure that it has gone from source 1 ,2 or 3 and arrived at 4. If for some reason its in 2 and not in 4 display that Doc ID in a table.

PART 2 - I can probably work this one out myself after I know how to link everything.
After they are linked I would like to compare the time between when it was at source 1 to when it arrived at source 3.

Re: Document Flow tracking over multiple sources. Linking unique field

ITWhisperer — Fri, 22 Jul 2022 06:27:50 GMT

Something like this: obviously you will have to put in the details of your indexes, sourcetypes, etc.

<your index(es) and sourcetype(s)> | eval time1=if(source is 1, _time, null()) | eval time2=if(source is 2, _time, null()) | eval time3=if(source is 3, _time, null()) | eval time4=if(source is 4, _time, null()) | stats values(time1) as time1 values(time2) as time2 values(time3) as time3 values(time4) as time4 values(description) as description by docID

Re: Document Flow tracking over multiple sources. Linking unique field

PickleRick — Fri, 22 Jul 2022 07:34:21 GMT

You need something like this (a general idea - you have to tweak it to your situation)

ID=* ((index=src1 sourcetype=src1) OR (index=src2 sourcetype=src2) OR (index=src3 sourcetype=src3) OR (index=src4 sourcetype=src4))
| eval starttime=if(index=src1,_time,null())
| eval middletime=if(index=src2 OR index=src3,_time,null())
| eval endtime=if(index=src3,_time,null())
| stats values(starttime) AS starttime values(middletime) AS middletime values(endtime) AS endtime by ID

Now you should have a table listing timestamps when the doc with given ID was registered at each of waypoints - start, middle and end. You can easily filter out some of the results and get, for example, only those which have middle time but don't have endtime

| search middletime=* NOT endtime=*

And of course when you have those timestamps you can calculate the delay between various steps. For example

| eval start_to_end=endtime - starttime

Re: Document Flow tracking over multiple sources. Linking unique field

MorphiusX — Mon, 25 Jul 2022 04:26:30 GMT

Hi,
Thanks for the help.

With some troubleshooting, I was able to get it working. The one thing that messed me up was putting quotes around the source in the if statements.

My final code is

ID=* index=ind ((source=start) OR (source=accept) OR (source=reject) OR (source=received)) | eval time1=if(source="start", _time, null()) | eval time2=if(source="accept", _time, null()) | eval time3=if(source="reject", _time, null()) | eval time4=if(source="received", _time, null()) | stats values(time1) AS time1 values(time2) AS time2 values(time3) AS time3 values(time4) AS time4 by ID