How to extract Job and Status fields and table their corresponding values?

[chrismok]

Hi,

I have some logs that look like the sample below. If I use .net or java or SQL, I can solve it, but I really don't know how to figure out how to get the final results in Splunk.....

Deploy.joblist=A,B,C,D,E,F
Deploy.job.A.start=true
Deploy.job.E.start=true
Deploy.job.B.start=true
Deploy.job.A.status=Completed
Deploy.job.C.start=true
Deploy.job.B.status=Failed

Expected Result

Job.    Status
A        Completed
B        Failed
C         In Progress
D        Not found
E       In Progress
F      Not found

[martin_mueller · Accepted solution]

Does your data in Splunk look like the dummy data generated by this?

| stats count | eval data = "Deploy.joblist=A,B,C,D,E,F;Deploy.job.A.start=true;Deploy.job.E.start=true;Deploy.job.B.start=true;Deploy.job.A.status=Completed;Deploy.job.C.start=true;Deploy.job.B.status=Failed" | makemv data delim=";" | mvexpand data | streamstats count | eval _time = now() + count | rename data as _raw | table _time _raw

_time                 _raw
2014-09-18 15:35:20   Deploy.joblist=A,B,C,D,E,F
2014-09-18 15:35:21   Deploy.job.A.start=true
2014-09-18 15:35:22   Deploy.job.E.start=true
2014-09-18 15:35:23   Deploy.job.B.start=true
2014-09-18 15:35:24   Deploy.job.A.status=Completed
2014-09-18 15:35:25   Deploy.job.C.start=true
2014-09-18 15:35:26   Deploy.job.B.status=Failed

If so, you can append this to calculate the result table you had in mind in the question:

... | rex "Deploy\\.joblist=(?<job>[\w,]+)" | rex "Deploy\\.job\\.(?<job>\w+)\.(?:status|start)=(?<status>\w+)" | replace true with "In Progress" in status | eval status = coalesce(status, "Not Found") | makemv job delim="," | stats latest(status) by job

job   latest(status)
A     Completed
B     Failed
C     In Progress
D     Not Found
E     In Progress
F     Not Found

The two rexes extract the job and status fields. The replace beautifies the "true" to read "In Progress" instead. The eval sets up the fallback "Not Found" for all jobs listed in the first event. The makemv splits the list of jobs into a multivalue field. The stats computes the most recent status for each job according to _time.

[theouhuios]

Use can use rex if the data has been indexed already.

rex :

job.(?P<job>\w+)\.status=(?P<status>\S+)

[kenliz]

You have to understand splunk is advanced in handle and present the clean and format log, but not easy and have a limitation for developer to do that.

For best practice
1. Make the log event clearly
2. If the log format can't be restructured, better to write the converter to handle.
3. Or like you said, create the web page using your known programming language to do.