Displaying results for events in same index but having different data format

[swarjs]

We had a report for which the input CSV data format was 'value-only', but the format was modified to 'key-value' pair. Data with both the formats still went to the same index. The problem that we are facing is the data which was in "value-only" format is not turning up in the results now.

Previous csv data format

2019-12-16 05:10:00,default,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,0,0,3,0,0,0,0

Present CSV data format

timestamp=2019-12-15 09:00:00,view=default,rr_xxx=2,rr_yy=0,rr_w=0,rr_e=0,rr_t=0,rr_r=0,rr_k=0,rr_b=0

The search query format I am using is,
sourcetype=records index=test_index
| rename rr_xxx as "xxx Records", rr_yy as "y Records", rr_w as "w Records", rr_e as "e Records", rr_t as "t Records", rr_r as "r Records", rr_k as "k Records", rr_b as "b Records"
| eval Timestamp = strftime(_time, "%Y-%m-%d %H:%M:%S %Z")

| table Timestamp, "xxx Records", "y Records", "w Records", "e Records", "t Records", "r Records", "k Records", "b Records"

Would appreciate advice on how to handle this scenario.

[woodcock]

The problem is your rename because it always does something. If you do rename foo AS bar and some events do not have a field named foo, but do have a field named bar, the value for the field named bar will be replaced by null() because that is what the value of foo is for that event. It actually makes perfect sense. It would be nice if we had a boolean argument to this command to force it to do nothing for the null-value case but for now, you can do this with coalesce():

index="test_index" AND sourcetype="records"
| foreach rr_* [eval "<<MATCHSEG>> Records" = coalesce(<<FIELD>>, '<<MATCHSEG>> Records') | fields - <<FIELD>> ]

[gcusello]

Hi @swarjs,
it's just a little long but you could do something like this:

| makeresults | eval my_field="2019-12-16 05:10:00,default,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,0,0,3,0,0,0,0"
| append [ | makeresults | eval my_field="timestamp=2019-12-15 09:00:00,view=default,rr_xxx=2,rr_yy=0,rr_w=0,rr_e=0,rr_t=0,rr_r=0,rr_k=0,rr_b=0" ]
| rex field=my_field "^(?<field1>[^,]+),(?<field2>[^,]+),(?<field3>[^,]+)"
| rex field=my_field "^\w+\=(?<field1bis>[^,]+),\w+\=(?<field2bis>[^,]+),\w+\=(?<field3bis>[^,]+)"
| eval field1=coalesce(field1bis,field1),field2=coalesce(field2bis,field2),field3=coalesce(field3bis,field3)
| table _time my_field field1 field2 field3

Ciao.
Giuseppe