I have a csv file input that is based on a data sampling method (takes the per-second average for a counter and records the result every 10 minutes), and needs to be multiplied by 600 to get the real number. For example, if I get a value for hits of 5.39, that means we actually had 3243 hits during that 10 minutes (5.39 * 600 = 3423).
Right now, when I do searches off the raw data, I have a very long macrothat multiplies each of the 12 counters by 600 for every event | eval hits=hits*600 | eval misses=misses*600... I'd like to move that to a .conf file, so it doesn't need to be done every time.
I have one possible solution (detailed below), but it's not perfect. Are there any other viable options?
One Solution (with a problem):
Paolo suggested that I try a scripted lookup to solve the problem. This worked, though it slowed the search down 25% (compared with 8% for the evals alone). That is perfectly reasonable for my needs, so it's not a problem. The other downside is that it seems you can't overwrite the original field. In effect, you can't do hits=hits*600 , but you can do myhits=hits*600 . You also can't fieldalias it afterward, or use OUTPUT myhits AS hits , or the lookup will balk. That's not ideal in general, but because I happen to be renaming the fields anyway, it meets my needs.
To implement, I put the following in props.conf:
[MySourceType]
Lookup-LookupField1 = LookupField1 Field1 OUTPUT MyField1
Lookup-LookupField2 = LookupField2 Field2 OUTPUT MyField2
...
Lookup-LookupField15 = LookupField15 Field15 OUTPUT MyField15
And in Transforms:
[LookupField1]
external_cmd = MultiplyAll.py Field1 MyField1
external_type = python
fields_list = Field1, MyField1
...
[LookupField15]
external_cmd = MultiplyAll.py Field15 MyField15
external_type = python
fields_list = Field15, MyField15
And then add MultiplyAll.py to your app's bin dir (this is probably filled with extra text -- I'm new to Python, so I grabbed most from external_lookup.py):
import sys,os,csv
def main():
if len(sys.argv) != 3:
print "Usage: python MultiplyAll.py [original field] [new field]"
sys.exit(0)
origf = sys.argv[1]
newf = sys.argv[2]
r = csv.reader(sys.stdin)
w = None
header = []
first = True
for line in r:
if first:
header = line
if origf not in header or newf not in header:
print "Original and New fields must exist in CSV data"
sys.exit(0)
csv.writer(sys.stdout).writerow(header)
w = csv.DictWriter(sys.stdout, header)
first = False
continue
# Read the result
result = {}
i = 0
while i < len(header):
if i < len(line):
result[header[i]] = line[i]
else:
result[header[i]] = ''
i += 1
# Do the math
if len(result[origf]) and len(result[newf]):
w.writerow(result)
elif len(result[origf]):
result[newf] = float(result[origf]) * 600
w.writerow(result)
elif len(result[newf]):
pass
main()
... View more
