https://community.splunk.com/t5/Getting-Data-In/How-to-use-K-anonymity-with-Splunk/m-p/347864#M63877 <P>I don't understand the reason for your business case, but here is what I would do to achieve your stated objective. Instead of running it through a standard indexing, I would bring it in, aggregate it, and then collect the aggregated data into a summary index. </P> <PRE><CODE> | inputcsv mystuff.csv | rename COMMENT as "Assign every pair of records to a group, then stats the group together " | streamstats count as recno | eval groupno = floor( ( 1+ recno ) / 2 ) | stats list(User) as User list(Age) as Age by groupno | rename COMMENT as "Set time, Get rid of unneeded fields, then copy them to the new index." | eval _time = now() | table _time User Age | collect .... send to desired index... </CODE></PRE> <P>If you want to break the link of order between each <CODE>User</CODE> and his <CODE>Age</CODE>, then do this to sort the fields after the <CODE>stats</CODE> command. This will break the relationship between any individual Age and its User.</P> <PRE><CODE> | stats list(User) as User list(Age) as Age by groupno | eval User=mvsort(User) | eval Age=mvsort(Age) </CODE></PRE> <P>If you want to change the number of records in each group to some number K, change line 5 to use your new K-1 and K as follows:</P> <PRE><CODE> | eval groupno = floor( ( K-1 + recno ) / K ) </CODE></PRE> <HR /> <P>Updated to remove suggestion to use <CODE>values</CODE>, since that would delete duplicates. </P> <HR /> <P>There is also an issue with this anonymization method if using K=2 or k=3 and all of the Users have the same Age. Sigh.</P> <PRE><CODE> | inputcsv mystuff.csv | rename COMMENT as "make sure that no two of the same Age are sequential." | streamstats count as ageno by Age | eventstats count as totalcount | eventstats max(ageno) as agecount by Age | eval myorder=round((ageno-0.5)/agecount,2) | sort 0 myorder User | rename COMMENT as "Assign every pair of records to a group, then stats the group together " | streamstats count as recno | eval groupno = floor( ( 1+ recno ) / 2 ) | stats list(User) as User list(Age) as Age by groupno | rename COMMENT as "Set time, Get rid of unneeded fields, then copy them to the new index." | eval _time = now() | table _time User Age | collect .... send to desired index... </CODE></PRE> <P>To work, the above depends on no one Age predominating in the data set. </P> Mon, 25 Sep 2017 15:19:09 GMT DalJeanis 2017-09-25T15:19:09Z https://community.splunk.com/t5/Getting-Data-In/How-to-use-K-anonymity-with-Splunk/m-p/347863#M63876 <P>Hello,<BR /> Let's say i have a csv file that contains sensitive data, I want on index to group multiple lines as one event in a way that it doesnt compromise my data. So let's say:<BR /> User - Age<BR /> U1 - 12<BR /> U2 - 13<BR /> U3 - 17<BR /> U4 - 15<BR /> U5 - 20<BR /> How can I group for example each 2 users as one event as so(of course before indexing and not on search time):<BR /> U1,U2 - 12,13<BR /> U3,U4 - 17-15<BR /> ...<BR /> Thanks in advance</P> Mon, 25 Sep 2017 12:21:27 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-use-K-anonymity-with-Splunk/m-p/347863#M63876 MarcHelou 2017-09-25T12:21:27Z https://community.splunk.com/t5/Getting-Data-In/How-to-use-K-anonymity-with-Splunk/m-p/347864#M63877 <P>I don't understand the reason for your business case, but here is what I would do to achieve your stated objective. Instead of running it through a standard indexing, I would bring it in, aggregate it, and then collect the aggregated data into a summary index. </P> <PRE><CODE> | inputcsv mystuff.csv | rename COMMENT as "Assign every pair of records to a group, then stats the group together " | streamstats count as recno | eval groupno = floor( ( 1+ recno ) / 2 ) | stats list(User) as User list(Age) as Age by groupno | rename COMMENT as "Set time, Get rid of unneeded fields, then copy them to the new index." | eval _time = now() | table _time User Age | collect .... send to desired index... </CODE></PRE> <P>If you want to break the link of order between each <CODE>User</CODE> and his <CODE>Age</CODE>, then do this to sort the fields after the <CODE>stats</CODE> command. This will break the relationship between any individual Age and its User.</P> <PRE><CODE> | stats list(User) as User list(Age) as Age by groupno | eval User=mvsort(User) | eval Age=mvsort(Age) </CODE></PRE> <P>If you want to change the number of records in each group to some number K, change line 5 to use your new K-1 and K as follows:</P> <PRE><CODE> | eval groupno = floor( ( K-1 + recno ) / K ) </CODE></PRE> <HR /> <P>Updated to remove suggestion to use <CODE>values</CODE>, since that would delete duplicates. </P> <HR /> <P>There is also an issue with this anonymization method if using K=2 or k=3 and all of the Users have the same Age. Sigh.</P> <PRE><CODE> | inputcsv mystuff.csv | rename COMMENT as "make sure that no two of the same Age are sequential." | streamstats count as ageno by Age | eventstats count as totalcount | eventstats max(ageno) as agecount by Age | eval myorder=round((ageno-0.5)/agecount,2) | sort 0 myorder User | rename COMMENT as "Assign every pair of records to a group, then stats the group together " | streamstats count as recno | eval groupno = floor( ( 1+ recno ) / 2 ) | stats list(User) as User list(Age) as Age by groupno | rename COMMENT as "Set time, Get rid of unneeded fields, then copy them to the new index." | eval _time = now() | table _time User Age | collect .... send to desired index... </CODE></PRE> <P>To work, the above depends on no one Age predominating in the data set. </P> Mon, 25 Sep 2017 15:19:09 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-use-K-anonymity-with-Splunk/m-p/347864#M63877 DalJeanis 2017-09-25T15:19:09Z