Home>

I am building freeradius on CentOS8.
I am trying to perform daily conversion processing from a csv file containing users, passwords, and attributes to the users format of freeradius.
Simple replacements, such as converting comma delimited to space delimited, are working fine, but I'm having trouble removing duplicate users.
For some reason, the csv file may have more than one line for the same user.

If there are duplicate users, I would like to adopt the last line (bottom line), but I didn't know how to do it.
I'm trying to achieve it with the awk command.

Corresponding source code

State until just before deleting duplicate lines by deleting commas etc. from the csv file
aaaaa Cleartext-Password: = "password1", Filter-Id: = "1001"
bbbbb Cleartext-Password: = "password2", Filter-Id: = "1002"
ccccc Cleartext-Password: = "password3", Filter-Id: = "1003"
ddddd Cleartext-Password: = "password4", Filter-Id: = "1004"
aaaaa Cleartext-Password: = "password5", Filter-Id: = "1005"
aaaaa Cleartext-Password: = "password6", Filter-Id: = "1006"

I want to do the following after executing the awk command
bbbbb Cleartext-Password: = "password2", Filter-Id: = "1002"
ccccc Cleartext-Password: = "password3", Filter-Id: = "1003"
ddddd Cleartext-Password: = "password4", Filter-Id: = "1004"
aaaaa Cleartext-Password: = "password6", Filter-Id: = "1006"

What I tried

When adopting the line that appears first ↓
aaaaa Cleartext-Password: = "password1", Filter-Id: = "1001"
bbbbb Cleartext-Password: = "password2", Filter-Id: = "1002"
ccccc Cleartext-Password: = "password3", Filter-Id: = "1003"
ddddd Cleartext-Password: = "password4", Filter-Id: = "1004"
We have confirmed that you can do the following.
awk'! colname [$1] ++ {print $1 "" $2 "" $3}'

Supplementary information (FW/tool version, etc.)

CentOS Linux release 8.2.2004
bash

  • Answer # 1

    You need to memorize everything once.
    It's easy if you don't have to save the line order.

    {X [$1] = $1 "" $2 "" $3} END {for (i in X) print X [i]}

  • Answer # 2

    How about this?

    tac input.txt | awk'! colname [$1] ++ {print $1 "" $2 "" $3}' | tac>output.txt