Форматирование вывода Masscan

Question

У меня есть следующий вывод из опции Masscan's -oG:

# Masscan 1.0.6 scan initiated Mon May  6 08:45:19 2019
# Ports scanned: TCP(13107;1-13107) UDP(0;) SCTP(0;) PROTOCOLS(0;)
Host: 192.168.1.1 ()  Ports: 8000/open/tcp//unknown//
Host: 192.168.1.2 ()  Ports: 3478/open/tcp//unknown//
Host: 192.168.1.3 ()   Ports: 8000/open/tcp//unknown//
Host: 192.168.1.1 ()        Ports: 80/open/tcp//http//
Host: 192.168.1.2 ()        Ports: 443/open/tcp//https//
Host: 192.168.1.4 () Ports: 443/open/tcp//https//
Host: 192.168.1.3 () Ports: 80/open/tcp//http//
Host: 192.168.1.4 () Ports: 80/open/tcp//http//

Как я могу манипулировать этим выводом, используя awk, cut, grep, sed и т. Д., Чтобы получить следующий формат:

192.168.1.1 80,8000
192.168.1.2 443,3478
192.168.1.3 80,8000
192.168.1.4 80,443

Answer 1

попробуйте это:

#!/bin/bash

# define testcontent
content=$(cat << EOT
# Masscan 1.0.6 scan initiated Mon May  6 08:45:19 2019
# Ports scanned: TCP(13107;1-13107) UDP(0;) SCTP(0;) PROTOCOLS(0;)
Host: 192.168.1.1 ()  Ports: 8000/open/tcp//unknown//
Host: 192.168.1.2 ()  Ports: 3478/open/tcp//unknown//
Host: 192.168.1.3 ()   Ports: 8000/open/tcp//unknown//
Host: 192.168.1.1 ()        Ports: 80/open/tcp//http//
Host: 192.168.1.2 ()        Ports: 443/open/tcp//https//
Host: 192.168.1.4 () Ports: 443/open/tcp//https//
Host: 192.168.1.3 () Ports: 80/open/tcp//http//
Host: 192.168.1.4 () Ports: 80/open/tcp//http//
EOT
)

# declare associative array
declare -A dict 

# loop over all ip lines
while read -r ip port; do
   # save ports
   dict[$ip]+="$port "
         # ignore lines start with #, grep ip an port from content 
done < <(sed '/^#/d;s/Host: \([^ ]*\).*Ports: \([0-9]*\).*/\1 \2/' <<< "$content") 

# loop over assocative array
for key in  "${!dict[@]}"; do

   # sort ports in string
   sorted=$(echo "${dict[$key]}" | tr " " "\n" | sort -n | tr "\n" ,)

   # extract leading ,
   ports="${sorted#*,}"

   # print key an ports without tailing ,
   printf "%s %s\n" "$key" "${ports%,*}"
done | sort  

Вывод

192.168.1.1 80,8000
192.168.1.2 443,3478
192.168.1.3 80,8000
192.168.1.4 80,443

Answer 2

Комментарии в коде:

#!/bin/bash

# create the input file:
cat <<EOF >file
Host: 192.168.1.1 ()  Ports: 8000/open/tcp//unknown//
Host: 192.168.1.2 ()  Ports: 3478/open/tcp//unknown//
Host: 192.168.1.3 ()   Ports: 8000/open/tcp//unknown//
Host: 192.168.1.1 ()        Ports: 80/open/tcp//http//
Host: 192.168.1.2 ()        Ports: 443/open/tcp//https//
Host: 192.168.1.4 () Ports: 443/open/tcp//https//
Host: 192.168.1.3 () Ports: 80/open/tcp//http//
Host: 192.168.1.4 () Ports: 80/open/tcp//http//
EOF

# extract fields 2 and 5
<file awk '{print $2,$5}' |
# remove all that /open/tcp//https... part
sed 's@/.*@@' |
# Now merging is the worst part...
# script from https://stackoverflow.com/questions/19823941/join-lines-with-the-same-value-in-the-first-column
# This outputs `field1 , field2, field3, field4`
awk -F' ' -v OFS=' ' '{x=$1;$1="";a[x]=a[x]","$0}END{for(x in a) print x,a[x]}' |
# subsitute `, ` for `,` and remove the only remaining first ` ,`
sed 's/, /,/g' | sed 's/ ,/ /'

Скрипт выведет:

192.168.1.1 8000,80
192.168.1.2 3478,443
192.168.1.3 8000,80
192.168.1.4 443,80

Есть ли способ сортировки портов по возрастанию?

Конечно.Произведите числовую сортировку по второму столбцу (или по первому, а затем по второму столбцу) перед awk -ing.awk сохранит порядок.

# extract fields 2 and 5
<file awk '{print $2,$5}' |
# remove all that /open/tcp//https... part
sed 's@/.*@@' |
# numeric sort using the second column (ie. port)
sort -t' ' -n -k2 |
# Now merging is the worst part...
# script from https://stackoverflow.com/questions/19823941/join-lines-with-the-same-value-in-the-first-column
# This outputs `field1 , field2, field3, field4`
awk -F' ' -v OFS=' ' '{x=$1;$1="";a[x]=a[x]","$0}END{for(x in a) print x,a[x]}' |
# subsitute `, ` for `,` and remove the only remaining first ` ,`
sed 's/, /,/g' | sed 's/ ,/ /'

выведет:

192.168.1.1 80,8000
192.168.1.2 443,3478
192.168.1.3 80,8000
192.168.1.4 80,443