Выберите значение из 11-го столбца в качестве переменной - PullRequest
0 голосов
/ 15 марта 2020

Цель - извлечь IP-адрес из 11-го столбца и передать его в whois. Исходный файл - это первая строка заголовков, поэтому их следует игнорировать. затем я пытаюсь выбрать с помощью awk 11-й столбец.

Поскольку пропуск первой строки кажется слишком сложным (для меня), сейчас я оставил его пока. Любое хорошее предложение приветствуется.

Код пока:


while IFS= read -r p
do
DESTIP=$(awk  'BEGIN{FS=OFS=";"} {print $11}' $p)
echo "$DESTIP; $p"

   ORGNAME=$(whois $DESTIP|grep 'OrgName')
   COUNTRY=$(whois $DESTIP|grep 'Country')
   echo "$p;$ORGNAME;$COUNTRY" >>whois-results.txt
done < working-sorted.csv

Первые строки исходного файла:

timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short fo
r Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42  has made a TCP connection towards the dest
ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest
to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monitore
d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist
2020-03-14 13:53:45;20809;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;144.217.92.167;55134;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short f
or Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 144.217.92.167  has made a TCP connection towards the de
stination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we sugges
t to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monito
red network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist

Результат на данный момент:

awk: cmd. line:1: fatal: cannot open file `2020-01-19' for reading (No such file or directory)
DESTINATION IP=
Variable P= 2020-01-19 20:42:56;43;3;Remote Administration Tool;::ffff:ac8:c8d0/128;101;00:0c:29:4c:20:37;Vmware;172.16.16.100;54552;52.174.64.84;443;TCP;Connection to blacklisted destination

После настройки awk на: DESTIP=$(awk -v TEST='$p' 'BEGIN{FS=OFS=";"} {print $9;}') Я получаю IP-адреса из правильного столбца, но они находятся в одном списке, а не построчно, и не передаются в команды whois

Желательно вывод:

timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description;OrgName;Country;
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short fo
r Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42  has made a TCP connection towards the dest
ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest
to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monitore
d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;SomeName;SomeCountry

Пока я немного застрял. Помощь будет оценена.

Ответы [ 2 ]

1 голос
/ 15 марта 2020

Рассмотрим этот подход вместо вашей оболочки l oop:

$ cat tst.awk
BEGIN {
    numFlds = split("OrgName Country",nr2name)
    FS=OFS=";"
}
{ delete name2val }
NR == 1 {
    for (fldNr=1; fldNr<=numFlds; fldNr++) {
        fldName = fldVal = nr2name[fldNr]
        name2val[fldName] = fldVal
    }
}
NR > 1 {
    cmd = "whois \047" $9 "\047"
    while ( (cmd | getline line) > 0 ) {
        fldName = fldVal = line
        sub(/[[:space:]]*:.*/,"",fldName)
        sub(/[^:]+:[[:space:]]*/,"",fldVal)
        name2val[fldName] = fldVal
    }
    close(cmd)
}
{
    printf "%s%s", $0, OFS
    for (fldNr=1; fldNr<=numFlds; fldNr++) {
        fldName = nr2name[fldNr]
        fldVal = name2val[fldName]
        printf "%s%s", fldVal, (fldNr<numFlds ? OFS : ORS)
    }
}

.

$ awk -f tst.awk file
timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description;OrgName;Country
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short for Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42  has made a TCP connection towards the dest ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monitore d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;;
2020-03-14 13:53:45;20809;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;144.217.92.167;55134;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short for Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 144.217.92.167  has made a TCP connection towards the de stination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we sugges t to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monito red network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;OVH Hosting, Inc.;CA

, поскольку он дает именно тот результат, который вы хотели, он не потерпит неудачу, когда Например, Country появляется в одном из значений (ваш текущий сценарий оболочки завершится неудачно из-за ложного совпадения, например, с указанием названия компании «Большая страна»), и с этим вы сможете получить доступ к любому из выходных значений от whois, просто ссылаясь на их имя. Поэтому, если вы хотите дополнительно напечатать «OrgAbuseEmail», все, что вам нужно сделать, это изменить это:

numFlds = split("OrgName Country",nr2name)

на это:

numFlds = split("OrgName Country OrgAbuseEmail",nr2name)

В качестве альтернативы, это позволяет избежать порождения оболочки один раз за IP-адрес и так МОЖЕТ быть немного более эффективным, чем указано выше:

$ cat tst.sh
#!/bin/env bash

file="$1"

awk 'BEGIN{FS=OFS=";"} {print $9, $0}' "$file" |
while IFS=';' read -r ip all; do
    whois "$ip"
    printf '%s\n---\n' "$all"
done |
awk '
BEGIN {
    numFlds = split("OrgName Country",nr2name)
    for (fldNr=1; fldNr<=numFlds; fldNr++) {
        fldName = nr2name[fldNr]
        name2val[fldName] = fldName
    }
    FS = OFS = ";"
}
/^[[:alpha:]]+:/ {
    fldName = fldVal = $0
    sub(/[[:space:]]*:.*/,"",fldName)
    sub(/[^:]+:[[:space:]]*/,"",fldVal)
    name2val[fldName] = fldVal
}
/^---$/ {
    printf "%s%s", prev, OFS
    for (fldNr=1; fldNr<=numFlds; fldNr++) {
        fldName = nr2name[fldNr]
        fldVal = name2val[fldName]
        printf "%s%s", fldVal, (fldNr<numFlds ? OFS : ORS)
    }
    delete name2val
}
{ prev = $0 }
'

.

$ ./tst.sh file
timestamp (UTC);ID;Threat Level;Category;Exporter IP address;Observation domain ID (ODID);Source MAC;Manufacturer;Source IP;Source Port;Destination IP;Destination Port;Protocol;Description;OrgName;Country
2020-03-14 13:54:10;20810;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;118.25.123.42;49420;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short fo r Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 118.25.123.42  has made a TCP connection towards the dest ination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we suggest to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monitore d network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;;
2020-03-14 13:53:45;20809;5;Ingress Traffic;::ffff:ac8:c8d0/128;101;00:1a:8c:f0:c2:c0;Sophos;144.217.92.167;55134;172.16.16.150;22;TCP;Ingress connection to common SSH port:  100% CertaintyHigh Severity Category: SSH Description: Short f or Secure Shell Description: This connection represents an encrypted channel (SSH), which is commonly used in IT environments to connect to remote machines. Observations: Source IP 144.217.92.167  has made a TCP connection towards the de stination IP 172.16.16.150 (Private) on destination port 22. Advice:We recommend to investigate the following conditions:  1) Verify if it is expected for your network environment to generate SSH connections. If it is expected, we sugges t to disable this category. A network where developers and sysadmins often host their machines is an example of a network where a significant amount of SSH connections is expected.  2) If you are not expecting SSH traffic from the monito red network, it is recommended to investigate the endpoint according to your company security policies. If the destination 172.16.16.150 is trusted, it is recommended to add that SSH destination IP to the whitelist;OVH Hosting, Inc.;CA
0 голосов
/ 15 марта 2020

Я нашел обходной путь. может быть, не так хорошо, но, по крайней мере, работает: (редактировать: обходной путь теперь немного приятнее и менее «обходной» :) Код ниже корректируется по совету Сайруса.)

while IFS= read -r p
do
DESTIP=$(echo "$p" | awk 'BEGIN{FS=OFS=";"} {print $9;}')
echo "DESTINATION IP= $DESTIP"
echo "Variable P= $p"
   ORGNAME=$(whois $DESTIP|grep 'OrgName')
   COUNTRY=$(whois $DESTIP|grep 'Country')
   echo "$p;$domain;$ORGNAME;$COUNTRY" >>working-whois.csv
done < working-sorted.csv

Благодаря Сайрусу в этом, давая понять, что ему нужен входной файл.

...