Как извлечь данные из массивов в массивах - PullRequest
Новая
       23

Как извлечь данные из массивов в массивах

0 голосов
/ 31 мая 2018

У меня есть JSON: 

{
  "count": 6918,
  "next": "https://otx.alienvault.com/api/v1/pulses/subscribed?limit=50&page=2",
  "results": [
    {
      "industries": [],
      "tlp": "white",
      "description": "With the massive ransomware campaigns of 2016 and 2017 taking a backseat to bankers and other malware families, information stealers made up 18% of malicious email payloads in the first part of this year. Proofpoint researchers recently discovered a new stealer, dubbed “Nocturnal Stealer,” most notable as an example of inexpensive commodity malware with significant potential for monetization.\n\nOn March 9, a user posted an advertisement for Nocturnal Stealer on an underground forum. The stealer sold for 1500 Rubles, or roughly US$25 at the time of analysis. Nocturnal Stealer is designed to steal the data found within multiple Chromium and Firefox based browsers. It can also steal many popular cryptocurrency wallets as well as any saved FTP passwords within FileZilla. Proofpoint researchers analyzed a sample being dropped in the wild by an unknown loader.",
      "created": "2018-05-31T12:25:14.636000",
      "tags": [],
      "modified": "2018-05-31T12:25:14.636000",
      "author_name": "AlienVault",
      "public": 1,
      "extract_source": [],
      "references": [
        "https://www.proofpoint.com/us/threat-insight/post/thief-night-new-nocturnal-stealer-grabs-data-cheap"
      ],
      "targeted_countries": [],
      "indicators": [
        {
          "indicator": "http://nctrnl.us/",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584318
        },
        {
          "indicator": "http://nctrnl.us/ara.exe",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584319
        },
        {
          "indicator": "http://nctrnl.us/ark.exe",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584320
        },
        {
          "indicator": "c9a834dde38c8b559d575ac61046e3a3fada97d2953d902b74cf8d5e51ada30f",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "FileHash-SHA256",
          "id": 981584321
        },
        {
          "indicator": "nctrnl.us",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "domain",
          "id": 981584322
        },
        {
          "indicator": "205def439aeb685d5a9123613e49f59d4cd5ebab9e933a1567a2f2972bda18c3",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "FileHash-SHA256",
          "id": 981584323
        },
        {
          "indicator": "ae7e5a7b34dc216e9da384fcf9868ab2c1a1d731f583f893b2d2d4009da15a4e",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "FileHash-SHA256",
          "id": 981584324
        },
        {
          "indicator": "http://nctrnl.us/server/gate.php",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584325
        }
      ],
      "more_indicators": false,
      "revision": 1,
      "adversary": "",
      "id": "5b0fe9aa8bc6a5498565929a",
      "name": "Thief in the night: New Nocturnal Stealer grabs data on the cheap"
    },
    ...
  ]
}

Основной массив - results, и я хотел бы получить поле name, а затем поля indicator и type из indicators array.

Я использовал команду JQ: 

$ jq -r '.results[] | [.name] | .indicators[] | [.type, .indicator] | @csv' \
    < /home/threat-intel/ThreatIntel/AV.json \
    >> /home/threat-intel/ThreatIntel/AV.csv

Команда не может сказать: 

jq: error (at <stdin>:0): Cannot index array with string "indicators"

Как бы я потянул поле name и назначилэто для каждого indicator в этом конкретном массиве?

Ответы [ 3 ]

0 голосов
/ 31 мая 2018

[.name] - это массив, и поэтому, когда вы передаете его в .indicators, вы получаете сообщение об ошибке:

Невозможно индексировать массив с помощью строки "Indicators"

Этот вариант вашего запроса, вероятно, соответствует вашим ожиданиям: 

.results[]
| .name as $name 
| .indicators[]
| [$name, .type, .indicator] | @csv

Сюда входит соответствующий .name в каждой строке: 

"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ara.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ark.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","c9a834dde38c8b559d575ac61046e3a3fada97d2953d902b74cf8d5e51ada30f"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","domain","nctrnl.us"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","205def439aeb685d5a9123613e49f59d4cd5ebab9e933a1567a2f2972bda18c3"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","ae7e5a7b34dc216e9da384fcf9868ab2c1a1d731f583f893b2d2d4009da15a4e"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/server/gate.php"
0 голосов
/ 01 июня 2018

Есть много способов, которыми это может быть достигнуто.При получении значений из дочерних массивов для генерации значений мне нравится использовать для этой цели foreach. 

$ jq -r '.results[] | foreach .indicators[] as $i ({name}; .;
    [.name,$i.type,$i.indicator]
) | @csv'
0 голосов
/ 31 мая 2018

jq решение: 

jq -r '.results[] | .name as $n | .indicators[] | [$n, .type, .indicator] | @csv' file.json

Выход: 

"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ara.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ark.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","c9a834dde38c8b559d575ac61046e3a3fada97d2953d902b74cf8d5e51ada30f"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","domain","nctrnl.us"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","205def439aeb685d5a9123613e49f59d4cd5ebab9e933a1567a2f2972bda18c3"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","ae7e5a7b34dc216e9da384fcf9868ab2c1a1d731f583f893b2d2d4009da15a4e"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/server/gate.php"
...