панды json_normalize все столбцы имеют уплощенные словари

Question

У меня есть вложенный словарь (json), который был возвращен из неофициального API словаря Google.

Что выглядит так:

{'word': 'slack',
 'phonetic': '/slak/',
 'meaning': {'adjective': [{'definition': 'Not taut or held tightly in position; loose.',
    'example': 'a slack rope',
    'synonyms': ['loose',
     'limp',
     'not taut',
     'not tight',
     'hanging',
     'flapping']},
   {'definition': '(of business) characterized by a lack of work or activity; quiet.',
    'example': 'business was rather slack'},
   {'definition': 'Having or showing laziness or negligence.',
    'example': 'slack accounting procedures',
    'synonyms': ['lax',
     'negligent',
     'neglectful',
     'remiss',
     'careless',
     'slapdash',
     'slipshod',
     'lackadaisical',
     'lazy',
     'inefficient',
     'incompetent',
     'inattentive',
     'offhand',
     'casual',
     'disorderly',
     'disorganized']},
   {'definition': '(of a tide) neither ebbing nor flowing.',
    'example': 'soon the water will become slack, and the tide will turn'}],
  'noun': [{'definition': 'The part of a rope or line which is not held taut; the loose or unused part.',
    'example': 'I picked up the rod and wound in the slack',
    'synonyms': ['looseness', 'play', 'give']},
   {'definition': 'Casual trousers.'},
   {'definition': 'A spell of inactivity or laziness.',
    'example': 'he slept deeply, refreshed by a little slack in the daily routine',
    'synonyms': ['lull',
     'pause',
     'respite',
     'spell of inactivity',
     'interval',
     'break',
     'hiatus',
     'breathing space']}],
  'verb': [{'definition': 'Loosen (something, especially a rope).'},
   {'definition': 'Decrease or reduce in intensity, quantity, or speed.',
    'example': 'the flow of blood slacked off',
    'synonyms': ['reduce',
     'lessen',
     'slacken',
     'slow',
     'ease off',
     'ease up']},
   {'definition': 'Work slowly or lazily.',
    'example': 'she reprimanded her girls if they were slacking',
    'synonyms': ['idle',
     'shirk',
     'be inactive',
     'be lazy',
     'be indolent',
     'sit back and do nothing',
     'waste time',
     'lounge about']},
   {'definition': 'Slake (lime).'}],
  'adverb': [{'definition': 'Loosely.',
    'example': 'their heads were hanging slack in attitudes of despair'}]}}

Это значение слова slack . Чтобы получить это значение, мы можем гуглить значение или просто использовать следующий код:

import numpy as np
import pandas as pd
import json
from pandas.io.json import json_normalize
from io import StringIO
import requests

word = 'slack'
url =  'https://googledictionaryapi.eu-gb.mybluemix.net/?define=' + word
response = requests.get(url)
content = response.content.decode('utf-8') # list of ugly strings
j = json.loads(content) # json list having nested dictionary
j = j[0]
j

Теперь в словаре j есть три ключа.

j.keys() # dict_keys(['word', 'phonetic', 'meaning'])

Меня в основном интересует значение:

j['meaning'].keys() # dict_keys(['adjective', 'noun', 'verb', 'adverb'])

Чтобы получить фрейм данных панд, я использовал следующий код:

json_normalize(data=j['meaning'])

Это дает фрейм данных только с 4 столбцами.

Здесь каждая часть речи (прилагательное, существительное и т. Д.) Должна иметь ключ «определения», а «пример» и «синонимы» являются необязательными.

j['meaning']['adjective'][0].keys() # dict_keys(['definition', 'example', 'synonyms'])

Как получить кадр данных с 4 * 3 = 12 столбцами, с именами столбцов, такими как adjective_definition, adjective_example, ...., verb_synonyms?

Я попытался получить некоторые идеи по следующим ссылкам:

http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.io.json.json_normalize.html
https://www.kaggle.com/jboysen/quick-tutorial-flatten-nested-json-in-pandas/notebook
pandas.io.json.json_normalize с очень вложенным json

Но не смог решить проблему. Помощь будет оценена.

Answer 1

Я думаю, что использование json_normalize ' record_path решит вашу проблему. Поскольку record_path предназначен для использования в качестве одного пути к списку json-объектов или записей, мне пришлось вызывать json_normalize более одного раза, а затем объединять результаты, чтобы получить фрейм данных с нужными вам данными. Вы также можете поэкспериментировать с параметром record_prefix , чтобы установить соглашение об именовании столбцов. Надеюсь, это поможет!

from pandas.io.json import json_normalize
from io import StringIO
import requests

word = 'slack'
url =  'https://googledictionaryapi.eu-gb.mybluemix.net/?define=' + word
response = requests.get(url)
content = response.content.decode('utf-8') # list of ugly strings
j = json.loads(content) # json list having nested dictionary
j = j[0]

df_adj = json_normalize(data=j['meaning'], record_path=["adjective"], record_prefix="adjective.")
df_verb = json_normalize(data=j['meaning'], record_path=["verb"], record_prefix="verb.")
df_adv = json_normalize(data=j['meaning'], record_path=["adverb"], record_prefix="adverb.")
df_noun = json_normalize(data=j['meaning'], record_path=["noun"], record_prefix="noun.")

df = pd.concat([df_adj, df_verb, df_adv, df_noun], axis=1)
print(df.head(3))

Answer 2

На самом деле идея 12 столбцов считалась не самой лучшей, тогда, немного поиграв с кодом, я получил гораздо лучший результат.

import numpy as np
import pandas as pd
import json
from pandas.io.json import json_normalize
import requests

word = 'slack'
url =  'https://googledictionaryapi.eu-gb.mybluemix.net/?define=' + word
response = requests.get(url)
content = response.content.decode('utf-8') # list of ugly strings
data = json.loads(content) # json list having nested dictionary
data = data[0]

df = pd.DataFrame()
for i in data['meaning'].keys():
    x = json_normalize(data=data['meaning'][i])
    x['part_of_speech'] = i
    df = df.append(x,sort=False)

df = df[['part_of_speech', 'definition', 'example',  'synonyms']]
df

Дает результат:

    part_of_speech  definition  example synonyms
0   adjective   Not taut or held tightly in position; loose.    a slack rope    [loose, limp, not taut, not tight, hanging, fl...
1   adjective   (of business) characterized by a lack of work ...   business was rather slack   NaN
2   adjective   Having or showing laziness or negligence.   slack accounting procedures [lax, negligent, neglectful, remiss, careless,...
3   adjective   (of a tide) neither ebbing nor flowing. soon the water will become slack, and the tide...   NaN
0   noun    The part of a rope or line which is not held t...   I picked up the rod and wound in the slack  [looseness, play, give]
1   noun    Casual trousers.    NaN NaN
2   noun    A spell of inactivity or laziness.  he slept deeply, refreshed by a little slack i...   [lull, pause, respite, spell of inactivity, in...
0   verb    Loosen (something, especially a rope).  NaN NaN
1   verb    Decrease or reduce in intensity, quantity, or ...   the flow of blood slacked off   [reduce, lessen, slacken, slow, ease off, ease...
2   verb    Work slowly or lazily.  she reprimanded her girls if they were slacking [idle, shirk, be inactive, be lazy, be indolen...
3   verb    Slake (lime).   NaN NaN
0   adverb  Loosely.    their heads were hanging slack in attitudes of...   NaN

Answer 3

Вид грязного решения, но я думаю, что оно работает. Начиная с j в качестве примера словаря:

res = pd.concat([json_normalize(v, meta=['definition', 'example', 'synonyms']).add_prefix(k + '_')
                 for k, v in j['meaning'].items()], 
                axis=1)

# The output is super wide and hard to read in console output,
# but hopefully this confirms the output is (close to) what you need
res
                                                adjective_definition  \
0                       Not taut or held tightly in position; loose.   
1  (of business) characterized by a lack of work or activity; quiet.   
2                          Having or showing laziness or negligence.   
3                            (of a tide) neither ebbing nor flowing.   

                                          adjective_example  \
0                                              a slack rope   
1                                 business was rather slack   
2                               slack accounting procedures   
3  soon the water will become slack, and the tide will turn   

                                                                adjective_synonyms  \
0                            [loose, limp, not taut, not tight, hanging, flapping]   
1                                                                              NaN   
2  [lax, negligent, neglectful, remiss, careless, slapdash, slipshod, lackadais...   
3                                                                              NaN   

                                                                noun_definition  \
0  The part of a rope or line which is not held taut; the loose or unused part.   
1                                                              Casual trousers.   
2                                            A spell of inactivity or laziness.   
3                                                                           NaN   

                                                        noun_example  \
0                         I picked up the rod and wound in the slack   
1                                                                NaN   
2  he slept deeply, refreshed by a little slack in the daily routine   
3                                                                NaN   

                                                                     noun_synonyms  \
0                                                          [looseness, play, give]   
1                                                                              NaN   
2  [lull, pause, respite, spell of inactivity, interval, break, hiatus, breathi...   
3                                                                              NaN   

                                        verb_definition  \
0                Loosen (something, especially a rope).   
1  Decrease or reduce in intensity, quantity, or speed.   
2                                Work slowly or lazily.   
3                                         Slake (lime).   

                                      verb_example  \
0                                              NaN   
1                    the flow of blood slacked off   
2  she reprimanded her girls if they were slacking   
3                                              NaN   

                                                                     verb_synonyms  \
0                                                                              NaN   
1                               [reduce, lessen, slacken, slow, ease off, ease up]   
2  [idle, shirk, be inactive, be lazy, be indolent, sit back and do nothing, wa...   
3                                                                              NaN   

  adverb_definition                                          adverb_example  
0          Loosely.  their heads were hanging slack in attitudes of despair  
1               NaN                                                     NaN  
2               NaN                                                     NaN  
3               NaN                                                     NaN