Надо разделить словарь

Question

У меня есть данные ниже в формате CSV (это файл, разделенный запятыми, первая строка - заголовки столбцов)

ID,ENV,dictionary_column
35702,name1,"{'Employee': 1.56, 'IC': 1.18}"
35700,nam22,"{'Quota': 3.06, 'ICS': 0.37}"
11765,quotation,"{'02 WSS': 12235, '44 HR Part': 485, '333 CNTL':1}"
22345,gamechanger,"{'02 Employee's': 5.1923513, '04 Participant': 0.167899}"
22345,supporter,"{'0': '31', 'Table': '5', 'NewAssignee': '1', 'Result': '5'}"

Столбец dictionary_column содержит несколько пар ключ-значение, которые мне нужно выделить и объединить с оставшимися столбцами

Желаемый вывод (csv или dataframe):

ID      ENV        dictionary_key       dictionary_value
35702   name1      Employee abc         1.56
35702   name1      IC                   1.18
35700   nam22      Quotation            3.06
35700   nam22      IC newer             0.37
35700   nam22      newmeansnew          0.001
11765   quotation  02 WSS               12235
11765   quotation  44 HR Part           485
11765   quotation  333 CNTL             1
........ .......   ...                  ... (likewise)

(Не обращайте внимания на пробелы в выводе, добавленные для форматирования или читабельности)

The dictionary_column values example : 

"{'0': '31', 'Table': '5', 'NewAssignee': '1', 'Result': '5'}"

this is the trouble part

Я пробовал мало вещи из функции ast, а также пытались преобразовать dict в json на json .normalize, но с 10k строками любой метод не дает правильных результатов

Answer 1

Решение для требуемого выхода

import json
import pandas as pd

with open("the.csv") as f:
    next(f)
    lines = [x.strip() for x in f]

vals = ""
valLst = []
for line in lines:
    parts = line.split(",") # file seems separated by 3 spaces or \t, adjust if needed

    flag = False
    nextParts = ""
    for part in parts: 
        if part.startswith('\"'): 
            flag = True
        if flag:
            nextParts = nextParts +','+ part


    nextParts = nextParts.strip(',')
    nextParts = nextParts.strip('"')
    nextParts = nextParts.replace('\'', "\"")


    for k, v in json.loads(nextParts).items(): # json.loads() excepts values enclosed in double quote, not single
        valLst.append([parts[0], parts[1], k, v])


df = pd.DataFrame(valLst, columns=["ID", "ENV", "dictionary_key", "dictionary_value"])

Answer 2

Вы можете использовать:

import json
import pandas as pd

with open("the.csv") as f:
    next(f)
    lines = [x.strip() for x in f]

vals = []
for line in lines:
    parts = line.split("   ") # file seems separated by 3 spaces or \t, adjust if needed
    for k, v in json.loads(parts[2].replace("'", "\"")).items(): # json.loads() excepts values enclosed in double quote, not single
        vals.append([parts[0], parts[1], k, v])

df = pd.DataFrame(vals, columns=["ID", "ENV", "dictionary_key", "dictionary_value"])

---

      ID    ENV dictionary_key  dictionary_value
0  35702  name1   Employee abc             1.560
1  35702  name1             IC             1.180
2  35700  nam22      Quotation             3.060
3  35700  nam22       IC newer             0.370
4  35700  nam22    newmeansnew             0.001

---

Демо

Answer 3

Более полное решение

import json
import pandas as pd

with open("the2.csv") as f:
    next(f)
    lines = [x.strip() for x in f]

vals = ""
valLst = []
for line in lines:
    parts = line.split(",") # file seems separated by 3 spaces or \t, adjust if needed

    flag = False
    nextParts = ""
    for part in parts: 
        if part.startswith('{') and part.endswith('}'):
            nextParts = nextParts +','+ part
            flag = False
        elif part.startswith('\"'):
            flag = True
        elif part.endswith('\"'):
            nextParts = nextParts +','+ part
            flag = False
        else:
            pass

        if flag:
            nextParts = nextParts +','+ part


    nextParts = nextParts.strip(',')
    nextParts = nextParts.strip('"')
    nextParts = nextParts.replace('\'', "\"")

    if nextParts.startswith('{"'):
        pass
    else:
        copyparts = nextParts.split(":")
        otherparts = ""

        for copypart in copyparts: 
            if copypart.startswith('{'):
                otherparts = otherparts + copypart
                otherparts = otherparts.replace('{', '{"')
                otherparts = otherparts + "\" :"
            else:
                otherparts = otherparts + "\"" + copypart.strip(' ')
                otherparts = otherparts.replace('}', '"}')

        otherparts = otherparts.strip('"')
        otherparts = otherparts.replace('\'', "\"")
        nextParts = otherparts

    for k, v in json.loads(nextParts).items(): # json.loads() excepts values enclosed in double quote, not single
        valLst.append([parts[0], parts[1], k, v])


df = pd.DataFrame(valLst, columns=["ID", "ENV", "dictionary_key", "dictionary_value"])