Я пытаюсь сгенерировать тепловую карту из набора данных SQUAD v1.1, заданного здесь .
Набор данных отряда изображен следующим образом:
Document/
├── Paragraph1/
│ ├── Question
│ ├── Answer1
│ ├── Answer2
│ └── Answer3
├── Paragraph2/
│ ├── Question
│ └── Answer1
Документ может иметь несколько абзацев / контекстов.Каждый абзац (контекст) может иметь несколько вопросов и ответов.Здесь изображено здесь .
Я планирую денормализовать JSON в CSV, что может быть неверным:
Context,Question,Answer
Context1,Question1,Answer1
Context1,Question1,Answer2
Context1,Question2,Answer1
...
Пока я нормализовал вложенный JSONв файл CSV, используя следующий код:
import json
import csv
with open(r'SQUAD v1.json') as squad_data_file_handle:
squad_data = json.load(squad_data_file_handle)
with open('SQUAD_11_CSV.csv', 'w', newline='', encoding='UTF-8') as squad_csv_handle:
writer = csv.writer(squad_csv_handle, dialect='excel', delimiter=',')
writer.writerow(["Context", "Question", "Answer"])
for data in squad_data["data"]:
for paragraph in data["paragraphs"]:
context = str(paragraph["context"])
question_answer_pairs = paragraph.get("qas", [])
for qa_pair in question_answer_pairs:
question = str(qa_pair["question"])
answers = list(set([str(answer.get("text")) for answer in qa_pair.get("answers", [])]))
for answer in answers:
writer.writerow([context, question, answer])
Таким образом, он генерирует CSV следующим образом (первые две строки):
Context,Question,Answer
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the ""golden anniversary"" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as ""Super Bowl L""), so that the logo could prominently feature the Arabic numerals 50.",Which NFL team represented the AFC at Super Bowl 50?,Denver Broncos
"Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the ""golden anniversary"" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as ""Super Bowl L""), so that the logo could prominently feature the Arabic numerals 50.",Which NFL team represented the NFC at Super Bowl 50?,Carolina Panthers
Это кодЯ использую для создания тепловую карту:
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.read_csv(r"SQUAD_11_CSV.csv")
df = df.pivot("Context", "Question", "Answer")
sns.heatmap(df)
plt.show()
Итак, когда я пытаюсь сгенерировать тепловую карту, возникает исключение:
ValueError: Index contains duplicate entries, cannot reshape
Итак, любые советы / указатели на любые очевидные ошибки в том, как сгенерировать тепловую карту и смоделировать данные SQUAD JSON в идеальный CSV, будут оценены.
Обновление:
Тепловая карта должна выглядеть примерно так:
