Загрузка данных и усреднение за одну операцию

Question

Я пытаюсь загрузить данные формы, показанной ниже, в фрейм данных.

popSize: 1000
numSurvivors: 0
tournamentSize: 10
probMutation: 0.1
probCrossover: 0.9
numIters: 100
Accuracy: 96.84 
Error Rate: 3.16 
Not Classified: 0.00
Total time: 5.367

popSize: 1000
numSurvivors: 0
tournamentSize: 10
probMutation: 0.1
probCrossover: 0.9
numIters: 100
Accuracy: 96.84 
Error Rate: 3.16 
Not Classified: 0.00
Total time: 4.472

popSize: 1000
numSurvivors: 0
tournamentSize: 10
probMutation: 0.1
probCrossover: 0.9
numIters: 100
Accuracy: 92.11 
Error Rate: 7.89 
Not Classified: 0.00
Total time: 4.46

Данные представляют собой несколько выполнений алгоритма.Есть ли способ загрузить эти данные в виде одной строки, используя усредненный результат для последних 4 значений?

Answer 1

# Test data 
text = """popSize: 1000
numSurvivors: 0
tournamentSize: 10
probMutation: 0.1
probCrossover: 0.9
numIters: 100
Accuracy: 96.84 
Error Rate: 3.16 
Not Classified: 0.00
Total time: 5.36

popSize: 1000
numSurvivors: 0
tournamentSize: 10
probMutation: 0.1
probCrossover: 0.9
numIters: 100
Accuracy: 96.84 
Error Rate: 3.16 
Not Classified: 0.00
Total time: 4.472

popSize: 1000
numSurvivors: 0
tournamentSize: 10
probMutation: 0.1
probCrossover: 0.9
numIters: 100
Accuracy: 92.11 
Error Rate: 7.89 
Not Classified: 0.00
Total time: 4.46
"""

import pandas as pd
from io import StringIO

import pandas as pd

# Function invoked by pandas read on each element, for conversion. Here
# just return the right hand side of the value delimited by ":"
def value(item):
    return item[item.find(':')+1:]

# Reformat the incoming text to comma delimited, each double line break is a record break.
text2 = text.replace(": ", ":").replace("\n\n", "$$break$$").replace("\n", ",").replace("$$break$$","\n").replace("\n,", "")


## Adapted from https://stackoverflow.com/questions/15907869/reading-key-value-pairs-into-pandas
## read table applying a coverter fucntion per element, and supplying the column names
df = pd.read_table(StringIO(text2), header=None, delimiter=',',
                   converters={i:value for i in range(11)},
                   names=['popSize','numSurvivors','tournamentSize','probMutation','probCrossover','numIters','Accuracy','Error Rate','Not Classified','Total time'])

Answer 2

Вот метод для преобразования ваших данных в фрейм данных с использованием itertools.groupby() и pandas:

from itertools import groupby
import pandas as pd

with open('test.txt', 'r') as f:

    chunks = [list(group) for k, group in groupby(f.readlines(), lambda x: x=='\n') if not k]

chunks = [dict([tuple(i.strip().split(': ')) for i in chunk]) for chunk in chunks]

df = pd.DataFrame(chunks).astype(float)

Возвращает:

  Accuracy Error Rate Not Classified Total time numIters numSurvivors popSize  \
0    96.84       3.16           0.00      5.367      100            0    1000   
1    96.84       3.16           0.00      4.472      100            0    1000   
2    92.11       7.89           0.00       4.46      100            0    1000   

  probCrossover probMutation tournamentSize  
0           0.9          0.1             10  
1           0.9          0.1             10  
2           0.9          0.1             10 

Затем вы можете легко рассчитать среднеевот так:

df[['Accuracy','Error Rate','Not Classified','Total time']].mean()

Возвращает:

Accuracy          95.263333
Error Rate         4.736667
Not Classified     0.000000
Total time         4.766333
dtype: float64