Среднемесячный набор данных временных рядов ежемесячно с Panda

Question

У меня есть очень большой набор данных временных рядов по минутам (за 3 месяца) в следующем формате

datetime,val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12
1/06/2017 0:00,0,0,0,0,0,0,0,0,0,0.011,0,0.036
1/06/2017 0:01,0,0,0,0,0,0,0,0,0,0.011,0,0.036
...
1/06/2017 23:59,0,0,0,0,0,0,0,0,0,0.011,0,0.035
2/06/2017 0:00,0,0,0,0,0,0,0,0,0,0.014,0,0.036
2/06/2017 0:01,0,0,0,0,0,0,0,0,0,0.011,0,0.036
...
2/06/2017 23:59,0,0,0,0,0,0,0,0,0,0.011,0,0.035
....
31/08/2017 0:00,0,0.2,0,0,0,0.56,0,0,0,0.014,0,0.036
31/08/2017 0:01,0,0.23,0,0,0,0,0,0,0,0.011,0,0.032
...
31/08/2017 23:59,0,0,0,0,0,0,.55,0,0,0.011,0,0.034

Каков наиболее эффективный способ получения среднего значения каждого столбца за каждый месяц с использованием панды? ожидаемый результат будет

month,val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12
06/2017,0,0,0,0,0,0,0,0,0,0.011,0,0.036
07/2017,0,0,0,0,0,0,0,0,0,0.014,0,0.036
08/2017,0,0,0.21,0,0,0,0,0.52,0,0.011,0,0.036

В данный момент я читаю набор данных день за днем, затем получаю накопленный набор данных дней и затем делю его на число дней в месяце. Но это очень неэффективно и отнимает много времени.

Answer 1

Панды read_csv и to_csv - это то, что вам нужно:

df = pd.read_csv('input.csv', parse_dates=['datetime'])
df.groupby(df.datetime.dt.strftime('%m/%Y')).mean().rename_axis('month').to_csv(out, float_format='%.06f')

С вашими входными данными (отфильтрованными по ...) это дает:

month,val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12
01/2017,0,0.000000,0,0,0,0.000000,0.000000,0,0,0.011000,0,0.035667
02/2017,0,0.000000,0,0,0,0.000000,0.000000,0,0,0.012000,0,0.035667
08/2017,0,0.143333,0,0,0,0.186667,0.183333,0,0,0.012000,0,0.034000

Answer 2

Сначала преобразуйте столбец по to_datetime, а затем DataFrame.resample по MS для начала месяцев, последнее изменение формата DatetimeIndex на MM/YYY на DatetimeIndex.strftime:

df['datetime'] = pd.to_datetime(df['datetime'], format='%d/%m/%Y %H:%M')

df = df.resample('MS', on='datetime').mean()
df.index = df.index.strftime('%m/%Y')
print (df)
         val1      val2  val3  val4  val5      val6      val7  val8  val9  \
06/2017   0.0  0.000000   0.0   0.0   0.0  0.000000  0.000000   0.0   0.0   
07/2017   NaN       NaN   NaN   NaN   NaN       NaN       NaN   NaN   NaN   
08/2017   0.0  0.143333   0.0   0.0   0.0  0.186667  0.183333   0.0   0.0   

          val10  val11     val12  
06/2017  0.0115    0.0  0.035667  
07/2017     NaN    NaN       NaN  
08/2017  0.0120    0.0  0.034000  

Или передать преобразованный столбец даты и времени Series.dt.strftime в groupby и объединить mean:

df = df.groupby(df['datetime'].dt.strftime('%m/%Y')).mean()
print (df)
          val1      val2  val3  val4  val5      val6      val7  val8  val9  \
datetime                                                                     
06/2017      0  0.000000     0     0     0  0.000000  0.000000     0     0   
08/2017      0  0.143333     0     0     0  0.186667  0.183333     0     0   

           val10  val11     val12  
datetime                           
06/2017   0.0115      0  0.035667  
08/2017   0.0120      0  0.034000