Я использую итеративный кластерный подход для заполнения нулевых значений в таблице. То есть я группирую по набору столбцов идентификаторов, беру среднее значение столбца назначенного значения и затем использую его для заполнения нулей. Если он все еще нулевой, то я удаляю последний столбец id в наборе и повторяю процесс снова.
Код, который я создал, работает нормально и делает то, что я хочу, но я ищу лучшего, быстрее и более оптимальный способ сделать это. Заранее спасибо.
Код
class FillByGroup(BaseEstimator, TransformerMixin):
def __init__(self, cols, groupedbycols, aggmeasure='median'):
self.cols = cols
self.groupedbycols = groupedbycols
self.aggmeasure = aggmeasure
def fit(self, X, y=None):
return self
def transform(self, X):
for col in self.cols:
keys = self.groupedbycols
for _ in range(0, len(self.groupedbycols)):
if X[col].isnull().values.any():
X[col] = X[col].fillna(X.groupby(keys)[col].transform(self.aggmeasure))
keys = keys[:-1]
return X**Sample Data**
num_pipeline = Pipeline([
('imputation', FillByGroup(['annualSalesRevenueUSD', 'decisionPowerScore'],
['trueSiteType', 'globalIndustry', 'customerId', 'primaryKey', 'hierarchyLevel'], 'median')),
])
data_tr = num_pipeline.fit_transform(data)
Пример данных
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| trueSiteType | globalIndustry | customerId | primaryKey | hierarchyLevel | annualSalesRevenueUSD | decisionPowerScore |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Government | 113403968 | 2118940 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Professional Services | 113403968 | 205703430 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 214964902 | | | 1.5 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 32184080 | | | 1.2 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 249459738 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 222774105 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 249921686 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | 1.3 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 157076496 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Wholesale/Distribution | 113403968 | 84875301 | | 1700000 | 1.3 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Wholesale/Distribution | 113403968 | 39856022 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Education- Public/Private | 113403968 | 34969725 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 90453245 | | | 1.1 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Retail | 113403968 | 215902607 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | 1000000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 303467675 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 84365783 | 2 | 200000 | 1.44 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 207744311 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Technical Services | 113403968 | 149942656 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 159035652 | | | 1.3 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 254727477 | | | 1.9 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 227306426 | | 600000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Service Provider | 113403968 | 168901981 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 84365783 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | 6.5 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Financial Services | 113403968 | 198800267 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 171432744 | | 400000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | 2.3 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 32943040 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Financial Services | 113403968 | 156575 | 2 | 357887458 | 7.69 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 193871010 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 200051280 | | | 3 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 218395043 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 165704385 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Manufacturing | 113403968 | 126218718 | 2 | | 1.53 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 113764764 | | 23000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 83829264 | | 500000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 176334974 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 153799333 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 236388836 | | 300000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 266620422 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Wholesale/Distribution | 113403968 | 198435875 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | 500000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 219714294 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Manufacturing | 113403968 | 207916067 | | 400000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 249459738 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 302778956 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 194833595 | | 390000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 212983687 | | | 2.8 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 9255363 | | | 5.6 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Financial Services | 113403968 | 2145686 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Professional Services | 113403968 | 2138916 | | 490000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8586539 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Manufacturing | 113403968 | 202951417 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 254824730 | | 49000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Wholesale/Distribution | 113403968 | 11500457 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Wholesale/Distribution | 113403968 | 2390614 | | 592500 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | 2.3 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Technical Services | 113403968 | 4229326 | | | 5.1 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Financial Services | 113403968 | 37387088 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | 57208392 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | 309300 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 145200107 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 205059370 | 1 | 1029005150 | 4.32 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Education- Public/Private | 113403968 | 6172545 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 5158785 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 265423245 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Education- Public/Private | 113403968 | 195759010 | | 200000000 | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Manufacturing | 113403968 | 146854 | 3 | | 6 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 247443455 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Manufacturing | 113403968 | 207916067 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 90453245 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 33600165 | | 383003000 | 1.1 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Professional Services | 113403968 | 140276652 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 302134182 | | | 1.2 |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Health Care | 113403968 | 206023815 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| BR | Unknown | 113403968 | 8000 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+
| GU | Unknown | 113403968 | 254825198 | | | |
+--------------+---------------------------+------------+------------+----------------+-----------------------+--------------------+