Поиск указанного c текста в pandas столбце данных

Question

У меня есть фрейм данных со столбцом, содержащим ссылки на документы, и я хочу найти любую ссылку, повторенную во всем столбце для всех ссылок.

Вот несколько строк из фрейма данных:

In [1]:

df4.iloc[0:2]

Out[2]:

 **cit2ref**    **reference**                                                                                                    **_id**
0   NaN     All about depression: Diagnosis. (2013). Retrieved December 7, 2016,from All About Depression,
            http://www.allaboutdepression.com/dia_03.html                                                                   Y17-1020
0   NaN     American Psychological Association. (2016). Center for epidemiological studies depression (CESD). 
            Retrieved December 7, 2016, from American Psychological Association, 
            http://www.apa.org/pi/ about/publications/caregivers/practice-settings/ assessment/tools/depression-scale.aspx  Y17-1020

Еще несколько строк:

 **cit2ref** **reference**                                                                                                                                 **_id**

0   NaN     All about depression: Diagnosis. (2013). Retrieved December 7, 2016, from All About Depression, http://www.allaboutdepression.com/dia_03.html   Y17-1020
0   NaN     American Psychological Association. (2016). Center for epidemiological studies depression (CESD). Retrieved December 7, 2016, from American Psychological Association, http://www.apa.org/pi/ about/publications/caregivers/practice-settings/ assessment/tools/depression-scale.aspx   Y17-1020
0   NaN     American Psychological Association. (2016). Patient health questionnaire (PHQ-9 %27 PHQ-2). Retrieved December 09, 2016, from http://www.apa.org/pi/ about/publications/caregivers/practice-settings/ assessment/tools/patient-health.aspx  Y17-1020
0   NaN     Beattie, G.S. (2005, November). Social Causes of Depression. Retrieved May 31, 2017, from http:// www.personalityresearch.org/papers/beattie.html   Y17-1020
0   Burton (2012)   Burton, N. (2012, June 5). Depressive Realism. Retrieved May 31, 2017, from https:// www.psychologytoday.com/blog/hide-and-seek/ 201206/depressive-realism  Y17-1020
0   NaN     Clark, P., Niblett, T. (1988, October 25). The CN2 induction Algorithm. Retireved May 10, 2017, from https://pdfs.semanticscholar.org/766f/ e3586bda3f36cbcce809f5666d2c2b96c98c.pdf    Y17-1020
0   Choudhury, 2014     De Choudhury, M., Counts, S., Horvits, E., %27 Hoff, A. (2014). Characterizing and Predicting Postpartum Depression from Shared Facebook Data.  Y17-1020
0   NaN     De Choudhury, M., Gamon, M., Couns, S., %27 Horvitz, E. (2013). Predicting Depression via Social Media.     Y17-1020
0   Gotlib and Joormann (2010)  Gotlib IH, Kasch KL, Traill S, Joormann J, Arnow BA, Johnson SL. (2010) Coherence and specificity of information-processing biases in depression and social phobia. J Abnorm Psychol. 2004;113(3): 386-98.  Y17-1020
0   NaN     Gotlib, I. H., %27 Hammen, C. L. (1992). Psychological aspects of depression: Toward a cognitive- interpersonal integration. New York: Wiley.   Y17-1020
0   NaN     Gotlib IH, Joormann J. Cognition and depression: current status and future directions. Annu Rev Clin Psychol. 2010;6:285-312.   Y17-1020
0   NaN     Hu, Quan, Ang Li, Fei Heng, Jianpeng Li, and Tingshao Zhu. "Predicting Depression of Social Media User on Different Observation Windows." 2015 IEEE/ WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI- IAT) (2015): n. pag. Web.   Y17-102

Здесь '0' - индекс для 1-й статьи, на которую имеется много ссылок, и есть 40 тыс. Статей с приблизительно ~ 20 ссылками для каждой.

Поиск любой ссылки, которая снова используется в другой статье (здесь разные индексы для каждой бумаги) с указателем и количеством повторений.

Пробовал с regular expression и методы сортировки pandas, такие как value_counts(sort=True).sort_index() и sort_values(), но это не помогает.

Вот снимок экрана информационного кадра с 2 бумагами с индексами «0» и «1»

Answer 1

IIU C, используйте pandas.DataFrame.index.groupby.

С псевдофреймом df: (обратите внимание, что для демонстрации я добавил последние три строки):

print(df)
   cit2ref                                          reference       _id
0      NaN  All about depression: Diagnosis. (2013). Retri...  Y17-1020
0      NaN  American Psychological Association. (2016). Ce...  Y17-1020
0      NaN  American Psychological Association. (2016). Pa...  Y17-1020
0      NaN  Beattie, G.S. (2005, November). Social Causes ...  Y17-1020
0      NaN  Burton   (2012)   Burton, N. (2012, June 5). D...  Y17-1020
0      NaN  Clark, P., Niblett, T. (1988, October 25). The...  Y17-1020
0      NaN  Choudhury, 2014     De Choudhury, M., Counts, ...  Y17-1020
0      NaN  De Choudhury, M., Gamon, M., Couns, S., %27 Ho...  Y17-1020
0      NaN  Gotlib and Joormann (2010)  Gotlib IH, Kasch K...  Y17-1020
0      NaN  Gotlib, I. H., %27 Hammen, C. L. (1992). Psych...  Y17-1020
0      NaN  Gotlib IH, Joormann J. Cognition and depressio...  Y17-1020
0      NaN  Hu, Quan, Ang Li, Fei Heng, Jianpeng Li, and T...   Y17-102
1      NaN  All about depression: Diagnosis. (2013). Retri...  Y17-1020
1      NaN  American Psychological Association. (2016). Ce...  Y17-1020
1      NaN                StackOverflow. Not to be grouped-by   Y17-102

Тогда groupby:

df.index.groupby(df['reference'])
# or
d = {k: list(v) for k, v in df.index.groupby(df['reference']).items()}
new_df = pd.DataFrame.from_dict(d, orient='index').reset_index()
print(new_df)
# this looks prettier

                                                index       0
0   All about depression: Diagnosis. (2013). Retri...  [0, 1]
1   American Psychological Association. (2016). Ce...  [0, 1]
2   American Psychological Association. (2016). Pa...     [0]
3   Beattie, G.S. (2005, November). Social Causes ...     [0]
4   Burton   (2012)   Burton, N. (2012, June 5). D...     [0]
5   Choudhury, 2014     De Choudhury, M., Counts, ...     [0]
6   Clark, P., Niblett, T. (1988, October 25). The...     [0]
7   De Choudhury, M., Gamon, M., Couns, S., %27 Ho...     [0]
8   Gotlib IH, Joormann J. Cognition and depressio...     [0]
9   Gotlib and Joormann (2010)  Gotlib IH, Kasch K...     [0]
10  Gotlib, I. H., %27 Hammen, C. L. (1992). Psych...     [0]
11  Hu, Quan, Ang Li, Fei Heng, Jianpeng Li, and T...     [0]
12                StackOverflow. Not to be grouped-by     [1]

Вы можете увидеть, какая бумага появилась в каких индексах. Если вы хотите считать, вы можете использовать len вместо list:

d = {k: len(v) for k, v in df.index.groupby(df['reference']).items()}
new_df = pd.DataFrame.from_dict(d, orient='index').reset_index()
print(new_df)

Выход:

                                                index  0
0   All about depression: Diagnosis. (2013). Retri...  2
1   American Psychological Association. (2016). Ce...  2
2   American Psychological Association. (2016). Pa...  1
3   Beattie, G.S. (2005, November). Social Causes ...  1
4   Burton   (2012)   Burton, N. (2012, June 5). D...  1
5   Choudhury, 2014     De Choudhury, M., Counts, ...  1
6   Clark, P., Niblett, T. (1988, October 25). The...  1
7   De Choudhury, M., Gamon, M., Couns, S., %27 Ho...  1
8   Gotlib IH, Joormann J. Cognition and depressio...  1
9   Gotlib and Joormann (2010)  Gotlib IH, Kasch K...  1
10  Gotlib, I. H., %27 Hammen, C. L. (1992). Psych...  1
11  Hu, Quan, Ang Li, Fei Heng, Jianpeng Li, and T...  1
12                StackOverflow. Not to be grouped-by  1