Свернуть и сосчитать идентичные твиты в большом фрейме данных - PullRequest
Новая
       16

Свернуть и сосчитать идентичные твиты в большом фрейме данных

0 голосов
/ 30 ноября 2018

У меня есть большой фрейм данных в RStudio, состоящий из данных Twitter (время публикации, идентификационный номер пользователя, текст твита и т. Д.).Я хочу объединить все наблюдения с идентичным текстом твита, text, в одно наблюдение, подсчитывая, сколько раз это сообщение появляется в кадре данных.Другими словами, если сообщение The cat is in the tree появляется в фрейме данных 12 раз, я хочу создать фрейм данных, в котором появляется только первый раз, когда оно публикуется, но со столбцом, который говорит 12 рядом с сообщением.

Как я могу это сделать?

Вот мои воспроизводимые данные: 

`structure(list(timestamp = structure(c(1446241090, 1446241086, 
1446241094, 1446241107, 1446241158, 1446241132, 1446241181, 1446241202, 
1446241209, 1446241304, 1446241318, 1446241327, 1446241297, 1446241345, 
1446241530, 1446241382, 1446241624, 1446241577, 1446241707, 1446241583, 
1446241739, 1446241739, 1446241602, 1446241682, 1446241687, 1446241773, 
1446241703, 1446241664, 1446241842, 1446241696), class = c("POSIXct", 
"POSIXt"), tzone = "UTC"), id_str = c(660209102790914048, 660209083505504256, 
660209119152893952, 660209170730258432, 660209385713573888, 660209278498824192, 
660209483080056832, 660209569935835136, 660209601502162944, 660209999600332800, 
660210059658432512, 660210094819311616, 660209967971115008, 660210170643922944, 
660210945285746688, 660210326596681728, 660211339168714752, 660211145849053184, 
660211690164744192, 660211169035161600, 660211824374231040, 660211825049497600, 
660211250324992000, 660211583772045312, 660211603237834752, 660211966766620672, 
660211670568988672, 660211508937363456, 660212253707358208, 660211641858916352
), user.id_str = c(68956490, 68956490, 949996219, 68956490, 1665986042, 
529591144, 20809182, 135909586, 20118515, 2327500422, 2382485564, 
1881559508, 2403408967, 949996219, 124533535, 14545416, 347334263, 
711042272, 68956490, 152240878, 1723563360, 1723563360, 135909586, 
68956490, 68956490, 419665502, 68956490, 17374940, 112219846, 
68956490), user.followers_count = c(15227, 15227, 2214, 15227, 
756, 3608, 1121, 721, 13484, 321, 188, 886, 1446, 2214, 1076, 
2310, 1754, 995, 15228, 1269, 7983, 7983, 721, 15228, 15228, 
2075, 15228, 955, 635, 15228), ideology = c(2.29286233202781, 
2.29286233202781, -0.309303177803536, 2.29286233202781, -0.778438324479111, 
2.16242522348951, -0.908875433017413, -0.699518393262659, 1.62423513699954, 
0.417417855481292, 1.12769723642936, 0.600468251497229, -0.907779322861629, 
-0.309303177803536, -0.59977236908631, 1.54860353625044, 1.76234501662833, 
-0.0111612154302728, 2.29286233202781, 0.112699232173325, -0.306014847336183, 
-0.306014847336183, -0.699518393262659, 2.29286233202781, 2.29286233202781, 
-0.749939460428726, 2.29286233202781, -0.83214772211253, -0.863934916630267, 
2.29286233202781), text = c("better dead than red bill gates says that only socialism can save us from climate change httpstcopypqmd1fok", "i see red people bill gates says that only socialism can save us from climate change httpstcopypqmd1fok", "this is an amusing headline bill gates says that only socialism can save us from climate change httpstcobfs5zbcijc", "communist core bill gates says that only socialism can save us from climate change httpstcogqm7k64f0r", "this is an amusing headline bill gates says that only socialism can save us from climate change httpstcobfs5zbcijc", "better dead than red bill gates says that only socialism can save us from climate change httpstcopypq", "lights camera climate change action showing of the carbon negative good allpowerlabs httpstcobpxdhjqw5i", "this is an amusing headline bill gates says that only socialism can save us from climate change httpstcobfs5zbcijc", "whether climate change is real or not we must all do our part to care for mother earth it is our gift from god httpstcotpjerkfu5u", "soot no doubt volcanic in origin greenland ice melt due to global warming found not so bad  httpstcoyqsmd6d4sm via", "this is an amusing headline bill gates says that only socialism can save us from climate change httpstcobfs5zbcijc", 
"the arctics got a serious chemtrails problem looks pretty bad again today\r\n\r\nclimatechange geoengineering httpstco1ls", "bill gates says that only socialism can save us from climate change httpstcoe5psgltj59 httpstcop9oye6sipx", "naomiaklein looks like the headline changed already bill gates says that capitalism cannot save us from climate chang", "bill gates says that only socialism can save us from climate change httpstcovjvklrncwq httpstcodtnjg7e0rz", "not to ruin halloween or anything but obama wants to take a moment to remind you that your jack olantern is causing globalwarming", 
"save on green hosting from hostgator use 25 off coupon code get25offhg httpstcohlk3yp1eew webhost webhosting climatechange", "bill gates says that only socialism can save us from climate change httpstcosspebdd3m9 httpstcovttupglukt", "bring back huac bill gates says that only socialism can save us from climate change httpstcospnurbgevy", "bill gates says that only socialism can save us from climate change httpstcovjvklrncwq httpstcodtnjg7e0rz", "what happens when you ingore climate good science amp management plans fail exhibit a atlantic cod science https", "what happens when you ingore climate good science amp management plans fail exhibit a atlantic cod science https", "naomiaklein looks like the headline changed already bill gates says that capitalism cannot save us from climate chang", "party like 1989 bill gates says that only socialism can save us from climate change httpstcopypqmd1fok", "dawn of the red bill gates says that only socialism can save us from climate change httpstcopypqmd1fok", "bill gates says that only socialism can save us from climate change httpstco8khrx6cgmd", "light up the tilt bill gates says that only socialism can save us from climate change httpstcopypqmd1fok", "annegalloway good  but maybe impossible to answer wout knowing the context ie war climate change other nonhumans that may be harmed", "bill gates says that only socialism can save us from climate change httpstcoe5psgltj59 httpstcop9oye6sipx", "unenjoyment line bill gates says that only socialism can save us from climate change httpstcobfstykngx4"), dup_text = c(FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, 
TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, 
FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, 
FALSE, FALSE, FALSE, TRUE, FALSE), dup_clean_text = c(FALSE, 
FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, 
TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, 
TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE, 
TRUE, FALSE), dup_user = c(FALSE, TRUE, FALSE, TRUE, FALSE, FALSE, 
FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE, 
FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, 
FALSE, TRUE, FALSE, FALSE, TRUE)), .Names = c("timestamp", "id_str", 
"user.id_str", "user.followers_count", "ideology", "text", "dup_text", 
"dup_clean_text", "dup_user"), row.names = c(NA, -30L), class = c("tbl_df", 
"tbl", "data.frame"))`

1 Ответ

0 голосов
/ 30 ноября 2018

Здесь с вашими данными и dplyr: 

data_text %>%
group_by(text) %>%                # group by tweet
summarise(freq = n(),             # count the occurencies
          date = min(timestamp))  # the first date the tweet appears

# A tibble: 23 x 3
   text                                                                                                                       freq date               
   <chr>                                                                                                                     <int> <dttm>             
 1 annegalloway good  but maybe impossible to answer wout knowing the context ie war climate change other nonhumans that ma~     1 2015-10-30 21:47:44
 2 better dead than red bill gates says that only socialism can save us from climate change httpstcopypq                         1 2015-10-30 21:38:52
 3 better dead than red bill gates says that only socialism can save us from climate change httpstcopypqmd1fok                   1 2015-10-30 21:38:10
 4 bill gates says that only socialism can save us from climate change httpstco8khrx6cgmd                                        1 2015-10-30 21:49:33
 5 bill gates says that only socialism can save us from climate change httpstcoe5psgltj59 httpstcop9oye6sipx                     2 2015-10-30 21:41:37
 6 bill gates says that only socialism can save us from climate change httpstcosspebdd3m9 httpstcovttupglukt                     1 2015-10-30 21:46:17
 7 bill gates says that only socialism can save us from climate change httpstcovjvklrncwq httpstcodtnjg7e0rz                     2 2015-10-30 21:45:30
 8 bring back huac bill gates says that only socialism can save us from climate change httpstcospnurbgevy                        1 2015-10-30 21:48:27
 9 communist core bill gates says that only socialism can save us from climate change httpstcogqm7k64f0r                         1 2015-10-30 21:38:27
10 dawn of the red bill gates says that only socialism can save us from climate change httpstcopypqmd1fok                        1 2015-10-30 21:48:07
# ... with 13 more rows

Возможно, вы захотите удалить все https, чтобы получить более согласованные результаты: 

data_text %>%
  mutate (cleaned_up = gsub("https\\w+ *", "", text)) %>%  # remove all the "https..things" adding a cleaned up column
  group_by(cleaned_up) %>%                                 # group by tweet
  summarise(freq = n(),                                    # count the occurencies
            date = min(timestamp))  %>% 
   arrange(-freq) %>%                                      # order desc 
  head()                                                   # take the top 6 

   # A tibble: 6 x 3
  cleaned_up                                                                                                                  freq date               
  <chr>                                                                                                                      <int> <dttm>             
1 "bill gates says that only socialism can save us from climate change "                                                         6 2015-10-30 21:41:37
2 "this is an amusing headline bill gates says that only socialism can save us from climate change "                             4 2015-10-30 21:38:14
3 "better dead than red bill gates says that only socialism can save us from climate change "                                    2 2015-10-30 21:38:10
4 naomiaklein looks like the headline changed already bill gates says that capitalism cannot save us from climate chang          2 2015-10-30 21:42:25
5 what happens when you ingore climate good science amp management plans fail exhibit a atlantic cod science https               2 2015-10-30 21:48:59
6 annegalloway good  but maybe impossible to answer wout knowing the context ie war climate change other nonhumans that may~     1 2015-10-30 21:47:44
...