Я использую 64-битный R / RStudio на 64-битной Windows 10. ПК имеет 16 ГБ оперативной памяти и работает на 8-ядерном.
Таким образом, при чтении большого набора данных RStudio дает сбой примерно на 1,6 / 7 ГБ памяти.
Поэтому я пытаюсь использовать параллельный пакет для выполнения операции с несколькими ядрами.Но где-то я ошибаюсь.
library("data.table")
library("lubridate")
library("parallel")
library("foreach")
library("doParallel")
cl <- makeCluster(detectCores() - 1)
registerDoParallel(cl, cores = detectCores() - 2)
files = list.files(pattern="public")
myfiles = do.call(rbind, lapply(files, function(x) fread(x, colClasses=c(ID="character"))))
У меня нет большого опыта параллельной обработки.
Подскажите, пожалуйста, где я ошибаюсь?
Обновление:
R не создает проблем при создании объекта 8 ГБ в памяти.
bigint <- integer(2^32 / 2)
Все еще не уверен, что ограничивает чтение данных.
Обновление 2:
Я сделал диагностический отчет.Это ошибки, которые я получаю.
24 Jan 2019 23:10:33 [rdesktop] ERROR system error 231 (All pipe instances are busy); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:11:47 [rdesktop] ERROR system error 231 (All pipe instances are busy); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:11:47 [rdesktop] ERROR system error 231 (All pipe instances are busy); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:13:39 [rdesktop] ERROR system error 231 (All pipe instances are busy); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:13:40 [rdesktop] ERROR system error 231 (All pipe instances are busy); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:13:41 [rdesktop] ERROR system error 232 (The pipe is being closed); OCCURRED AT: void rstudio::core::http::AsyncClient<SocketService>::handleWrite(const rstudio_boost::system::error_code&) [with SocketService = rstudio_boost::asio::windows::basic_stream_handle<>] C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/AsyncClient.hpp:342; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:13:42 [rdesktop] ERROR system error 2 (The system cannot find the file specified); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288
24 Jan 2019 23:13:42 [rdesktop] ERROR system error 2 (The system cannot find the file specified); OCCURRED AT: virtual void rstudio::core::http::NamedPipeAsyncClient::connectAndWriteRequest() C:/Users/Administrator/rstudio/src/cpp/core/include/core/http/NamedPipeAsyncClient.hpp:84; LOGGED FROM: void rstudio::desktop::NetworkReply::onError(const rstudio::core::Error&) C:\Users\Administrator\rstudio\src\cpp\desktop\DesktopNetworkReply.cpp:288