Я использую raspberry pi 2 B+ для запуска моего курса данных Python.
Я могу прочитать файл .csv (примерно маленький 300M ).
Но когда я пытаюсь прочитать файл 500M + .csv, я получаю memory error.
Я использую:
- Панды 0.23.4
- python 3.5.6
оболочка
$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
import pandas
pandas.read_csv('ccf_online_stage1_train.csv', nrows=11429827)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pi/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 678, in parser_f
return _read(filepath_or_buffer, kwds)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 446, in _read
data = parser.read(nrows)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 1051, in read
df = DataFrame(col_dict, columns=columns, index=index)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/frame.py", line 348, in __init__
mgr = self._init_dict(data, index, columns, dtype=dtype)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/frame.py", line 459, in _init_dict
return _arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/frame.py", line 7364, in _arrays_to_mgr
return create_block_manager_from_arrays(arrays, arr_names, axes)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/internals.py", line 4872, in create_block_manager_from_arrays
blocks = form_blocks(arrays, names, axes)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/internals.py", line 4918, in form_blocks
int_blocks = _multi_blockify(items_dict['IntBlock'])
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/internals.py", line 4995, in _multi_blockify
values, placement = _stack_arrays(list(tup_block), dtype)
File "/home/pi/.local/lib/python3.5/site-packages/pandas/core/internals.py", line 5037, in _stack_arrays
stacked = np.empty(shape, dtype=dtype)
MemoryError