Я использовал функцию ЗАГРУЗИТЬ ОТ S3 в Авроре Mysql. Чтобы максимально использовать возможности и пропускную способность, я отправил 30 одновременных запросов на загрузку S3. При проверке списка процессов в Mysql я вижу только 4 запроса S3 LOAD, выполняющихся в любой момент времени. Это жесткий предел? Или я что-то не так делаю?
Вот мои сценарии для отправки нагрузок:
Однократная загрузка из S3:
cat load-one-table.sh
export MYSQL_PWD=[REDACTED]
echo "Shard #$1"
mysql -u [REDACTED] -B -e "load data from s3 's3://A/B/C/D' into table DB.table_"$1;
Одновременная загрузка:
cat parallel-load.sh
for i in $(seq 1 $1); do
echo "Starting Task #$i"
nohup ./load-one-table.sh $i &
echo "Submitted Task #$i"
done
Триггер:
./parallel-load 30
Я вижу все 30 запросов, отправленных в журналах nohup.
Точная проверка, все ли загрузки выполняются на клиенте:
ps auxww | grep -c "load-one-table"
31
# The result shows 31, since there is one extra process match for the grep
Проверка принятых запросов на сервере:
show full processlist;
+-----+----------+---------------------+--------------------+---------+------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+----------+---------------------+--------------------+---------+------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| 606 | REDACTED | localhost | NULL | Query | 60 | executing | load data from s3 's3://REDACTED' into table REDACTED_2 |
| 607 | REDACTED | localhost | NULL | Query | 60 | executing | load data from s3 's3://REDACTED' into table REDACTED_4 |
| 608 | REDACTED | localhost | NULL | Query | 60 | executing | load data from s3 's3://REDACTED' into table REDACTED_1 |
| 609 | REDACTED | localhost | NULL | Query | 60 | executing | load data from s3 's3://REDACTED' into table REDACTED_3 |
| 610 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 611 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 612 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 613 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 614 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 615 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 616 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 617 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 618 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 619 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 620 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 621 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 622 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 623 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 624 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 625 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 626 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 627 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 628 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 629 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 630 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 631 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 632 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 633 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 634 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
| 635 | REDACTED | localhost | NULL | Sleep | 60 | cleaning up | NULL |
+-----+----------+---------------------+--------------------+---------+------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+