Я хочу получить 100 из 1000 самых просматриваемых лотов на прошлых аукционах (в случайном порядке). У меня есть база данных MySQL, все таблицы, использующие InnoDB.
Аппаратное обеспечение: Intel i7 6700, 32 ГБ ОЗУ, SSD (не знаю, сколько IOPS).
Некоторые сведения о моем my.cnf:
innodb_log_buffer_size = 256M
innodb_log_file_size = 1G
innodb_write_io_threads = 16
innodb_flush_log_at_trx_commit = 0
# mysqltuner
query_cache_type = 1
join_buffer_size = 256K
tmp_table_size = 32M
max_heap_table_size = 32M
innodb_buffer_pool_size = 11G
Вывод MySQL-Tuner:
>> MySQLTuner 1.6.0 - Major Hayden <major@mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.7.24-0ubuntu0.16.04.1
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in InnoDB tables: 28G (Tables: 456)
[!!] Total fragmented tables: 18
-------- Security Recommendations -------------------------------------------
[OK] There is no anonymous account in all database users
[OK] All database users have passwords assigned
[--] There is 605 basic passwords in the list.
-------- Performance Metrics -------------------------------------------------
[--] Up for: 18d 5h 43m 18s (80M q [51.384 qps], 827K conn, TX: 539B, RX: 28B)
[--] Reads / Writes: 58% / 42%
[--] Binary logging is disabled
[--] Total buffers: 11.3G global + 1.1M per thread (151 max threads)
[OK] Maximum reached memory usage: 11.3G (36.31% of installed RAM)
[OK] Maximum possible memory usage: 11.5G (36.76% of installed RAM)
[OK] Slow queries: 0% (0/80M)
[OK] Highest usage of available connections: 11% (17/151)
[OK] Aborted connections: 0.00% (11/827974)
[OK] Query cache efficiency: 74.7% (50M cached / 67M selects)
[!!] Query cache prunes per day: 650750
[OK] Sorts requiring temporary tables: 0% (1K temp sorts / 744K sorts)
[!!] Temporary tables created on disk: 26% (160K on disk / 593K total)
[OK] Thread cache hit rate: 99% (70 created / 827K connections)
[!!] Table cache hit rate: 0% (416 open / 169K opened)
[OK] Open file limit used: 2% (21/1K)
[OK] Table locks acquired immediately: 100% (624 immediate / 624 locks)
-------- MyISAM Metrics -----------------------------------------------------
[!!] Key buffer used: 18.2% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/43.0K
[!!] Read Key buffer hit rate: 77.3% (172 cached / 39 reads)
-------- InnoDB Metrics -----------------------------------------------------
[--] InnoDB is enabled.
[!!] InnoDB buffer pool / data size: 11.0G/28.3G
[!!] InnoDB buffer pool instances: 8
[OK] InnoDB Used buffer: 98.87% (712671 used/ 720852 total)
[OK] InnoDB Read buffer efficiency: 99.77% (20288463287 hits/ 20335997890 total)
[!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total)
[OK] InnoDB log waits: 0.00% (0 waits / 733034 writes)
-------- AriaDB Metrics -----------------------------------------------------
[--] AriaDB is disabled.
-------- Replication Metrics -------------------------------------------------
[--] No replication slave(s) for this server.
[--] This is a standalone server..
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64:
Beware that open_files_limit (1024) variable
should be greater than table_open_cache ( 431)
Variables to adjust:
query_cache_size (> 16M)
tmp_table_size (> 32M)
max_heap_table_size (> 32M)
table_open_cache (> 431)
innodb_buffer_pool_size (>= 28G) if possible.
innodb_buffer_pool_instances(=11)
Я управляю случайной частью с помощью рандомизированного OFFSET в PHP:
$offsetRand = rand(0, 1000-100);
Затем рандомизированное смещение добавляется к запросу sql.
Это приводит к следующему запросу:
SELECT l.id FROM Auction a
INNER JOIN Lot l ON a.id = l.auction_id
WHERE a.startDate < "2018-11-09"
ORDER BY l.views DESC LIMIT 100 OFFSET 543
Explain результат запроса:
+----+-------------+-------+------------+-------+-------------------------------+-----------------------+---------+--------------------+-------+----------+-----------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+-------------------------------+-----------------------+---------+--------------------+-------+----------+-----------------------------------------------------------+
| 1 | SIMPLE | a | NULL | range | PRIMARY,auction_startDate_idx | auction_startDate_idx | 5 | NULL | 33864 | 100.00 | Using where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | l | NULL | ref | IDX_33CC6FFB57B8F0DE | IDX_33CC6FFB57B8F0DE | 5 | lotsearch_new.a.id | 320 | 100.00 | NULL |
+----+-------------+-------+------------+-------+-------------------------------+-----------------------+---------+--------------------+-------+----------+-----------------------------------------------------------+
Таблица Auction: 70000 строк
Таблица Lot: 27 миллионов строк
Индексы Lot Таблица:
+-------+------------+-----------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-----------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| lot | 0 | PRIMARY | 1 | id | A | 23945704 | NULL | NULL | | BTREE | | |
| lot | 0 | UNIQ_33CC6FFB989D9B62 | 1 | slug | A | 23945900 | NULL | NULL | YES | BTREE | | |
| lot | 1 | IDX_33CC6FFB57B8F0DE | 1 | auction_id | A | 74748 | NULL | NULL | YES | BTREE | | |
| lot | 1 | lot_visible_idx | 1 | visible | A | 1 | NULL | NULL | | BTREE | | |
| lot | 1 | lot_hammerPrice_idx | 1 | hammerPrice | A | 59499 | NULL | NULL | YES | BTREE | | |
| lot | 1 | lot_views_idx | 1 | views | A | 3440 | NULL | NULL | YES | BTREE | | |
+-------+------------+-----------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Индексы Auction Таблица:
+---------+------------+-----------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+-----------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| auction | 0 | PRIMARY | 1 | id | A | 67729 | NULL | NULL | | BTREE | | |
| auction | 0 | UNIQ_1159CC0FED9CD316 | 1 | md5Image_id | A | 57263 | NULL | NULL | YES | BTREE | | |
| auction | 1 | IDX_1159CC0F38248176 | 1 | currency_id | A | 24 | NULL | NULL | YES | BTREE | | |
| auction | 1 | IDX_1159CC0F47EE7BD5 | 1 | auctionhouse_id | A | 752 | NULL | NULL | YES | BTREE | | |
| auction | 1 | IDX_1159CC0F8BAC62AF | 1 | city_id | A | 100 | NULL | NULL | YES | BTREE | | |
| auction | 1 | auction_visible_idx | 1 | visible | A | 1 | NULL | NULL | | BTREE | | |
| auction | 1 | auction_startDate_idx | 1 | startDate | A | 8810 | NULL | NULL | | BTREE | | |
+---------+------------+-----------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Таблица Lot:
CREATE TABLE `Lot` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`auction_id` int(11) DEFAULT NULL,
`title` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`description` longtext COLLATE utf8mb4_unicode_ci,
`visible` tinyint(1) NOT NULL DEFAULT '1',
`url` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`number` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created` datetime NOT NULL,
`views` int(11) DEFAULT '0',
`startPrice` bigint(20) NOT NULL DEFAULT '0',
`endPrice` bigint(20) DEFAULT NULL,
`hammerPrice` bigint(20) DEFAULT NULL,
`trained` tinyint(1) NOT NULL DEFAULT '0',
`classifiedByCategory` tinyint(1) NOT NULL DEFAULT '0',
`goldStandard` tinyint(1) NOT NULL DEFAULT '0',
`slug` varchar(150) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_33CC6FFB989D9B62` (`slug`),
KEY `IDX_33CC6FFB57B8F0DE` (`auction_id`),
KEY `lot_visible_idx` (`visible`),
KEY `lot_hammerPrice_idx` (`hammerPrice`),
KEY `lot_views_idx` (`views`),
KEY `id` (`id`,`auction_id`,`views`),
CONSTRAINT `FK_33CC6FFB57B8F0DE` FOREIGN KEY (`auction_id`) REFERENCES `Auction` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=39363610 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Стол Аукцион:
CREATE TABLE `Auction` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`currency_id` int(11) DEFAULT NULL,
`auctionhouse_id` int(11) DEFAULT NULL,
`title` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`startDate` datetime NOT NULL,
`endDate` datetime DEFAULT NULL,
`created` datetime NOT NULL,
`visible` tinyint(1) NOT NULL,
`url` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`priceType` smallint(6) NOT NULL COMMENT '0: Aufrufpreis | 1: Schätzpreis | 2: Limitpreis; Standard: 0',
`identifier` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`address` varchar(1000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`viewing` longtext COLLATE utf8mb4_unicode_ci,
`useLocalImages` tinyint(1) NOT NULL DEFAULT '0',
`md5Image_id` int(11) DEFAULT NULL,
`city_id` int(11) DEFAULT NULL,
`importedViaApi` tinyint(1) NOT NULL DEFAULT '0',
`salecode` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_1159CC0FED9CD316` (`md5Image_id`),
KEY `IDX_1159CC0F38248176` (`currency_id`),
KEY `IDX_1159CC0F47EE7BD5` (`auctionhouse_id`),
KEY `IDX_1159CC0F8BAC62AF` (`city_id`),
KEY `auction_visible_idx` (`visible`),
KEY `auction_startDate_idx` (`startDate`),
CONSTRAINT `FK_1159CC0F38248176` FOREIGN KEY (`currency_id`) REFERENCES `Currency` (`id`),
CONSTRAINT `FK_1159CC0F47EE7BD5` FOREIGN KEY (`auctionhouse_id`) REFERENCES `Auctionhouse` (`id`) ON DELETE CASCADE,
CONSTRAINT `FK_1159CC0F8BAC62AF` FOREIGN KEY (`city_id`) REFERENCES `City` (`id`),
CONSTRAINT `FK_1159CC0FED9CD316` FOREIGN KEY (`md5Image_id`) REFERENCES `AuctionMd5Image` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB AUTO_INCREMENT=116337 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Ограничение FK: Lot.auction_id <=> Auction.id (только FK, необходимый для запроса, подлежащего оптимизации)
Этот запрос не завершается в течение разумного времени (<500 мс), он выполняется в течение нескольких минут.
Если я выполню <code>SELECT id FROM Auction WHERE startDate < :date;, запрос завершится в течение 80 мс.
Если я выполню SELECT id FROM Lot ORDER BY views DESC LIMIT 100 OFFSET 50, запрос завершится также через 20 мс. Похоже, что операция объединения занимает много времени.
Так почему это так медленно при объединении с объединением? Как я могу закрепить запрос?