Hello,

Please check following link for thread pool performance, i am not getting such performace through sysbench.

So from which command or from which tool we can check such benchmark performance.

http://www.mysql.com/products/enterprise/scalability.html

I have download Enterprise version of mysql for checking thread pool performance , but still i am not getting such performance

Thanks in advance

Hello,

Please check following link for thread pool performance, i am not getting such performace through sysbench.

So from which command or from which tool we can check such benchmark performance.

http://www.mysql.com/products/enterprise/scalability.html

I am interested to buy Enterprise version thats why I have download Enterprise version of mysql 5.6 for checking thread pool performance , but still i am not getting such performance

Thanks in advance

Machine with 32GB of RAM, SSD hard disk, win 8.1 64bit.

I had a preconfigued WAMP setup that was flying but needed to do my own setup. Thought I would try 5.7.3 but a query that was taking about 10 minutes in 5.1 or 5.2 (can't remember exactly which is now 6 hours and still going. I'll post my config file at the bottom to see if anyone has a suggestion. I swear I've covered everything that I have read but I think I am missing one key thing...

The query:

SELECT * FROM temp_company tc WHERE tc.number NOT IN (SELECT number FROM company c)

The equivalent query that is also slow:

SELECT tc.* FROM temp_company tc
LEFT JOIN company c on c.number = tc.number
WHERE c.number IS NULL

The tables are the same, both have a key on the number field and the cardinality of the column is equal to the number of rows. There are approx 3.1 million in each table.

Specifically number is formed by
`number` varchar(12) CHARACTER SET ascii COLLATE ascii_bin DEFAULT NULL
with its index KEY `number` (`number`)


EXPLAIN for the first query
+---------+------+---------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key
| key_len | ref | rows | filtered | Extra |
+----+--------------------+-------+------------+-------+---------------+--------
+---------+------+---------+----------+--------------------------+
| 1 | PRIMARY | tc | NULL | index | NULL | number
| 39 | NULL | 3131848 | 100.00 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | c | NULL | index | number | number
| 15 | NULL | 3169475 | 100.00 | Using where; Using index |
+----+--------------------+-------+------------+-------+---------------+--------
+---------+------+---------+----------+--------------------------+

Profile for the first query (limit 5):
+-------------------------------+----------+
| Status | Duration |
+-------------------------------+----------+
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000111 |
| executing | 0.000001 |
| Sending data | 0.000110 |
| executing | 0.000001 |
| Sending data | 0.000110 |
| executing | 0.000001 |
| Sending data | 0.000110 |
| executing | 0.000001 |
| Sending data | 0.000111 |
| executing | 0.000001 |
| Sending data | 0.000111 |
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000112 |
| executing | 0.000001 |
| Sending data | 0.000113 |
| executing | 0.000001 |
| Sending data | 0.000114 |
| executing | 0.000001 |
| Sending data | 0.000114 |
| executing | 0.000001 |
| Sending data | 0.000114 |
| executing | 0.000001 |
| Sending data | 0.000115 |
| executing | 0.000001 |
| Sending data | 0.000116 |
| executing | 0.000001 |
| Sending data | 0.000115 |
| executing | 0.000001 |
| Sending data | 0.000115 |
| executing | 0.000001 |
| Sending data | 0.000116 |
| executing | 0.000001 |
| Sending data | 0.000116 |
| executing | 0.000001 |
| Sending data | 0.000115 |
| executing | 0.000001 |
| Sending data | 0.000115 |
| executing | 0.000001 |
| Sending data | 0.000116 |
| executing | 0.000001 |
| Sending data | 0.000116 |
| executing | 0.000001 |
| Sending data | 0.000117 |
| executing | 0.000001 |
| Sending data | 0.000117 |
| executing | 0.000001 |
| Sending data | 0.000117 |
| executing | 0.000001 |
| Sending data | 0.000118 |
| executing | 0.000001 |
| Sending data | 0.000118 |
| executing | 0.000001 |
| Sending data | 0.000118 |
| executing | 0.000001 |
| Sending data | 0.000118 |
| executing | 0.000001 |
| Sending data | 0.000118 |
| executing | 0.000001 |
| Sending data | 0.000118 |
| executing | 0.000001 |
| Sending data | 0.000120 |
| executing | 0.000001 |
| Sending data | 0.000120 |
| executing | 0.000001 |
| Sending data | 0.000121 |
| executing | 0.000001 |
| Sending data | 0.000123 |
| executing | 0.000001 |
| Sending data | 0.000121 |
| executing | 0.000001 |
| Sending data | 0.000120 |
| executing | 0.000001 |
| Sending data | 0.000121 |
| executing | 0.000001 |
| Sending data | 0.000121 |
| executing | 0.000001 |
| Sending data | 0.000121 |
| executing | 0.000001 |
| Sending data | 0.000122 |
| executing | 0.000001 |
| Sending data | 0.000123 |
| executing | 0.000001 |
| Sending data | 0.000124 |
| executing | 0.000001 |
| Sending data | 1.063880 |
| end | 0.000009 |
| query end | 0.000008 |
| closing tables | 0.000009 |
| freeing items | 0.000007 |
| Waiting for query cache lock | 0.000002 |
| freeing items | 0.000062 |
| Waiting for query cache lock | 0.000002 |
| freeing items | 0.000001 |
| storing result in query cache | 0.000002 |
| cleaning up | 0.000028 |
+-------------------------------+----------+

The my.ini file:

[client]
no-beep

# pipe
# socket=mysql
port=3306

[mysql]

default-character-set=utf8


# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.
#
# server_type=3
[mysqld]

# The next three options are mutually exclusive to SERVER_PORT below.
# skip-networking
# enable-named-pipe
# The Pipe the MySQL Server will use
# socket=mysql

# The TCP/IP Port the MySQL Server will listen on
port=3306

# Path to installation directory. All paths are usually resolved relative to this.
# basedir="C:/WEB/MySQL/MySQL Server 5.7/"

# Path to the database root
datadir="C:/ProgramData/MySQL/MySQL Server 5.7/data\"

# The default character set that will be used when a new schema or table is
# created and no character set is defined
character-set-server=utf8

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

# Enable Windows Authentication
# plugin-load=authentication_windows.dll

# General and Slow logging.
log-output=FILE
general-log=0
general_log_file="NAME-PC.log"
slow-query-log=1
slow_query_log_file="NAME-PC-slow.log"
long_query_time=10

# Binary Logging.
# log-bin

# Error Logging.
log-error="NAME-PC.err"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=50

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=2048M

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache=2000

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=6144M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before. This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=10

#*** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method. This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=474M

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=16M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=2048K
read_rnd_buffer_size=12288K

# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=12288K

#*** INNODB Specific options ***
# innodb_data_home_dir=0.0

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb

# Additional memory pool that is used by InnoDB to store metadata
# information. If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS. As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size=32M

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=16M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system. Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=16G

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=4G

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=17

# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64

# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=8

# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=5000

# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000

# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
innodb_open_files=300

# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0

# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
innodb_file_per_table=1

# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm=0

# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=80

# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0

# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K

# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=104857600

# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=100

# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
open_files_limit=4161

# Set the query cache type. 0 for OFF, 1 for ON and 2 for DEMAND.
query_cache_type=1

# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size=12288K

# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400

# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.
binlog_row_event_max_size=8K

# If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_master_info events.
sync_master_info=10000

# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000

# If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000

#extra ones
max_heap_table_size=6144M
query_cache_limit=1024M

I'd like to thank everyone in advance for helping me out and being understanding. This is my first post in the forum so please correct me if I do something wrong or don't give enough information.

A little while ago I upgraded MySQL on my server to version 5.5.31 and since then MySQL has been running so slow that's it's become unusable. A script which used to take around 3 minutes on average, now takes almost an hour.

I run a very basic VPS with 256MB of RAM. Everything was working fine until the upgrade, so I'm guessing that's where the problem is--but it could be something else as other software was upgraded at the same time. The server runs Debian 7 (Wheezy) and I've been using PHP 5.4.22 with PDO to access MyISAM tables, though like I said the same exact code started running *much* slower after the upgrade.

Here's some other info that may help. Let me know if there's absolutely anything else I can do to help you help me.


mysql> SHOW VARIABLES LIKE 'sync%';
+---------------------+-------+
| Variable_name | Value |
+---------------------+-------+
| sync_binlog | 0 |
| sync_frm | ON |
| sync_master_info | 0 |
| sync_relay_log | 0 |
| sync_relay_log_info | 0 |
+---------------------+-------+
5 rows in set (0.00 sec)


mysql> SHOW VARIABLES LIKE '%buffer%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| bulk_insert_buffer_size | 8388608 |
| innodb_buffer_pool_instances | 1 |
| innodb_buffer_pool_size | 8388608 |
| innodb_change_buffering | all |
| innodb_log_buffer_size | 8388608 |
| join_buffer_size | 131072 |
| key_buffer_size | 8388608 |
| myisam_sort_buffer_size | 8388608 |
| net_buffer_length | 16384 |
| preload_buffer_size | 32768 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| sort_buffer_size | 2097152 |
| sql_buffer_result | OFF |
+------------------------------+---------+
14 rows in set (0.00 sec)


mysql> SHOW VARIABLES LIKE '%log%';
+-----------------------------------------+------------------------------------+
| Variable_name | Value |
+-----------------------------------------+------------------------------------+
| back_log | 50 |
| binlog_cache_size | 32768 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_format | STATEMENT |
| binlog_stmt_cache_size | 32768 |
| expire_logs_days | 0 |
| general_log | OFF |
| general_log_file | /var/lib/mysql/craigwatch.log |
| innodb_flush_log_at_trx_commit | 2 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 8388608 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | ./ |
| innodb_mirrored_log_groups | 1 |
| log | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_error | /var/lib/mysql/craigwatch.com.err |
| log_output | FILE |
| log_queries_not_using_indexes | OFF |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| max_relay_log_size | 0 |
| relay_log | |
| relay_log_index | |
| relay_log_info_file | relay-log.info |
| relay_log_purge | ON |
| relay_log_recovery | OFF |
| relay_log_space_limit | 0 |
| slow_query_log | OFF |
| slow_query_log_file | /var/lib/mysql/craigwatch-slow.log |
| sql_log_bin | ON |
| sql_log_off | OFF |
| sync_binlog | 0 |
| sync_relay_log | 0 |
| sync_relay_log_info | 0 |
+-----------------------------------------+------------------------------------+
41 rows in set (0.01 sec)


mysql> SHOW VARIABLES LIKE 'query_cache%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 0 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
+------------------------------+---------+
5 rows in set (0.00 sec)


mysql> SHOW GLOBAL STATUS LIKE 'Qc%';
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Qcache_free_blocks | 0 |
| Qcache_free_memory | 0 |
| Qcache_hits | 0 |
| Qcache_inserts | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 0 |
| Qcache_queries_in_cache | 0 |
| Qcache_total_blocks | 0 |
+-------------------------+-------+
8 rows in set (0.08 sec)


Once again, I really appreciate your patience and assistance!

Hi All - hoping someone can help shed some light on what i'm experiencing. Basically, I have two tables (below) with common field of CompanyID. When I join the two tables together and just select "email" from the first, explain shows using indexes from both tables. If I add a field company 2nd table (companyname), explain shows full table scan.

Also, Company has around 600,000 records. If I run the query that uses table scan against the Contact table with 300,000 records, it takes about 1 minute. If the Contact table has 2,000,000 records, it's taking about 1 hour.

I'm stumped as to what I can do, so would appreciate any insight.


CREATE TABLE `contact_temp_seg` (
`Email` varchar(128) DEFAULT NULL,
`ContactID` int(20) NOT NULL DEFAULT '0',
`CompanyID` int(20) DEFAULT NULL,
KEY `cont_seg_idx1` (`CompanyID`),
KEY `cont_seg_idx2` (`Email`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1


CREATE TABLE `company_temp_seg` (
`CompanyID` int(20) NOT NULL DEFAULT '0',
`CompanyName` varchar(128) DEFAULT NULL,
UNIQUE KEY `comp_temp_idx1` (`CompanyID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1


explain
Select a.Email, b.CompanyName
from Contact_Temp_Seg a, Company_Temp_Seg b
Where a.CompanyID = b.CompanyID

id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE b ALL comp_temp_idx1 NULL NULL NULL 597440 NULL
1 SIMPLE a ref cont_seg_idx1 cont_seg_idx1 5 hip.b.CompanyID 1 NULL

explain
Select a.Email
from Contact_Temp_Seg a, Company_Temp_Seg b
Where a.CompanyID = b.CompanyID

id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE b index comp_temp_idx1 comp_temp_idx1 4 NULL 597440 Using index
1 SIMPLE a ref cont_seg_idx1 cont_seg_idx1 5 hip.b.CompanyID 1 NULL

Hi All

We are facing performance related problem with select query.
We have reports table which contains approximate 2 Crore records. When we are executing simple count(*) from to check count, its taking more than 1 minute to display result.

Here is the info about mysql, server and query,

System info

OS : Debian 6.0.7
Model : AMD Opteron(tm) Processor 6172
cpu MHz : 2100.154
cache size : 512 KB
processor : 2

Memory total used free shared buffers cached
Mem: 16083 6335 9747 0 153 5323


Mysql info

mysql Ver 14.14 Distrib 5.1.66, for debian-linux-gnu (x86_64) using readline 6.1


my.conf settings

key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
max_connections = 1000
table_cache = 128
innodb_buffer_pool_size = 3G
query_cache_limit = 512M
query_cache_size = 3G

mysql> select count(*) from reports;
+-----------+
| count(*) |
+-----------+
| 23311587 |
+-----------+
1 row in set (67.07 sec)

DB engine : Innodb.

Can any body please help me to improve performance of table?

Thanks in Advance

Im using MySQL server 5.5.27, for Linux (x86_64) using readline 5.1. Only one of the table heavily used to dump data. It has 3 indexes and paritioned on timestamp.
Inserts are done from java code, batch inserts, 3000 records per second. But, while monitoring mysql server using any profiling application, we could see inserts rate are not going beyond 2000 per second. Eventually application starts dropping records. Below are the VM configuration and my.cnf. Using InnoDB. I have tried all the configuration tuning, but could not increase insert rate more than 2000 inserts.
Kindly suggest will any configiuration change help to increase performacne of mysql server.

(Note, in same VM i'm running Java application which has 2G of ram allocated for JVM)
VM configuration is 32GB RAM, 4 CPUs. Here is my.cnf
[mysqld]

innodb_buffer_pool_size=24G
bulk_insert_buffer_size=3G
thread_cache_size=15000

thread_stack = 512K
sort_buffer_size = 2M
read_buffer_size = 1M
read_rnd_buffer_size = 2M
join_buffer_size = 2M
binlog_cache_size = 128K
query_cache_size = 64M
query_cache_limit = 512K
max_connections = 1000
max_connect_errors = 100
concurrent_insert = 2
connect_timeout = 30
max_allowed_packet = 32M
max_heap_table_size = 1G
bulk_insert_buffer_size = 1G
tmp_table_size = 1G
table_cache = 512
table_open_cache = 512
key_buffer = 1M
key_buffer_size = 128M
innodb_write_io_threads = 64
innodb_read_io_threads = 64
innodb_io_capacity = 2000000
#innodb_file_io_threads = 16
#innodb_log_files_in_group = 3
innodb_buffer_pool_instances = 8
innodb_additional_mem_pool_size = 4M
innodb_flush_log_at_trx_commit = 2

transaction-isolation=READ-COMMITTED
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=8
innodb_log_buffer_size=4M


max_prepared_stmt_count = 100000
lower_case_table_names=1
expire_logs_days=1
innodb_support_xa = 0
log_warnings=5

Hi ALL,

I am new person for MySQL Performance Tuning.I had an slow query which is given the slow response time and I need some help to improve the response time. In that query I am creating one temporary tables and I get the datas from transaction tables. I insert into the temporary tables after that I fetch(select statement with some left joins) the result from that temporary tables. Is there to improve the slow performance time. Thank you

I want to reduce the time taken by the query in mysql.

There are three tables say A ~600k rows, B ~2K rows, and C ~100K rows having 2 columns each.

A has one column which is used in aggregation and other to join with table B. B has one column to join with A and other with C C has one column to join with B and other to group by.

What should be the indexing plan to reduce the run time. As of now it is using temporary tables and then file sort. Is there any way we could avoid temporary tables.

Sample query : select sum(revenue_facts.total_price) as m0 from category_groups as category_groups , revenue_facts as revenue_facts , dim_products as dim_products where dim_products.product_category_group_sk = category_groups.product_category_group_sk and revenue_facts.product_sk = dim_products.product_sk
group by category_groups.category_name ; I already have indexes on group by column and the columns in join.

I've got a fairly large application (>1M LoC) with a number of tables with 750k records though the normal implementation would only have 1k - 25k records. There is a cache table that holds text blobs with the keys of these records that are a group cache. I'm trying to figure out what was happening recently when mysqld went crazy/thrashed for a few days. Obviously the truncation of this cache table was taking huge amounts of time. It also appears that by the time it was done truncating it needed to start the truncation again, if I am understanding the repeated timestamp below. Can someone explain to me what it means that these timestamps on different queries are all the same?


# User@Host: thissite[thissite] @ localhost []
# Query_time: 160106.517713 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
SET timestamp=1391090819;
TRUNCATE my_table_cache;
# User@Host: thissite[thissite] @ localhost []
# Query_time: 160126.774491 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
SET timestamp=1391090819;
DELETE g
FROM my_table_cache g
WHERE g.group_id IN ( '205' );
# User@Host: thissite[thissite] @ localhost []
# Query_time: 160329.190431 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
SET timestamp=1391090819;
TRUNCATE my_table_cache;
# User@Host: thissite[thissite] @ localhost []
# Query_time: 160543.276354 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
SET timestamp=1391090819;T
RUNCATE my_table_cache;
# User@Host: thissite[thissite] @ localhost []
# Query_time: 160909.070926 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
SET timestamp=1391090819;
TRUNCATE my_table_cache;
# User@Host: thissite[thissite] @ localhost []
# Query_time: 161107.651733 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0
SET timestamp=1391090819;
TRUNCATE my_table_cache;

Hello,

I have installed a fresh version of a Wordpress site on my web host's server, deactivated plugins, installed WP Super Cache, limited certain countries' IPs from hitting my site, and altered code to up the memory limits to 256MB. I still get 100% CPU usage stats on the server.

My web host says my PHP memory limit is 256MB and my CPU RAM is 1 GB.

I look at phpMyAdmin and run the status/advisor and it returns 17 issues and recommendations. Since I am a newbie, I don't know how to begin to implement the recommendations. One such issue I show is:

"The rate of reading the first index entry is high. This usually indicates frequent full index scans. Full index scans are faster than table scans but require lots of CPU cycles in big tables, if those tables that have or had high volumes of UPDATEs and DELETEs, running 'OPTIMIZE TABLE' might reduce the amount of and/or speed up full index scans. Other than that full index scans can only be reduced by rewriting queries."

Could someone please point me in the right direction of how to do this/troubleshoot?

Thanks very much for taking a look!

Hello,

since our InnoDB database size (75GB) exceeds the RAM size (32GB), we plan to use SSD drives to speed up our MySQL Server.
Hardware specs:
- 32GB ECC RAM
- 2x 1 TB RAID 1
- RAID Controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)

MySQL settings:
innodb_flush_method = O_DIRECT
innodb_buffer_pool_size = 26624M
innodb_log_file_size = 256M
innodb_log_buffer_size = 8M

We want to use the free RAID ports to install two 240 GB 6 Gb/s SSD (RAID 1).

After reading some articles about MySQL SSD/HDD performance, I would use the SDD for all innoDB data except for Redo log and ibdata, which I would put on HDD.
Is this a good setup? Is it better to leave the OS (Ubuntu Linux) on HDD, or will performance be better if the OS is installed on SSD?

Thanks!

I have a partitioned raw data table, one partition is created per day, and only the last 30 days of data is kept. Each partition of the table contains over 200M rows.
The following query is executed:
SELECT device_key, component_key
FROM `raw_data` as raw
WHERE
raw.fct_ts >= "2014-02-10 00:00:00"
AND raw.fct_ts < "2014-02-11 00:00:00"
AND raw.device_key =? AND raw.component_key = ?

The query timed out after 10 minutes of execution. Only when I replaced the search criteria raw.fc_ts range with a specific fct_ts value, the results came back immediately.

Here is table definition:
CREATE TABLE `raw_data` (
`device_key` int(11) NOT NULL COMMENT 'Foreign key for the device dimension table.',
`component_key` int(11) NOT NULL COMMENT 'Foreign key for a component dimension table.',
`fct_ts` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' COMMENT 'Timestamp of the outputoctets__bytes measurement.',
`fct_resolution` int(10) unsigned NOT NULL COMMENT 'Seconds over which the outputoctets__bytes measurement was taken.',
`fct_value` double NOT NULL COMMENT 'Value of the outputoctets__bytes measurement.',
PRIMARY KEY (`device_key` , `component_key` , `fct_ts`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 PARTITION BY RANGE (unix_timestamp(fct_ts)) (PARTITION p735638 VALUES LESS THAN (1392012000) ENGINE = InnoDB , PARTITION p735639 VALUES LESS THAN (1392098400) ENGINE = InnoDB , PARTITION pEOW VALUES LESS THAN MAXVALUE ENGINE = InnoDB)$$

EXPLAIN PARTITION-
'1', 'SIMPLE', 'raw', 'p735639', 'range', 'PRIMARY', 'PRIMARY', '12', NULL, '229101935', 'Using where; Using index'

EXPLAIN PLAN
'1', 'SIMPLE', 'raw', 'range', 'PRIMARY', 'PRIMARY', '12', NULL, '229101935', 'Using where; Using index'


The mysql(version 5.6) configuration
innodb_buffer_pool_size = 100G
# log file size should be 25% of of buffer pool size
innodb_log_file_size = 1892M
innodb_additional_mem_pool_size = 32M
innodb_log_buffer_size = 256M
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 2
# In previous releases of MySQL, this was recommended to be set to 2 times the
# number of CPUs, however the default and recommended option in 5.5 is to not
# set a bound on the thread pool size.
innodb_thread_concurrency = 0

Any suggestions for improving the query performance?
Will changing the order of the PK from (fct_ts, device_key, component_key) to (device_key, component_key, fct_ts) help?
Any known issue of having timestamp as part of the PK?


Thanks in advance.

Hallo together,

a simpel select * from Table takes over 3 Minutes to execute.
The resulat has 19210 rows. A lock at the proviel shows me that sending data takes 182.743367 second.

Sorry I just start with MySQL (and Suse Linux) and I thing that this is my vault.But im not quit sure where to start.

To explane what I have doen.
I did make a Backup from the old databas (5.0) with MySQL Workbench.
Deinstall the old MYSQL with zypper. (rename the file directory)
Install MYSQL 5.6 with zypper and import the Backup with Workench.

Till yet it try some solution from google and change some values in the my.cnf .

The my.cnf looks no lik this:

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html
# *** DO NOT EDIT THIS FILE. It's a template which will be copied to the
# *** default location during install, and will be replaced if you
# *** upgrade to a newer version of MySQL.

[mysqld]

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
innodb_buffer_pool_size = 5500M
innodb_autoextend_increment = 512

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
# basedir = .....
# datadir = .....
# port = .....
# server_id = .....
# socket = .....

# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

#Externer zugriff Host
bind-address = 0.0.0.0 # Alle



I would be happy fore every help.

p.s.

please excuse my terrieble english, i am not a native speaker.

As in the code
"
void update_server_status()
{
ulonglong end_utime_of_query= current_utime();
if (end_utime_of_query > utime_after_lock + variables.long_query_time)
server_status|= SERVER_QUERY_WAS_SLOW;
}
"

This means that when deciding whether the query is a slow_query, the lock time is excluded.

Is it reasonable?
Will it be better to use start_utime to instead utime_after_lock here?

Need some help from MySQL experts to improve the execution time of the following query. It currently takes about 1 hour 25 minutes each night and growing a few minutes each day as more data is added to the rsdata and trdata tables. The other 3 tables are static.

Any help greatly appreciated.

Table sizes are:
rsdata - 480,566 rows / 130.4 MiB
trdata - 522,506 rows / 64.3 MiB
module - 1,458 rows / 164 KiB
variantgates - 87 rows / 8 KiB
testatussequence - 8 rows / 2.3 KiB

The query:
select distinct
rsdata.rs_variant,
rsdata.weeknumber,
rsdata.rs_moduleid,
rsdata.rs_objectidentifier,
trdata.tr_objectidentifier,
trdata.tr_testversion,
trdata.tr_plannedgate,
trdata.tr_vnvmethod,
testatussequence.tesequencenr,
trdata.tr_testexecutionstatus
from rsdata,
trdata,
module,
variantgates,
testatussequence
where rsdata.weeknumber='1408'
and rsdata.rs_moduleid = module.moduleid
and rsdata.rs_reviewstatus != 'Obsolete'
and rsdata.rs_reviewstatus != 'Rejected'
and rsdata.rs_introbjectidentifieralllevels != ''
and rsdata.rs_introbjectidentifieralllevels != 'Unknown'
and trdata.weeknumber='1408'
and trdata.tr_plannedgate != ''
and trdata.tr_plannedgate != 'Unknown'
and trdata.tr_plannedgate = variantgates.gate
and trdata.tr_variant = variantgates.variant
and trdata.tr_testexecutionstatus = testatussequence.testatus
and find_in_set(trdata.tr_objectidentifier, (select rsdata.rs_introbjectidentifieralllevels))
order by
rs_variant ASC,
weeknumber ASC,
rs_moduleid ASC,
rs_objectidentifier ASC,
tr_testversion ASC,
tr_plannedgate ASC,
tr_vnvmethod ASC,
tesequencenr DESC;


Output from explain query:

id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY variantgates ALL Variant_2,Variant NULL NULL NULL 87 Using where; Using temporary; Using filesort
1 PRIMARY trdata ref TR_Variant,WeekNumber TR_Variant 23 remets.variantgates.Variant 5226 Using where
1 PRIMARY testatussequence eq_ref PRIMARY,TEStatus PRIMARY 22 remets.trdata.TR_TestExecutionStatus 1
1 PRIMARY rsdata ref RSDataSI1,RSDataUI2,WeekNumber RSDataUI2 7 const 75658 Using where
1 PRIMARY module ref ModuleUI1 ModuleUI1 11 remets.rsdata.RS_ModuleID 10 Using where; Using index
2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL No tables used



The details of the tables:

CREATE TABLE `rsdata` (
`RS_Variant` varchar(20) default 'Unknown',
`RS_ModuleID` varchar(8) default 'Unknown',
`RS_ObjectIdentifier` varchar(50) default 'Unknown',
`RS_ObjectType` varchar(20) default 'Unknown',
`WeekNumber` varchar(4) default '9999',
`RS_ReviewStatus` varchar(12) default 'Unknown',
`RS_Priority` varchar(20) default 'Unknown',
`RS_PlannedGate` varchar(10) default 'Unknown',
`RS_VnVMethod` varchar(50) default 'Unknown',
`RS_Datasheet` varchar(1) default NULL,
`RS_RSLinks` int(11) default '0',
`RS_TSLinks` int(11) default '0',
`RS_ReqLinked` varchar(10) default 'Unknown',
`RS_VnVLinked` varchar(10) default 'Unknown',
`RS_InRSObjectIdentifierOneLevel` varchar(500) default 'Unknown',
`RS_InTSObjectIdentifierOneLevel` varchar(500) default 'Unknown',
`RS_InTSObjectIdentifierAllLevels` varchar(750) default 'Unknown',
`RS_InTRObjectIdentifierAllLevels` varchar(750) default 'Unknown',
UNIQUE KEY `RSDataUI1` (`RS_Variant`,`RS_ModuleID`,`RS_ObjectIdentifier`,`WeekNumber`),
KEY `RSDataSI1` (`RS_ModuleID`,`WeekNumber`),
KEY `RSDataUI2` (`WeekNumber`,`RS_Variant`,`RS_ModuleID`,`RS_ObjectIdentifier`),
KEY `WeekNumber` (`WeekNumber`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE `trdata` (
`TR_Variant` varchar(20) default NULL,
`TR_ModuleID` varchar(8) default NULL,
`TR_ObjectIdentifier` varchar(50) default NULL,
`TR_ObjectType` varchar(20) default 'Unknown',
`WeekNumber` varchar(4) default '9999',
`TR_ReviewStatus` varchar(12) default 'Unknown',
`TR_Priority` varchar(20) default 'Unknown',
`TR_PlannedGate` varchar(10) default 'Unknown',
`TR_VnVMethod` varchar(50) default 'Unknown',
`TR_TestVersion` varchar(12) default 'Unknown',
`TR_TestExecutionStatus` varchar(20) default 'Unknown',
UNIQUE KEY `TR_Variant` (`TR_Variant`,`TR_ModuleID`,`TR_ObjectIdentifier`,`WeekNumber`),
KEY `TRDataSI1` (`TR_ModuleID`,`WeekNumber`),
KEY `WeekNumber` (`WeekNumber`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE `module` (
`ModuleID` varchar(8) default NULL,
`ModuleType` varchar(5) default NULL,
`ModuleName` varchar(50) default NULL,
`ModuleNameLng` varchar(250) default NULL,
UNIQUE KEY `ModuleUI1` (`ModuleID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE `variantgates` (
`Variant` varchar(20) default NULL,
`GateGroup` varchar(20) default NULL,
`GateSequenceNr` int(11) default NULL,
`Gate` varchar(10) default NULL,
UNIQUE KEY `Variant_2` (`Variant`,`GateGroup`,`GateSequenceNr`),
KEY `Variant` (`Variant`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE `testatussequence` (
`TESequenceNr` int(11) NOT NULL,
`TEStatus` varchar(20) NOT NULL default '',
PRIMARY KEY (`TEStatus`),
UNIQUE KEY `TEStatus` (`TEStatus`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

Hi all. I've been reading this forums for years, and until now I've solved every problem related to MySQl since version 4. But now I'm stuck and so this is my first post here. Thanks to everyone in advance.

I run a machine with MySQL 5.6.16 and Tomcat 7.0.35. The traffic is around 50000 daily pageviews, 25000 daily unique visitors. Apart from serving pages, there are complex applications that do background, parallel processing in the database.

Machine specs
-------------

2xCPU E5-2630 2.3 ghz (12 physical cores, 24 logical cores), 32GB RAM, 4x600GB SAS in RAID 10, Red Hat Enterprise Linux 6.4

The problem
-----------

Load average is high, around 10. Iowait between 5%-15%. I don't know the reason. I thought it could be queries without indexes, but it doesn't seem to be that (see below). Almost every query is using indexes, with the exception of some UNION ALL that are infrequent.

Load average has been increased in the last weeks.

top - 10:21:41 up 174 days, 18:24, 1 user, load average: 10.60, 9.31, 8.60
Tasks: 637 total, 1 running, 635 sleeping, 0 stopped, 1 zombie
Cpu(s): 6.5%us, 0.6%sy, 0.0%ni, 82.9%id, 9.9%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 32687288k total, 32451548k used, 235740k free, 17308k buffers
Swap: 2097144k total, 7412k used, 2089732k free, 5882968k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5780 mysql 20 0 33.2g 22g 5832 S 91.8 73.6 2040:56 mysqld
13729 root 20 0 14.6g 1.7g 8328 S 83.9 5.4 655:03.48 java
11206 root 20 0 15420 1748 968 R 0.7 0.0 0:06.48 top
160 root 20 0 0 0 0 S 0.3 0.0 167:12.04 kblockd/6
259 root 20 0 0 0 0 S 0.3 0.0 146:53.47 kswapd1
338 root 39 19 0 0 0 S 0.3 0.0 215:51.21 kipmi0
1 root 20 0 19228 480 292 S 0.0 0.0 0:26.77 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 15:57.16 migration/0


top - 10:32:36 up 174 days, 18:35, 2 users, load average: 7.45, 8.71, 8.71
Tasks: 543 total, 1 running, 542 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.9%us, 0.4%sy, 0.0%ni, 86.7%id, 7.8%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 32687288k total, 32475072k used, 212216k free, 18128k buffers
Swap: 2097144k total, 7412k used, 2089732k free, 5963572k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5780 mysql 20 0 33.2g 22g 5832 S 67.5 73.5 2051:44 mysqld
13729 root 20 0 14.6g 1.7g 8332 S 65.2 5.4 665:41.57 java
99 root 20 0 0 0 0 S 0.3 0.0 56:33.75 events/0
122 root 20 0 0 0 0 S 0.3 0.0 10:09.10 events/23
259 root 20 0 0 0 0 S 0.3 0.0 146:54.01 kswapd1
338 root 39 19 0 0 0 S 0.3 0.0 215:51.77 kipmi0
13512 root 20 0 15424 1716 1008 R 0.3 0.0 0:00.17 top
1 root 20 0 19228 480 292 S 0.0 0.0 0:26.77 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd


top - 12:04:54 up 174 days, 20:07, 1 user, load average: 10.51, 9.24, 9.09
Tasks: 619 total, 1 running, 618 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 0.5%sy, 0.0%ni, 91.9%id, 4.6%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 32687288k total, 32488104k used, 199184k free, 20200k buffers
Swap: 2097144k total, 7412k used, 2089732k free, 6297620k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16032 root 20 0 14.7g 1.4g 12m S 58.3 4.4 61:26.96 java
5780 mysql 20 0 33.8g 22g 5840 S 25.8 73.5 2138:04 mysqld
116 root 20 0 0 0 0 S 0.3 0.0 9:50.24 events/17
154 root 20 0 0 0 0 S 0.3 0.0 175:36.99 kblockd/0
1 root 20 0 19228 480 292 S 0.0 0.0 0:26.78 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd

top - 12:07:25 up 174 days, 20:10, 1 user, load average: 10.56, 9.55, 9.22
Tasks: 619 total, 1 running, 617 sleeping, 0 stopped, 1 zombie
Cpu(s): 7.4%us, 0.6%sy, 0.0%ni, 80.4%id, 11.4%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 32687288k total, 32341388k used, 345900k free, 18172k buffers
Swap: 2097144k total, 7412k used, 2089732k free, 6079752k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16032 root 20 0 14.7g 1.4g 12m S 113.8 4.5 63:55.32 java
5780 mysql 20 0 33.8g 22g 5840 S 82.4 73.7 2140:07 mysqld
20596 root 20 0 15428 1768 1008 R 0.7 0.0 0:00.70 top
338 root 39 19 0 0 0 S 0.3 0.0 215:56.71 kipmi0
1598 root 20 0 243m 5236 428 S 0.3 0.0 2:48.35 rsyslogd
1 root 20 0 19228 480 292 S 0.0 0.0 0:26.78 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd



Performance is good, enabling slow_query_log with long_query_time=1 shows few queries every 3-5 minutes.

Enabling slow_query_log with log_queries_not_using_indexes, shows many UNION ALL queries, like these:

# Query_time: 0.000544 Lock_time: 0.000213 Rows_sent: 0 Rows_examined: 0
SET timestamp=1392804389;
select A.* from administradores_absolutos A,administradores_absolutos_multiples_nombres AN where AN.NOMBRE='XXX YYY ZZZ' and A.id=AN.id_administrador and AN.id_administrador!=1143381 UNION ALL select A.* from administradores_absolutos A,administradores_absolutos_multiples_nombres AN where AN.id_administrador=1143381 and AN.nombre=A.nombre and AN.pais=A.pais and AN.NOMBRE!='XXX YYY ZZZ';

I'm concerned about the future, because the database is growing. The current size is 462 GB. Largest table is 42 GB.

The speed of the background processing is not important. We used to perform INSERT and UPDATE in N parallel threads, but now we use a single thread and process them in a batch.

There are sometimes extremely slow insert & updates (background). There are sometimes extremely slow queries like this (10GB table)

# Query_time: 5.354870 Lock_time: 0.000035 Rows_sent: 251 Rows_examined: 251
SET timestamp=1392806972;
SELECT DATE,PAIS FROM STATS_DC_T_1 WHERE OBJECT_ID=3097122;

Explain plan:

id;select_type;table;type;possible_keys;key;key_len;ref;rows;Extra
1;SIMPLE;STATS_DC_T_1;ref;OBJECT_ID;OBJECT_ID;8;const;251;

Configuration files
---------------------

my.cnf:

[mysqld]
datadir = /opt/db
lower_case_table_names=1
innodb_large_prefix=true
innodb_file_format=barracuda
innodb_file_per_table=true
innodb_buffer_pool_size=20G
innodb_buffer_pool_instances=2
transaction-isolation=READ-COMMITTED
table-definition-cache=2000
default-time-zone='Europe/Madrid'
max_connections=130
query_cache_type=0
max_allowed_packet=64M
query_cache_size=0
key_buffer_size=512M
innodb_flush_log_at_trx_commit=2
innodb_read_io_threads = 32
innodb_write_io_threads = 32
thread_cache_size=128
innodb_log_file_size=128M
tmpdir=/dev/shm
table_open_cache=3000
skip-name-resolve
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

The Java JVM uses 2GB memory.
I've tried with innodb_buffer_pool_size of 16GB, and the situation is the same, with slightly worse performance.
I've tried different innodb_buffer_pool_instances values (between 2 and 8), currently 2 gives the best performance
I've tried different innodb_flush_method values, all of them give worse performance
I've tried innodb_flush_log_at_trx_commit=0, and updates are slower (innodb_flush_log_at_trx_commit=1 much slower)
I've tried different innodb_io_capacity values, with no luck

Database is in a lvm volume, mount info:

/dev/mapper/VolGroup-lvroot on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda3 on /boot type ext4 (rw)
/dev/sda1 on /boot/efi type vfat (rw,umask=0077,shortname=winnt)
/dev/mapper/VolGroup-lvdata on /opt/data type ext4 (rw,noatime,data=writeback,barrier=0,nobh)
/dev/mapper/VolGroup-lvdb on /opt/db type ext4 (rw,noatime,data=writeback,barrier=0,nobh)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

df-k output:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lvroot
10321208 4177700 5619220 43% /
tmpfs 16343644 4 16343640 1% /dev/shm
/dev/sda3 198337 68681 119416 37% /boot
/dev/sda1 204580 260 204320 1% /boot/efi
/dev/mapper/VolGroup-lvdata
209859040 64405708 134793084 33% /opt/data
/dev/mapper/VolGroup-lvdb
928910792 589935960 291788912 67% /opt/db

MySQLTuner information:

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.6.16
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
Warning: Using a password on the command line interface can be insecure.
Warning: Using a password on the command line interface can be insecure.
[--] Data in MyISAM tables: 2G (Tables: 121)
[--] Data in InnoDB tables: 232G (Tables: 436)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 52)
[!!] Total fragmented tables: 220

-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 7h 3m 36s (177M q [1K qps], 713K conn, TX: 158B, RX: 26B)
[--] Reads / Writes: 96% / 4%
[--] Total buffers: 20.5G global + 1.1M per thread (130 max threads)
[OK] Maximum possible memory usage: 20.7G (66% of installed RAM)
[OK] Slow queries: 0% (2K/177M)
[OK] Highest usage of available connections: 69% (90/130)
[OK] Key buffer size / total MyISAM indexes: 512.0M/320.9M
[OK] Key buffer hit rate: 100.0% (27M cached / 7K reads)
[!!] Query cache is disabled
[OK] Sorts requiring temporary tables: 2% (19K temp sorts / 736K sorts)
[OK] Temporary tables created on disk: 0% (880 on disk / 4M total)
[OK] Thread cache hit rate: 99% (90 created / 713K connections)
[OK] Table cache hit rate: 98% (1K open / 1K opened)
[OK] Open file limit used: 0% (85/16K)
[OK] Table locks acquired immediately: 99% (111M immediate / 111M locks)
[!!] InnoDB data size / buffer pool: 232.9G/20.0G

-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Variables to adjust:
query_cache_size (>= 8M)
innodb_buffer_pool_size (>= 232G)

Variables:

Variable_name;Value
auto_increment_increment;1
auto_increment_offset;1
autocommit;ON
automatic_sp_privileges;ON
back_log;76
basedir;/usr
big_tables;OFF
bind_address;*
binlog_cache_size;32768
binlog_checksum;CRC32
binlog_direct_non_transactional_updates;OFF
binlog_format;STATEMENT
binlog_max_flush_queue_time;0
binlog_order_commits;ON
binlog_row_image;FULL
binlog_rows_query_log_events;OFF
binlog_stmt_cache_size;32768
bulk_insert_buffer_size;8388608
character_set_client;utf8
character_set_connection;utf8
character_set_database;latin1
character_set_filesystem;binary
character_set_results;utf8
character_set_server;latin1
character_set_system;utf8
character_sets_dir;/usr/share/mysql/charsets/
collation_connection;utf8_general_ci
collation_database;latin1_spanish_ci
collation_server;latin1_swedish_ci
completion_type;NO_CHAIN
concurrent_insert;AUTO
connect_timeout;10
core_file;OFF
datadir;/opt/db/
date_format;%Y-%m-%d
datetime_format;%Y-%m-%d %H:%i:%s
default_storage_engine;InnoDB
default_tmp_storage_engine;InnoDB
default_week_format;0
delay_key_write;ON
delayed_insert_limit;100
delayed_insert_timeout;300
delayed_queue_size;1000
disconnect_on_expired_password;ON
div_precision_increment;4
end_markers_in_json;OFF
enforce_gtid_consistency;OFF
eq_range_index_dive_limit;10
error_count;0
event_scheduler;OFF
expire_logs_days;0
explicit_defaults_for_timestamp;OFF
external_user;
flush;OFF
flush_time;0
foreign_key_checks;ON
ft_boolean_syntax;+ -><()~*:""&|
ft_max_word_len;84
ft_min_word_len;4
ft_query_expansion_limit;20
ft_stopword_file;(built-in)
general_log;OFF
general_log_file;/opt/db/instnt.log
group_concat_max_len;1024
gtid_executed;
gtid_mode;OFF
gtid_next;AUTOMATIC
gtid_owned;
gtid_purged;
have_compress;YES
have_crypt;YES
have_dynamic_loading;YES
have_geometry;YES
have_openssl;DISABLED
have_profiling;YES
have_query_cache;YES
have_rtree_keys;YES
have_ssl;DISABLED
have_symlink;YES
host_cache_size;258
hostname;xxxx.com
identity;0
ignore_builtin_innodb;OFF
ignore_db_dirs;
init_connect;
init_file;
init_slave;
innodb_adaptive_flushing;ON
innodb_adaptive_flushing_lwm;10
innodb_adaptive_hash_index;ON
innodb_adaptive_max_sleep_delay;150000
innodb_additional_mem_pool_size;8388608
innodb_api_bk_commit_interval;5
innodb_api_disable_rowlock;OFF
innodb_api_enable_binlog;OFF
innodb_api_enable_mdl;OFF
innodb_api_trx_level;0
innodb_autoextend_increment;64
innodb_autoinc_lock_mode;1
innodb_buffer_pool_dump_at_shutdown;OFF
innodb_buffer_pool_dump_now;OFF
innodb_buffer_pool_filename;ib_buffer_pool
innodb_buffer_pool_instances;2
innodb_buffer_pool_load_abort;OFF
innodb_buffer_pool_load_at_startup;OFF
innodb_buffer_pool_load_now;OFF
innodb_buffer_pool_size;21474836480
innodb_change_buffer_max_size;25
innodb_change_buffering;all
innodb_checksum_algorithm;innodb
innodb_checksums;ON
innodb_cmp_per_index_enabled;OFF
innodb_commit_concurrency;0
innodb_compression_failure_threshold_pct;5
innodb_compression_level;6
innodb_compression_pad_pct_max;50
innodb_concurrency_tickets;5000
innodb_data_file_path;ibdata1:12M:autoextend
innodb_data_home_dir;
innodb_disable_sort_file_cache;OFF
innodb_doublewrite;ON
innodb_fast_shutdown;1
innodb_file_format;Barracuda
innodb_file_format_check;ON
innodb_file_format_max;Barracuda
innodb_file_per_table;ON
innodb_flush_log_at_timeout;1
innodb_flush_log_at_trx_commit;2
innodb_flush_method;
innodb_flush_neighbors;1
innodb_flushing_avg_loops;30
innodb_force_load_corrupted;OFF
innodb_force_recovery;0
innodb_ft_aux_table;
innodb_ft_cache_size;8000000
innodb_ft_enable_diag_print;OFF
innodb_ft_enable_stopword;ON
innodb_ft_max_token_size;84
innodb_ft_min_token_size;3
innodb_ft_num_word_optimize;2000
innodb_ft_result_cache_limit;2000000000
innodb_ft_server_stopword_table;
innodb_ft_sort_pll_degree;2
innodb_ft_total_cache_size;640000000
innodb_ft_user_stopword_table;
innodb_io_capacity;200
innodb_io_capacity_max;2000
innodb_large_prefix;ON
innodb_lock_wait_timeout;50
innodb_locks_unsafe_for_binlog;OFF
innodb_log_buffer_size;8388608
innodb_log_compressed_pages;ON
innodb_log_file_size;134217728
innodb_log_files_in_group;2
innodb_log_group_home_dir;./
innodb_lru_scan_depth;1024
innodb_max_dirty_pages_pct;75
innodb_max_dirty_pages_pct_lwm;0
innodb_max_purge_lag;0
innodb_max_purge_lag_delay;0
innodb_mirrored_log_groups;1
innodb_monitor_disable;
innodb_monitor_enable;
innodb_monitor_reset;
innodb_monitor_reset_all;
innodb_old_blocks_pct;37
innodb_old_blocks_time;1000
innodb_online_alter_log_max_size;134217728
innodb_open_files;3000
innodb_optimize_fulltext_only;OFF
innodb_page_size;16384
innodb_print_all_deadlocks;OFF
innodb_purge_batch_size;300
innodb_purge_threads;1
innodb_random_read_ahead;OFF
innodb_read_ahead_threshold;56
innodb_read_io_threads;32
innodb_read_only;OFF
innodb_replication_delay;0
innodb_rollback_on_timeout;OFF
innodb_rollback_segments;128
innodb_sort_buffer_size;1048576
innodb_spin_wait_delay;6
innodb_stats_auto_recalc;ON
innodb_stats_method;nulls_equal
innodb_stats_on_metadata;OFF
innodb_stats_persistent;ON
innodb_stats_persistent_sample_pages;20
innodb_stats_sample_pages;8
innodb_stats_transient_sample_pages;8
innodb_status_output;OFF
innodb_status_output_locks;OFF
innodb_strict_mode;OFF
innodb_support_xa;ON
innodb_sync_array_size;1
innodb_sync_spin_loops;30
innodb_table_locks;ON
innodb_thread_concurrency;0
innodb_thread_sleep_delay;10000
innodb_undo_directory;.
innodb_undo_logs;128
innodb_undo_tablespaces;0
innodb_use_native_aio;ON
innodb_use_sys_malloc;ON
innodb_version;5.6.16
innodb_write_io_threads;32
insert_id;0
interactive_timeout;28800
join_buffer_size;262144
keep_files_on_create;OFF
key_buffer_size;536870912
key_cache_age_threshold;300
key_cache_block_size;1024
key_cache_division_limit;100
large_files_support;ON
large_page_size;0
large_pages;OFF
last_insert_id;0
lc_messages;en_US
lc_messages_dir;/usr/share/mysql/
lc_time_names;en_US
license;GPL
local_infile;ON
lock_wait_timeout;31536000
locked_in_memory;OFF
log_bin;OFF
log_bin_basename;
log_bin_index;
log_bin_trust_function_creators;OFF
log_bin_use_v1_row_events;OFF
log_error;/opt/db/instnt.com.err
log_output;FILE
log_queries_not_using_indexes;OFF
log_slave_updates;OFF
log_slow_admin_statements;OFF
log_slow_slave_statements;OFF
log_throttle_queries_not_using_indexes;0
log_warnings;1
long_query_time;1.000000
low_priority_updates;OFF
lower_case_file_system;OFF
lower_case_table_names;1
master_info_repository;FILE
master_verify_checksum;OFF
max_allowed_packet;67108864
max_binlog_cache_size;18446744073709547520
max_binlog_size;1073741824
max_binlog_stmt_cache_size;18446744073709547520
max_connect_errors;100
max_connections;130
max_delayed_threads;20
max_error_count;64
max_heap_table_size;16777216
max_insert_delayed_threads;20
max_join_size;18446744073709551615
max_length_for_sort_data;1024
max_prepared_stmt_count;16382
max_relay_log_size;0
max_seeks_for_key;18446744073709551615
max_sort_length;1024
max_sp_recursion_depth;0
max_tmp_tables;32
max_user_connections;0
max_write_lock_count;18446744073709551615
metadata_locks_cache_size;1024
metadata_locks_hash_instances;8
min_examined_row_limit;0
multi_range_count;256
myisam_data_pointer_size;6
myisam_max_sort_file_size;9223372036853727232
myisam_mmap_size;18446744073709551615
myisam_recover_options;OFF
myisam_repair_threads;1
myisam_sort_buffer_size;8388608
myisam_stats_method;nulls_unequal
myisam_use_mmap;OFF
net_buffer_length;16384
net_read_timeout;30
net_retry_count;10
net_write_timeout;60
new;OFF
old;OFF
old_alter_table;OFF
old_passwords;0
open_files_limit;16384
optimizer_prune_level;1
optimizer_search_depth;62
optimizer_switch;index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on
optimizer_trace;enabled=off,one_line=off
optimizer_trace_features;greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit;1
optimizer_trace_max_mem_size;16384
optimizer_trace_offset;-1
performance_schema;ON
performance_schema_accounts_size;100
performance_schema_digests_size;10000
performance_schema_events_stages_history_long_size;10000
performance_schema_events_stages_history_size;10
performance_schema_events_statements_history_long_size;10000
performance_schema_events_statements_history_size;10
performance_schema_events_waits_history_long_size;10000
performance_schema_events_waits_history_size;10
performance_schema_hosts_size;100
performance_schema_max_cond_classes;80
performance_schema_max_cond_instances;4620
performance_schema_max_file_classes;50
performance_schema_max_file_handles;32768
performance_schema_max_file_instances;9539
performance_schema_max_mutex_classes;200
performance_schema_max_mutex_instances;21780
performance_schema_max_rwlock_classes;40
performance_schema_max_rwlock_instances;12660
performance_schema_max_socket_classes;10
performance_schema_max_socket_instances;280
performance_schema_max_stage_classes;150
performance_schema_max_statement_classes;168
performance_schema_max_table_handles;6000
performance_schema_max_table_instances;12500
performance_schema_max_thread_classes;50
performance_schema_max_thread_instances;360
performance_schema_session_connect_attrs_size;512
performance_schema_setup_actors_size;100
performance_schema_setup_objects_size;100
performance_schema_users_size;100
pid_file;/opt/db/instint.com.pid
plugin_dir;/usr/lib64/mysql/plugin/
port;3306
preload_buffer_size;32768
profiling;OFF
profiling_history_size;15
protocol_version;10
proxy_user;
pseudo_slave_mode;OFF
pseudo_thread_id;698665
query_alloc_block_size;8192
query_cache_limit;1048576
query_cache_min_res_unit;4096
query_cache_size;0
query_cache_type;OFF
query_cache_wlock_invalidate;OFF
query_prealloc_size;8192
rand_seed1;0
rand_seed2;0
range_alloc_block_size;4096
read_buffer_size;131072
read_only;OFF
read_rnd_buffer_size;262144
relay_log;
relay_log_basename;
relay_log_index;
relay_log_info_file;relay-log.info
relay_log_info_repository;FILE
relay_log_purge;ON
relay_log_recovery;OFF
relay_log_space_limit;0
report_host;
report_password;
report_port;3306
report_user;
rpl_stop_slave_timeout;31536000
secure_auth;ON
secure_file_priv;
server_id;0
server_id_bits;32
server_uuid;d112791a-7160-11e2-a0d8-6cae8b61ce6a
skip_external_locking;ON
skip_name_resolve;ON
skip_networking;OFF
skip_show_database;OFF
slave_allow_batching;OFF
slave_checkpoint_group;512
slave_checkpoint_period;300
slave_compressed_protocol;OFF
slave_exec_mode;STRICT
slave_load_tmpdir;/dev/shm
slave_max_allowed_packet;1073741824
slave_net_timeout;3600
slave_parallel_workers;0
slave_pending_jobs_size_max;16777216
slave_rows_search_algorithms;TABLE_SCAN,INDEX_SCAN
slave_skip_errors;OFF
slave_sql_verify_checksum;ON
slave_transaction_retries;10
slave_type_conversions;
slow_launch_time;2
slow_query_log;ON
slow_query_log_file;/opt/db/instint-slow.log
socket;/var/lib/mysql/mysql.sock
sort_buffer_size;262144
sql_auto_is_null;OFF
sql_big_selects;ON
sql_buffer_result;OFF
sql_log_bin;ON
sql_log_off;OFF
sql_mode;STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION
sql_notes;ON
sql_quote_show_create;ON
sql_safe_updates;OFF
sql_select_limit;18446744073709551615
sql_slave_skip_counter;0
sql_warnings;OFF
ssl_ca;
ssl_capath;
ssl_cert;
ssl_cipher;
ssl_crl;
ssl_crlpath;
ssl_key;
storage_engine;InnoDB
stored_program_cache;256
sync_binlog;0
sync_frm;ON
sync_master_info;10000
sync_relay_log;10000
sync_relay_log_info;10000
system_time_zone;CET
table_definition_cache;2000
table_open_cache;3000
table_open_cache_instances;1
thread_cache_size;128
thread_concurrency;10
thread_handling;one-thread-per-connection
thread_stack;262144
time_format;%H:%i:%s
time_zone;Europe/Madrid
timed_mutexes;OFF
timestamp;1392808236.865061
tmp_table_size;16777216
tmpdir;/dev/shm
transaction_alloc_block_size;8192
transaction_allow_batching;OFF
transaction_prealloc_size;4096
tx_isolation;READ-COMMITTED
tx_read_only;OFF
unique_checks;ON
updatable_views_with_limit;YES
version;5.6.16
version_comment;MySQL Community Server (GPL)
version_compile_machine;x86_64
version_compile_os;Linux
wait_timeout;28800
warning_count;0

Status:

Variable_name;Value
Aborted_clients;71
Aborted_connects;14
Binlog_cache_disk_use;0
Binlog_cache_use;0
Binlog_stmt_cache_disk_use;0
Binlog_stmt_cache_use;0
Bytes_received;10982
Bytes_sent;465207
Com_admin_commands;474
Com_assign_to_keycache;0
Com_alter_db;0
Com_alter_db_upgrade;0
Com_alter_event;0
Com_alter_function;0
Com_alter_procedure;0
Com_alter_server;0
Com_alter_table;0
Com_alter_tablespace;0
Com_alter_user;0
Com_analyze;0
Com_begin;0
Com_binlog;0
Com_call_procedure;0
Com_change_db;1
Com_change_master;0
Com_check;0
Com_checksum;0
Com_commit;0
Com_create_db;0
Com_create_event;0
Com_create_function;0
Com_create_index;0
Com_create_procedure;0
Com_create_server;0
Com_create_table;0
Com_create_trigger;0
Com_create_udf;0
Com_create_user;0
Com_create_view;0
Com_dealloc_sql;0
Com_delete;0
Com_delete_multi;0
Com_do;0
Com_drop_db;0
Com_drop_event;0
Com_drop_function;0
Com_drop_index;0
Com_drop_procedure;0
Com_drop_server;0
Com_drop_table;0
Com_drop_trigger;0
Com_drop_user;0
Com_drop_view;0
Com_empty_query;0
Com_execute_sql;0
Com_flush;0
Com_get_diagnostics;0
Com_grant;0
Com_ha_close;0
Com_ha_open;0
Com_ha_read;0
Com_help;0
Com_insert;0
Com_insert_select;0
Com_install_plugin;0
Com_kill;0
Com_load;0
Com_lock_tables;0
Com_optimize;0
Com_preload_keys;0
Com_prepare_sql;0
Com_purge;0
Com_purge_before_date;0
Com_release_savepoint;0
Com_rename_table;0
Com_rename_user;0
Com_repair;0
Com_replace;0
Com_replace_select;0
Com_reset;0
Com_resignal;0
Com_revoke;0
Com_revoke_all;0
Com_rollback;0
Com_rollback_to_savepoint;0
Com_savepoint;0
Com_select;13
Com_set_option;5
Com_signal;0
Com_show_binlog_events;0
Com_show_binlogs;0
Com_show_charsets;0
Com_show_collations;1
Com_show_create_db;0
Com_show_create_event;0
Com_show_create_func;0
Com_show_create_proc;0
Com_show_create_table;9
Com_show_create_trigger;0
Com_show_databases;0
Com_show_engine_logs;0
Com_show_engine_mutex;0
Com_show_engine_status;0
Com_show_events;1
Com_show_errors;0
Com_show_fields;1
Com_show_function_code;0
Com_show_function_status;1
Com_show_grants;0
Com_show_keys;0
Com_show_master_status;0
Com_show_open_tables;0
Com_show_plugins;0
Com_show_privileges;0
Com_show_procedure_code;0
Com_show_procedure_status;1
Com_show_processlist;1
Com_show_profile;0
Com_show_profiles;0
Com_show_relaylog_events;0
Com_show_slave_hosts;0
Com_show_slave_status;0
Com_show_status;3
Com_show_storage_engines;1
Com_show_table_status;5
Com_show_tables;0
Com_show_triggers;1
Com_show_variables;4
Com_show_warnings;0
Com_slave_start;0
Com_slave_stop;0
Com_stmt_close;0
Com_stmt_execute;0
Com_stmt_fetch;0
Com_stmt_prepare;0
Com_stmt_reprepare;0
Com_stmt_reset;0
Com_stmt_send_long_data;0
Com_truncate;0
Com_uninstall_plugin;0
Com_unlock_tables;0
Com_update;2
Com_update_multi;0
Com_xa_commit;0
Com_xa_end;0
Com_xa_prepare;0
Com_xa_recover;0
Com_xa_rollback;0
Com_xa_start;0
Compression;ON
Connection_errors_accept;0
Connection_errors_internal;0
Connection_errors_max_connections;0
Connection_errors_peer_address;0
Connection_errors_select;0
Connection_errors_tcpwrap;0
Connections;751138
Created_tmp_disk_tables;5
Created_tmp_files;38576
Created_tmp_tables;72
Delayed_errors;0
Delayed_insert_threads;0
Delayed_writes;0
Flush_commands;1
Handler_commit;12
Handler_delete;0
Handler_discover;0
Handler_external_lock;36
Handler_mrr_init;0
Handler_prepare;0
Handler_read_first;2
Handler_read_key;1030
Handler_read_last;4
Handler_read_next;1029
Handler_read_prev;2826
Handler_read_rnd;0
Handler_read_rnd_next;2549
Handler_rollback;0
Handler_savepoint;0
Handler_savepoint_rollback;0
Handler_update;2
Handler_write;2532
Innodb_buffer_pool_dump_status;not started
Innodb_buffer_pool_load_status;not started
Innodb_buffer_pool_pages_data;1252604
Innodb_buffer_pool_bytes_data;20522663936
Innodb_buffer_pool_pages_dirty;20404
Innodb_buffer_pool_bytes_dirty;334299136
Innodb_buffer_pool_pages_flushed;7782403
Innodb_buffer_pool_pages_free;2005
Innodb_buffer_pool_pages_misc;56110
Innodb_buffer_pool_pages_total;1310719
Innodb_buffer_pool_read_ahead_rnd;0
Innodb_buffer_pool_read_ahead;1163621
Innodb_buffer_pool_read_ahead_evicted;9
Innodb_buffer_pool_read_requests;22041438899
Innodb_buffer_pool_reads;42474547
Innodb_buffer_pool_wait_free;0
Innodb_buffer_pool_write_requests;106010393
Innodb_data_fsyncs;1896949
Innodb_data_pending_fsyncs;1
Innodb_data_pending_reads;2
Innodb_data_pending_writes;0
Innodb_data_read;732948140032
Innodb_data_reads;44737350
Innodb_data_writes;13616761
Innodb_data_written;267467585536
Innodb_dblwr_pages_written;7782403
Innodb_dblwr_writes;227552
Innodb_have_atomic_builtins;ON
Innodb_log_waits;17
Innodb_log_write_requests;24735436
Innodb_log_writes;5568625
Innodb_os_log_fsyncs;131688
Innodb_os_log_pending_fsyncs;0
Innodb_os_log_pending_writes;0
Innodb_os_log_written;12448931328
Innodb_page_size;16384
Innodb_pages_created;270343
Innodb_pages_read;44738563
Innodb_pages_written;7782402
Innodb_row_lock_current_waits;0
Innodb_row_lock_time;372756
Innodb_row_lock_time_avg;46
Innodb_row_lock_time_max;51004
Innodb_row_lock_waits;7962
Innodb_rows_deleted;476002
Innodb_rows_inserted;10079517
Innodb_rows_read;7521242058
Innodb_rows_updated;4611614
Innodb_num_open_files;471
Innodb_truncated_status_writes;0
Innodb_available_undo_logs;128
Key_blocks_not_flushed;0
Key_blocks_unused;420105
Key_blocks_used;8689
Key_read_requests;29874549
Key_reads;8220
Key_write_requests;3323642
Key_writes;17199
Last_query_cost;10.499000
Last_query_partial_plans;1
Max_used_connections;99
Not_flushed_delayed_rows;0
Open_files;308
Open_streams;0
Open_table_definitions;689
Open_tables;1661
Opened_files;43412
Opened_table_definitions;0
Opened_tables;0
Performance_schema_accounts_lost;0
Performance_schema_cond_classes_lost;0
Performance_schema_cond_instances_lost;0
Performance_schema_digest_lost;0
Performance_schema_file_classes_lost;0
Performance_schema_file_handles_lost;0
Performance_schema_file_instances_lost;0
Performance_schema_hosts_lost;0
Performance_schema_locker_lost;0
Performance_schema_mutex_classes_lost;0
Performance_schema_mutex_instances_lost;0
Performance_schema_rwlock_classes_lost;0
Performance_schema_rwlock_instances_lost;0
Performance_schema_session_connect_attrs_lost;0
Performance_schema_socket_classes_lost;0
Performance_schema_socket_instances_lost;0
Performance_schema_stage_classes_lost;0
Performance_schema_statement_classes_lost;0
Performance_schema_table_handles_lost;0
Performance_schema_table_instances_lost;0
Performance_schema_thread_classes_lost;0
Performance_schema_thread_instances_lost;0
Performance_schema_users_lost;0
Prepared_stmt_count;0
Qcache_free_blocks;0
Qcache_free_memory;0
Qcache_hits;0
Qcache_inserts;0
Qcache_lowmem_prunes;0
Qcache_not_cached;0
Qcache_queries_in_cache;0
Qcache_total_blocks;0
Queries;187455074
Questions;52
Select_full_join;0
Select_full_range_join;0
Select_range;2
Select_range_check;0
Select_scan;21
Slave_heartbeat_period;
Slave_last_heartbeat;
Slave_open_temp_tables;0
Slave_received_heartbeats;
Slave_retried_transactions;
Slave_running;OFF
Slow_launch_threads;0
Slow_queries;3
Sort_merge_passes;0
Sort_range;0
Sort_rows;0
Sort_scan;0
Ssl_accept_renegotiates;0
Ssl_accepts;0
Ssl_callback_cache_hits;0
Ssl_cipher;
Ssl_cipher_list;
Ssl_client_connects;0
Ssl_connect_renegotiates;0
Ssl_ctx_verify_depth;0
Ssl_ctx_verify_mode;0
Ssl_default_timeout;0
Ssl_finished_accepts;0
Ssl_finished_connects;0
Ssl_server_not_after;
Ssl_server_not_before;
Ssl_session_cache_hits;0
Ssl_session_cache_misses;0
Ssl_session_cache_mode;NONE
Ssl_session_cache_overflows;0
Ssl_session_cache_size;0
Ssl_session_cache_timeouts;0
Ssl_sessions_reused;0
Ssl_used_session_cache_entries;0
Ssl_verify_depth;0
Ssl_verify_mode;0
Ssl_version;
Table_locks_immediate;117792284
Table_locks_waited;43
Table_open_cache_hits;601
Table_open_cache_misses;0
Table_open_cache_overflows;0
Tc_log_max_pages_used;0
Tc_log_page_size;0
Tc_log_page_waits;0
Threads_cached;74
Threads_connected;25
Threads_created;99
Threads_running;19
Uptime;117556
Uptime_since_flush_status;117556


=====================================
2014-02-19 12:13:33 7f8ec11b0700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 25 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 92543 srv_active, 0 srv_shutdown, 39 srv_idle
srv_master_thread log flush and writes: 92582
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 3909832
OS WAIT ARRAY INFO: signal count 5847856
Mutex spin waits 12196760, rounds 93452389, OS waits 2736187
RW-shared spins 2257375, rounds 29148794, OS waits 705893
RW-excl spins 722218, rounds 21939901, OS waits 417583
Spin rounds per wait: 7.66 mutex, 12.91 RW-shared, 30.38 RW-excl
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (read thread)
I/O thread 7 state: waiting for completed aio requests (read thread)
I/O thread 8 state: waiting for completed aio requests (read thread)
I/O thread 9 state: waiting for completed aio requests (read thread)
I/O thread 10 state: waiting for completed aio requests (read thread)
I/O thread 11 state: waiting for completed aio requests (read thread)
I/O thread 12 state: waiting for completed aio requests (read thread)
I/O thread 13 state: waiting for completed aio requests (read thread)
I/O thread 14 state: waiting for completed aio requests (read thread)
I/O thread 15 state: waiting for completed aio requests (read thread)
I/O thread 16 state: waiting for completed aio requests (read thread)
I/O thread 17 state: waiting for completed aio requests (read thread)
I/O thread 18 state: waiting for completed aio requests (read thread)
I/O thread 19 state: waiting for completed aio requests (read thread)
I/O thread 20 state: waiting for completed aio requests (read thread)
I/O thread 21 state: waiting for completed aio requests (read thread)
I/O thread 22 state: waiting for completed aio requests (read thread)
I/O thread 23 state: waiting for completed aio requests (read thread)
I/O thread 24 state: waiting for completed aio requests (read thread)
I/O thread 25 state: waiting for completed aio requests (read thread)
I/O thread 26 state: waiting for completed aio requests (read thread)
I/O thread 27 state: waiting for completed aio requests (read thread)
I/O thread 28 state: waiting for completed aio requests (read thread)
I/O thread 29 state: waiting for completed aio requests (read thread)
I/O thread 30 state: waiting for completed aio requests (read thread)
I/O thread 31 state: waiting for completed aio requests (read thread)
I/O thread 32 state: waiting for completed aio requests (read thread)
I/O thread 33 state: waiting for completed aio requests (read thread)
I/O thread 34 state: waiting for completed aio requests (write thread)
I/O thread 35 state: waiting for completed aio requests (write thread)
I/O thread 36 state: waiting for completed aio requests (write thread)
I/O thread 37 state: waiting for completed aio requests (write thread)
I/O thread 38 state: waiting for completed aio requests (write thread)
I/O thread 39 state: waiting for completed aio requests (write thread)
I/O thread 40 state: waiting for completed aio requests (write thread)
I/O thread 41 state: waiting for completed aio requests (write thread)
I/O thread 42 state: waiting for completed aio requests (write thread)
I/O thread 43 state: waiting for completed aio requests (write thread)
I/O thread 44 state: waiting for completed aio requests (write thread)
I/O thread 45 state: waiting for completed aio requests (write thread)
I/O thread 46 state: waiting for completed aio requests (write thread)
I/O thread 47 state: waiting for completed aio requests (write thread)
I/O thread 48 state: waiting for completed aio requests (write thread)
I/O thread 49 state: waiting for completed aio requests (write thread)
I/O thread 50 state: waiting for completed aio requests (write thread)
I/O thread 51 state: waiting for completed aio requests (write thread)
I/O thread 52 state: waiting for completed aio requests (write thread)
I/O thread 53 state: waiting for completed aio requests (write thread)
I/O thread 54 state: waiting for completed aio requests (write thread)
I/O thread 55 state: waiting for completed aio requests (write thread)
I/O thread 56 state: waiting for completed aio requests (write thread)
I/O thread 57 state: waiting for completed aio requests (write thread)
I/O thread 58 state: waiting for completed aio requests (write thread)
I/O thread 59 state: waiting for completed aio requests (write thread)
I/O thread 60 state: waiting for completed aio requests (write thread)
I/O thread 61 state: waiting for completed aio requests (write thread)
I/O thread 62 state: waiting for completed aio requests (write thread)
I/O thread 63 state: waiting for completed aio requests (write thread)
I/O thread 64 state: waiting for completed aio requests (write thread)
I/O thread 65 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
44793995 OS file reads, 13633525 OS file writes, 1899611 OS fsyncs
4 pending preads, 0 pending pwrites
425.42 reads/s, 16384 avg bytes/read, 80.28 writes/s, 16.64 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 5909, free list len 55168, seg size 61078, 1595654 merges
merged operations:
insert 3332111, delete mark 1293065, delete 386475
discarded operations:
insert 5, delete mark 0, delete 0
Hash table size 42499631, node heap has 56124 buffer(s)
81103.16 hash searches/s, 5544.98 non-hash searches/s
---
LOG
---
Log sequence number 1880598171372
Log flushed up to 1880598141532
Pages flushed up to 1880574289557
Last checkpoint at 1880573892822
0 pending log writes, 0 pending chkp writes
5587307 log i/o's done, 28.04 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 21978152960; in additional pool allocated 0
Dictionary memory allocated 9544167
Buffer pool size 1310719
Free buffers 1693
Database pages 1252902
Old database pages 462535
Modified db pages 17740
Pending reads 4
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 27257573, not young 607122492
221.83 youngs/s, 5617.74 non-youngs/s
Pages read 44795222, created 270601, written 7793375
425.26 reads/s, 1.36 creates/s, 50.20 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 20 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1252902, unzip_LRU len: 0
I/O sum[43420]:cur[880], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 655360
Free buffers 841
Database pages 626478
Old database pages 231277
Modified db pages 9225
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 13930350, not young 306054081
115.00 youngs/s, 2823.69 non-youngs/s
Pages read 22695071, created 162262, written 4001055
217.31 reads/s, 1.08 creates/s, 27.44 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 20 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 626478, unzip_LRU len: 0
I/O sum[21710]:cur[440], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 655359
Free buffers 852
Database pages 626424
Old database pages 231258
Modified db pages 8515
Pending reads 4
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 13327223, not young 301068411
106.84 youngs/s, 2794.05 non-youngs/s
Pages read 22100151, created 108339, written 3792320
207.95 reads/s, 0.28 creates/s, 22.76 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 20 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 626424, unzip_LRU len: 0
I/O sum[21710]:cur[440], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
3 read views open inside InnoDB
Main thread process no. 5780, id 140226909579008, state: sleeping
Number of rows inserted 10080740, updated 4617793, deleted 476016, read 7525336775
7.60 inserts/s, 20.32 updates/s, 0.08 deletes/s, 105339.35 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================


Again, many thanks in advance

Q:
I have been running this select count(*) from batch table over night !
it's been sitting for 10 hours and does not returns any count for me !!

WHY?

>>select count(*) from batch ;

PRIMARY KEY (`batch_id`),
KEY `idx_batch_user_date` (`user_id`,`creation_date`)

>> explain select count(*) from batch ;
+------+-------------+-----------+-------+---------------+---------------------+---------+------+-----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-----------+-------+---------------+---------------------+---------+------+-----------+-------------+
| 1 | SIMPLE | batch | index | NULL | idx_batch_user_date | 14 | NULL | 674576723 | Using index |
+------+-------------+-----------+-------+---------------+---------------------+---------+------+-----------+-------------+
1 row in set (0.00 sec)

+-----------+
>>Show create table batch | CREATE TABLE `batch` (
`batch_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`batch_name` varchar(32) DEFAULT NULL,
`batch_desc` varchar(128) DEFAULT NULL,
`num_files` tinyint(3) unsigned DEFAULT NULL,
`type` varchar(3) DEFAULT NULL,
`status` varchar(3) DEFAULT NULL,
`user_id` int(10) unsigned DEFAULT NULL,
`mark_delete` tinyint(1) DEFAULT NULL,
`creation_date` datetime DEFAULT NULL,
`expiration_date` datetime DEFAULT NULL,
`batch_lock` int(16) DEFAULT NULL,
`expiration_interval` int(30) DEFAULT NULL,
PRIMARY KEY (`batch_id`),
KEY `idx_batch_user_date` (`user_id`,`creation_date`)
) ENGINE=InnoDB AUTO_INCREMENT=2047622768 DEFAULT CHARSET=utf8 |
+-----------+
)>> SHOW TABLE STATUS LIKE 'batch' \G
*************************** 1. row ***************************
Name: batch
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 674576234
Avg_row_length: 129
Data_length: 87298146304
Max_data_length: 0
Index_length: 20997619712
Data_free: 114294784
Auto_increment: 2047621060
Create_time: 2013-12-19 15:57:25
Update_time: NULL
Check_time: NULL
Collation: utf8_general_ci
Checksum: NULL
Create_options:
Comment:
1 row in set (0.14 sec)


>> select 87298146304/1024/1024/1024 as DATA_GB from dual ;
+-----------------+
| DATA_GB |
+-----------------+
| 81.302734375000 |
+-----------------+
1 row in set (0.00 sec)

>> select 20997619712/1024/1024/1024 as INDEX_GB from dual ;
+-----------------+
| INDEX_GB |
+-----------------+
| 19.555557250977 |
+-----------------+
1 row in set (0.00 sec)

I am using MySQL as queue / dispatching system for workers that really works very well for hundreds of workers executing concurrently on very fast task. Very rarely however, the performance of queries comes to a grinding halt, and queries back up and will just stay there. It isn't a slow trickle of progression, it doesn't seem like a single query moves (and occassionally even killing the processes sending them, doesn't free the resources, I have to issue a KILL QUERY manually).

What is particularly odd, is that this issue then disappears later and isn't reproducable.

Here is the table:

CREATE TABLE `runConfigs` (
`runConfigID` int(11) NOT NULL AUTO_INCREMENT,
`runConfigUUID` char(48) NOT NULL,
`execConfigID` int(11) NOT NULL,
`problemInstance` varchar(2048) NOT NULL,
`instanceSpecificInformation` varchar(2048) NOT NULL DEFAULT '0',
`seed` bigint(20) NOT NULL,
`cutoffTime` double NOT NULL,
`paramConfiguration` varchar(2048) NOT NULL,
`cutoffLessThanMax` tinyint(1) NOT NULL,
`status` enum('NEW','ASSIGNED','COMPLETE','PAUSED') NOT NULL DEFAULT 'NEW',
`priority` enum('LOW','NORMAL','HIGH','UBER') NOT NULL DEFAULT 'NORMAL',
`workerUUID` char(48) NOT NULL DEFAULT '0',
`killJob` tinyint(1) NOT NULL DEFAULT '0',
`retryAttempts` int(11) NOT NULL DEFAULT '0',
`runPartition` int(11) NOT NULL,
`noop` tinyint(1) NOT NULL DEFAULT '0',
`runResult` enum('TIMEOUT','SAT','UNSAT','CRASHED','ABORT','KILLED') NOT NULL DEFAULT 'ABORT',
`runLength` double NOT NULL DEFAULT '0',
`quality` double NOT NULL DEFAULT '0',
`resultSeed` bigint(20) NOT NULL DEFAULT '1',
`runtime` double NOT NULL DEFAULT '0',
`walltime` double NOT NULL DEFAULT '0',
`additionalRunData` varchar(2048) NOT NULL DEFAULT '',
`worstCaseEndtime` datetime NOT NULL DEFAULT '1900-01-01 00:00:00',
`lastModified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`runConfigID`),
UNIQUE KEY `runConfigUUID` (`runConfigUUID`),
KEY `status2` (`status`,`priority`),
KEY `status` (`status`,`workerUUID`,`priority`,`retryAttempts`,`runConfigID`),
KEY `statusCutoff` (`status`,`cutoffTime`),
KEY `statusEndtime` (`status`,`worstCaseEndtime`)
) ENGINE=InnoDB AUTO_INCREMENT=12557823 DEFAULT CHARSET=latin1

This is a query that grabs 5 jobs from it and is executing:

UPDATE runConfigs A JOIN (
SELECT runConfigID, priority FROM (
(SELECT runConfigID,0 AS priority FROM runConfigs WHERE status="NEW" AND priority="LOW" ORDER BY runConfigID LIMIT 5) UNION ALL
(SELECT runConfigID,1 AS priority FROM runConfigs WHERE status="NEW" AND priority="NORMAL" ORDER BY runConfigID LIMIT 5) UNION ALL
(SELECT runConfigID,2 AS priority FROM runConfigs WHERE status="NEW" AND priority="HIGH" ORDER BY runConfigID LIMIT 5) UNION ALL
(SELECT runConfigID,3 AS priority FROM runConfigs WHERE status="NEW" AND priority="UBER" ORDER BY runConfigID LIMIT 5) ) innerTable
ORDER BY priority
DESC LIMIT 5 ) B
ON B.runConfigID=A.runConfigID SET status="ASSIGNED", workerUUID="b4572d-8448-4729-bf7d-9d1da100dfa1", retryAttempts = retryAttempts+1;

Here is an explain:
mysql> EXPLAIN EXTENDED SELECT runConfigID, priority FROM (
-> (SELECT runConfigID,0 AS priority FROM runConfigs WHERE status="NEW" AND priority="LOW" ORDER BY runConfigID LIMIT 5) UNION ALL
-> (SELECT runConfigID,1 AS priority FROM runConfigs WHERE status="NEW" AND priority="NORMAL" ORDER BY runConfigID LIMIT 5) UNION ALL
-> (SELECT runConfigID,2 AS priority FROM runConfigs WHERE status="NEW" AND priority="HIGH" ORDER BY runConfigID LIMIT 5) UNION ALL
-> (SELECT runConfigID,3 AS priority FROM runConfigs WHERE status="NEW" AND priority="UBER" ORDER BY runConfigID LIMIT 5) ) innerTable
-> ORDER BY priority;


+----+--------------+---------------------------+------+-------------------------------------------+---------+---------+------+--------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+---------------------------+------+-------------------------------------------+---------+---------+------+--------+----------+--------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 5 | 100.00 | Using filesort |
| 2 | DERIVED | runConfigs | ref | status2,status,statusCutoff,statusEndtime | status2 | 2 | | 1 | 100.00 | Using where; Using index |
| 3 | UNION | runConfigs | ref | status2,status,statusCutoff,statusEndtime | status2 | 2 | | 377126 | 100.00 | Using where; Using index |
| 4 | UNION | runConfigs | ref | status2,status,statusCutoff,statusEndtime | status2 | 2 | | 1 | 100.00 | Using where; Using index |
| 5 | UNION | runConfigs | ref | status2,status,statusCutoff,statusEndtime | status2 | 2 | | 1 | 100.00 | Using where; Using index |
| NULL | UNION RESULT | <union2,3,4,5> | ALL | NULL | NULL | NULL | NULL | NULL | NULL | |
+----+--------------+---------------------------+------+-------------------------------------------+---------+---------+------+--------+----------+--------------------------+
6 rows in set, 1 warning (0.04 sec)

When the table is in this state, if I kill every other process on the MySQL server except my one console, and then in a bash script launch 100 processes that execute the above query (and just the above query), not a single query completes in 2 minutes 30 seconds. If I put a GET_LOCK and RELEASE_LOCK around the query, they can complete in about 1 minute. While debugging this, I just moved rows back from ASSIGNED to NEW, and re-ran, and the first time I did this, the performance still didn't change, after a few more attempts however the table eventually fixed itself. Rerunning the 100 concurrent calls when fixed causes them all to finish in about 5 seconds. Incidentally with the GET_LOCK and RELEASE_LOCK approach, each of the UPDATE queries takse about 1.09 seconds, but once the problem disappears it goes back down to 0.00 seconds.


I am 100% sure that there is no other query hitting the database at the same time. A show process list that occurs when this is happening is just pages and pages of this:

: *************************** 130. row ***************************
ID: 65659
USER: mysql_db_tae
COMMAND: Query
TIME: 2067
STATE: Sending data
INFO: UPDATE runConfigs A JOIN (
SELECT runConfigID, priority FROM (
(SELECT runConfigID,0 AS priority runConfigs WHERE status="NEW" AND priority="LOW" ORDER BY runConfigID LIMIT 5)
UNION ALL
(SELECT runConfigID,1 AS priority FROM runConfigs WHERE status="NEW" AND priority="NORMAL" ORDER BY runConfigID LIMIT 5)
UNION ALL
(SELECT runConfigID,2 AS priority FROM runConfigs WHERE status="NEW" AND priority="HIGH" ORDER BY runConfigID LIMIT 5)
UNION ALL
(SELECT runConfigID,3 AS priority FROM runConfigs WHERE status="NEW" AND priority="UBER" ORDER BY runConfigID LIMIT 5)

) innerTable ORDER BY priority DESC LIMIT 5
) B ON B.runConfigID=A.runConfigID SET status="ASSIGNED", workerUUID="40ecb2c0-ac95-4df2-be8b-d16005c6b58f", retryAttempts = retryAttempts+1


Of the 199 queries executing, 197 are identical to the above, all in the STATE: Sending data, all essentially the same query (except different workerUUIDs). One additionaly query is in the Sleep state, and one other query is in the Query state executing "SELECT * FROM PROCESSLIST". I'm not sure where to begin debugging this, it happens sporadically and ideally I would like a way to recover from this in my application. I don't think it's a hardware issue, as this MySQL server (running 5.5.35) is brand new and this issue has also happened on another MySQL server (running 5.5.28), both running linux one Ubuntu Server, the other some OpenSUSE (otherwise different machines located in different countries).

Anyway I'm looking for ideas of what to check when this happens, subject to the fact that the problem is rare, and even updating rows in the database (moving runs from ASSIGNED to NEW, seems to cause it to disappear).

How can I customize an algorithm if I want to make changes to the Query Evaluation Plan?