Projecte

General

Perfil

Accions

Tasca #5963

tancat
FJ FJ

Treure el servidor MariaDB de Mompou

Tasca #5963: Treure el servidor MariaDB de Mompou

Afegit per Ferran Jorba fa més de 5 anys. Actualitzat fa aproximadament 5 anys.

Estat:
Tancada
Prioritat:
Alta
Assignat a:
Categoria:
Tecnologia
Inici:
02-11-2020
Data de venciment:
Paraula clau:

Descripció

Amb el continu creixement de l'ús del DDD, augmenten també les incidències de consulta per sobrecàrrega del servidor. Hem estat fent tota mena de canvis de paràmetres per millorar-ne el seu rendiment, però crec que la cosa ja no dóna més de si i caldrà pensar en una estratègia a mig termini. El que pretenc és fer el canvi menys costós per mantenir la instal·lació actual en el mateix servidor mentre estiguem en Invenio 1.1.6.

La meva proposta és treure el servidor MariaDB (MySQL) de Mompou i traslladar-lo a una màquina externa. En principi això hauria de suposar un canvi mínim en la configuració d'Invenio, on li diem on tenim el servidor. En comptes de dir-li la màquina local, en seria una de remota.

La motivació d'aquest canvi és doble. D'una banda, les estadístiques de creixement de l'ús, en brut. Mai no baixen, i sempre pugen:

D'altra banda, observant-ne el comportament, me n'adono que els errors de Gateway timeout i similars es donen, sobretot, en els processos d'indexació d'Invenio (a la base de dades MariaDB), que els tenim programats cada 15 minuts.

Els valors actuals de la base de dades (/etc/mysql/mariadb.conf.d/invenio.cnf), que haurien de servir per dimensionar el futur servidor, són aquests:

# UAB MySQL config for 3 Invenios (DDD, Traces and IFMuC)
# Fist, copy my-huge.cnf to /etc/mysql/my.cnf

# Based on
# * http://invenio-software.org/wiki/Tools/MySQL/Tuning
# * tuning-primer.sh
# * mysqltuner
# * mysqlreport

[mysqld]
key_buffer_size         = 10G
max_allowed_packet      = 1G
query_cache_size        = 256M
max_heap_table_size     = 512M
tmp_table_size          = 64M
max_connections         = 50
table_definition_cache  = 1600
table_cache             = 6000
character_set_server    = utf8
collation_server        = utf8_unicode_ci
skip-character-set-client-handshake

[mysqldump]
max_allowed_packet      = 1G

[myisamchk]
key_buffer_size         = 5G

[client]
default-character-set   = utf8

La RAM usada pel servidor de base de dades, segons htop, és de 14.3 GB, que és la meitat dels 32 GB que té Mompou.

L'espai de disc per a cadascuna de les tres bases de dades és:

  • ddd 42 GB
  • traces 13 GB
  • ifmuc 400 MB

En total són 55 GB. Per tant, amb un disc de 100 GB en tenim de sobres, fins i tot si hem de reorganitzar.

Els bolcats de la base de dades, com que els fa un script d'Invenio, es podrien continuar fent a Mompou (/var/backups/mysql/, que amb una rotació de 3 generacions, ara ocupa 70 GB), i així els canvis que caldria fer en producció serien mínims.

Com que del que es tracta és d'accel·lerar l'entrada/sortida, el servidor on hauria d'anar a parar MariaDB hauria de ser amb disc flash.

La versió de MariaDB millor que sigui la mateixa que la del client. Per tant, hauria de ser Debian stable.


Tasques relacionades 2 (0 obertes2 tancades)

relacionat amb DDD - Incidència #5771: Gateway errors al DDD: augmentar número de processos?TancadaFerran Jorba20-05-2020Accions
relacionat amb DDD - Tasca #6117: Minimitzar els accessos a disc de l'Apache via caché de miniaturesTancadaFerran Jorba03-12-2020Accions

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #1

Sortida del mysqltuner d'ara mateix:

$ mysqltuner --skippassword                        
 >>  MySQLTuner 1.7.13 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/                         
 >>  Run with '--help' for additional options and output filtering 

[--] Skipped version check for MySQLTuner script            
[!!] failed to execute: SHOW SLAVE STATUS\G                 
[!!] FAIL Execute SQL / return code: 256                                                                                             
[!!] failed to execute: SHOW SLAVE HOSTS                                                                                             
[!!] FAIL Execute SQL / return code: 256                                                                                             
[OK] Currently running supported MySQL version 10.3.25-MariaDB-0+deb10u1 
[OK] Operating on 64-bit architecture                          

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysql/error.log(619B)                                                                                        
[OK] Log file /var/log/mysql/error.log exists                                                                                        
[OK] Log file /var/log/mysql/error.log is readable.              
[OK] Log file /var/log/mysql/error.log is not empty                                                                                  
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[!!] /var/log/mysql/error.log contains 4 warning(s).
[!!] /var/log/mysql/error.log contains 3 error(s).
[--] 0 start(s) detected in /var/log/mysql/error.log
[--] 0 shutdown(s) detected in /var/log/mysql/error.log

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE 
[--] Data in MyISAM tables: 38.9G (Tables: 533)
[!!] InnoDB is enabled but isn't being used
[!!] Total fragmented tables: 1                                   

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[--] Skipped due to --skippassword option

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 11d 7h 10m 47s (1B q [1K qps], 227K conn, TX: 1386G, RX: 107G)
[--] Reads / Writes: 96% / 4%
[--] Binary logging is disabled
[--] Physical Memory     : 31.3G
[--] Max MySQL memory    : 10.7G
[--] Other process memory: 5.3G
[--] Total buffers: 10.6G global + 2.9M per thread (50 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 10.7G (34.27% of installed RAM)
[OK] Maximum possible memory usage: 10.7G (34.26% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (43K/1B)
[!!] Highest connection usage: 100%  (51/50)
[!!] Aborted connections: 17.17%  (39030/227366)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 38.7% (539M cached / 1B selects)
[!!] Query cache prunes per day: 16663627
[OK] Sorts requiring temporary tables: 0% (5 temp sorts / 85M sorts)
[OK] No joins without indexes
[!!] Temporary tables created on disk: 27% (75M on disk / 274M total)
[OK] Thread cache hit rate: 99% (734 created / 227K connections)
[OK] Table cache hit rate: 93% (2K open / 2K opened)
[OK] Open file limit used: 22% (3K/16K)
[OK] Table locks acquired immediately: 99% (474M immediate / 474M locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 40 thread(s).
[--] Using default value is good enough for your version (10.3.25-MariaDB-0+deb10u1)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 63.2% (6B used / 10B cache)
[OK] Key buffer size / total MyISAM indexes: 10.0G/9.9G
[OK] Read Key buffer hit rate: 100.0% (52B cached / 4M reads)
[!!] Write Key buffer hit rate: 64.6% (43M cached / 27M writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is disabled.
[!!] InnoDB Storage engine is disabled. InnoDB is the default storage engine

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[!!] Aria pagecache hit rate: 76.0% (315M cached / 75M reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/log/mysql/error.log file
    Control error line(s) into /var/log/mysql/error.log file
    Add skip-innodb to MySQL configuration to disable InnoDB
    Run OPTIMIZE TABLE to defragment tables for better performance
      OPTIMIZE TABLE `ddduab`.`rnkWORD01R`; -- can free 17.620906829834 MB
    Total freed space after theses OPTIMIZE TABLE : 17.620906829834 Mb
    Reduce or eliminate persistent connections to reduce connection usage
    Reduce or eliminate unclosed connections and network issues
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Increasing the query_cache size over 128M may reduce performance
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Performance schema should be activated for better diagnostics
    Consider installing Sys schema from https://github.com/mysql/mysql-sys
Variables to adjust:
    max_connections (> 50)
    wait_timeout (< 28800)
    interactive_timeout (< 28800)
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_size (> 256M) [see warning above]
    tmp_table_size (> 64M)
    max_heap_table_size (> 512M)
    performance_schema = ON enable PFS

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #2

Sortida del tuning-primer.sh:


 -- MYSQL PERFORMANCE TUNING PRIMER --
      - By: Matthew Montgomery -

MySQL Version 10.3.25-MariaDB-0+deb10u1 x86_64

Uptime = 11 days 7 hrs 13 min 11 sec
Avg. qps = 1103                                                   
Total Questions = 1077690521                                      
Threads Connected = 32                                                                                                               

Server has been running for over 48hrs.                                                                                              
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.3/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service

SLOW QUERIES
The slow query log is NOT enabled.

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.3/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 50
Current threads_cached = 3
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 50
Current threads_connected = 32
Historic max_used_connections = 51
The number of used connections is 102% of the configured maximum.
You should raise max_connections

INNODB STATUS
Current InnoDB index space = 0 bytes
Current InnoDB data space = 0 bytes
Current InnoDB buffer pool free = 96 %
Current innodb_buffer_pool_size = 128 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE
Max Memory Ever Allocated : 10.53 G
Configured Max Per-thread Buffers : 145 M
Configured Max Global Buffers : 10.39 G
Configured Max Memory Limit : 10.53 G
Physical Memory : 31.28 G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = 9.93 G
Current key_buffer_size = 10.00 G
Key cache miss rate is 1 : 11783
Key buffer free ratio = 36 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere

QUERY CACHE
Query cache is enabled
Current query_cache_size = 256 M
Current query_cache_used = 160 M
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 62.56 %
Current query_cache_min_res_unit = 4 K
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 2 M
Current read_rnd_buffer_size = 256 K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 260.00 K
You have had 1698 queries where a join could not use an index properly
You should enable "log-queries-not-using-indexes" 
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.

Note! This script will still suggest raising the join_buffer_size when
ANY joins not using indexes are found.

OPEN FILES LIMIT
Current open_files_limit = 16384 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 6000 tables
Current table_definition_cache = 1600 tables
You have a total of 533 tables
You have 2201 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current max_heap_table_size = 512 M
Current tmp_table_size = 64 M
Of 274360208 temp tables, 21% were created on disk
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 128 K
Current table scan ratio = 12 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 2773
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=ALWAYS'.

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #3

  • S'ha afegit relacionat amb Incidència #5771: Gateway errors al DDD: augmentar número de processos?

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #4

  • S'ha actualitzat Descripció (diferències)
  • Prioritat ha canviat de Normal a Alta

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #5

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #6

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #7

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #8

FJ Actualitzat per Ferran Jorba fa més de 5 anys Accions #9

  • S'ha afegit relacionat amb Tasca #6117: Minimitzar els accessos a disc de l'Apache via caché de miniatures

FJ Actualitzat per Ferran Jorba fa aproximadament 5 anys Accions #10

  • Estat ha canviat de Creada a Tancada

Com ja he descrit a la tasca mare #5771, de moment sembla que no caldrà prendre cap mesura, ja que la solució #6117 sembla ser suficient.

Tanco aquesta tasca fins que no es demostri que cal tornar-la a obrir.

Accions

També disponible a: PDF Atom