Clever Geek Handbook

Visualización de datos de rendimiento en RHEL 8

Hoy analizaremos la recopilación y visualización de métricas de rendimiento en Red Hat Enterprise Linux 8 utilizando las herramientas Performance Co-Pilot (PCP), Grafana y Bpftrace, así como las nuevas capacidades de monitoreo de rendimiento introducidas en la consola web RHEL 8.4.





Conceptos básicos del trabajo

En nuestros ejemplos, usaremos la siguiente configuración:





  1. Dos servidores RHEL 8.3 (denominados servidor-1 y servidor-2).





  2. Versión de Grafana grafana-6.7.4-3.el8.x86_64.





  3. Versión PCP pcp-5.1.1-3.el8.x86_64.





De forma predeterminada, Performance Co-Pilot no se instala con RHEL 8, por lo que debe instalarlo usted mismo. Instálelo en server-1 y server-2 ejecutando el siguiente comando en ellos:





yum install pcp-zeroconf -y

Una vez que Performance Co-Pilot está instalado y en ejecución, puede ver qué métricas de rendimiento integradas ofrece:





pminfo | wc -l
3101
pminfo | grep kernel | head -n 15
kernel.all.load
kernel.all.intr
kernel.all.pswitch
kernel.all.sysfork
kernel.all.running
kernel.all.blocked
kernel.all.boottime
kernel.all.hz 
kernel.all.uptime
kernel.all.idletime
kernel.all.nusers
kernel.all.nroots kernel.all.nsessions
kernel.all.lastpid
kernel.all.runnable

Puede mostrar los valores de la métrica requerida con el comando pmrep:





pmrep kernel.all.load -s 5
  k.a.load  k.a.load  k.a.load
  1 minute  5 minute  15 minut
  0.020  0.020 0.080
  0.020  0.020 0.080
  0.020  0.020 0.080
  0.020  0.020 0.080
  0.020  0.020 0.080

La opción -s 5 en el ejemplo anterior dice que solo queremos mostrar los primeros 5 valores métricos.





Para que otras aplicaciones utilicen las métricas, debe habilitar pmproxy y abrir el puerto correspondiente en el firewall:





systemctl enable pmproxy
systemctl start pmproxy
firewall-cmd --add-service=pmproxy --permanent
firewall-cmd --reload

, Performance Co-Pilot , Grafana, .





grafana server-1, :





yum install grafana grafana-pcp -y
systemctl enable grafana-server
systemctl start grafana-server
firewall-cmd –add-service=grafana –permanent
firewall-cmd –reload

http://server-1:3000 admin admin. .





, , (Configuration), Plugins. «Performance Co-Pilot App», Enable.





Configuration, Data Sources, Add Data Source, PCP Vector Select. , HTTP «http://localhost:44322» URL, Save & Test . , «Data source is working».





(Dashboard), Manage PCP Vector Host Overview.





server-1, :





  • CPU Percentage





  • Load Average





  • Memory Utilization





  • Disk Utilization





  • Per-CPU busy (user/system)





  • CPU user%,system%,interrupt%,wait%





  • Scheduler context switches per second and runnable processes.





  • Memory Used/Cached/Free/Page Fault Rate/Hard Fault Rate





  • Network throughput in/out





  • Network drops in/out





  • Network packets in/out





  • TCP connections/timeouts/close-waits/time wait/established/listen errors/retransmits





  • Disk latency/iops/throughput/utilization





, .





, pmseries Redis, .





, . server-1 Redis PCP ( ). server-1 Redis. PCP 100 200 . , Redis, pmseries, server-1, /etc/pcp/pmproxy/pmproxy.conf, [pmproxy] :





# support Redis protocol proxying
redis.enabled = true

, , [pmseries] :





# allow REST API queries of fast, scalable time series
enabled = true

, , Redis server-1, :





yum install redis -y
systemctl start redis
systemctl enable redis

:





systemctl restart pmcd pmlogger pmproxy

, server-1 , pmlogger. pmlogger server-1, server-2. 





server-2 /etc/sysconfig/pmcd PMCD_LOCAL 1 0, pmcd:





# Behaviour regarding listening on external-facing interfaces;
# unset PMCD_LOCAL to allow connections from remote hosts.
# A value of 0 permits remote connections, 1 permits local only.
PMCD_LOCAL=0

:





firewall-cmd --add-service=pmcd --permanent
firewall-cmd --reload

pcp unreserved:





setsebool -P pcp_bind_all_unreserved_ports on

, pcp:





systemctl restart pmcd pmlogger

server-2 , pmlogger .





server-1 pmlogger, server-2. /etc/pcp/pmlogger/control.d/remote  :





server-2 n n PCP_LOG_DIR/pmlogger/server-2 -r -T24h10m -c config.remote

pmlogger:





systemctl restart pmlogger

:





cd /var/log/pcp/pmlogger/server-2 
ls *.0

, , «.0»:





20200923.16.41.0

, , server-1 pcp- server-2.





Grafana (http://server-1:3000)  , , redis.





Configuration, Data Sources, Add Data Source. PCP Redis Select. , HTTP «http://localhost:44322» URL, Save & Test . , «Data source is working».





, . Dashboard, Manage PCP Redis Host Overview.





PCP Redis Host Overview , . :





  • load average





  • memory utilization





  • per-cpu busy (user/sys)





  • context switches per second





  • runnable processes





  • memory (used/cached/free)





  • network throughput in/out





  • network drops in/out





  • network packets in/out





  • tcp timeouts/listen errors/retransmits





  • disk latency/iops/throughput/utilization





, , Redis. , , , Host . , , .





, , ansible – rhel-system-roles.metrics , :





https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/administration_and_configuration_tasks_using_system_roles_in_rhel/monitoring-performance-using-rhel-system-roles_administration-and-configuration-tasks-using-system-roles-in-rhel#introduction-to-the-metrics-role_monitoring-performance-using-rhel-system-roles





Performance Co-Pilot Bpftrace

, Performance Co-Pilot bpftrace , Linux. , bpftrace Linux - eBPF ( «-, eBPF), Performance Co-Pilot.





, dev- bpftrace Performance Co-Pilot. , Performance Co-Pilot Grafana . server-1 pcp-pmda-bpftrace, :





yum install bpftrace pcp-pmda-bpftrace -y
cd /var/lib/pcp/pmdas/bpftrace
./Install

, bpftrace pmda , :





pmrep bpftrace.scripts.runqlat.data_bytes -s 5

, ( ):





b.s.r.data_bytes
            byte/s
               N/A
           586.165
           590.276
           588.141
           589.113

, bpftrace pmda Performance Co-Pilot, Grafana. dev-, . 





, Performance Co-Pilot metrics:





yum install cyrus-sasl-scram cyrus-sasl-lib -y
useradd -r metrics
passwd metrics
saslpasswd2 -a pmcd metrics
chown root:pcp /etc/pcp/passwd.db
chmod 640 /etc/pcp/passwd.db

saslpasswd2 , , metrics sasl- pmcd, pcp.





, /etc/sasl2/pmcd.conf :





mech_list: scram-sha-256
sasldb_path: /etc/pcp/passwd.db

pcp:





systemctl restart pmcd

bpftrace pmda , . : , . /var/lib/pcp/pmdas/bpftrace/bpftrace.conf , [dynamic_scripts] :





enabled=true
allowed_users=root,metrics

bpftrace pmda:





cd /var/lib/pcp/pmdas/bpftrace
./Remove
./Install

Grafana (http://server-1:3000) admin ( ).





Grafana Configuration, Data Sources, Add Data Source, PCP bpftrace, Select.





URL «http://localhost:44322», Auth Basic auth, Basic Auth Details metrics , saslpasswd2. Save & Test, , , «Data source is working».





Dashboards Manage. PCP bpftrace System Analysis, . . CPU usage Edit. , bpftrace-, bpf- @cpu . Grafana bpf- .





, bpftrace , pid , fork. pidpersec.bt , , :





tracepoint:sched:sched_process_fork
{
     @ = count();
}

, sched_process_fork bpf, Grafana.





, . PCP bpftrace System Analysis Add Panel, , Add Query.





Query , PCP bpftrace, Query A bpftrace:





tracepoint:sched:sched_process_fork
{
     @ = count();
}

General – «Processes Per Second». PCP bpftrace System Analysis , . .





, grafana pcp, bpftrace. dev-, . bpftrace-, bpftrace.





bpftrace pmda. bpftrace, /var/lib/pcp/pmdas/bpftrace/autostart, Performance Co-Pilot , bpftrace pmda. 





/var/lib/pcp/pmdas/bpftrace/autostart/pidpersec.bt :





tracepoint:sched:sched_process_fork
{
     @ = count();
}

, :





cd /var/lib/pcp/pmdas/bpftrace/
./Remove
./Install
pminfo | grep pidpersec

:





bpftrace.scripts.pidpersec.data.root
bpftrace.scripts.pidpersec.data_bytes
bpftrace.scripts.pidpersec.code
bpftrace.scripts.pidpersec.probes
bpftrace.scripts.pidpersec.error
bpftrace.scripts.pidpersec.exit_code
bpftrace.scripts.pidpersec.pid
bpftrace.scripts.pidpersec.status

bpf-, pmda root. , :





pmrep bpftrace.scripts.pidpersec.data.root -s 5

:





  b.s.p.d.root
            /s
           N/A
         3.984
        31.049
         0.000
         0.000

grafana, grafana- - , Performance Co-Pilot.





Grafana, Dashboards, Manage, PCP Vector Host Overview, Add Panel Add Query.





Query PCP Vector A- :





bpftrace.scripts.pidpersec.data.root

General «Processes per Second».





PCP Vector Host Overview bpf-, bpftrace - . .





pcp-pmda-bpftrace Linux- Grafana.





, bpftrace , , . .





- RHEL 8.4

RHEL 8.4 , , , .





-

, , , , , (saturation) , . ?





, . - RHEL 8.4 , , , , .





, System -> Overview - Usage card. cockpit-pcp (. ), Usage card. 





. Usage card, , , :





CPU , . , , - . IO- -, . - .





, , pcp ( ), :





, , . , ; .





– , , . , –  (saturation). . , , , .





flame graph

, , , - RHEL. RHEL , flame graphs.





, , , . : , , , .





, , perf js-d3-flame-graph.





flame graph perf. , , , , 30 , , Ctrl+C .





[root@bigdemo1 ~]# perf script record flamegraph -F 99 -g           
^C

[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 1.194 MB perf.data (3448 samples) ]

flame graph, :





[root@bigdemo1 ~]# perf script report flamegraph

:





dumping data to flamegraph.html

flamegraph.html, :





30 , perf. -, . , . , user space – .





root, . -, : SetroubleshootP, mysqld, apache2, 240998, rpm, ksoftirqd, kworker, grafana-server redis-server.





( ), , «» mysqld.







More articles:


All Articles