Elastic — . — : ELK EFK. Elasticsearch, Logstash, Kibana ( — Beats, , : «Do we call it BELK? BLEK? ELKB?»). — Elasticsearch, Fluentd Kibana ( ). . ELK-, — : Logstash Elasticsearch.
. - , , ELK. , , - ( Logstash) - : core-.
Logstash
Logstash, , , . , , logstash. , (, Elasticsearch ), Logstash , OOM- , «».
Persistent Queue
Logstash , (, consumer' - ) — , . — ( Elasticsearch), persistent queue (). Logstash, .
, : ( ) Logstash (SIGKILL). , PV ( Logstash Kubernetes-). path.queue
path_to_data/data/queue
.
persistent queue queue.checkpoint.writes
. , , fsync
. — 1024 . 1 , .
queue.max_bytes
— . — 1 . (.. 1 ) Logstash , . PV persistent queue , , 4 , .
, queue.max_events
, , , ( 0, .. ). queue.max_bytes
, queue.max_events
, Logstash , .
: persistent queue , . — queue.page_capacity
( 64 ), — , .. (, ). , , queue.page_capacity
… - , , ( , queue.max_bytes
). queue.page_capacity
.
. Persistent queue , , production-, .. ( ) . Elastic.
Dead Letter Queue
persistent queue, , dead letter queue (). . , . «»:
DLQ , Elasticsearch, .. DLQ , output Elasticsearch.
Logstash . Elasticsearch input- DLQ, .
DLQ . Logstash’ Elasticsearch , Elasticsearch , Elasticsearch , Logstash « » . , DLQ, Logstash path_to_data/data/dead_letter_queue/_
.
DLQ, Logstash’, , , : 1.log.tmp
.lock
. tmp
— , .
DLQ . : dead_letter_queue.flush_interval
— , , tmp , — 5 (5000 ). 1.log.tmp
. 1.log.tmp
1.log
( tmp
, , ) 2.log.tmp
, 1.log.tmp
5 . , - ( ).
— dead_letter_queue.max_bytes
. , . (1.log
, 2.log
, ...) , .
, DQL , Elasticsearch. , main
, Elasticsearch, DLQ :
1. dead-letter-queue-main.conf
:
input {
dead_letter_queue {
path => "/usr/share/logstash/data/dead_letter_queue"
commit_offsets => true
pipeline_id => "main"
}
}
output {
elasticsearch {
hosts => [ "{{ .Values.elasticsearch.host }}:{{ .Values.elasticsearch.port }}" ]
index => "logstash-dlq-%{+YYYY.MM.dd}"
}
}
input
, DLQ . pipeline_id
, main
. .
2. , DLQ . pipelines.yml
main- DLQ dead-letter-queue-main.conf
:
- pipeline.id: main
path.config: "/usr/share/logstash/pipeline/pipeline-main.conf"
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1024mb
dead_letter_queue.flush_interval: 5000
- pipeline.id: main-dlq
path.config: "/usr/share/logstash/pipeline/dead-letter-queue-main.conf"
DLQ main
. logstash.yaml
, pipelines.yml
, DLQ Logstash’. persistent queue - , .
DLQ max_bytes
flush_interval
. , *.log.tmp
- DLQ (, , Elasticsearch), ( tmp) *.log.tmp
. dead_letter_queue.flush_interval
.
DLQ , , - Elasticsearch . , Elasticsearch . DLQ , , ( DLQ ).
Elasticsearch
Elasticsearch — . , .
, Elastic- X-Pack, , . 6.3 (basic- ).
, X-Pack, . «BASIC — FREE AND OPEN» , .
NB. — - , . , self-hosted «Contact us».
, : , .
, — . xpack.monitoring.collection.enabled: true
elasticsearch.yml
, , . , http://_/app/monitoring
. :
Elasticsearch — , Elasticsearch’ .monitoring-es-7-%{+YYYY.MM.dd}
. .
Logstash, xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: "_:9200"
( legacy- 7.9.0 ).
, Kibana. Security. X-Pack , , - basic-.
, , Elasticsearch’ .monitoring-es-7-%{+YYYY.MM.dd}
. xpack.monitoring.history.duration
, basic- .
, X-Pack’, — , , . , .. Elasticsearch , Elasticsearch. - ( Elasticsearch , ).
, — production- . Elasticsearch, , unsigned-, ... . watermark.
Watermark
watermark. Elasticsearch, , Elasticsearch . :
low — - Elasticsearch .
cluster.routing.allocation.disk.watermark.low
, 85%. — , — ;
high — - Elasticsearch .
cluster.routing.allocation.disk.watermark.high
, 90%;
flood_stage — - Elasticsearch
read_only_allow_delete
.cluster.routing.allocation.disk.watermark.flood_stage
, 95%.
(flood_stage) , .. read_only_allow_delete , — .
read_only_allow_delete
. , number_of_shards
1, (, - Java-). , ( high), , , .
, number_of_shards
, . ( number_of_replicas
. , 0, Elasticsearch - unassigned_shards
.)
, Elasticsearch API curl’ ( . ) UI — Cerebro.
, Elasticsearch, watermark. , watermark , . , : , 85% , 150 .
Cheat sheet
cheat sheet Elasticsearch API — . , Elasticsearch NODE_IP
. :
NODE_IP=$(netstat -tulnp |grep 9200 |awk '{print $4}') && echo $NODE_IP
… : NODE_IP="ip__:9200"
1. Elasticsearch:
curl -s -X GET "$NODE_IP"
2. . green
— . , , unassigned shards:
curl -s -X GET "$NODE_IP/_cluster/health?pretty"
3. :
curl -s -X GET "$NODE_IP/_nodes/stats?pretty" | head -6
4. , :
curl -s -X GET "$NODE_IP/_cat/nodes?v=true"
node.role
« » . :
Master eligible node (
m
);
Data node (
d
);
Ingest node (
i
);
Coordinating node only (
-
).
5. :
curl -s -X GET "$NODE_IP/_cat/allocation?v"
6. :
curl -s -X GET "$NODE_IP/_cat/plugins?v=true&s=component&h=name,component,version,description"
7. , :
curl -s -X GET "$NODE_IP/_all/_settings?pretty&include_defaults=true"
8. watermark:
curl -s -X GET "$NODE_IP/_cluster/settings?pretty&include_defaults=true" | grep watermark -A5
:
curl -s -X GET "$NODE_IP/_cluster/settings?pretty&include_defaults=true" | jq .defaults.cluster.routing.allocation.disk.watermark
,
1. :
curl -s -X GET "$NODE_IP/_cat/indices"
2. , :
curl -s -X GET "$NODE_IP/_cat/indices?pretty&s=store.size"
3. :
curl -s -X GET "$NODE_IP/<_>/_settings?pretty"
4. read-only :
curl -X PUT "$NODE_IP/<_>/_settings?pretty" -H 'Content-Type: application/json' -d'
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}'
5. :
curl -X GET "$NODE_IP/_cat/shards?pretty"
6. unassigned shards:
curl -s -X GET $NODE_IP/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
7. , :
curl -s $NODE_IP/_cluster/allocation/explain | jq
8. - :
curl -X PUT "$NODE_IP/<_>/_settings" -H 'Content-Type: application/json' -d'
{
"index" : {
"number_of_replicas" : 0
}
}'
9. /:
curl -X DELETE $NODE_IP/<>
1. :
curl -s -X GET "$NODE_IP/_cat/templates?pretty"
2. :
curl -X GET "$NODE_IP/_index_template/<_>?pretty"
3. :
curl -X PUT "$NODE_IP/_index_template/<_>" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["<__>-*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
}'
4. :
curl -X DELETE "$NODE_IP/_index_template/<_>?pretty"
Logstash . — (filter
) (output
). Persistent Dead Letter Queue , , — .
Elasticsearch — . , , — . X-Pack . watermark: .
P.S.
: