ELK-6.0.0升级到7.15.0
本文基于二进制6.0单节点安装升级到7.15.0 rpm包 基于：Centos6.x 注意：centos6.x只支持 <= 7.10.0
升级策略
- 7.0–7.14 -> 7.15.0
- 6.8 -> 7.15.0
- 6.0–6.7 -> 6.8 -> 7.15.0
- 5.6 -> 6.8 -> 7.15.0
- 5.0–5.5 - > 5.6 -> 6.8 -> 7.15.0
备份
添加备份路径
重启生效
# vim config/elasticsearch.yml
path.repo: /data/backup/elastic
注册并备份索引
Dev Tools -> console
# console操作
# 注册仓库名称为：backup，对应路径为：/data/backup/elastic
PUT _snapshot/backup
{
"type": "fs",
"settings": {
"location": "/data/backup/elastic"
}
}
# 查看仓库
GET _snapshot/backup
# 查看备份仓库列表
GET _cat/repositories?v
# 备份
PUT _snapshot/backup/snapshot_20211012?wait_for_completion=false
# 查看状态
GET _snapshot/backup/snapshot_20211012/_status?pretty
# 备份中状态
# "state" : "STARTED", # 运行状态
# "shards_stats" : {
# "initializing" : 3044,
# "started" : 4,
# "finalizing" : 0,
# "done" : 639, # 完成数量
# "failed" : 0,
# "total" : 3687
# },
# 完成状态
# "state": "SUCCESS", # 完成状态
# "shards_stats": {
# "initializing": 0,
# "started": 0,
# "finalizing": 0,
# "done": 3687,
# "failed": 0,
# "total": 3687
# },
备份代码
tar czf /data/backup/elasticsearch_6.0.0.tar.gz /usr/local/elasticsearch
准备
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/rolling-upgrades.html
具体版本升级准备参考最新的相关文档
- Disable shard allocation.
- Stop non-essential indexing and perform a synced flush. (Optional)
- Stop any machine learning jobs that are running.
- Shut down a single node.
- Upgrade the node you shut down.
- Upgrade any plugins.
- Start the upgraded node.
升级到6.8.x
安装
cd /usr/local/src
# kibana需要升级为elasticsearch同版本(如果纯粹通过api接口升级，可以不安装kibana)
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.8.19-linux-x86_64.tar.gz
tar -xzf kibana-6.8.19-linux-x86_64.tar.gz
# elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.19.tar.gz
# 因为是升级的临时包，所以放在此位置即可
# 注意授权为运行用户
tar xzf elasticsearch-6.8.19.tar.gz && chown -R zaza:zaza elasticsearch-6.8.19 && cd elasticsearch-6.8.19
配置
config/elasticsearch.yml
# 和6.0的配置一样的
cluster.name: plat
node.name: node-1
path.data: /data/elastic/data
path.logs: /data/elastic/logs
path.repo: /data/backup/elastic
network.host: 0.0.0.0
action.destructive_requires_name: true # 禁止匹配删除
bootstrap.system_call_filter: false
# 添加用户
xpack.security.enabled: true
config/kibana.yml
i18n.locale: "zh-CN"
server.host: "0.0.0.0"
elasticsearch.username: "elastic"
elasticsearch.password: "passwd"
运行elasticsearch
su - zaza -c "/usr/local/src/elasticsearch-6.8.19/bin/elasticsearch"
创建用户
curl -XPUT -u elastic:changeme -H 'Content-Type: application/json' 'http://localhost:9200/_xpack/security/user/elastic/_password' -d '{ "password" : "your_passwd" }'
添加license
# 查看集群状态
curl -XGET -u 'elastic:passwd' 'http://localhost:9200/_cat/health?v'
# 查看授权（"license" : null这个状态需要添加授权）
curl -XGET -u 'elastic:passwd' 'http://localhost:9200/_xpack?pretty'
# 添加基础授权
curl -XPOST -H 'Content-Type: application/json' -u 'elastic:passwd' 'http://localhost:9200/_xpack/license/start_basic?acknowledge=true'
# 确认授权
curl -X GET -u 'elastic:passwd' 'http://localhost:9200/_xpack?pretty'
运行kibana
依赖license
# 备份 .kibana_2
curl -XPUT -H 'Content-Type: application/json' --user 'elastic:passwd' "localhost:9200/_snapshot/backup/snapshot_kibana_2_20211012?wait_for_completion=false" -d '
{
"indices": ".kibana_2"
}
'
# 查看
curl --user 'elastic:passwd' -X GET "localhost:9200/_snapshot/backup/snapshot_kibana_2_20211012?pretty"
# 删除
curl -XDELETE --user 'elastic:passwd' http://localhost:9200/.kibana_2
# 启动
cd /usr/local/src/kibana-6.8.19-linux-x86_64 && NODE_OPTIONS="--max-old-space-size=512" bin/kibana
升级到7.15.0
下载
cd /usr/local/src/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.0-x86_64.rpm
# centos6.x只支持kibana-7.10.0-x86_64.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.15.0-x86_64.rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.15.0-x86_64.rpm
elasticsearch
安装
rpm -ivh elasticsearch-7.15.0-x86_64.rpm
配置
/etc/elasticsearch/elasticsearch.yml
cluster.name: plat
node.name: node-1
cluster.initial_master_nodes: ["node-1"]
path.data: /data/elastic/data
path.logs: /data/elastic/logs
path.repo: /data/backup/elastic
network.host: 0.0.0.0
action.destructive_requires_name: true # 禁止匹配删除
bootstrap.system_call_filter: false
# 密码验证
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
# 分片数量限制
cluster.max_shards_per_node: 10000
# 内存调整(根据需求调整)
vim /etc/elasticsearch/jvm.options
# 日志调整(根据需求调整)
vim /etc/elasticsearch/log4j2.properties
启动
# 修改权限
chown -R elasticsearch:elasticsearch /data/elastic/ /data/backup/elastic/
# 开机自启动
chkconfig --add elasticsearch
# 启动
service elasticsearch start
# 日志
tailf /data/elastic/logs/plat.log
kibana
centos6.x 只支持 <= 7.10.0
备份清理数据
因为kibana和elasticsearch版本不匹配，导致403权限报错，如果是同版本，不需要清理数据
升级elasticsearch会同事升级kibana的数据到对应的版本导致的？
# 备份
curl -XPUT -H 'Content-Type: application/json' --user 'elastic:passwd' "localhost:9200/_snapshot/backup/snapshot_kibana_2_20211013?wait_for_completion=false" -d '
{
"indices": ".kibana*"
}
# 查看(这里查看需要清理的索引)
curl -XGET --user 'elastic:passwd' "localhost:9200/_snapshot/backup/snapshot_kibana_2_20211013?pretty"
# 清理(这里可能会丢失索引，如果被删除那么需要重建索引)
curl -XDELETE --user 'elastic:passwd' http://localhost:9200/.kibana_7.15.0_001
curl -XDELETE --user 'elastic:passwd' http://localhost:9200/.kibana_task_manager_2
......
安装
rpm -ivh kibana-7.10.0-x86_64.rpm
配置
/etc/kibana/kibana.yml
i18n.locale: "zh-CN"
server.host: "0.0.0.0"
elasticsearch.username: "elastic"
elasticsearch.password: "passwd"
启动
chkconfig kibana on
/etc/init.d/kibana start
# 查看日志
tailf /var/log/kibana/kibana.stdout
开启相关功能
- Reenable shard allocation.
- Wait for the node to recover.
- Repeat
- Restart machine learning jobs.
logstash
安装
# 安装
rpm -ivh logstash-7.15.0-x86_64.rpm
# logstash配置用户密码(elasticsearch生成的用户密码)
cp /etc/logstash/logstash.yml{,.bak}
logstash配置
/etc/logstash/logstash.yml
path.data: /data/logstash/data
path.logs: /data/logstash/log
# 监控
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: password
xpack.monitoring.elasticsearch.hosts: ["http://127.0.0.1:9200"]
授权
mkdir -pv /data/logstash/data /data/logstash/log
chown -R logstash: /data/logstash
pipelines配置
# pipelines配置
cat > /etc/logstash/pipelines.yml << 'EOF'
- pipeline.id: nginx
pipeline.workers: 2
pipeline.batch.size: 3000
pipeline.batch.delay: 200
path.config: /etc/logstash/conf.d/nginx.conf
- pipeline.id: php
pipeline.workers: 2
pipeline.batch.size: 3000
pipeline.batch.delay: 200
path.config: /etc/logstash/conf.d/php.conf
EOF
# client配置
cat > /etc/logstash/conf.d/nginx.conf << 'EOF'
# 客户端配置
input {
beats{
port => 5026
}
}
filter {
......
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
manage_template => false
index => "%{project}-%{[fields][document_type]}-%{+YYYY-MM}"
user => "elastic"
password => "123456"
}
#stdout{codec => rubydebug}
}
EOF
# game配置
cat > /etc/logstash/conf.d/php.conf << 'EOF'
# 配置说:此logstash设计目标适用于所有项目服务端数据录入(应用场景为多项目公用服务器)
# logstash收集各种类型日志-建议使用pipeline
input {
beats {
port => 5027
}
}
filter {
# 处理消息日志
......
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
manage_template => false
index => "%{project}-server-%{+YYYY.MM.dd}"
user => "elastic"
password => "123456"
}
#stdout {
# codec => rubydebug
#}
}
EOF
测试
# 测试
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf -t
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/php.conf -t
启动
# centos6.x管理脚本
/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv
chkconfig logstash on
/etc/init.d/logstash start
# 查看日志
tailf /var/log/logstash-stdout.log
filebeat
暂时使用6.0.0版本，升级到7.15.0的时候注意source字段改成[log][file][path]
参考
- 原文作者：zaza
- 原文链接：https://zazayaya.github.io/2021/10/12/elk-6.0.0-upgrade-to-7.15.0.html
- 说明：转载本站文章请标明出处，部分资源来源于网络，如有侵权请及时与我联系!