https://github.com/goharbor/harbor/releases
下载安装包
提前部署好docker和docker-compose
解压后执行install.sh

一些工作及日常随笔
OPENVPNserver:
内网地址:172.17.244.130
打开阿里云安全组的1194/udp端口
CAserver:
内网地址:172.17.244.131
操作系统均为ubuntu18.04
apt install openvpn
wget -P ~/ https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.4/EasyRSA-3.0.4.tgz
cd ~
tar xvf EasyRSA-3.0.4.tgz
登录CAserver
cd ~/EasyRSA-3.0.4/
cp vars.example vars
编辑vars文件
#set_var EASYRSA_REQ_COUNTRY "US"
#set_var EASYRSA_REQ_PROVINCE "California"
#set_var EASYRSA_REQ_CITY "San Francisco"
#set_var EASYRSA_REQ_ORG "Copyleft Certificate Co"
#set_var EASYRSA_REQ_EMAIL "me@example.net"
#set_var EASYRSA_REQ_OU "My Organizational Unit"
修改成公司对应的信息
使用easyrsa脚本,构建管理各种任务
./easyrsa init-pki
输出如下:
init-pki complete; you may now create a CA or requests.
Your newly created PKI dir is: /home/sammy/EasyRSA-3.0.4/pki
使用build-ca选项调用easyrsa脚本。 这将构建CA并创建两个重要文件 – ca.crt和ca.key – 构成SSL证书。其中ca.crt是CA的公共证书文件,ca.key是CA机器用于为服务器和客户端签名密钥和证书的私钥。
命令如下:
./easyrsa build-ca nopass
输出如下:
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:
选择默认即可
CA服务器部署完成
登录VPNserver
cd ~/EasyRSA-3.0.4/
./easyrsa init-pki
为了确保不出现权限及路径等等,都以默认选项为主,选择nopass模式
./easyrsa gen-req server nopass
为服务器和一个名为server.req的证书请求文件创建一个私钥。 将服务器密钥复制到/etc/openvpn/目录中:
cp ~/EasyRSA-3.0.4/pki/private/server.key /etc/openvpn/
将server.req传输到CAserver
scp ~/EasyRSA-3.0.4/pki/reqs/server.req CAserver:/tmp
登入CAserver
cd ~/EasyRSA-3.0.4/
在CAserver导入server.req文件
./easyrsa import-req /tmp/server.req server
通过使用sign-req选项运行easyrsa脚本,然后签署请求类型和公用名称,对于VPNserver来说,需要用server选项,其中,请求类型可以是client或者server
./easyrsa sign-req server server
输出如下:
You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.
Request subject, to be signed as a server certificate for 3650 days:
subject=
commonName = server
Type the word 'yes' to continue, or any other input to abort.
Confirm request details: yes
输入yes
将生成的证书传递回VPNserver
scp pki/issued/server.crt VPNserver:/tmp
scp pki/ca.crt VPNserver:/tmp
返回VPNserver
cp /tmp/{server.crt,ca.crt} /etc/openvpn/
进入EasyRSA目录:
cd ~/EasyRSA-3.0.4/
创建Diffie-Hellman密钥
./easyrsa gen-dh
生成一个HMAC签名以加强服务器的TLS完整性验证功能
openvpn --genkey --secret ta.key
将这两个新文件复制到/etc/openvpn/目录中
cp ~/EasyRSA-3.0.4/ta.key /etc/openvpn/
cp ~/EasyRSA-3.0.4/pki/dh.pem /etc/openvpn/
我们可以创建一个脚本,它将自动生成包含所有必需密钥和证书的客户端配置文件
先手工创建一个客户端密钥和证书对,假设证书/密钥对名称为client1
登入VPNserver
mkdir -p ~/client-configs/keys
chmod -R 700 ~/client-configs
进入到EasyRSA目录并使用gen-req和nopass选项运行easyrsa脚本
cd ~/EasyRSA-3.0.4/
./easyrsa gen-req client1 nopass
将client1.key文件复制到之前创建的/client-configs/keys/目录中
cp pki/private/client1.key ~/client-configs/keys/
将client1.req文件传输到您的CA机器
scp pki/reqs/client1.req CAserver:/tmp
登入CAserver
导入证书请求
cd ~/EasyRSA-3.0.4/
./easyrsa import-req /tmp/client1.req client1
进行签名,选择client
./easyrsa sign-req client client1
Type the word 'yes' to continue, or any other input to abort.
Confirm request details: yes
输入yes
这将创建一个名为client1.crt的客户端证书文件。 将该文件传回服务器
scp pki/issued/client.crt VPNserver:/tmp
登入VPNserver
cp /tmp/client1.crt ~/client-configs/keys/
将ca.crt和ta.key文件复制到/client-configs/keys/目录中
cp ~/EasyRSA-3.0.4/ta.key ~/client-configs/keys/
cp /etc/openvpn/ca.crt ~/client-configs/keys/
将示例OpenVPN配置文件复制到配置目录中,然后解压缩以作为安装的基础
cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
gzip -d /etc/openvpn/server.conf.gz
编辑server.conf文件
查找tls-auth指令来查找HMAC部分。 此行应该已被取消注释,但如果不是,则删除“ ; ”以取消注释。 在该行下方,添加key-direction参数,设置为“0”
tls-auth ta.key 0 # This file is secret
key-direction 0
查找注释掉的cipher行来查找加密密码部分,删除“;”,取消注释
cipher AES-256-CBC
在此之下,添加一个auth指令来选择HMAC消息摘要算法,源文件是dh dh2048.pem
dh dh.pem
找到user和group设置,并在每个开始处删除“ ; ”以取消注释这些行
user nobody
group nogroup
增加VPNserver路由转发功能
/etc/sysctl.conf
增加如下配置
net.ipv4.ip_forward=1
使配置生效
sysctl -p
修改server.conf,推送内网地址到客户端
push "route 172.17.0.0 255.255.0.0"
增加iptables规则
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
创建iptables配置文件的文件夹
mkdir /etc/iptables
保存配置文件
iptables-save > /etc/iptables/rules.v4
修改/etc/iptables/rules.v4,仅保留如下内容
*nat
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
COMMIT
安装自动恢复路由表的软件
apt-get install iptables-persistent
查看iptables路由表
iptables -nL -t nat
通过在systemd单元文件名后指定配置文件名作为实例变量来启动OpenVPN服务器
systemctl start openvpn@server
查看服务状态
● openvpn@server.service - OpenVPN connection to server
Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
Active: active (running) since Sun 2021-01-31 16:16:38 CST; 6min ago
Docs: man:openvpn(8)
https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
https://community.openvpn.net/openvpn/wiki/HOWTO
Main PID: 805 (openvpn)
Status: "Initialization Sequence Completed"
Tasks: 1 (limit: 4915)
CGroup: /system.slice/system-openvpn.slice/openvpn@server.service
└─805 /usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/server.conf --writepid /run/openvpn/server.pid
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: UDPv4 link remote: [AF_UNSPEC]
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: GID set to nogroup
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: UID set to nobody
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: MULTI: multi_init called, r=256 v=256
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: IFCONFIG POOL: base=10.8.0.4 size=62, ipv6=0
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: ifconfig_pool_read(), in='client1,10.8.0.4', TODO: IPv6
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: succeeded -> ifconfig_pool_set()
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: IFCONFIG POOL LIST
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: client1,10.8.0.4
Jan 31 16:16:38 iZ2ze0i59st92lrx3gwcnpZ ovpn-server[805]: Initialization Sequence Completed
设置OpenVPN自启动
systemctl enable openvpn@server
登入VPNSERVER
将客户端配置文件存储在之前创建的client-configs目录中
mkdir -p ~/client-configs/files
将示例客户端配置文件复制到client-configs目录中以用作基本配置
cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf ~/client-configs/base.conf
编辑base.conf
# The hostname/IP and port of the server.
# You can have multiple remote entries
# to load balance between the servers.
### 修改以下内容 ###
remote VPNserverIP/域名 1194
### 注释掉以下内容 ###
user nobody
group nogroup
### 注释掉以下内容 ###
#ca ca.crt
#cert client.crt
#key client.key
### 增加以下内容 ###
cipher AES-256-CBC
auth SHA256
### 末尾追加一条 ###
key-direction 1
### 再为linux客户端准备一些配置项,保留注释状态 ###
# script-security 2
# up /etc/openvpn/update-resolv-conf
# down /etc/openvpn/update-resolv-conf
创建一个简单的脚本,它将使用相关的证书,密钥和加密文件编译您的基本配置,然后将生成的配置置于~/client-configs/files目录中
vim ~/client-configs/make_config.sh
添加以下内容
#!/bin/bash
# First argument: Client identifier
KEY_DIR=~/client-configs/keys
OUTPUT_DIR=~/client-configs/files
BASE_CONFIG=~/client-configs/base.conf
cat ${BASE_CONFIG} \
<(echo -e '<ca>') \
${KEY_DIR}/ca.crt \
<(echo -e '</ca>\n<cert>') \
${KEY_DIR}/${1}.crt \
<(echo -e '</cert>\n<key>') \
${KEY_DIR}/${1}.key \
<(echo -e '</key>\n<tls-auth>') \
${KEY_DIR}/ta.key \
<(echo -e '</tls-auth>') \
> ${OUTPUT_DIR}/${1}.ovpn
修改make_config.sh的权限
chmod 700 ~/client-configs/make_config.sh
在生成客户端证书和密钥对时,我们已经创建了client1的客户端证书及密钥对。使用该客户端证书及密钥对来生成配置文件
cd ~/client-configs
sudo ./make_config.sh client1
在~/client-configs/files目录下创建了client1.ovpn
把这个文件拷贝到客户端
从https://openvpn.net/community-downloads/下载windows客户端并安装(使用默认路径)
把client1.ovpn文件拷贝到C:\Program Files\OpenVPN\config
以管理员身份运行运行OpenVPN客户端,选择yes,连接即可。
登入CAserver
cd EasyRSA-3.0.4/
./easyrsa revoke client1
Type the word 'yes' to continue, or any other input to abort.
Continue with revocation: yes
输入yes
在CAserver上生成废除列表
./easyrsa gen-crl
将该列表文件cp至VPNserver
scp ~/EasyRSA-3.0.4/pki/crl.pem VPNserver:/tmp
登入VPNserver
cp /tmp/crl.pem /etc/openvpn
编辑server.conf,增加如下行:
crl-verify crl.pem
最后,重启vpnserver
systemctl restart openvpn@server
如果需要增加废除的证书,重复以上步骤即可。
图形数据库
version: '3'
services:
neo4j:
image: neo4j:3.5.28-community
ports:
- "7474:7474"
- "7687:7687"
restart: always
container_name: neo4j
volumes:
- neo4j:/var/lib/neo4j/data
- /var/log/neo4j:/logs
volumes:
neo4j:
external: true
如果不需要密码验证,则添加如下内容
environment:
- NEO4J_AUTH=none
初始用户名/密码
neo4j/neo4j
version: '2'
services:
zoo1:
image: zookeeper
container_name: zoo
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
volumes:
- $PWD/conf:/conf
ports:
- 2181:2181
kafka1:
image: 'bitnami/kafka:latest'
ports:
- '9092:9092'
container_name: kafka1
volumes:
- "kafka_data1:/bitnami"
environment:
- KAFKA_ZOOKEEPER_CONNECT=zoo1:2181
- KAFKA_BROKER_ID=1
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- LOG.RETENTION.HOURS=6400 ## 这句是日志保存时间
depends_on:
- zoo1
kafka2:
image: 'bitnami/kafka:latest'
ports:
- '9093:9092'
container_name: kafka2
volumes:
- "kafka_data2:/bitnami"
environment:
- KAFKA_ZOOKEEPER_CONNECT=zoo1:2181
- KAFKA_BROKER_ID=2
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9093
depends_on:
- zoo1
kafka3:
image: 'bitnami/kafka:latest'
ports:
- '9094:9092'
container_name: kafka3
volumes:
- "kafka_data3:/bitnami"
environment:
- KAFKA_ZOOKEEPER_CONNECT=zoo1:2181
- KAFKA_BROKER_ID=3
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9094
depends_on:
- zoo1
volumes:
kafka_data1:
external: true
kafka_data2:
external: true
kafka_data3:
external: true
基于UBUNTU
#!/bin/env bash
apt update
apt upgrade -y
apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
apt update
apt install docker-ce docker-ce-cli containerd.io -y
#!/bin/env bash
COMPOSEVERSION=$(curl -s https://github.com/docker/compose/releases/latest/download 2>&1 | grep -Po [0-9]+\.[0-9]+\.[0-9]+)
curl -L "https://get.daocloud.io/docker/compose/releases/download/$COMPOSEVERSION/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
目前docker-compose已经是2.x版本了,把COMPOSEVERSION赋值1.29.2
/etc/docker/daemon.json
{
"registry-mirrors": ["https://0xj5rnq5.mirror.aliyuncs.com"],
"log-driver":"json-file",
"log-opts": {"max-size":"500m", "max-file":"3"}
}
建立start.sh
#!/bin/env bash
java -jar -Duser.timezone=GMT+08 -Xms512m -Xmx512m xxx.jar >> /var/log/xxx/xxx.log 2>&1
建立Dockerfile
FROM java:8
RUN mkdir /opt/xxx
COPY ./xxx.jar /opt/xxx/xxx.jar
COPY ./start.sh /opt/xxx/start.sh
WORKDIR /opt/xxx
ENTRYPOINT ["sh","/opt/xxx/start.sh"]
EXPOSE 8080
编辑docker-compose.yml
version: '2'
services:
xxx:
image: xxx:openjdk-8
container_name: xxx
ports:
- "8080:8080"
mem_limit: 1024m
restart: always
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/log/xxx:/var/log/xxx
import hmac
import os
from flask import Flask, request, jsonify
import json
import base64
from urllib.parse import quote
import logging
logging.basicConfig(filename='webhook.log', level=logging.DEBUG, format='%(asctime)s - %(message)s')
app = Flask(__name__)
secret = '密码'
def encryption(data):
"""
Step1:把timestamp+"\n"+密钥当做签名字符串,使用HmacSHA256算法计算签名。
Setp2:对上述得到的结果进行 Base64 encode。
Setp3:对上述得到的结果进行 urlEncode,得到最终的签名(需要使用UTF-8字符集)。
"""
key = secret.encode('utf-8')
obj = hmac.new(key, msg=data, digestmod='sha256')
return base64.b64encode(obj.digest()).decode("utf-8")
@app.route('/', methods=['POST'])
def post_data():
"""
githee加密是将post提交的timestamp +'\n' + WebHooks的secret通过hmac的sha256加密,放到HTTP headers的
X-Gitee-Token参数中
"""
logging.info(request.url)
post_data = json.loads(request.data)
sign_string = post_data["timestamp"] + "\n" + secret
token = encryption(sign_string.encode('utf-8'))
# 认证签名是否有效
signature = request.headers.get('X-Gitee-Token', '')
if signature != token:
return "token认证无效", 401
# Push:"push_hooks"/"tag_push_hooks"。 Pull Request:"merge_request_hooks"
if post_data["hook_name"] in ["push_hooks", "tag_push_hooks", "merge_request_hooks"]:
try:
branch_name = post_data["ref"].split("/")[-1]
except KeyError:
branch_name = post_data["target_branch"].split("/")[-1]
# 运行shell脚本,更新代码
logging.info(os.popen('/opt/webhook/command.sh ' + branch_name).read())
return jsonify({"status": 200})
if __name__ == '__main__':
app.run(host="0.0.0.0", port=端口)
启动命令:
nohup python3 /opt/webhook/webhook.py &
prometheus的系统结构是server+agent模式,prometheus自己是server,各种exporter就相当于agent,运行在各个被监控主机上,通过定时从server去pull各个exporter的metric,形成统计输出。
version: '3'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
restart: always
network_mode: 'host'
- job_name: 'testwork'
static_configs:
- targets: ['172.17.244.131:9100']
node-exporter的安装,官方推荐不要使用docker,个人还是给出docker-compose.yml
version: '3'
services:
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
command:
- '--path.rootfs=/host'
restart: always
network_mode: 'host'
pid: host
volumes:
- '/:/host:ro,rslave'
version: '3'
services:
mysqld-exporter:
image: prom/mysqld-exporter:latest
container_name: mysqld-exporter
volumes:
- ./.my.cnf:/home/.my.cnf
ports:
- 9104:9104
network_mode: 'host'
[client]
user = xxx
password = xxx
host = xxx
port = xxx
version: '3'
services:
grafana:
image: grafana/grafana:latest
container_name: grafana
network_mode: 'host'
访问hostname:3000,进入grafana,import dashboard即可。
docker volume create data1
docker volume create data2
docker-compose.yml重点配置
注意顶级标签一定要写成这样:
volumes:
data1:
external: true #如果没有这句,就会自动创建以docker-compose.yml所在目录为名字的volumes,每个已有卷都要增加这句。
data2:
external: true
一个共享示例:
第一个docker-compose.yml
version: '3'
services:
container1:
image: busybox:latest
volumes:
- data1:/opt/data1
- data2:/opt/data2
volumes:
data1:
external: true
data2:
external: true
第二个docker-compose.yml
version: '3'
services:
container2:
image: busybox:latest
volumes:
- data1:/opt/data1
- data2:/opt/data2
volumes:
data1:
external: true
data2:
external: true
这样就实现了两个容器的数据共享
最优:
本地写一个shell,内容如下:
#!/bin/bash
ssh root@远程主机 << remotessh
cd /opt/shell;
sh opendata.sh;
exit
remotessh
说明:本地shell会在远程主机上执行在remotessh标记中间的所有脚本
远程主机上的脚本执行完了,必须使用exit退出