nginx的安装配置及通过keepalived实现高可用

虚拟机VirtualBox
淘宝nginx http://tengine.taobao.org/
centos6.5 http://mirror.nsc.liu.se/centos-store/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso
keepalived http://www.keepalived.org/

单个nginx的安装配置

安装需要的库

gcc: GNU Compiler Collection,一套编程语言编译器 https://gcc.gnu.org/
openssl-devel: OpenSSL 是一个安全套接字层密码库 https://www.openssl.org/
pcre-devel: PCRE(Perl Compatible Regular Expressions)是一个Perl库,包括perl兼容的正则表达式库 http://www.pcre.org/
zlib-devel: 用于文件压缩的库 https://zlib.net/

1
yum -y install gcc openssl-devel pcre-devel zlib-devel

配置检查

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
./configure \
--prefix=/work/bigdata/tengine-2.2.0/ \
--error-log-path=/work/log/nginx/error.log \
--http-log-path=/work/log/nginx/access.log \
--pid-path=/work/run/nginx/nginx.pid \
--lock-path=/work/lock/nginx.lock \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/work/tmp/nginx/client/ \
--http-proxy-temp-path=/work/tmp/nginx/proxy/ \
--http-fastcgi-temp-path=/work/tmp/nginx/fcgi/ \
--http-uwsgi-temp-path=/work/tmp/nginx/uwsgi \
--http-scgi-temp-path=/work/tmp/nginx/scgi \
--with-pcre
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
......
Configuration summary
+ using system PCRE library
+ using system OpenSSL library
+ md5: using OpenSSL library
+ sha1: using OpenSSL library
+ using system zlib library
+ jemalloc library is disabled

nginx path prefix: "/work/bigdata/tengine-2.2.0/"
nginx binary file: "/work/bigdata/tengine-2.2.0//sbin/nginx"
nginx configuration prefix: "/work/bigdata/tengine-2.2.0//conf"
nginx configuration file: "/work/bigdata/tengine-2.2.0//conf/nginx.conf"
nginx pid file: "/work/run/nginx/nginx.pid"
nginx error log file: "/work/log/nginx/error.log"
nginx http access log file: "/work/log/nginx/access.log"
nginx http client request body temporary files: "/work/tmp/nginx/client/"
nginx dso module path: "/work/bigdata/tengine-2.2.0//modules/"
nginx http proxy temporary files: "/work/tmp/nginx/proxy/"
nginx http fastcgi temporary files: "/work/tmp/nginx/fcgi/"
nginx http uwsgi temporary files: "/work/tmp/nginx/uwsgi"
nginx http scgi temporary files: "/work/tmp/nginx/scgi"

编译安装

1
make && make install

在/etc/init.d下面建立nginx文件

拷贝下面的内容到nginx文件,注意路径

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
#!/bin/bash
#
# chkconfig: - 85 15
# description: nginx is a World Wide Web server. It is used to serve
# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/work/bigdata/tengine-2.2.0/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/work/bigdata/tengine-2.2.0/conf/nginx.conf"

#[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/work/lock/subsys/nginx

start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
# make_dirs
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}

stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -rf $lockfile
return $retval
}

restart() {
configtest || return $?
stop
sleep 1
start
}

reload() {
configtest || return $?
echo -n $"Reloading $prog: "
# -HUP是nginx平滑重启参数
killproc $nginx -HUP
RETVAL=$?
echo
}

force_reload() {
restart
}

configtest() {
$nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
status $prog
}

rh_status_q() {
rh_status >/dev/null 2>&1
}

case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac

修改nginx文件执行权限

1
chmod a+x nginx

配置chkconfig

服务脚本必须存放在/etc/ini.d/目录下
在chkconfig工具服务列表中增加此服务,此时服务会被在/etc/rc.d/rcN.d中赋予K/S入口了

1
chkconfig --add nginx

修改服务的默认启动等级

1
chkconfig --level 35 nginx on

启动nginx

如果提示目录不存在或找不到,手动创建,如

1
2
3
4
mkdir -p /work/tmp/nginx/client/
mkdir -p /work/lock/subsys/nginx

service nginx start

查看nginx服务

1
2
3
4
5
6
7
8
9
# service nginx start
# ps -ef|grep nginx
root 16721 1 0 23:57 ? 00:00:00 nginx: master process /work/bigdata/tengine-2.2.0/sbin/nginx -c /work/bigdata/tengine-2.2.0/conf/nginx.conf
nobody 16723 16721 0 23:57 ? 00:00:00 nginx: worker process
root 16757 11369 0 23:59 pts/0 00:00:00 grep nginx
# lsof -i:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 16721 root 6u IPv4 215609 0t0 TCP *:http (LISTEN)
nginx 16723 nobody 6u IPv4 215609 0t0 TCP *:http (LISTEN)

浏览器访问

http://192.168.99.100

image

高可用配置和实现

四台虚拟机
192.168.99.100 nginx+keepalived master
192.168.99.101 nginx+keepalived slave
192.168.99.102 tomcat(real server eg:spring boot-node etc.)
192.168.99.103 tomcat(real server)
虚拟IP 192.168.99.111
tomcat的安装过程省略

安装配置keepalived

警告不支持IPv6

1
2
*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
yum -y install libnl3.x86_64 libnl3-devel.x86_64

报错缺库

1
2
configure: error: libnfnetlink headers missing
yum -y install libnfnetlink.x86_64 libnfnetlink-devel.x86_64

安装

1
2
3
4
5
6
wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
tar -zxvf keepalived-1.3.5.tar.gz
cd keepalived-1.3.5
./configure --prefix=/work/bigdata/keepalived-1.3.5 --sysconf=/etc/ --with-kernel-dir=/usr/src/kernels/2.6.32-642.11.1.el6.x86_64
make && make install
ln -s /work/bigdata/keepalived-1.3.5/sbin/keepalived /sbin/

keepalived的启动停止和重启

将源文件夹下init.d中的keepalived拷贝到/etc/init.d中

1
2
3
4
5
6
7
8
9
10
11
12
13
# cp /work/bigdata/software/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/
# service keepalived start
正在启动 keepalived: [确定]
# ps -ef|grep keepalived
root 8194 1 0 07:49 ? 00:00:00 keepalived -D
root 8196 8194 0 07:49 ? 00:00:00 keepalived -D
root 8197 8194 0 07:49 ? 00:00:00 keepalived -D
root 8216 7243 0 07:49 pts/1 00:00:00 grep keepalived
# service keepalived stop
停止 keepalived: [确定]
# service keepalived restart
停止 keepalived: [确定]
正在启动 keepalived: [确定]

master机器keepalived.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_NGINX
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
# 此处关键
lvs_sync_daemon_inteface eth0
virtual_router_id 51
priority 110
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
# 192.168.99.111/24 dev eth0 label eth0:1
192.168.99.111/24 brd 192.168.99.255
}
}

vrrp_script chk_http_port {
script "/usr/local/src/check_nginx_pid.sh"
interval 2
weight 2
}

track_script {
chk_http_port
}

slave机器keepalived.conf

不同之处在于priority的值不同

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_NGINX
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
lvs_sync_daemon_inteface eth0
virtual_router_id 51
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
# 192.168.99.111/24 dev eth0 label eth0:1
192.168.99.111/24 brd 192.168.99.255
}
}

vrrp_script chk_http_port {
script "/usr/local/src/check_nginx_pid.sh"
interval 2
weight 2
}

track_script {
chk_http_port
}

nginx进程检测脚本

1
2
3
4
5
6
7
8
9
vi /usr/local/src/check_nginx_pid.sh
#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
/work/bigdata/tengine-2.2.0/sbin/nginx
if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
killall keepalived
fi
fi

相同nginx.conf配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
user  nobody;
worker_processes 1;
error_log /work/log/nginx/logs/error.log notice;
pid /work/log/nginx/logs/nginx.pid;
worker_rlimit_nofile 51200;
events {
use epoll;
worker_connections 51200;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main;
server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
client_max_body_size 8m;
sendfile on;
tcp_nopush on;
server_tokens off;
keepalive_timeout 60;
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
fastcgi_buffer_size 64k;
fastcgi_buffers 4 64k;
fastcgi_busy_buffers_size 128k;
fastcgi_temp_file_write_size 128k;
gzip on;

upstream backend {
server 192.168.99.102:8080;
server 192.168.99.103:8080;
}

# server_name虚拟IP
server {
listen 80;
server_name 192.168.99.111;
location / {
root html;
index index.php index.html index.htm;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
#后端的Web服务器可以通过X-Forwarded-For获取用户真实IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://backend;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}

location /nginx_status {
stub_status on;
auth_basic "NginxStatus";
auth_basic_user_file /usr/local/nginx/htpasswd;
#allow 127.0.0.1;
#deny all;
}
location ~* \.(ini|docx|txt|doc|pdf)$ {
#禁止访问文档性文件
root /usr/share/nginx/html;
deny all;
}
location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|js|html|htm|css)$ {
root /home/image;
proxy_store on;
proxy_store_access user:rw group:rw all:rw;
proxy_temp_path /home/image;
if ( !-e $request_filename) {
proxy_pass http://backend;
}
}
}
}

tomcat的index.jsp修改

不改变原有内容,只修改部分和添加几句,用于观察
tomcat1 page 和 tomcat2 page
session设为true

1
2
3
4
5
6
7
8
9
10
11
......
<%@ page session="true" pageEncoding="UTF-8" contentType="text/html; charset=UTF-8" %>
<body>
SessionID:<%=session.getId()%>
<br/>
SessionIP:<%=request.getServerName()%>
<br/>
<h1>tomcat1 page</h1>

<div id="wrapper">
......

高可用测试

查看vip绑定

依次启动两个tomcat、两个nginx和两个keepalived,查看主机上是否有虚拟IP地址
ip a 和 ping

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:11:73:65 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.4/24 brd 10.0.2.255 scope global eth0
inet 192.168.99.111/24 brd 192.168.99.255 scope global eth0
inet6 fe80::a00:27ff:fe11:7365/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:63:d9:52 brd ff:ff:ff:ff:ff:ff
inet 192.168.99.100/24 brd 192.168.99.255 scope global eth1
inet6 fe80::a00:27ff:fe63:d952/64 scope link
valid_lft forever preferred_lft forever
1
2
3
4
5
# ping 192.168.99.111
PING 192.168.99.111 (192.168.99.111) 56(84) bytes of data.
64 bytes from 192.168.99.111: icmp_seq=1 ttl=64 time=0.018 ms
64 bytes from 192.168.99.111: icmp_seq=2 ttl=64 time=0.035 ms
64 bytes from 192.168.99.111: icmp_seq=3 ttl=64 time=0.027 ms

浏览器访问

没有共享session的服务切换

image
image

邵志鹏 wechat
扫一扫上面的二维码关注我的公众号
0%