本文共 4632 字，大约阅读时间需要 15 分钟。

一、启动一个tcp server，把送来的数据返回。

$ more echoserver.py 

#!/usr/bin/env python """Simple server that listens on port 16000 and echos back every input to the client. Connect to it with:   telnet localhost 16000 Terminate the connection by terminating telnet (typically Ctrl-] and then 'quit'). """ from __future__ import print_function from gevent.server import StreamServer # this handler will be run for each incoming connection in a dedicated greenlet def echo(socket, address):     print('New connection from %s:%s' % address)     socket.sendall(b'Welcome to the echo server! Type quit to exit.\r\n')     # using a makefile because we want to use readline()     rfileobj = socket.makefile(mode='rb')     while True:         line = rfileobj.readline()         if not line:             print("client disconnected")             break         if line.strip().lower() == b'quit':             print("client quit")             break         socket.sendall(line)         print(line)     rfileobj.close() if __name__ == '__main__':     # to make the server use SSL, pass certfile and keyfile arguments to the constructor     server = StreamServer(('0.0.0.0', 16000), echo)     # to start the server asynchronously, use its start() method;     # we use blocking serve_forever() here because we have no other jobs     print('Starting echo server on port 16000')     server.serve_forever()

二、连接server，发送单个字符，并记录，100个做一次统计。

$ more tcptest.py 

import subprocess, time, sys, errno import socket import numpy as np def tcptest(host, port, interval=.1, count=None):   print host, port   s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)   s.connect((host, port))   i = 0   err = 0   t = []   while True:     try:       i += 1       t1 = time.time()       s.sendall(str(i)+'\n')        data = s.recv(1024)       t2 = time.time()       t.append(t2-t1)       if i % 100 == 0:         summary(t)         t = []       if count and i >= count:         break       time.sleep(interval)     except IOError as e:       print e       if e.errno == errno.EPIPE:         break     except Exception, e:       print e       err += 1       if err > 100:         break   summary(t)   s.close() def summary(t):   if len(t) > 0:     print '%s %d packets(min/mean/median/max/std):%f/%f/%f/%f/%f' % (time.asctime(), len(t), np.min(t), np.mean(t),np.median(t) , np.max(t), np.std(t))     sys.stdout.flush() if __name__ == '__main__':   host, port = sys.argv[1], int(sys.argv[2])   tcptest(host, port)

------

日志类似于：

xxx.xxx.xxx.xx 16000

Fri Sep  2 16:59:26 2016 100 packets(min/mean/median/max/std):0.000338/0.000647/0.000624/0.002038/0.000269 Fri Sep  2 16:59:36 2016 100 packets(min/mean/median/max/std):0.000329/0.001173/0.000483/0.063784/0.006295 Fri Sep  2 16:59:46 2016 100 packets(min/mean/median/max/std):0.000332/0.000498/0.000477/0.000846/0.000102 Fri Sep  2 16:59:56 2016 100 packets(min/mean/median/max/std):0.000325/0.000546/0.000487/0.001736/0.000201 Fri Sep  2 17:00:06 2016 100 packets(min/mean/median/max/std):0.000371/0.000633/0.000576/0.005159/0.000474 Fri Sep  2 17:00:16 2016 100 packets(min/mean/median/max/std):0.000307/0.000663/0.000547/0.008853/0.000851

并在多个服务器启动这个进程。

三、从各个服务器获取日志

服务器记录在

$ more loghost.txt 

msg0=172.41.17.127 msg1=172.41.17.128 msg2=172.41.17.129 msg3=172.41.17.130 msg4=172.41.17.131 msg5=172.41.17.132 msg6=172.41.17.135 msg7=172.41.17.136

循环从服务器获取日志：

$ more fetchlog.sh 

#!/bin/bash basedir=/tmp/ana remotelogdir=/tmp loghostfile=$basedir/loghost.txt cat $loghostfile|while read line;do dirname=`echo $line|awk -F = '{print $1}'` ip=`echo $line|awk -F = '{print $2}'` echo mkdir -p $basedir/$dirname; mkdir -p $basedir/$dirname; echo scp $ip:$remotelogdir/tcptest.log $basedir/$dirname; scp $ip:$remotelogdir/tcptest.log $basedir/$dirname; done

四、对日志文件做summary

$ more summary.sh 

#!/bin/bash basedir=/tmp/ana datadir=$1 if [ "$datadir" == "" ];then   echo input datadir,pls.   exit 2; esle   echo do summary datadir=$datadir fi cat /dev/null > $datadir/summary.log #find dataline and date string grep "min\/mean\/median\/max\/std" $datadir/tcptest.log |cut -c 1-10 |uniq|while read date;do    echo date="$date" ;   day=`echo $date|awk '{print $3}'`;   month=`echo $date|awk '{print $2}'`;   datefile=$month$day.log   echo split into datefile=$datefile;   echo "min/mean/median/max/std" > $datadir/$datefile ;   grep "$date" $datadir/tcptest.log |awk -F : '{print $4}' >> $datadir/$datefile;   echo $date >> $datadir/summary.log   python $basedir/summary.py $datadir/$datefile / >> $datadir/summary.log done

--------

$ more summary.py

import sys, pandas; if __name__ == '__main__':   file = sys.argv[1];   separator = sys.argv[2];   data = pandas.read_csv(file,sep = separator);   summary = data.describe();   print(summary.head(10));   print("end");

五、获取日志和分析全部过程串联在一起

]$ more doall.sh 

#!/bin/bash basedir=/tmp/ana loghostfile=$basedir/loghost.txt $basedir/fetchlog.sh cat $loghostfile|while read line;do dirname=`echo $line|awk -F = '{print $1}'` $basedir/summary.sh $dirname done

转载地址：http://fhbws.baihongyu.com/