Odoo中文社区可以通过以下三个域名访问:shine-it.net , odoocn.org,odoo.net.cn

原论坛用户的基本信息和发帖这里都予以保留,请注意:原论坛用户无需重新注册新用户,但是您的密码需要重置

开发人员可以登录gitter讨论组: http://gitter.im/odoo-china/Talk, 需要github账号

如果您登录系统碰到问题,请在微信公众号留言:

Openerp压力测试:多线程直连OE Server NET-RPC/XML-RPC端口测试



  •         之前发表的一篇文章 Openerp压力测试:Openerp到底能支撑多大的用户数? ,转贴到OpenERP中文社区和OpenERP QQ群(69195329)之后,得到了许多童靴的关注,尤其是很多前辈的鼓励和转贴。这里就不一一道谢了。总之,谢谢大家!
            也有善意的质疑,某童靴就指出,“不代表 OE server 的 xmlrpc 性能的.  中间或者有 cherrypy 的缓存在起作用.  并发100 已经相当牛X了.”,某童靴同时也提出,“期待 清沙出个 xmlrpc/netrpc 直接性能测试版本....射射....”。我表示,完全没有鸭梨。

            今天偷得半日浮生,想起某童靴的教导,不由的觉得一种鸭梨油然而生,!^@!A*&* 今天天气晴朗,万里乌云,小朋友们高高兴兴的。。。(原谅我吧,一有鸭梨,总是想起当年有个可怜的小朋友在憋课堂作文的时候)。废话少说,今天写了一小段Python代码,用于测试OpenERP NET-XML 和 XML-RPC性能。本人才疏学浅,代码中如有臭虫或错误之处,还望各位童靴斧正。

    <br />#! /usr/bin/python#! /usr/bin/python<br /># -*- coding: utf-8 -*-<br />#<br /># OE压力测试小工具&nbsp; www.360yun.info<br />#<br /><br /># 导入 openerprpc 模块。<br /># 纯属个人习惯,如果您是使用apt或者python setup.py 等方式安装的话,<br /># 可以无视下面2行<br />import sys,os<br />sys.path.append (os.path.abspath(os.path.join(os.path.dirname(__file__), &#039;openerprpc-1.0.1&#039;)))<br />import openerprpc<br /><br />import threading<br />import Queue<br />import datetime<br />import time<br /><br />#参数设置<br />JOBS_COUNT = 1000 #任务总数<br />THREAD_LIMIT = 100 #并发线程数<br />HOST = &#039;127.0.0.1&#039; #主机IP<br />PROTOCOL = &#039;netrpc&#039; #可选值 netrpc xmlrpc<br />PORT = &#039;auto&#039; #端口<br />DATABASE = &#039;test&#039; #数据库<br />LOGIN = &#039;admin&#039; #用户名<br />PASSWORD = &#039;admin&#039; #密码<br />USER_ID = None #用户id<br /><br />#队列<br />jobs = Queue.Queue(0)<br />def thread(num):<br />&nbsp; &nbsp; while True:<br />&nbsp; &nbsp; &nbsp; &nbsp; try:<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; job = jobs.get(False)<br />&nbsp; &nbsp; &nbsp; &nbsp; except Queue.Empty:<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; return<br /><br />&nbsp; &nbsp; &nbsp; &nbsp; #连接OE Server<br />&nbsp; &nbsp; &nbsp; &nbsp; conn = openerprpc.get_connection(HOST, protocol=PROTOCOL, port=PORT,<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; database=DATABASE, login=LOGIN,<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; password=PASSWORD, user_id=USER_ID)<br />&nbsp; &nbsp; &nbsp; &nbsp; #取第一个partner 记录<br />&nbsp; &nbsp; &nbsp; &nbsp; company = conn.get_object(&#039;res.partner&#039;).read([(1)],[(&#039;name&#039;)])<br />&nbsp; &nbsp; &nbsp; &nbsp; print &#039;job %s thread %s : %s &#039; % (str(job) , str(num), str(company))<br />&nbsp; &nbsp; &nbsp; &nbsp; time.sleep(1/1000) # 休息1毫秒<br /><br /><br />def main():<br />&nbsp; &nbsp; #将任务压入队列<br />&nbsp; &nbsp; for job in range(JOBS_COUNT):<br />&nbsp; &nbsp; &nbsp; &nbsp; jobs.put(job)<br /><br />&nbsp; &nbsp; start_time = datetime.datetime.now()<br /><br />&nbsp; &nbsp; #启动线程<br />&nbsp; &nbsp; for n in range(THREAD_LIMIT):<br />&nbsp; &nbsp; &nbsp; &nbsp; t = threading.Thread(target = thread, kwargs = {&#039;num&#039;: n})<br />&nbsp; &nbsp; &nbsp; &nbsp; t.start()<br /><br />&nbsp; &nbsp; #等待线程结束<br />&nbsp; &nbsp; while threading.activeCount() &gt; 1:<br />&nbsp; &nbsp; &nbsp; &nbsp; pass<br /><br />&nbsp; &nbsp; end_time = datetime.datetime.now()<br /><br />&nbsp; &nbsp; print &#039;\r\nStart at %s&#039; % str(start_time)<br />&nbsp; &nbsp; print &#039;End&nbsp;  at %s&#039; % str(end_time)<br />&nbsp; &nbsp; print &#039;\r\nTotal Time: %s&#039; % str(end_time - start_time)<br /><br />if __name__ == &quot;__main__&quot;:<br />&nbsp; &nbsp; main()<br />
    



            上面的程序使用官方的openerprpc模块连接到OE Server,支持NET-RPC和XML-RPC连接方式,使用很简单,例子请看上面的代码。下载地址是http://pypi.python.org/pypi/openerprpc 。下载之后用 python setup.py 安装。当然,也可以不安装,代码中用sys.path.append添加openerprpc的路径即可,如同上面的代码。

            OpenERP中文社区论坛里面也介绍了其他如oersted 等,不过下载来看了下只支持NET-RPC。GOOGLE之后发现原来官方也有出rpc客户端模块,缺点就是官方的openerprpc模块没有文档也没有样例。呵呵,有机会在来写写openerprpc模块介绍。

            按照上面的程序代码,就可以运行了么?当然可以运行,如果你没有改动我的程序的话,很快你就会得到以下错误:

    <br />error: [Errno 104] Connection reset by peer<br />
    


      神马?不会吧,这明显是Socket Server 撑不住啊。难道连100并发都搞不定?难道某童靴一语言中?莫慌,打开OE的源码来看看(这就是开源的好处啊,哇哈哈)......中间省略数万字....  下面是解决办法:

            1、编辑 openerp-server-6.0.2/bin/service/netrpc_server.py ,找到以下代码(112行):

    <br />class TinySocketServerThread(threading.Thread,netsvc.Server):<br />&nbsp; &nbsp; def __init__(self, interface, port, secure=False):<br />&nbsp; &nbsp; &nbsp; &nbsp; threading.Thread.__init__(self, name=&quot;NetRPCDaemon-%d&quot;%port)<br />&nbsp; &nbsp; &nbsp; &nbsp; netsvc.Server.__init__(self)<br />&nbsp; &nbsp; &nbsp; &nbsp; self.__port = port<br />&nbsp; &nbsp; &nbsp; &nbsp; self.__interface = interface<br />&nbsp; &nbsp; &nbsp; &nbsp; self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)<br />&nbsp; &nbsp; &nbsp; &nbsp; self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)<br />&nbsp; &nbsp; &nbsp; &nbsp; self.socket.bind((self.__interface, self.__port))<br />&nbsp; &nbsp; &nbsp; &nbsp; #self.socket.listen(5)<br />&nbsp; &nbsp; &nbsp; &nbsp; #此处修改为128 或更大 这个是Socket 队列,不是监听端口,详情请Google之。<br />&nbsp; &nbsp; &nbsp; &nbsp; self.socket.listen(128)<br />&nbsp; &nbsp; &nbsp; &nbsp; self.threads = &#91;]<br />&nbsp; &nbsp; &nbsp; &nbsp; netsvc.Logger().notifyChannel(&quot;web-services&quot;, netsvc.LOG_INFO, <br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  &quot;starting NET-RPC service at %s port %d&quot; % (interface or &#039;0.0.0.0&#039;, port,))<br />
    



            2、编辑 openerp-server-6.0.2/bin/service/http_server.py ,找到以下代码(80 行):

    <br />class ThreadedHTTPServer(ConnThreadingMixIn, SimpleXMLRPCDispatcher, HTTPServer):<br /><br />&nbsp; &nbsp; encoding = None<br />&nbsp; &nbsp; allow_none = False<br />&nbsp; &nbsp; allow_reuse_address = 1<br />&nbsp; &nbsp; _send_traceback_header = False<br />&nbsp; &nbsp; i = 0<br /><br />&nbsp; &nbsp; def __init__(self, addr, requestHandler, proto=&#039;http&#039;,<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  logRequests=True, allow_none=False, encoding=None, bind_and_activate=True):<br />&nbsp; &nbsp; &nbsp; &nbsp; self.logRequests = logRequests<br /><br />&nbsp; &nbsp; &nbsp; &nbsp; SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding)<br />&nbsp; &nbsp; &nbsp; &nbsp; HTTPServer.__init__(self, addr, requestHandler)<br /><br />&nbsp; &nbsp; &nbsp; &nbsp; self.numThreads = 0<br />&nbsp; &nbsp; &nbsp; &nbsp; self.proto = proto<br />&nbsp; &nbsp; &nbsp; &nbsp; self.__threadno = 0<br /><br />&nbsp; &nbsp; &nbsp; &nbsp; #此处添加一行<br />&nbsp; &nbsp; &nbsp; &nbsp; self.socket.listen(128)<br /><br />&nbsp; &nbsp; &nbsp; &nbsp; # [Bug #1222790] If possible, set close-on-exec flag; if a<br />&nbsp; &nbsp; &nbsp; &nbsp; # method spawns a subprocess, the subprocess shouldn&#039;t have<br />&nbsp; &nbsp; &nbsp; &nbsp; # the listening socket open.<br />&nbsp; &nbsp; &nbsp; &nbsp; if fcntl is not None and hasattr(fcntl, &#039;FD_CLOEXEC&#039;):<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD)<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; flags |= fcntl.FD_CLOEXEC<br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags)<br />
    


            3、编辑OE Server 的配置文件openerp-server.conf,在最后添加以下代码:

    <br />; 数据库连接池数量<br />db_maxconn = 128<br />
    


            这个是数据库的连接池数量,出处请察看源码openerp-server-6.0.2/bin/sql_db.py 第258行处,默认值是64。如果你启动OE没有使用配置文件,那么你只好修改源码了。sql_db.py 的代码节选如下:(使用配置文件的话,此处无须修改,只是举例说明db_maxconn这个参数不是凭空得来)

    <br />class ConnectionPool(object):<br />#################中间省略无数代码######################<br />&nbsp; &nbsp; def __init__(self, maxconn=64):<br />&nbsp; &nbsp; &nbsp; &nbsp; self._connections = &#91;]<br />&nbsp; &nbsp; &nbsp; &nbsp; self._maxconn = max(maxconn, 1)<br />&nbsp; &nbsp; &nbsp; &nbsp; self._lock = threading.Lock()<br />#################中间省略无数代码######################<br />_Pool = ConnectionPool(int(tools.config&#91;&#039;db_maxconn&#039;]))<br /><br />
    


            经过三个参数的修改,我们的测试程序终于可以正常运行了。但,程序很有可能输出异常,虽然程序可以运行。异常情况如下:

    UnpicklingError: Global and instance pickles are not supported.
    


            尤其是第一次启动测试程序,再次启动,就没事了。貌似线程占用资源问题,不知道是不是要加个线程锁。百思不得其解啊,求各位高手给个说法。

            测试结果

    1、NET-RPC 100并发 1000总任务

    <br />Start at 2011-08-12 20:16:45.058085<br />End&nbsp;  at 2011-08-12 20:16:52.233411<br /><br />Total Time: 0:00:07.175326<br />
    


    2、XML-RPC 100并发 1000总任务 (运行这个测试请把测试代码中参数修改为 PROTOCOL = 'xmlrpc' #可选值 netrpc xmlrpc)

    <br />Start at 2011-08-12 20:20:21.405651<br />End&nbsp;  at 2011-08-12 20:20:32.149523<br /><br />Total Time: 0:00:10.743872<br />
    


            从上面的测试结果看,XML-RPC所花时间是NET-RPC时间的  1.497335731 倍。与我上一篇OE测试结果相近。

            结论

    1、某童靴,你赢了!”不代表 OE server 的 xmlrpc 性能的.  中间或者有 cherrypy 的缓存在起作用“,您的观点是正确的。OE Web Client 在通过RPC获取对象之后,会把对象缓存在OE Web Client 自己的 cache中。官方文档 http://doc.openerp.com/v6.0/book/1/1_1_Inst_Config/1_1_Inst_Config_architecture.html 中也提到这点。

    <br />When you are changing the structure of your OpenERP installation (adding and<br />removing modules, perhaps changing labels), you might find the web client to be<br />irritating because of its use of caching.<br />
    



    2、如果在部署OE应用是,如果需要高并发,建议调整OE Server 中netrpc_server 和 http_server 的 socket.listen,数据库连接池 db_maxconn也建议你调整到更高。在默认情况下,OE Server 的确不能支持100并发或更多。(如果你也同时使用OE Web Client,请参照之前发表的一篇文章 Openerp压力测试:Openerp到底能支撑多大的用户数? 进行调整)

    3、开放源代码的力量是无穷的,本文调整参数的例子就很好的说明一切。假如你部署的ERP系统,随着业务量的增长遇上性能瓶颈是,估计大部分商业公司给你的建议都是升级到更高、更新的版本,或是购买更强的硬件服务器。没有开放源代码,你甚至都不知道哪儿出问题。

    以上转自本人博客 [检测到链接无效,已移除] 。测试代码打包的下载,请到本人博客。

    谢谢!



  • 沙发~~~辛苦!


  • 管理员

    LZ我用你代码测试过,没改self.socket.listen(5)和db_maxconn = 128也可以运行,但是多次出现了下面错误(而且就算改了源码,错误依旧),

    error: (10061, &#039;Connection refused&#039;)
    





  • [quote author=Joshua link=topic=2556.msg8470#msg8470 date=1313225229]
    LZ我用你代码测试过,没改self.socket.listen(5)和db_maxconn = 128也可以运行,但是多次出现了下面错误(而且就算改了源码,错误依旧),

    error: (10061, &#039;Connection refused&#039;)
    




    [/quote]

    error: (10061, 'Connection refused')

    连接被拒绝,socket 的等待队列默认值才5,多于5个并发就挂了啊,所以才需要修改 socket.listen。


  • 管理员

    LZ 很奇怪,我改了还是一样出现的错误



  • 三个地方都改了吗?改完后要重启OE Server

    另外,麻烦你把测试程序的参数设置和完整的错误提示贴出来。谢谢


  • 管理员

    lz参数设置就是按照你上面所设置的,这个是其中一个出错信息。
    我用的源码也是6.0.2

    Exception in thread Thread-99:<br />Traceback (most recent call last):<br />&nbsp; File &quot;C:\Python25\lib\threading.py&quot;, line 486, in __bootstrap_inner<br />&nbsp; &nbsp; self.run()<br />&nbsp; File &quot;C:\Python25\lib\threading.py&quot;, line 446, in run<br />&nbsp; &nbsp; self.__target(*self.__args, **self.__kwargs)<br />&nbsp; File &quot;D:\eclipse_workspace\oes602\bin\test\yaliceshi.py&quot;, line 44, in thread<br />&nbsp; &nbsp; company = conn.get_object(&#039;res.partner&#039;).read([(1)],[(&#039;name&#039;)])<br />&nbsp; File &quot;D:\eclipse_workspace\oes602\bin\test\openerprpc-1.0.1\openerprpc\__init__.py&quot;, line 240, in proxy<br />&nbsp; &nbsp; self.connection.check_login()<br />&nbsp; File &quot;D:\eclipse_workspace\oes602\bin\test\openerprpc-1.0.1\openerprpc\__init__.py&quot;, line 209, in check_login<br />&nbsp; &nbsp; self.user_id = Service(self.connector,&quot;common&quot;).login(self.database, self.login, self.password)<br />&nbsp; File &quot;D:\eclipse_workspace\oes602\bin\test\openerprpc-1.0.1\openerprpc\__init__.py&quot;, line 178, in proxy<br />&nbsp; &nbsp; result = self.connector.send(self.service_name, method, *args)<br />&nbsp; File &quot;D:\eclipse_workspace\oes602\bin\test\openerprpc-1.0.1\openerprpc\__init__.py&quot;, line 76, in send<br />&nbsp; &nbsp; return getattr(service, method)(*args)<br />&nbsp; File &quot;C:\Python25\lib\xmlrpclib.py&quot;, line 1147, in __call__<br />&nbsp; &nbsp; return self.__send(self.__name, args)<br />&nbsp; File &quot;C:\Python25\lib\xmlrpclib.py&quot;, line 1437, in __request<br />&nbsp; &nbsp; verbose=self.__verbose<br />&nbsp; File &quot;C:\Python25\lib\xmlrpclib.py&quot;, line 1183, in request<br />&nbsp; &nbsp; self.send_content(h, request_body)<br />&nbsp; File &quot;C:\Python25\lib\xmlrpclib.py&quot;, line 1297, in send_content<br />&nbsp; &nbsp; connection.endheaders()<br />&nbsp; File &quot;C:\Python25\lib\httplib.py&quot;, line 860, in endheaders<br />&nbsp; &nbsp; self._send_output()<br />&nbsp; File &quot;C:\Python25\lib\httplib.py&quot;, line 732, in _send_output<br />&nbsp; &nbsp; self.send(msg)<br />&nbsp; File &quot;C:\Python25\lib\httplib.py&quot;, line 699, in send<br />&nbsp; &nbsp; self.connect()<br />&nbsp; File &quot;C:\Python25\lib\httplib.py&quot;, line 683, in connect<br />&nbsp; &nbsp; raise socket.error, msg<br />error: (10061, &#039;Connection refused&#039;)
    


  • openerprpc 报错,非测试程序问题。很有可能是你的python 版本偏低,性能不佳导致。

    建议你先调低并发数试试,THREAD_LIMIT = 10 ,先从10并发开始试试看。

    我注意到你的python 版本是 2.5,建议你升级到2.6 或 2.X 系列的更高版本。(我使用的是Python 2.6.6 on Debian 6.1)。



  • 不错.... 赞一个先......

    windows 下的线程不太好....

    OE 应该是有线程锁的... 我没有仔细阅读过代码.. 我关注 addons 比较多些...



  • 另外, 别某童鞋某童鞋了... 我叫 mrshelly 叫我 shelly 也成..

    贴子LZ可以 Add Tag 的. 加一些 Tags 会比较利于BBS检索...



  • 谢谢shelly 的关注。有您的首肯,以后我在文章中就直接引用您的名字。谢谢!



  • opnerprpc  官方 貌似木有了啊