• 企业400电话
  • 微网小程序
  • AI电话机器人
  • 电商代运营
  • 全 部 栏 目

    企业400电话 网络优化推广 AI电话机器人 呼叫中心 网站建设 商标✡知产 微网小程序 电商运营 彩铃•短信 增值拓展业务
    支持python的分布式计算框架Ray详解

    项目地址:https://github.com/ray-project/ray

    1、简介

    Ray为构建分布式应用程序提供了一个简单、通用的API。Ray是一种分布式执行框架,便于大规模应用程序和利用先进的机器学习库。

    Ray通过以下方式完成这项任务:

    为构建和运行分布式应用程序提供简单的原语。

    使最终用户能够并行化单个机器代码,而代码更改很少到零。

    在核心Ray之上包含大量应用程序、库和工具,以支持复杂的应用程序。

    2、安装

    安装方式比较简单: pip install ray==1.4.1

    [root@node2 ~]# pip install 'ray[default]'
    Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
    Collecting ray[default]
      Downloading https://mirrors.aliyun.com/pypi/packages/13/ec/f727ddd3fbcdc6102eace62c9d5dd9d9ad8112d40eeb7de8783676aca24d/ray-1.4.1-cp36-cp36m-manylinux2014_x86_64.whl (51.6 MB)
         |████████████████████████████████| 51.6 MB 269 kB/s 
    Collecting aiohttp
      Downloading https://mirrors.aliyun.com/pypi/packages/31/28/7b49246d7825c61e1a14306ab050e8eec48fcf5b009b19c8f6ea4e312467/aiohttp-3.7.4.post0-cp36-cp36m-manylinux2014_x86_64.whl (1.3 MB)
         |████████████████████████████████| 1.3 MB 121.5 MB/s 
    Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/site-packages (from ray[default]) (0.8)
    Collecting protobuf>=3.15.3
      Downloading https://mirrors.aliyun.com/pypi/packages/53/4e/e2db88d0bb0bda6a879eea62fddbaf813719ce3770d458bc5580512d9c95/protobuf-3.17.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
         |████████████████████████████████| 1.0 MB 136.1 MB/s 
    Collecting jsonschema
      Downloading https://mirrors.aliyun.com/pypi/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
         |████████████████████████████████| 56 kB 7.1 MB/s 
    Collecting aioredis
      Downloading https://mirrors.aliyun.com/pypi/packages/b0/64/1b1612d0a104f21f80eb4c6e1b6075f2e6aba8e228f46f229cfd3fdac859/aioredis-1.3.1-py3-none-any.whl (65 kB)
         |████████████████████████████████| 65 kB 6.3 MB/s 
    Collecting pydantic>=1.8
      Downloading https://mirrors.aliyun.com/pypi/packages/2b/7c/7d0b3f2d7959b7193018896db236ded165f9bca1bb75f46f4c32fa6f4f9d/pydantic-1.8.2-cp36-cp36m-manylinux2014_x86_64.whl (10.2 MB)
         |████████████████████████████████| 10.2 MB 420 kB/s 
    Collecting pyyaml
      Downloading https://mirrors.aliyun.com/pypi/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
         |████████████████████████████████| 640 kB 19.1 MB/s 
    Collecting numpy>=1.16
      Downloading https://mirrors.aliyun.com/pypi/packages/14/32/d3fa649ad7ec0b82737b92fefd3c4dd376b0bb23730715124569f38f3a08/numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
         |████████████████████████████████| 14.8 MB 296 kB/s 
    Collecting gpustat
      Downloading https://mirrors.aliyun.com/pypi/packages/b4/69/d8c849715171aeabd61af7da080fdc60948b5a396d2422f1f4672e43d008/gpustat-0.6.0.tar.gz (78 kB)
         |████████████████████████████████| 78 kB 7.3 MB/s 
    Collecting redis>=3.5.0
      Downloading https://mirrors.aliyun.com/pypi/packages/a7/7c/24fb0511df653cf1a5d938d8f5d19802a88cef255706fdda242ff97e91b7/redis-3.5.3-py2.py3-none-any.whl (72 kB)
         |████████████████████████████████| 72 kB 968 kB/s 
    Collecting aiohttp-cors
      Downloading https://mirrors.aliyun.com/pypi/packages/13/e7/e436a0c0eb5127d8b491a9b83ecd2391c6ff7dcd5548dfaec2080a2340fd/aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)
    Collecting py-spy>=0.2.0
      Downloading https://mirrors.aliyun.com/pypi/packages/9d/4d/1a9cbe9a0b543e6733cb38afe26451522a9ef8e4897b59e74cc76838f245/py_spy-0.3.7-py2.py3-none-manylinux1_x86_64.whl (3.1 MB)
         |████████████████████████████████| 3.1 MB 131 kB/s 
    Requirement already satisfied: requests in /usr/local/lib/python3.6/site-packages (from ray[default]) (2.24.0)
    Collecting colorama
      Downloading https://mirrors.aliyun.com/pypi/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl (16 kB)
    Collecting filelock
      Downloading https://mirrors.aliyun.com/pypi/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl (7.6 kB)
    Collecting prometheus-client>=0.7.1
      Downloading https://mirrors.aliyun.com/pypi/packages/09/da/4e8471ff825769581593b5b84769d32f58e5373b59fccaf355d3529ad530/prometheus_client-0.11.0-py2.py3-none-any.whl (56 kB)
         |████████████████████████████████| 56 kB 4.2 MB/s 
    Collecting msgpack2.0.0,>=1.0.0
      Downloading https://mirrors.aliyun.com/pypi/packages/0c/0d/b1d9d32d03ce38ba5e2a37fbae850afd4530a14cc441e8335f1865a03705/msgpack-1.0.2-cp36-cp36m-manylinux1_x86_64.whl (272 kB)
         |████████████████████████████████| 272 kB 127.6 MB/s 
    Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.6/site-packages (from ray[default]) (7.1.2)
    Collecting opencensus
      Downloading https://mirrors.aliyun.com/pypi/packages/18/59/12044123133d000f705383ad98579aeb0dd82d66b33a254a21b54bf0d6bb/opencensus-0.7.13-py2.py3-none-any.whl (127 kB)
         |████████████████████████████████| 127 kB 128.9 MB/s 
    Collecting grpcio>=1.28.1
      Downloading https://mirrors.aliyun.com/pypi/packages/eb/32/332a2bb0cee0d4040331952c5ef3d040d119a15b3c622b60bdcbe768bded/grpcio-1.39.0-cp36-cp36m-manylinux2014_x86_64.whl (4.3 MB)
         |████████████████████████████████| 4.3 MB 417 kB/s 
    Collecting colorful
      Downloading https://mirrors.aliyun.com/pypi/packages/b0/8e/e386e248266952d24d73ed734c2f5513f34d9557032618c8910e605dfaf6/colorful-0.5.4-py2.py3-none-any.whl (201 kB)
         |████████████████████████████████| 201 kB 113.9 MB/s 
    Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/site-packages (from grpcio>=1.28.1->ray[default]) (1.15.0)
    Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.6/site-packages (from pydantic>=1.8->ray[default]) (3.10.0.0)
    Collecting multidict7.0,>=4.5
      Downloading https://mirrors.aliyun.com/pypi/packages/a1/35/b22524d6b9cacfb4c5eff413a069bbc17c6ea628e54da5c6c989998ced5f/multidict-5.1.0-cp36-cp36m-manylinux2014_x86_64.whl (141 kB)
         |████████████████████████████████| 141 kB 120.6 MB/s 
    Collecting idna-ssl>=1.0
      Downloading https://mirrors.aliyun.com/pypi/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz (3.4 kB)
    Collecting attrs>=17.3.0
      Downloading https://mirrors.aliyun.com/pypi/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl (53 kB)
         |████████████████████████████████| 53 kB 3.9 MB/s 
    Requirement already satisfied: chardet5.0,>=2.0 in /usr/local/lib/python3.6/site-packages (from aiohttp->ray[default]) (3.0.4)
    Collecting yarl2.0,>=1.0
      Downloading https://mirrors.aliyun.com/pypi/packages/da/08/52b26b44bce7b818b410aee37c5e424c9ea420c557bca97dc2adac29b151/yarl-1.6.3-cp36-cp36m-manylinux2014_x86_64.whl (293 kB)
         |████████████████████████████████| 293 kB 130.8 MB/s 
    Collecting async-timeout4.0,>=3.0
      Downloading https://mirrors.aliyun.com/pypi/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
    Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.6/site-packages (from idna-ssl>=1.0->aiohttp->ray[default]) (2.10)
    Collecting hiredis
      Downloading https://mirrors.aliyun.com/pypi/packages/82/72/313fb6d30b7e413c8662447f9a0abe9105494ceb9266ecb08d442c14afbb/hiredis-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl (84 kB)
         |████████████████████████████████| 84 kB 4.8 MB/s 
    Collecting nvidia-ml-py3>=7.352.0
      Downloading https://mirrors.aliyun.com/pypi/packages/6d/64/cce82bddb80c0b0f5c703bbdafa94bfb69a1c5ad7a79cff00b482468f0d3/nvidia-ml-py3-7.352.0.tar.gz (19 kB)
    Collecting psutil
      Downloading https://mirrors.aliyun.com/pypi/packages/da/82/56cd16a4c5f53e3e5dd7b2c30d5c803e124f218ebb644ca9c30bc907eadd/psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl (291 kB)
         |████████████████████████████████| 291 kB 131.6 MB/s 
    Collecting blessings>=1.6
      Downloading https://mirrors.aliyun.com/pypi/packages/03/74/489f85a78247609c6b4f13733cbf3ba0d864b11aa565617b645d6fdf2a4a/blessings-1.7-py3-none-any.whl (18 kB)
    Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from jsonschema->ray[default]) (39.2.0)
    Collecting pyrsistent>=0.14.0
      Downloading https://mirrors.aliyun.com/pypi/packages/6c/19/1af501f6f388a40ede6d0185ba481bdb18ffc99deab0dd0d092b173bc0f4/pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (117 kB)
         |████████████████████████████████| 117 kB 138.2 MB/s 
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.6/site-packages (from jsonschema->ray[default]) (4.3.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata->jsonschema->ray[default]) (3.4.1)
    Collecting opencensus-context==0.1.2
      Downloading https://mirrors.aliyun.com/pypi/packages/f1/33/990f1bd9e7ee770fc8d3c154fc24743a96f16a0e49e14e1b7540cc2fdd93/opencensus_context-0.1.2-py2.py3-none-any.whl (4.4 kB)
    Collecting google-api-core2.0.0,>=1.0.0
      Downloading https://mirrors.aliyun.com/pypi/packages/aa/51/629a31d4e7db2bbc0b0b789ac4034108c68647bea9cf5b54c8366df47b8e/google_api_core-1.31.0-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.6 MB/s 
    Collecting contextvars
      Downloading https://mirrors.aliyun.com/pypi/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz (9.6 kB)
    Collecting google-auth2.0dev,>=1.25.0
      Downloading https://mirrors.aliyun.com/pypi/packages/d7/13/35b1e0a63e160ecad04985e60339382e48e6020d22f8328a2ab47226c910/google_auth-1.33.1-py2.py3-none-any.whl (152 kB)
         |████████████████████████████████| 152 kB 127.7 MB/s 
    Collecting packaging>=14.3
      Downloading https://mirrors.aliyun.com/pypi/packages/3c/77/e2362b676dc5008d81be423070dd9577fa03be5da2ba1105811900fda546/packaging-21.0-py3-none-any.whl (40 kB)
         |████████████████████████████████| 40 kB 6.1 MB/s 
    Collecting googleapis-common-protos2.0dev,>=1.6.0
      Downloading https://mirrors.aliyun.com/pypi/packages/55/08/796a6bc0b550e2b7116041c953d3d5100016abea106131d71e5651826e7b/googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
         |████████████████████████████████| 198 kB 136.3 MB/s 
    Collecting setuptools
      Downloading https://mirrors.aliyun.com/pypi/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl (819 kB)
         |████████████████████████████████| 819 kB 127.0 MB/s 
    Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from google-api-core2.0.0,>=1.0.0->opencensus->ray[default]) (2020.1)
    Collecting cachetools5.0,>=2.0.0
      Downloading https://mirrors.aliyun.com/pypi/packages/bf/28/c4f5796c67ad06bb91d98d543a5e01805c1ff065e08871f78e52d2a331ad/cachetools-4.2.2-py3-none-any.whl (11 kB)
    Collecting pyasn1-modules>=0.2.1
      Downloading https://mirrors.aliyun.com/pypi/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
         |████████████████████████████████| 155 kB 21.5 MB/s 
    Collecting rsa5,>=3.1.4
      Downloading https://mirrors.aliyun.com/pypi/packages/e9/93/0c0f002031f18b53af7a6166103c02b9c0667be528944137cc954ec921b3/rsa-4.7.2-py3-none-any.whl (34 kB)
    Collecting pyparsing>=2.0.2
      Downloading https://mirrors.aliyun.com/pypi/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
         |████████████████████████████████| 67 kB 7.2 MB/s 
    Collecting pyasn10.5.0,>=0.4.6
      Downloading https://mirrors.aliyun.com/pypi/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
         |████████████████████████████████| 77 kB 7.0 MB/s 
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->ray[default]) (2020.6.20)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests->ray[default]) (1.25.10)
    Collecting immutables>=0.9
      Downloading https://mirrors.aliyun.com/pypi/packages/4a/52/e64a14a99c509cbdfe0405e9f076aef0331cb9548a3efa1d5bacd524978a/immutables-0.15-cp36-cp36m-manylinux1_x86_64.whl (100 kB)
         |████████████████████████████████| 100 kB 18.2 MB/s 
    Using legacy 'setup.py install' for idna-ssl, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for gpustat, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for nvidia-ml-py3, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for contextvars, since package 'wheel' is not installed.
    Installing collected packages: pyasn1, setuptools, rsa, pyparsing, pyasn1-modules, protobuf, multidict, immutables, cachetools, yarl, packaging, idna-ssl, googleapis-common-protos, google-auth, contextvars, attrs, async-timeout, pyrsistent, psutil, opencensus-context, nvidia-ml-py3, hiredis, google-api-core, blessings, aiohttp, redis, pyyaml, pydantic, py-spy, prometheus-client, opencensus, numpy, msgpack, jsonschema, grpcio, gpustat, filelock, colorama, aioredis, aiohttp-cors, ray, colorful
      Attempting uninstall: setuptools
        Found existing installation: setuptools 39.2.0
        Uninstalling setuptools-39.2.0:
          Successfully uninstalled setuptools-39.2.0
        Running setup.py install for idna-ssl ... done
        Running setup.py install for contextvars ... done
        Running setup.py install for nvidia-ml-py3 ... done
        Running setup.py install for gpustat ... done
    Successfully installed aiohttp-3.7.4.post0 aiohttp-cors-0.7.0 aioredis-1.3.1 async-timeout-3.0.1 attrs-21.2.0 blessings-1.7 cachetools-4.2.2 colorama-0.4.4 colorful-0.5.4 contextvars-2.4 filelock-3.0.12 google-api-core-1.31.0 google-auth-1.33.1 googleapis-common-protos-1.53.0 gpustat-0.6.0 grpcio-1.39.0 hiredis-2.0.0 idna-ssl-1.1.0 immutables-0.15 jsonschema-3.2.0 msgpack-1.0.2 multidict-5.1.0 numpy-1.19.5 nvidia-ml-py3-7.352.0 opencensus-0.7.13 opencensus-context-0.1.2 packaging-21.0 prometheus-client-0.11.0 protobuf-3.17.3 psutil-5.8.0 py-spy-0.3.7 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydantic-1.8.2 pyparsing-2.4.7 pyrsistent-0.18.0 pyyaml-5.4.1 ray-1.4.1 redis-3.5.3 rsa-4.7.2 setuptools-57.4.0 yarl-1.6.3

    3、单机

    下面是不使用分布式的代码示例:

    # test_ray.py
    import ray
    import cv2
    import numpy
     
    import time
     
    #@ray.remote
    def test(img):
        orb = cv2.AKAZE_create()
        kb = orb.detect(img, None)
        kp, des = orb.compute(img, kp)
        return des
     
    # 不使用注解的方式
    test_remote = ray.remote(test)
     
    # 函数test 加注解时使用方式
    # test_remote = test.remote(test)
     
    img = cv2.imread("/tmp/test.jpg")
     
    start_time = time.time()
     
    for i in range(10):
        test(img)
     
    print(time.time() - start_time)

    运行:python test_ray.py

    4、集群

    如果需要启动集群模式,则需要先启动服务,选择一台机器作为主服务器,然后按虾米那命令启动:

    [root@node1 ~]# ray start --head --port=8888      
    Local node IP: 192.168.0.81
    2021-07-28 09:58:44,051 INFO services.py:1274 -- View the Ray dashboard at http://127.0.0.1:8265
     
    --------------------
    Ray runtime started.
    --------------------
     
    Next steps
      To connect to this Ray runtime from another node, run
        ray start --address='192.168.0.81:8888' --redis-password='5241590000000000'
      
      Alternatively, use the following Python code:
        import ray
        ray.init(address='auto', _redis_password='5241590000000000')
      
      If connection fails, check your firewall settings and network configuration.
      
      To terminate the Ray runtime, run
        ray stop

    可以看到里面提示具体的用法。此时通过ray的web界面,端口8265访问,看到已有一台机器。

    然后在另外两台机启动客户端节点,保证各个节点的ray版本一致,查看版本:

    # pip freeze |grep ray

    版本不一致的话,要升级为一致:pip install -U ray, 此处选择的版本为1.4.1

    从节点启动方式(密码可以在主节点启动时显示):

    ray start --address='192.168.0.81:8888' --redis-password='5241590000000000'

    改写程序,用于分布式:

    # test_ray.py
    import ray
    import cv2
    import numpy
     
    import time
     
    @ray.remote
    def test(img):
        orb = cv2.AKAZE_create()
        kb = orb.detect(img, None)
        kp, des = orb.compute(img, kp)
        return des
     
    # 不使用注解的方式
    #test_remote = ray.remote(test)
     
    # 函数test 加注解时使用方式
    test_remote = test.remote(test)
     
    if __name__ == "__main__":
        ray.init(address='auto', _redis_password='5241590000000000')
        start_time = time.time()
        img = cv2.imread("/tmp/test.jpg")
        futures = [test_remote.remote(img) for i in range(100)]
        ray.get(futures)
        start_time = time.time()
        print(time.time() - start_time)

    在主节点运行程序,程序会在3台机器上启动。停止ray服务的方式:ray stop.

    到此这篇关于支持python的分布式计算框架Ray的文章就介绍到这了,更多相关python的分布式计算框架Ray内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家!

    您可能感兴趣的文章:
    • python 常用的异步框架汇总整理
    • 5分钟快速掌握Python定时任务框架的实现
    • Python编程pydantic触发及访问错误处理
    上一篇:OpenCV霍夫圆变换cv2.HoughCircles()
    下一篇:python用户自定义异常的实例讲解
  • 相关文章
  • 

    © 2016-2020 巨人网络通讯 版权所有

    《增值电信业务经营许可证》 苏ICP备15040257号-8

    支持python的分布式计算框架Ray详解 支持,python,的,分布式,计算,