10 Star 57 Fork 23

HuaweiCloudDeveloper / huaweicloud-solution-ploto

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

huaweicloud-solution-ploto

PLOTO:自动驾驶研发平台,集合业界优秀伙伴提供专业的自动驾驶研发工具。
前端使用vue框架,后端采用Django框架。
为自动驾驶全链路提供可靠服务(数据采集、数据传输、数据脱敏、场景数据提取、数据标注、AI训练、仿真测试、评估评价、OTA升级全链路流程)。

组织结构

├── huaweicloud-solution-ploto --主目录
    ├── config
    ├── initutils			--初始化项目目录	
    ├── ploto				--后端项目目录
    |  ├── ae_trace			--事件告警功能目录
    |  ├── commmon			--通用函数工具
    |  ├── conf				--配置目录
    |  ├── data_mgt			--数据管理目录
    |  ├── logs				--日志文件目录
    |  ├── monitor			--大屏数据目录
    |  ├── ploto			--ploto应用目录
    |  ├── __init__.py		--模块声明文件
    |  ├── user				--用户管理目录
    |  ├── manage.py		--Django管理文件
    |  └── requirements.txt --python环境依赖列表
    ├── src					--前端源码目录			
    ├── public
    ├── vue.config.js
    └── README.md

架构图

image-20221010191916938

server环境配置

Python版本:Python 3.7及以上版本

运行环境:CentOS 7.6

环境安装

安装python 3.7

  • 安装python编译相关的依赖包
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make libffi-devel

yum -y install epel-release 
yum install python-pip
yum install -y python3-pip
pip install wget
  • 下载python3.7的源码
wget https://mirrors.huaweicloud.com/python/3.7.2/Python-3.7.2.tgz
  • 编译生成
# 解压缩
tar -zxvf Python-3.7.2.tgz
# 进入解压后的目录
cd Python-3.7.2
# 手动编译
./configure prefix=/usr/local/python3 
make && make install
  • 添加软连接
ln -s /usr/local/python3/bin/python3.7 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3.7 /usr/bin/pip3
  • 测试是否安装成功
[root@loaclhost ~]# python3 --version
Python 3.7.2
[root@loaclhost ~]# pip3 --version
pip 10.0.1 

环境依赖安装

  • cd到项目主目录/ploto下,找到requirements.txt,使用批量安装的方式安装python相关的环境依赖
pip3 install -r requirements.txt

功能组件安装

  • Redis:用于当作Monitor查询的缓存数据,Celery的任务队列,以及登录查询和告警相关功能

    推荐使用华为云分布式缓存服务DCS服务快速部署,操作指引参考https://support.huaweicloud.com/qs-dcs/index.html

  • Mysql,用于存储平台的用户、车辆、任务等相关信息

    推荐使用华为云数据库RDS快速部署,操作指引参考https://support.huaweicloud.com/qs-rds/zh-cn_topic_0046585334.html

  • ElasticSesrch:用于加速事件告警功能的查询

    推荐使用华为云搜索服务CSS,操作指引参考https://support.huaweicloud.com/qs-css/index.html#toTop

  • 安装node.js,用于在服务器运行前端环境

    下载文件并解压,推荐使用华为云镜像加速https://repo.huaweicloud.com/nodejs/

    wget https://repo.huaweicloud.com/nodejs/v16.16.0/node-v16.16.0-linux-arm64.tar.gz
    tar -xzf node-v16.16.0-linux-arm64.tar.gz

    设置软连接(根据实际安装的node路径修改),连接到/usr/local/bin/

    ln -s /root/node-v16.16.0-linux-x64/bin/npm /usr/local/bin/
    ln -s /root/node-v16.16.0-linux-x64/bin/node /usr/local/bin/

    在/root/目录下验证是否安装成功,打印出版本号则安装成功

    node -v
    npm -v
  • 安装nginx,用于转发前端收到的请求

    yum install nginx

前后端部署

1、前端部署

1.1 配置Node.js

  • 切换到主目录huaweicloud-solution-ploto执行

    npm install
  • 修改服务器地址信息

    在主目录下vue.config.js中host修改前端服务器所在的地址,端口默认为80

    devServer: {
        host: '127.0.0.1',  # 前端服务器地址,若为本机运行则使用本机地址
        port: '80',
        open: true
    }
  • 主目录下编译运行,生成dist文件(任何对前端文件的修改都需要重新编译生成)

    npm run build

1.2 配置Nginx

  • 若nginx安装路径为**/usr/local**目录下,修改 /usr/local/nginx/conf/nginx.conf配置文件,监听80端口的服务,**location/**中root对应build之后生成的dist文件目录,将前端的访问路径通过nginx代理转发到后端服务的8000端口

    server {
            listen       80;
            server_name  localhost;
            location / {
            	root  /root/huaweicloud-solution-ploto/dist/; # 修改为上一步生成的dist目录
            	index  index.html index.htm;
            }
            
            location /login{
            	proxy_pass http://localhost:8000;
          	}
    
          	location /monitor{
            	proxy_pass http://localhost:8000;
          	}
    
            location /data{
            	proxy_pass http://localhost:8000;
            }
    
            location /ADlogin{
            	proxy_pass http://localhost:8000;
            }
    
            location /logout{
            	proxy_pass http://localhost:8000;
            }
    
            location /admin{
            	proxy_pass http://localhost:8000;
            }
    
            error_page   500 502 503 504  /50x.html;
            	location = /50x.html {
            	root   html;
            }
    }
  • 启动或重启nginx使配置生效

    /usr/local/nginx/sbin/nginx start
    /usr/local/nginx/sbin/nginx -s reload

2、后端部署

2.1 配置连接信息

  1. ploto/ploto/settings/dev.py中填入数据库,Redis,ElasticSearch的连接信息

    其中DATABASES 为数据库配置信息,先在mysql中创建一个数据库ploto_test,作为项目使用的数据库,并将其配置参数添加至字段中

    # 数据库配置
    DATABASES = {
        "default": {
            "ENGINE": "django.db.backends.mysql",
            "NAME": "ploto_test",	# 创建的数据库名
            "USER": "root",			# 用户名默认为root		
            "PASSWORD": "****",		# 数据库登录密码
            "HOST": "127.0.0.1",	# 数据库服务器地址
            "PORT": "3306",			# 数据库连接端口,默认3306
        }
    }
    
    # ElasticSearch 配置
    ES_HOST = '127.0.0.1'			# CSS服务中的服务器地址
    ES_PORT = '9200'				# 服务器连接端口,默认9200
    ES_USERNAME = '*****'			
    ES_PASSWORD = '*****'
    
    # Redis 配置
    REDIS_HOST = '127.0.0.1'		# Redis服务器地址
    REDIS_PORT = '6379'				# Redis服务器连接端口,默认6379
    REDIS_PASSWORD = '*****'
  2. 配置IAM认证信息

    存放海量自动驾驶数据,推荐使用华为云对象存储服务OBS,操作指引https://support.huaweicloud.com/qs-obs/obs_qs_1000.html,同时华为云平台提供强大的模型训练平台ModelArts,并且提供了丰富的API可供用户调用。

    • 在华为云的控制台中的用户名下拉菜单中->“ 我的凭证”,在访问密钥中新增访问密钥,获得Access Key和Secret Key(请妥善保管并定期更新)

    • 在华为云的对象存储服务OBS中,创建一个桶(桶名为ploto-test)

    • ploto/conf/conf.py填写ak,sk,以及OBS的桶名bucket_name和终端节点endpoint,场景提取输出路径,IAM账号认证信息

obs_ak = "*****" 							# Access Key
obs_sk = "*****"						 	# Secret Key
obs_endpoint = "obs.XXX.myhuaweicloud.com"  # OBS终端节点地址
obs_bucket_name = ploto-test
# 场景提取输出路径,根据实际情况修改
scene_cut_output_dir = "obs://{}/ploto_file/scene".format(obs_bucket_name)

# IAM账号认证信息,用于模型训练相关操作
IAMDomain = "*********"
IAMUser = "*********"
IAMPassword = "*********"
IAMproject = "*********"

2.2 运行服务

  • 数据库迁移

切换到manage.py所在的目录下,通过终端启动数据库迁移服务

python3 manage.py makemigrations
python3 manage.py migrate
  • 创建超级用户,填入用户名与密码
python3 manage.py createsuperuser
  • 运行服务,指定运行在0.0.0.0,端口为8000,
python3 manage.py runserver 0.0.0.0:8000
  • 创建账户与登录

    本地访问 http://0.0.0.0:8000/admin/auth/user/,若在服务器外访问则使用服务器地址,使用刚刚创建的超级用户登录Django后台

  • 创建用户ploto_test,在用户权限中给予相应权限;

  • 增加Ploto users,选择新建的用户ploto_test,保存后该用户即被激活(不进行此操作无法登录),并给用户配置相应权限

  • 通过 http://0.0.0.0:8000/ 访问系统页面,或者通过服务器地址进行访问

  • 创建ES索引

    python3 manage.py serach_index --rebuild
  • 启动Celery的异步任务

    celery -A ploto worker -l info -P eventlet
  • 启动Celery定时任务

    celery -A ploto beat
  • 可以运行initutils工具类中的脚本对平台数据进行填充、删除

    fill_obs.py:对obs的填充测试数据

    fill_db.py : 对数据库进行填充测试数据

    delete_test_data.py:删除测试数据

2.3 uwsgi托管启动

使用python3的启动方式适合进行开发调试,log信息直接打印在控制台,上线运行可选择用uWSGI托管Django,log记录在uwsgi.log中,不输出至终端。

先安装uwsgi模块:

pip3 install uwsgi

主目录/ploto文件夹下创建uwsgi.ini文件,文件内容如下:

[uwsgi]
# 应用目录
chdir = /usr/local/huaweicloud-solution-ploto/ploto
# 服务访问链接
http = 0.0.0.0:8000
# wsgi文件目录
wsgi-file = /usr/local/huaweicloud-solution-ploto/ploto/wsgi.py
# 工作进程数
processes = 4
# 启动主进程来管理其他子进程
master = true
http-timeout = 60
# 退出时清空环境
vacuum = true
# 指定pid文件
pidfile = /usr/local/huaweicloud-solution-ploto/ploto/uwsgi.pid
# log文件路径
daemonize = /usr/local/huaweicloud-solution-ploto/ploto/uwsgi.log
# 加载wsgi模块
module=ploto.wsgi:application

配置完成后cd到uwsgi.ini所在文件夹,使用以下命令启动或停止

uwsgi --ini uwsgi.ini # 启动
uwsgi --stop uwsgi.pid # 停止
uwsgi --reload uwsgi.pid # 重启
killall -8 uwsgi # 强制停止

加密信息配置

如果需要对密码明文的敏感信息进行加密,可参考本小结实现加密。

对密码明文信息进行加密,并实现密钥密文分离,使用AEC_GCM加密算法,以数据库密码加密为例:

  • 加密算法于ploto/common/common_crypte.py
  • 密钥存放于ploto/conf/conf.py
  • 密文存放于ploto/ploto/dev.py

生成密钥与密文

  • 生成加密密钥aes_gcm_key AES GCM加密算法的秘钥配置,秘钥长度32字节,填写密钥ascii码的base64格式字符串。可以使用以下代码生成加密密钥,把获取的aes_gcm_key填写到ploto/conf/conf.py中。建议周期性替换秘钥,修改密码。

    import binascii
    from Crypto import Random
    aes_gcm_key = binascii.b2a_base64(Random.get_random_bytes(32))
  • 生成密文信息

    在ploto/common/common_crypt.py中,填入“密码明文”,使用之前生成的aes_gcm_key进行加密,输出加密后的密文信息(bytes格式),使用后删除明文信息

    from conf.conf import aes_gcm_key 		# 加密密钥
    mysql_pwd, mysql_aes_gcm_tag, mysql_aes_gcm_iv = encrypt_aes256gcm("密码明文", key=aes_gcm_key, tag_len=16)

    输出:

    crypt_text: b'*****' #加密密文
    tag: b'*****'		 #密文校验信息
    iv: b'*****'		 #随机初始向量	

数据库配置

ploto/ploto/dev.py中数据库连接配置可参考以下方式

import sys
sys.path.append("..")
from common import common_crypte
mysql_pwd: 	b"*****"						 # 密码密文
mysql_aes_gcm_tagb"*****"					# 密文校验信息
mysql_aes_gcm_iv b"*****"					# 随机初始向量

# 数据库配置
DATABASES = {
  "default": {
      "ENGINE": "django.db.backends.mysql",
      "NAME": "ploto_test",
      "USER": "root",
      # 调用解密函数获得密码
      "PASSWORD": common_crypte.decrypt_aes256gcm(mysql_pwd, aes_gcm_key, 																mysql_aes_gcm_iv, mysql_aes_gcm_tag),
      "HOST": "127.0.0.1",
      "PORT": "3306",
  },

数据可视化

部署Webviz工具,针对ros打包的bag格式数据在Web端进行可视化预览,并可实现远端数据在线预览。

1、部署

推荐使用Docker镜像部署,下载官方镜像后运行

docker run -p 8080:8080 cruise/webviz

源码部署方式参考官方教程:https://github.com/cruise-automation/webviz

2、访问

2.1 主服务链接

部署后浏览器访问http://127.0.0.1:8080即可使用服务,可将本地bag文件拖拽进页面预览,支持读取3D点云数据,图像数据,以及数据Plot等功能;

部署在远程主机上则在主目录/ploto/conf/conf.py中配置Webviz服务地址

webviz_url = 'http://localhost:8080' # webviz服务路径
rosbridge_url = 'ws://localhost:9090' #(可选)rosbridge 链接
layout_key = 'ploto_file/layout.json' #(可选)自定义layout在桶中的对象key

2.2 访问远程数据

支持通过授权链接访问远程数据,以OBS为例,需要配置桶的CORS规则:

配置项 参数
允许的来源 *
允许的方法 GET, HEAD
允许的头域 *
补充头域 ETag
Content-Type
Content-Length
Cache-Control
Content-Disposition
Content-Encoding
Content-Language
Expires
x-obs-request-id
Accept-Ranges
缓存时间 100

获取桶内数据的带授权链接,可参考OBS的SDK(https://support.huaweicloud.com/sdk-python-devg-obs/obs_22_1301.html),或使用OBS Browser+进行分享,在服务链接后带参数“remote-bag-url=授权链接”,如

http://127.0.0.1:8080?/remote-bag-url=http://remote-bag-url.com

注意:授权链接需要进行urlEncode编码,对部分特殊字符(如&)转义

2.3 布局管理

支持布局的保存和读取,可对这个页面或单个panel板块的布局进行保存或导入,生成layout.json文件,也可访问桶内的布局文件,需携带参数"layout-url"在访问链接中,如

http://127.0.0.1:8080?/remote-bag-url=http://remote-bag-url.com&layout-url=http://layout_json.com

更多工具相关功能参考官方链接或服务页面的帮助文档。

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

PLOTO:自动驾驶研发平台,集合业界优秀伙伴提供专业的自动驾驶研发工具。 前端使用vue框架,后端采用Django框架。 为自动驾驶全链路提供可靠服务(数据采集、数据传输、数据脱敏、场景数据提取、数据标注、AI训练、仿真测试、评估评价、OTA升级全链路流程)。 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
1
https://gitee.com/HuaweiCloudDeveloper/huaweicloud-solution-ploto.git
git@gitee.com:HuaweiCloudDeveloper/huaweicloud-solution-ploto.git
HuaweiCloudDeveloper
huaweicloud-solution-ploto
huaweicloud-solution-ploto
master

搜索帮助

14c37bed 8189591 565d56ea 8189591