1 Star 0 Fork 23

没什么好笑的 / xl-lighthouse

forked from xl-xueling / xl-lighthouse 
加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

A general-purpose streaming big data statistics system.
Easier to use, supports a larger amount of data, and can complete more statistical indicators faster.

LICENSE Language Language GitHub release GitHub release date


一键部署,一行代码接入,无需大数据研发运维经验,轻松驾驭海量数据实时统计。

使用XL-LightHouse后:

  • 1、再也不需要用Flink、Spark、ClickHouse或者基于Redis这种臃肿笨重的方案跑数了;
  • 2、再也不需要疲于应付对个人价值提升没有多大益处的数据统计需求了,能够帮助您从琐碎反复的数据统计需求中抽身出来,从而专注于对个人提升、对企业发展更有价值的事情;
  • 3、轻松帮您实现任意细粒度的监控指标,是您监控服务运行状况,排查各类业务数据波动、指标异常类问题的好帮手;
  • 4、培养数据思维,辅助您将所从事的工作建立数据指标体系,量化工作产出,做专业严谨的职场人,创造更大的个人价值;

概述

  • XL-LightHouse是针对互联网领域繁杂的流式数据统计需求而开发的一套集成了数据写入、数据运算、数据存储和数据可视化等一系列功能,支持超大数据量,支持超高并发的【通用型流式大数据统计平台】。
  • XL-LightHouse目前已涵盖了各种流式数据统计场景,包括count、sum、max、min、avg、distinct、topN/lastN等多种运算,支持多维度计算,支持分钟级、小时级、天级多个时间粒度的统计,支持自定义统计周期的配置。
  • XL-LightHouse内置丰富的转化类函数、支持表达式解析,可以满足各种复杂的条件筛选和逻辑判断。
  • XL-LightHouse是一套功能完备的流式大数据统计领域的数据治理解决方案,它提供了比较友好和完善的可视化查询功能,并对外提供API查询接口,此外还包括数据指标管理、权限管理、统计限流等多种功能。
  • XL-LightHouse支持时序性数据的存储和查询。

产品优势

  • XL-LightHouse面向企业繁杂的流式数据统计需求,可以帮助企业在极短时间内快速实现数以万计、数十万计的数据指标,而这是Flink、ClickHouse之类技术所远不能比拟的,XL-LightHouse帮助企业低成本实现数据化运营,数据指标体系可遍布企业运转的方方面面;
  • 对单个流式统计场景的数据量无限制,可以非常庞大,也可以非常稀少,您既可以使用它完成十亿级用户量APP的DAU统计、十几万台服务器的运维监控、一线互联网大厂数据量级的日志统计、也可以用它来统计一天只有零星几次的接口调用量、耗时状况;
  • 支持高并发查询统计结果;
  • 一键部署、一行代码接入,无需专业的大数据研发人员,普通工程人员就可以轻松驾驭;
  • 有完善的数据指标可视化以及数据指标管理维护等功能;

XL-LightHouse与Flink和ClickHouse之类技术对比

收益

XL-LightHouse可以帮助企业以尽可能低的成本,更快速的搭建起一套较为完善的、稳定可靠的数据化运营体系,节省企业在数据化运营方面的投入,主要体现在以下几个方面:

  • 减少企业在流式大数据统计方面的研发成本和数据维护成本。
  • 帮助企业节省时间成本,辅助互联网产品的快速迭代。
  • 为企业节省较为可观的服务器运算资源。
  • 便于数据在企业内部的共享和互通。

此外,XL-LightHouse对中小企业友好,它大大降低了中小企业使用流式大数据统计的技术门槛,通过简单的页面配置和数据接入即可应对繁杂的流式数据统计需求。

一键部署

Hello World 使用范例

完整版使用示例请查阅:HelloWorld

范例一:首页ICON区域用户行为数据统计

该区域包含3个Tab,每个Tab有多个业务ICON图标,用户手动滑动可切换Tab,假设针对该ICON区域我们有如下数据指标需求:

点击量:
1、每5分钟_点击量
2、每5分钟_各ICON_点击量
3、每小时_点击量
4、每小时_各ICON_点击量
5、每天_总点击量
6、每天_各Tab_总点击量
7、每天_各ICON_总点击量

点击UV:
1、每5分钟_点击UV
2、每小时_点击UV
3、每小时_各ICON_点击UV
4、每天_总点击UV
5、每天_各ICON_总点击UV
  • 定义元数据结构:
字段 字段类型 描述
user_id string 用户标识
tab_id string Tab栏 tab1、tab2、tab3
icon_id string 美食团购、酒店民宿、休闲玩乐、打车 ...
  • 上报元数据时机

用户点击ICON图标时上报相应埋点数据

  • 配置统计项
  • 查看统计结果
范例二:移动支付订单数据统计
1、 支付成功订单数据统计
  • 统计需求梳理
订单量:
1、每10分钟_订单量
2、每10分钟_各商户_订单量
3、每10分钟_各省份_订单量
4、每10分钟_各城市_订单量
5、每小时_订单量
6、每天_订单量
7、每天_各商户_订单量
8、每天_各省份_订单量
9、每天_各城市_订单量
10、每天_各价格区间_订单量
11、每天_各应用场景_订单量

交易金额:
1、每10分钟_成交金额
2、每10分钟_各商户_成交金额top100
3、每10分钟_各省份_成交金额
4、每10分钟_各城市_成交金额
5、每小时_成交金额
6、每小时_各商户_成交金额
7、每天_成交金额
8、每天_各商户_成交金额
9、每天_各省份_成交金额
10、每天_各城市_成交金额
11、每天_各应用场景_成交金额

下单用户数:
1、每10分钟_下单用户数
2、每10分钟_各商户_下单用户数
3、每10分钟_各省份_下单用户数
4、每10分钟_各城市_下单用户数
5、每小时_下单用户数
6、每天_下单用户数
7、每天_各商户_下单用户数
8、每天_各省份_下单用户数
9、每天_各城市_下单用户数
10、每天_各价格区间_下单用户数
11、每天_各应用场景_下单用户数
  • 定义元数据
字段 字段类型 描述
userId string 用户ID
orderId string 订单ID
province string 用户所在省份
city string 用户所在城市
dealerId string 商户ID
scene string 支付场景 电商、外卖、餐饮、娱乐、游戏 ...
amount numeric 订单金额
  • 消息上报时机

用户支付成功后上报原始消息数据。

  • 配置统计消息
2、 订单支付状态数据监控

我这里假设订单有四种状态:支付成功、支付失败、超时未支付、订单取消。

订单量:
1、每10分钟_各状态_订单量
2、每10分钟_各商户_各状态_订单量
1、每天_各状态_订单量
2、每天_各商户_各状态_订单量

订单异常率:
1、每10分钟_订单异常率
2、每10分钟_各商户_订单异常率
3、每小时_订单异常率
4、每天_订单异常率
5、每天_各商户_订单异常率

支付失败用户数统计:
1、每5分钟_支付失败用户数
  • 定义元数据
字段 字段类型 描述
userId string 用户ID
province string 用户所在省份
city string 用户所在城市
dealerId string 商户ID
orderId string 订单ID
state string 订单支付状态 1:支付成功、2:支付失败、3:超时未支付 4:订单取消
  • 配置统计项
  • 查看统计结果

更多适用场景举例

版权声明

为保障创作者的合法权益以及支持XL-LightHouse项目的发展,本项目在Apache2.0开源协议的基础上,增加如下补充条款,如果以下条款与Apache2.0协议内容有所冲突,以该补充条款为准。

  • 1、企业或机构内部使用XL-LightHouse源程序不受任何限制,但不可删除源程序中的版权声明等信息。
  • 2、企业、机构或个人销售XL-LightHouse相关软硬件产品或服务需向原作者支付一定比例的授权费用。所述的“服务”指为购买者提供统计数据类服务或提供相应产品的技术支持维护服务,在销售相关产品或服务前请您查阅【版权声明】。

相关文档

1、项目介绍
2、Git地址
3、交流社区
4、项目设计
5、一键部署
6、XL-Formula使用
7、Web服务操作说明
8、Hello World
9、适用场景
10、版权声明
11、使用反馈
12、依赖组件
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

XL-LightHouse是一套支持大数据量、支持超高并发的通用型流式大数据统计平台,常见的应用场景比如:PV、UV统计,电商销售额统计、日志数据统计、接口调用量、耗时情况等统计,支持多维度统计,支持各种复杂的条件筛选和逻辑判断,一键部署,一行代码接入,轻松实现各种海量数据实时统计,帮助企业以更低的成本快速搭建起数据指标体系,是企业降本增效的好帮手! 展开 收起
Java
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Java
1
https://gitee.com/xintao/xl-lighthouse.git
git@gitee.com:xintao/xl-lighthouse.git
xintao
xl-lighthouse
xl-lighthouse
master

搜索帮助