加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

该仓库未声明开源许可证文件（LICENSE），使用请关注具体项目描述及其代码上游依赖。

克隆/下载

Spark实战.md 11.33 KB

# Spark实战进阶文章合集

1. [如果你在准备面试，好好看看这130道题](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486986&idx=1&sn=422d1a3c11c72ff97b32cc01142839f4&chksm=fd3d489fca4ac1895242ab94b932b12c65dc57b5f3a16acc7084dc8a189e9026290245a64c4f&token=1999457569&lang=zh_CN#rd)
2. [ORC文件存储格式的深入探究](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486981&idx=1&sn=9c8fc4c127d7e6108ac4e171e750d490&chksm=fd3d4890ca4ac186614f0dda8ffb2d35693a925b03861a01769c898652b53d0d436bca05ea12&token=1999457569&lang=zh_CN#rd)
3. [基于SparkStreaming+Kafka+HBase实时点击流案例](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486938&idx=1&sn=83347b444fc721442d2b4e1a58eca0e8&chksm=fd3d4b4fca4ac2597abd4a39a21dea83220858ae5efd1ae287b471bb9970f165b4854426f4c2&token=1999457569&lang=zh_CN#rd)
4. [HyperLogLog函数在Spark中的高级应用](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486931&idx=1&sn=9c3f3d6a677ed2aa6cc508046ca6da78&chksm=fd3d4b46ca4ac25044d6033458d3b43e7d1f50f447bded4eed8b246991cf620c91919c51e35b&token=1999457569&lang=zh_CN#rd)
5. [我们常说的海量小文件的根源是什么？](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486931&idx=2&sn=1c5987f3bad7a805895484ebfd683e11&chksm=fd3d4b46ca4ac250d31502a76f0cfd02ebea29d1161b9e9faec23ea6aa8947d0dd7d4268427f&token=1999457569&lang=zh_CN#rd)
6. [Structured Streaming | Apache Spark中处理实时数据的声明式API](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486904&idx=1&sn=5f4b673a87497a9c1dc9d7ce253994f3&chksm=fd3d4b2dca4ac23b9850c54d62ebe8be5920cbd4a491a46a0325e1fbbd91f6b1aef9adf2f638&token=1999457569&lang=zh_CN#rd)
7. [Spark面对OOM问题的解决方法及优化总结](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486765&idx=1&sn=516a32e8c1e9842606a7670862ec7e97&chksm=fd3d4bb8ca4ac2ae24e315083cdb195fdf897e1c3cce69d2c94183d5c20c1af1bfbe0e5480fd&token=1999457569&lang=zh_CN#rd)
8. [Spark 动态资源分配(Dynamic Resource Allocation) 解析](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486761&idx=1&sn=959aaa5266307a64181631ba1ae46e86&chksm=fd3d4bbcca4ac2aad8a3958da40c22e565997180c635bb6c32a9b4aa7e82a74a56423320c65a&token=1999457569&lang=zh_CN#rd)
9. [Apache Spark在海致大数据平台中的优化实践](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486668&idx=1&sn=981028fdcf937ba45b3914e2d48b94db&chksm=fd3d4a59ca4ac34f089ff1dce9542a0f481a73439acec9223fa2d776ff3f2eaec498cc543d06&token=1999457569&lang=zh_CN#rd)
10. [Spark/Flink广播实现作业配置动态更新](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486644&idx=1&sn=d2637a1e918c2b1be4c9fe3d74f75a92&chksm=fd3d4a21ca4ac3377cc8836939cc041cf934bb57f73b6b618fd1de608495d86e278c1c7e4cdc&token=1999457569&lang=zh_CN#rd)
11. [Spark SQL读数据库时不支持某些数据类型的问题](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486629&idx=2&sn=54d05858d659756a46b9be36b7b63ee5&chksm=fd3d4a30ca4ac326f69ba9b7cf6a82418afd31193e96aa28346779264a7d8a1df4c7a3a31fba&token=1999457569&lang=zh_CN#rd)
12. [这个面试问题很难么 | 如何处理大数据中的数据倾斜](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486586&idx=1&sn=e35def6429adaada0910b91eed8d7b1f&chksm=fd3d4aefca4ac3f9918d57b947d4bc8f71afd9d7666ffe28e3660e5efab332c714bef0879a90&token=1999457569&lang=zh_CN#rd)
13. [Spark难点 | Join的实现原理](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486582&idx=1&sn=7b6291dedb2e6892342e1ed705bdfb2e&chksm=fd3d4ae3ca4ac3f591f297635c0ff8e63e3deb6d6aec72c8fb18e19f8f6fcdb5cd2abcd37d09&token=1999457569&lang=zh_CN#rd)
14. [面试注意点 | Spark&Flink的区别拾遗](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486577&idx=1&sn=49fd0138ad9837192b6eb78cbdc478e6&chksm=fd3d4ae4ca4ac3f2aec91f40435d1eb55a9e497fe7d8ab9ad52396838f467879ac56cb83c9c2&token=1999457569&lang=zh_CN#rd)
15. [Spark Checkpoint的运行原理和源码实现](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486514&idx=1&sn=a3c3084b8a797a0889d7583fb10d70c9&chksm=fd3d4aa7ca4ac3b12416ba8269041032a515b7a4e71bda1709e359a15a04df4f638738c6e01a&token=1999457569&lang=zh_CN#rd)
16. [阿里云Spark Shuffle的优化](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486505&idx=1&sn=91316d3aa5a99945ccd992c0a98e500d&chksm=fd3d4abcca4ac3aa26f051504244239ff1ab48eb71cf01c14f689eb4767c121b1477d93b60ea&token=1999457569&lang=zh_CN#rd)
17. [使用Kafka+Spark+Cassandra构建实时处理引擎](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486485&idx=1&sn=999b84e0ac87f2aa1be4870921279a21&chksm=fd3d4a80ca4ac39695d9307582ff58938180d0c5b12f6db84ddae933edb345b0cbe46f6284ed&token=1999457569&lang=zh_CN#rd)
18. [基于HBase和Spark构建企业级数据处理平台](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486477&idx=1&sn=21394cca8fe279bbc2032d48f65672f6&chksm=fd3d4a98ca4ac38e3bf9700cfe65131fffadbbf9493ccc658b3e7be97ddb6665ad41a70f67f9&token=1999457569&lang=zh_CN#rd)
19. [SparkSQL在字节跳动的应用实践和优化实战](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486419&idx=1&sn=3bc8af144370a602817ca87415fe525d&chksm=fd3d4d46ca4ac450ec6f1ac84b2f3162071eb85221464bdb1976178ca2363cd2acb3d63d401c&token=1999457569&lang=zh_CN#rd)
20. [SparkRDD转DataSet/DataFrame的一个深坑](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486369&idx=1&sn=9a760114ed2c7509e191ad29370eecce&chksm=fd3d4d34ca4ac4222d310c37fb67bcaafd5425e1a16c43434c4d1fea403756e266de21b83181&token=1999457569&lang=zh_CN#rd)
21. [Spark和Flink的状态管理State的区别和应用](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486332&idx=1&sn=ecbe21981c5c36f6c420755e8d63fb8f&chksm=fd3d4de9ca4ac4ff1b11093432f444c9e4f66c45a6e01155d69cface70456fb418a98163757d&token=1999457569&lang=zh_CN#rd)
22. [Kafka+Spark Streaming管理offset的几种方法](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486328&idx=1&sn=b5d53e0007032114fb277e440e5ce4bf&chksm=fd3d4dedca4ac4fbb5026338af7dbbfe1d845fdc11e7e2279035dddc8ff0b6dd24c32be129ce&token=1999457569&lang=zh_CN#rd)
23. [从 PageRank Example谈Spark应用程序调优](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247486322&idx=2&sn=00ddcd16109249e45a70233d5ef959ba&chksm=fd3d4de7ca4ac4f15f85d9a2873c5d1070af3bb929479bd66f7a0dc89ac0777b6960cbce5970&token=1999457569&lang=zh_CN#rd)
24. [Spark调优|SparkSQL参数调优](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485574&idx=1&sn=723c02562ee1f44e88c389d6ac8a2c87&chksm=fd3d4e13ca4ac705ad892ada2c68792c906946f271bb1e6afd15a93749f8f87d0ea967ba2d37&token=1999457569&lang=zh_CN#rd)
25. [Flink/Spark 如何实现动态更新作业配置](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485507&idx=1&sn=2cab7a3714ce4e16351b8edcde95e777&chksm=fd3d4ed6ca4ac7c058d820f36b2d03c3ccddc293e1035b6897e99544e55931f6cd6a2f343647&token=1999457569&lang=zh_CN#rd)
26. [Stream SQL的执行原理与Flink的实现](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485361&idx=1&sn=8348203b6f17662a64fa5d412de97296&chksm=fd3d4124ca4ac8329bac00fae705f37c08236d583e9900f268796d463be9323d82ba4b8b1554&token=1999457569&lang=zh_CN#rd)
27. [Spark将Dataframe数据写入Hive分区表的方案](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485321&idx=1&sn=13e73673fb29bd134ab79e03a369288c&chksm=fd3d411cca4ac80ac68204d18aee55b95cadde7bf18cc2f8a549fc0b6ff4ec4db570190566c3&token=1999457569&lang=zh_CN#rd)
28. [Spark中几种ShuffleWriter的区别你都知道吗？](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485316&idx=1&sn=8a7a02023f15324885de7a5c93d4dd94&chksm=fd3d4111ca4ac807f34e1fa03023494d46f3770c035290ddefe1333419453a9258f504584aee&token=1999457569&lang=zh_CN#rd)
29. [SparkSQL的3种Join实现](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485217&idx=1&sn=3ce9fa8ad179c008754873129e51fbe7&chksm=fd3d41b4ca4ac8a25957fc9541437e6546fc2926df2908bad5adbd30a6cecf9f797fef74f894&token=1999457569&lang=zh_CN#rd)
30. [周期性清除Spark Streaming流状态的方法](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485138&idx=1&sn=8f71070470c8963e7c973b5f10bf3c03&chksm=fd3d4047ca4ac951981f7f0fa08f9f6a1821270441b5008da28115b67020b01ef2936b5f1e88&token=1999457569&lang=zh_CN#rd)
31. [Structured Streaming之状态存储解析](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485119&idx=1&sn=fd172b1f9c9ef99eac2ed976a7d4459f&chksm=fd3d402aca4ac93ced929dfb9b3d785fa5e00d82939b4ea7ed31e68096b87f31a0d11c5f7414&token=1999457569&lang=zh_CN#rd)
32. [Spark SQL重点知识总结](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485105&idx=1&sn=0bea228e6845d04739937b75bd2f8d9a&chksm=fd3d4024ca4ac9322cae55569b7bdc6d8546dc9c4583f25801d3820ab3b30094c50816c7c1c8&token=1999457569&lang=zh_CN#rd)
33. [SparkSQL极简入门](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485058&idx=1&sn=4d3b5c25ca1fdf1f0fb0cd99959d2371&chksm=fd3d4017ca4ac90140442dfc6032346d5841d6705bd92441ad3d0a0366db346752a6b154b976&token=1999457569&lang=zh_CN#rd)
34. [Spark Shuffle在网易的优化](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247485051&idx=1&sn=a1a70cad450634ceae44d4e14a4fc3ef&chksm=fd3d40eeca4ac9f8901a93271683811da05c62e7a6f1d2825378c32542ef2edf9947273a601a&token=1999457569&lang=zh_CN#rd)
35. [广告点击数实时统计：Spark StructuredStreaming + Redis Streams](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247484999&idx=1&sn=f9cf6eae39bc1d54faaa144357731d2f&chksm=fd3d40d2ca4ac9c4e886bd0208521c45dbab98e9cfd34826e6808d414bcd66f203f0359382e1&token=1999457569&lang=zh_CN#rd)
36. [Spark内存调优](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247484972&idx=1&sn=ff9a2925c31e07b558504be17937872b&chksm=fd3d40b9ca4ac9afe2cce8ff4a6c50724a146b87d2d05d87d6dcdfb34a177f3be4d5ba79b492&token=1999457569&lang=zh_CN#rd)
37. [Structured Streaming 实现思路与实现概述](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247484959&idx=1&sn=2173f71a32e16b510f047fa716549bc2&chksm=fd3d408aca4ac99cef4ec3079d9c7376c14f457e80f814e1494f5324612b553fb5487783a486&token=1999457569&lang=zh_CN#rd)
38. [Spark之数据倾斜调优](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247484956&idx=1&sn=9182a40fcf1fced04acee81aa9261bfe&chksm=fd3d4089ca4ac99f23952f0d627db4600a81808d98b1a635ae40e06c939e4a229a47d666de47&token=1999457569&lang=zh_CN#rd)
39. [你不得不知道的知识-零拷贝](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247484766&idx=1&sn=8d0aeaa1166a9338df9f28bb47959f4a&chksm=fd3d43cbca4acadd0dfc9e753ca4fe1cc2ca49060d886b359bdf3537809015b04f82b4be27ac&token=1999457569&lang=zh_CN#rd)
40. [Spark Streaming消费Kafka数据的两种方案](https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247484751&idx=1&sn=11315f599b39eac96c17a78da2fa1258&chksm=fd3d43daca4acaccc624947f5fa84f650e7f61d8638feda67db07cacf51bc985faae74cc94c3&token=1999457569&lang=zh_CN#rd)
41. [Spark Shuffle在网易的优化](https://mp.weixin.qq.com/s/1WNda357-UtClsVbwWvDfA)
42. [SparkSQL极简入门](https://mp.weixin.qq.com/s/r6s3a4hqAjxsrRy49aNS_w)
43. [Spark SQL重点知识总结](https://mp.weixin.qq.com/s/MFWziDtpIjRBXkdgoNQ3Fw)
44. [Apache Spark3.0什么样？一文读懂Apache Spark最新技术发展与展望](https://mp.weixin.qq.com/s/rOzg23blA6efKLAcMZ18nw)
45. [Flink/Spark 如何实现动态更新作业配置](https://mp.weixin.qq.com/s/sjRV_F9tXEfqKL_00rJc7w)
46. [[Spark调优 | Spark SQL参数调优](https://mp.weixin.qq.com/s/jkncoXo5fn_GQtp-eImFiA)]

一键复制编辑原始数据按行查看历史

提交于 2020-11-16 20:16 . 新增实战系列

Spark实战进阶文章合集

Java

1

https://gitee.com/coder_wz/God-Of-BigData.git

git@gitee.com:coder_wz/God-Of-BigData.git

coder_wz

God-Of-BigData

God-Of-BigData

master