通过对Spark并行计算框架以及投影树算法的分析,提出基于Spark的投影树频繁项集挖掘算法(projection tree algorithm based on Spark,PTBS)。为解决重复遍历数据集带来的损耗,改变数据的存储结构;利用广播变量对频繁二项集的产生进行改...通过对Spark并行计算框架以及投影树算法的分析,提出基于Spark的投影树频繁项集挖掘算法(projection tree algorithm based on Spark,PTBS)。为解决重复遍历数据集带来的损耗,改变数据的存储结构;利用广播变量对频繁二项集的产生进行改进,降低运算量,减少集群节点间通信的消耗;将Apriori先验性质添加到候选项集的筛选过程中,减少节点间的通信量。将算法与Spark平台所具备的特殊性能相结合,得到基于Spark的投影树关联规则挖掘算法,实验结果表明,该算法具备良好的可扩展性,适宜于处理大规模数据。展开更多
Frequent Pattern mining plays an essential role in data mining. Most of the previous studies adopt an Apriori-like candidate set generation-and-test approach. However, candidate set generation is still costly, especia...Frequent Pattern mining plays an essential role in data mining. Most of the previous studies adopt an Apriori-like candidate set generation-and-test approach. However, candidate set generation is still costly, especially when there exist prolific patterns and/or long patterns.In this study, we introduce a novel frequent pattern growth (FP-growth)method, which is efficient and scalable for mining both long and short frequent patterns without candidate generation. And build a new project frequent pattern growth (PFP-tree)algorithm on this study, which not only heirs all the advantages in the FP-growth method, but also avoids it's bottleneck in database size dependence. So increase algorithm's scalability efficiently.展开更多
文摘通过对Spark并行计算框架以及投影树算法的分析,提出基于Spark的投影树频繁项集挖掘算法(projection tree algorithm based on Spark,PTBS)。为解决重复遍历数据集带来的损耗,改变数据的存储结构;利用广播变量对频繁二项集的产生进行改进,降低运算量,减少集群节点间通信的消耗;将Apriori先验性质添加到候选项集的筛选过程中,减少节点间的通信量。将算法与Spark平台所具备的特殊性能相结合,得到基于Spark的投影树关联规则挖掘算法,实验结果表明,该算法具备良好的可扩展性,适宜于处理大规模数据。
文摘Frequent Pattern mining plays an essential role in data mining. Most of the previous studies adopt an Apriori-like candidate set generation-and-test approach. However, candidate set generation is still costly, especially when there exist prolific patterns and/or long patterns.In this study, we introduce a novel frequent pattern growth (FP-growth)method, which is efficient and scalable for mining both long and short frequent patterns without candidate generation. And build a new project frequent pattern growth (PFP-tree)algorithm on this study, which not only heirs all the advantages in the FP-growth method, but also avoids it's bottleneck in database size dependence. So increase algorithm's scalability efficiently.
