옛날 블로그

do you want some reshape in spark?well here it is in.groupBy("SALES_ID").pivot("SOMETHING").sum("FEATURE")

if you have two map type data collections, that have the same structure.just use ++ will

b.groupBy(identity).mapValues(s => s.size)

say we have string type array like below,val arrayy = Array("AA","BB","CCAB","AABA")We have to split into code that consists of just TWO characters.Using just a flatmap function will solve your problema.flatMap{ _.grouped(2) }

A.join(B, Seq("id"), "left").orderBy("id").groupBy("id", "name").agg(concat_ws(" ", collect_list("text")) as "texts") concat string into single column using agg collect_list returns array in that case,so if you want to use it inside the udf definition, you should take as Seq[String}

import breeze.linalg._ val df = spark.sparkContext.parallelize(List((1.0,3.0,5.0), (2.0,4.0,6.0))).toDF() val darr = df.collect() val da = darr.map{ row => Array(row.getAs[Double](0), row.getAs[Double](1), row.getAs[Double](2) )} val ncol = da.transpose.length val nrow = da.length val dd = da.transpose.flatten val dv = new DenseMatrix(nrow, ncol, dd) // val rdd1 = spark.sparkContext.parallelize(..

Spark Dataframe UDF - Schema for type Any is not supported When implementing udf (user defined function), don't forget to take care of all possible situation that can happen in the code block.when IF comes, remember that ELSE always follows.So simple, So Stupid.

When using udf, it is hard to understand what is happening inside the function, because udf() returns a function, not a value.The benefit of using function using udf() is, you can just declare code block inside udf() just like declaring function, and use your customer parameter (function type) when calling withColumn.This may help advanced programmer, but when it comes to customizing the functio..

아래처럼 만들면 된다. 참쉽쥬

출처http://docs.scala-lang.org/overviews/collections/performance-characteristics.htmlPerformance characteristics of sequence types: headtailapplyupdateprependappendinsertimmutable ListCCLLCL-StreamCCLLCL-VectoreCeCeCeCeCeC-StackCCLLCLLQueueaCaCLLCC-RangeCCC----StringCLCLLL-mutable ArrayBufferCLCCLaCLListBufferCLLLCCLStringBuilderCLCCLaCLMutableListCLLLCCLQueueCLLLCCLArraySeqCLCC---StackCLLLCLLArra..