옛날 블로그

say we have string type array like below,val arrayy = Array("AA","BB","CCAB","AABA")We have to split into code that consists of just TWO characters.Using just a flatmap function will solve your problema.flatMap{ _.grouped(2) }

A.join(B, Seq("id"), "left").orderBy("id").groupBy("id", "name").agg(concat_ws(" ", collect_list("text")) as "texts") concat string into single column using agg collect_list returns array in that case,so if you want to use it inside the udf definition, you should take as Seq[String}

Linde–Buzo–Graykmeans와 이진분류의 강점만 가져왔다고 하는데 실무에서 써본적도 본적도 없다.논문은 돈을 내야 보인다.

import breeze.linalg._ val df = spark.sparkContext.parallelize(List((1.0,3.0,5.0), (2.0,4.0,6.0))).toDF() val darr = df.collect() val da = darr.map{ row => Array(row.getAs[Double](0), row.getAs[Double](1), row.getAs[Double](2) )} val ncol = da.transpose.length val nrow = da.length val dd = da.transpose.flatten val dv = new DenseMatrix(nrow, ncol, dd) // val rdd1 = spark.sparkContext.parallelize(..

Spark Dataframe UDF - Schema for type Any is not supported When implementing udf (user defined function), don't forget to take care of all possible situation that can happen in the code block.when IF comes, remember that ELSE always follows.So simple, So Stupid.

When using udf, it is hard to understand what is happening inside the function, because udf() returns a function, not a value.The benefit of using function using udf() is, you can just declare code block inside udf() just like declaring function, and use your customer parameter (function type) when calling withColumn.This may help advanced programmer, but when it comes to customizing the functio..

if [ -n "$1" ]; thenecho "$1"elseecho "else"fi

sudo service cron reload/etc/init.d/cron reloadubuntu sudo service cron restart

pip installation fails: Double requirement given: numpy==1.12.1..ddpip install 'pandas < 0.21'로 설치

아래처럼 만들면 된다. 참쉽쥬