[MechCoder]

The densities in KernelDensity are scaled down by
(number of parallel processes X number of points). It should be just no.of samples. This results in broken tests in KernelDensitySuite which haven't been tested properly.

[MechCoder]

Note: In master

import org.apache.commons.math3.distribution.NormalDistribution
import org.apache.spark.mllib.stat.KernelDensity

val rdd = sc.parallelize(Array(5.0))
val evaluationPoints = Array(5.0, 6.0)
val densities = new KernelDensity().setSample(rdd).setBandwidth(3.0).estimate(evaluationPoints)
val normal = new NormalDistribution(5.0, 3.0)

densities(0) - normal.density(5.0)
res1: Double = -0.06649038006690546

val rdd = sc.parallelize(Array(5.0, 10.0))
val evaluationPoints = Array(5.0, 6.0)
val densities = new KernelDensity().setSample(rdd).setBandwidth(3.0).estimate(evaluationPoints)
val normal1 = new NormalDistribution(5.0, 3.0)
val normal2 = new NormalDistribution(10.0, 3.0)


densities(0) - (normal1.density(5.0) + normal2.density(5.0)) / 2
res2: Double = -0.04153495159951512

Hence the tests pass

cc @jkbradley @mengxr

[SparkQA]

Test build #33426 has finished for PR 6383 at commit a92fe50.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

Please aggregate densities and count in a single pass.

[MechCoder]

@mengxr fixed!

[MechCoder]

Also, we might probably want to generalize this n dimensions (Some code can be borrowed from the GaussianMixture ) and add a user guide. Is anyone actively working on these?

[SparkQA]

Test build #33501 has finished for PR 6383 at commit 9b8ed50.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

minor: Math is deprecated. Use math instead.

[mengxr]

LGTM. Also ping @sryza for verification.

[SparkQA]

Test build #33530 has finished for PR 6383 at commit ab81302.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

Merged into master and branch-1.4. Please update #6387 with a doctest. Thanks!