Geo spatial interfaces by pranavbhole · Pull Request #16029 · apache/druid

pranavbhole · 2024-03-02T04:28:39Z

Description

This PR creates an interface for ImmutableRTree and moved the existing implementation to new class which represent 32 bit implementation (stores coordinate as floats). This PR makes the ImmutableRTree extendable to create higher precision implementation as well (64 bit).
In all spatial bound filters, we accept float as input which might not be accurate in the case of high precision implementation of ImmutableRTree. This PR changed the bound filters to accepts the query bounds as double instead of float and it is backward compatible change as it compares double to existing float values in RTree. Previously it was comparing input float to RTree floats which can cause precision loss, now it is little better as it compares double to float which is still not 100% accurate.
There are no changes in the way that we query spatial dimension today except input bound parsing. There is little improvement in string filter predicate which now parse double strings instead of float and compares double to double which is 100% accurate but string predicate is only called when we dont have spatial index.
With allowing the interface to extend ImmutableRTree, we allow to create high precision (HP) implementation and defines new search strategies to perform HP search Iterable<ImmutableBitmap> search(ImmutableDoubleNode node, Bound bound);
With possible HP implementations, Radius bound filter can not really focus on accuracy, it is calculating Euclidean distance in comparing. As EARTH 🌍 is round and not flat, Euclidean distances are not accurate in geo system. This PR adds new param called 'radiusUnit' which allows you to specify units like meters, km, miles etc. It uses https://en.wikipedia.org/wiki/Haversine_formula to check if given geo point falls inside circle or not. Added a test that generates set of points inside and outside in RadiusBoundTest.

PR is fully backward compatible. Let me know if you have any questions.

Release Notes:
Adding radiusUnit in Radius Bound filter. radiusUnit : String value of radius unit in lowercase, default value is 'euclidean'. Allowed units are euclidean, meters, miles, kilometers.

This PR has:

+      Assert.assertFalse(bound.containsDouble(geoPoint));
+      float[] floatPoint = new float[]{
+          Float.parseFloat(String.valueOf(geoPoint[0])), // this is how legacy sptial index
+          Float.parseFloat(String.valueOf(geoPoint[1]))


pranavbhole · 2024-03-06T06:20:13Z

Can you please take look at this?
@clintropolis @gianm @cheddar

+      Assert.assertTrue(distance < circleRadius);
+      Assert.assertTrue(bound.containsDouble(geoPoint));
+      float[] floatPoint = new float[]{
+          Float.parseFloat(String.valueOf(geoPoint[0])),


+      Assert.assertTrue(haversineDistance > circleRadius); // asserts that point is outside
+      Assert.assertFalse(bound.containsDouble(geoPoint));
+      float[] floatPoint = new float[]{
+          Float.parseFloat(String.valueOf(geoPoint[0])),


clintropolis

👍

clintropolis · 2024-03-07T23:42:04Z

+    double lat1 = coord1[0];
+    double lon1 = coord1[1];
+    double lat2 = coord2[0];
+    double lon2 = coord2[1];


nit: maybe just inline these in the next 4 double declarations since they aren't used separately

(also a lot of these could be marked final)

clintropolis · 2024-03-07T23:45:08Z

+import org.apache.druid.collections.spatial.hp.ImmutableDoublePoint;

 /**
+ *


nit: maybe either add real javadoc or delete this

clintropolis · 2024-03-14T20:37:45Z

+/**
+ * An immutable representation of an {@link RTree} for spatial indexing.
+ */
+public final class ImmutableRTree32BitImpl implements ImmutableRTree


nit: how about ImmutableFloatRTree? or this is fine too, naming things is the worst

+      Assert.assertTrue(bound.contains(geoPoint));
+      float[] floatPoint = new float[]{
+          Float.parseFloat(String.valueOf(geoPoint[0])),
+          Float.parseFloat(String.valueOf(geoPoint[1]))


clintropolis · 2024-03-22T20:32:47Z

+  }
+
+  @Test
+  public void testDeSerForDoubleArrays() throws JsonProcessingException


nit: test name is stale i guess. Also instead of doing it with a string, it would be nicer to make a RectangularBound serialize it to string, and then deserialize back to an object to make sure round tripping works ok

modified test

clintropolis · 2024-03-22T20:35:06Z

      @JsonProperty("radius") float radius,
-      @JsonProperty("limit") int limit
+      @JsonProperty("limit") int limit,
+      @JsonProperty("radiusUnit") RadiusUnit radiusUnit


should probably add a serde test to RadiusBoundTest with the new parameter. also it should be marked @Nullable

added more tests for nullable

abhishekagarwal87

Minor comments but not a blocker so I will merge the PR.

abhishekagarwal87 · 2024-04-01T09:16:15Z

+    this.childrenOffset = initialOffset
+                          + offsetFromInitial
+                          + HEADER_NUM_BYTES
+                          + 2 * numDims * Float.BYTES
+                          + Integer.BYTES
+                          + bitmapSize;


Suggested change

this.childrenOffset = initialOffset

+ offsetFromInitial

+ HEADER_NUM_BYTES

+ 2 * numDims * Float.BYTES

+ Integer.BYTES

+ bitmapSize;

this.childrenOffset = sizePosition

+ Integer.BYTES

+ bitmapSize;

abhishekagarwal87 · 2024-04-01T09:18:12Z

+    final int sizePosition = initialOffset + offsetFromInitial + HEADER_NUM_BYTES + 2 * numDims * Float.BYTES;
+    int bitmapSize = data.getInt(sizePosition);
+    this.childrenOffset = initialOffset
+                          + offsetFromInitial
+                          + HEADER_NUM_BYTES
+                          + 2 * numDims * Float.BYTES
+                          + Integer.BYTES
+                          + bitmapSize;
+
+    this.data = data;


Suggested change

final int sizePosition = initialOffset + offsetFromInitial + HEADER_NUM_BYTES + 2 * numDims * Float.BYTES;

int bitmapSize = data.getInt(sizePosition);

this.childrenOffset = initialOffset

+ offsetFromInitial

+ HEADER_NUM_BYTES

+ 2 * numDims * Float.BYTES

+ Integer.BYTES

+ bitmapSize;

this.data = data;

this(numDims, initialOffset, offsetFromInitial, numChildren, leaf, data, bitmapFactory)

This does not work as this( ) calls needs to first statement. I just renamed the class here with no other changes.

abhishekagarwal87 · 2024-04-01T09:21:25Z

+import java.util.Iterator;
+
+/**
+ * Byte layout:


What is the child here and what is the dim?

Nums of dims = size of float array, needs it here esp for byte layout. Child here in byte layout represents the child node stored at the childOffeset
this.childrenOffset = initialOffset
+ offsetFromInitial
+ HEADER_NUM_BYTES
+ 2 * numDims * Float.BYTES
+ Integer.BYTES
+ bitmapSize;

abhishekagarwal87 · 2024-04-01T09:26:03Z

+{
+  private static Random random = ThreadLocalRandom.current();
+
+  public static float[][] generateGeoCoordinatesAroundCircle(


since its only used in test, the method should be moved to test code

it is already in test package.

This PR creates an interface for ImmutableRTree and moved the existing implementation to new class which represent 32 bit implementation (stores coordinate as floats). This PR makes the ImmutableRTree extendable to create higher precision implementation as well (64 bit). In all spatial bound filters, we accept float as input which might not be accurate in the case of high precision implementation of ImmutableRTree. This PR changed the bound filters to accepts the query bounds as double instead of float and it is backward compatible change as it compares double to existing float values in RTree. Previously it was comparing input float to RTree floats which can cause precision loss, now it is little better as it compares double to float which is still not 100% accurate. There are no changes in the way that we query spatial dimension today except input bound parsing. There is little improvement in string filter predicate which now parse double strings instead of float and compares double to double which is 100% accurate but string predicate is only called when we dont have spatial index. With allowing the interface to extend ImmutableRTree, we allow to create high precision (HP) implementation and defines new search strategies to perform HP search Iterable<ImmutableBitmap> search(ImmutableDoubleNode node, Bound bound); With possible HP implementations, Radius bound filter can not really focus on accuracy, it is calculating Euclidean distance in comparing. As EARTH 🌍 is round and not flat, Euclidean distances are not accurate in geo system. This PR adds new param called 'radiusUnit' which allows you to specify units like meters, km, miles etc. It uses https://en.wikipedia.org/wiki/Haversine_formula to check if given geo point falls inside circle or not. Added a test that generates set of points inside and outside in RadiusBoundTest.

github-actions Bot added the Area - Segment Format and Ser/De label Mar 2, 2024

github-advanced-security AI found potential problems Mar 2, 2024

View reviewed changes

github-actions Bot added the Area - Documentation label Mar 4, 2024

clintropolis added the Design Review label Mar 6, 2024

github-advanced-security AI found potential problems Mar 11, 2024

View reviewed changes

clintropolis reviewed Mar 15, 2024

View reviewed changes

pranavbhole added 3 commits March 20, 2024 11:01

Applying patch from clint

04e3323

geo interfaces

5c2a995

Merge branch 'master' into geoSpatialInterfaces1

f827ea7

pranavbhole force-pushed the geoSpatialInterfaces branch from 48f71dd to f827ea7 Compare March 21, 2024 05:22

github-advanced-security AI found potential problems Mar 21, 2024

View reviewed changes

pranavbhole added 3 commits March 21, 2024 14:31

Adding base ImmutableTree interface for handling SpatialIndex

ba2b956

Merge branch 'master' into geoSpatialInterfaces

70a2ce3

Adding more test for meter radius bound filter

5b7e39f

clintropolis approved these changes Mar 22, 2024

View reviewed changes

Adding more test

233cca0

abhishekagarwal87 reviewed Apr 1, 2024

View reviewed changes

abhishekagarwal87 merged commit 20de7fd into apache:master Apr 1, 2024

adarshsanjeev added this to the 30.0.0 milestone May 6, 2024

adarshsanjeev mentioned this pull request May 28, 2024

[DRAFT] 30.0.0 release notes #16505

Closed

Conversation

pranavbhole commented Mar 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

pranavbhole commented Mar 6, 2024

Uh oh!

Uh oh!

Uh oh!

Check notice

Check notice

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Check notice

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhishekagarwal87 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pranavbhole commented Mar 2, 2024 •

edited

Loading