Add documentations for config to filter internal Druid-related messages from error response by maytasm · Pull Request #11755 · apache/druid

maytasm · 2021-09-29T06:51:53Z

Add documentations for #11711

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

techdocsmith · 2021-09-29T23:16:28Z

 |`druid.indexer.logs.kill.initialDelay`| Optional. Number of milliseconds after Overlord start when first auto kill is run. |random value less than 300000 (5 mins)|
 |`druid.indexer.logs.kill.delay`|Optional. Number of milliseconds of delay between successive executions of auto kill run. |21600000 (6 hours)|

+### API Error Response


Suggested change

### API Error Response

### API error response

no caps

techdocsmith · 2021-09-29T23:17:03Z


+### API Error Response
+
+Druid API error responses can be configured to hide internal information (such as Druid class name, stack trace, thread name, servlet name, code, line/column number, and host/ip). 


Suggested change

Druid API error responses can be configured to hide internal information (such as Druid class name, stack trace, thread name, servlet name, code, line/column number, and host/ip).

You can configure Druid API error responses to hide internal information like the Druid class name, stack trace, thread name, servlet name, code, line/column number, or host/ip.

techdocsmith · 2021-09-29T23:19:14Z

+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.server.http.showDetailedJettyErrors`|When set to true, any error resulted from the Jetty layer / Jetty filter will have `servlet`, `message`, `url`, `status` and (if exist) `cause` fields in the JSON response. When set to false, the JSON response will only contains `message`, `url`, and `status`. Note that the value of the fields remain unchanged.|true|


Suggested change

|`druid.server.http.showDetailedJettyErrors`|When set to true, any error resulted from the Jetty layer / Jetty filter will have `servlet`, `message`, `url`, `status` and (if exist) `cause` fields in the JSON response. When set to false, the JSON response will only contains `message`, `url`, and `status`. Note that the value of the fields remain unchanged.|true|

|`druid.server.http.showDetailedJettyErrors`|When set to true, any error from the Jetty layer / Jetty filter includes `servlet`, `message`, `url`, `status`, and `cause`, if it exists, fields in the JSON response. When set to false, the JSON response only includes `message`, `url`, and `status`. The field values remain unchanged.|true|

techdocsmith · 2021-09-29T23:25:01Z

+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.server.http.showDetailedJettyErrors`|When set to true, any error resulted from the Jetty layer / Jetty filter will have `servlet`, `message`, `url`, `status` and (if exist) `cause` fields in the JSON response. When set to false, the JSON response will only contains `message`, `url`, and `status`. Note that the value of the fields remain unchanged.|true|
+|`druid.server.http.errorResponseTransformStrategy.strategy`|Error response transform strategy to use to transform error response produced within Druid services.|`none`|


Suggested change

|`druid.server.http.errorResponseTransformStrategy.strategy`|Error response transform strategy to use to transform error response produced within Druid services.|`none`|

|`druid.server.http.errorResponseTransformStrategy.strategy`|Error response transform strategy. The strategy controls how Druid transforms error responses from Druid services. When empty, Druid leaves error responses unchanged.|`none`|

Is

techdocsmith · 2021-09-30T00:16:30Z

+
+##### Error response transform strategy
+
+Error response transform strategy can be use to transform error response produced within Druid services.


Suggested change

Error response transform strategy can be use to transform error response produced within Druid services.

You can use an error response transform strategy to transform error responses from within Druid services to hide internal information.

Suggest re-iterating why you want to use a transform strategy.

techdocsmith · 2021-09-30T00:21:12Z

+Error response transform strategy can be use to transform error response produced within Druid services.
+Currently, when error response transform strategy is used (not `none`), the error responses from the following Druid services will be transformed as follow:
+ - Any query API that failed in the router service. The fields `errorClass` and `host` will always be null while the field `errorMessage` will be transformed based on the set strategy.
+ - Any SQL qeury API (`/druid/v2/sql/` POST) that failed. The fields `errorClass` and `host` will always be null while the field `errorMessage` will be transformed based on the set strategy.


Suggested change

- Any SQL qeury API (`/druid/v2/sql/` POST) that failed. The fields `errorClass` and `host` will always be null while the field `errorMessage` will be transformed based on the set strategy.

- For Any SQL query API, for example `POST /druid/v2/sql/...` that fails, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.

techdocsmith · 2021-09-30T00:23:12Z

+
+###### No error response transform strategy
+
+In this mode, error response produced within Druid services are left unchanged and returned to the API client as is. 


Suggested change

In this mode, error response produced within Druid services are left unchanged and returned to the API client as is.

In this mode, Druid leaves error responses from underlying services unchanged and returns the unchanged errors to the API client.

techdocsmith · 2021-09-30T00:23:45Z

+###### No error response transform strategy
+
+In this mode, error response produced within Druid services are left unchanged and returned to the API client as is. 
+This is the default Druid error response mode. Enable this strategy explicitly by setting druid.server.http.errorResponseTransformStrategy.strategy to none.


Suggested change

This is the default Druid error response mode. Enable this strategy explicitly by setting druid.server.http.errorResponseTransformStrategy.strategy to none.

This is the default Druid error response mode. To explicitly enable this strategy, set `druid.server.http.errorResponseTransformStrategy.strategy` to "none".

techdocsmith · 2021-09-30T00:26:03Z

+In this mode, the error response produced within Druid services will be validated against a list of regular expressions. 
+The error message of the response will only be returned to the client if it matches any of the regular expressions configured.
+This strategy can be enabled by setting druid.server.http.errorResponseTransformStrategy.strategy=allowedRegex.
+


Suggested change

In this mode, the error response produced within Druid services will be validated against a list of regular expressions.

The error message of the response will only be returned to the client if it matches any of the regular expressions configured.

This strategy can be enabled by setting druid.server.http.errorResponseTransformStrategy.strategy=allowedRegex.

In this mode, Druid validates the error responses from underlying services will against a list of regular expressions. Only error messages that match a configured regular expression are returned. To enable this strategy, set `druid.server.http.errorResponseTransformStrategy.strategy` to `allowedRegex`.

nit on suggested rewrite (which otherwise lgtm):

...Druid validates the error responses from underlying services will against a list of regular expressions...

should instead be

...Druid validates the error responses from underlying services against a list of regular expressions...

techdocsmith · 2021-09-30T00:28:34Z

+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.server.http.errorResponseTransformStrategy.allowedRegex`|The list of regular expressions used to check against the error message. If the error message match any of the regular expressions, then it will be included in the response unchanged. If the error message does not match any of the regular expressions then the error message will either be null or replaced with a default message depedning on the type of underlying Exception. |`[]`|


Suggested change

|`druid.server.http.errorResponseTransformStrategy.allowedRegex`|The list of regular expressions used to check against the error message. If the error message match any of the regular expressions, then it will be included in the response unchanged. If the error message does not match any of the regular expressions then the error message will either be null or replaced with a default message depedning on the type of underlying Exception. |`[]`|

|`druid.server.http.errorResponseTransformStrategy.allowedRegex`|The list of regular expressions Druid uses to validate error messages. If the error message matches any of the regular expressions, then Druid includes it in the response unchanged. If the error message does not match any of the regular expressions, Druid replaces the error message with null or with a default message depending on the type of underlying Exception. |`[]`|

This could use an example.

techdocsmith · 2021-09-30T16:36:55Z

+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.server.http.showDetailedJettyErrors`|When set to true, any error from the Jetty layer / Jetty filter includes `servlet`, `message`, `url`, `status`, and `cause`, if it exists, fields in the JSON response. When set to false, the JSON response only includes `message`, `url`, and `status`. The field values remain unchanged.|true|


Suggested change

|`druid.server.http.showDetailedJettyErrors`|When set to true, any error from the Jetty layer / Jetty filter includes `servlet`, `message`, `url`, `status`, and `cause`, if it exists, fields in the JSON response. When set to false, the JSON response only includes `message`, `url`, and `status`. The field values remain unchanged.|true|

|`druid.server.http.showDetailedJettyErrors`|When set to true, any error from the Jetty layer / Jetty filter includes the following fields in the JSON response: `servlet`, `message`, `url`, `status`, and `cause`, if it exists. When set to false, the JSON response only includes `message`, `url`, and `status`. The field values remain unchanged.|true|

sorry the way I edited this before was a little confusing.

techdocsmith · 2021-09-30T16:38:26Z

+You can use an error response transform strategy to transform error responses from within Druid services to hide internal information.
+When you specify an error response transform strategy other than `none`, Druid transforms the error responses from Druid services as follows:
+ - For any query API that fails in the Router service, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.
+ - For Any SQL query API, for example `POST /druid/v2/sql/...` that fails, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.


Suggested change

- For Any SQL query API, for example `POST /druid/v2/sql/...` that fails, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.

- For Any SQL query API that fails, for example `POST /druid/v2/sql/...`, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.

sorry another minor edit

btw For Any should be For any?

techdocsmith

@maytasm thank you for the great example! It really helps. I tweaked some of my previous suggestions and made a couple of suggestions for the wording of the example. Otherwise LGTM.

techdocsmith · 2021-09-30T16:39:46Z

+|--------|-----------|-------|
+|`druid.server.http.errorResponseTransformStrategy.allowedRegex`|The list of regular expressions Druid uses to validate error messages. If the error message matches any of the regular expressions, then Druid includes it in the response unchanged. If the error message does not match any of the regular expressions, Druid replaces the error message with null or with a default message depending on the type of underlying Exception. |`[]`|
+
+For example, if the original query error response is:


Suggested change

For example, if the original query error response is:

For example, consider the following error response:

techdocsmith · 2021-09-30T16:40:32Z

+```
+{"error":"Plan validation failed","errorMessage":"org.apache.calcite.runtime.CalciteContextException: From line 1, column 15 to line 1, column 38: Object 'nonexistent-datasource' not found","errorClass":"org.apache.calcite.tools.ValidationException","host":null}
+```
+if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[]` then the query error response will be transformed to


Suggested change

if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[]` then the query error response will be transformed to

If `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[]`, Druid transforms the query error response to the following:

techdocsmith · 2021-09-30T16:41:04Z

+```
+{"error":"Plan validation failed","errorMessage":null,"errorClass":null,"host":null}
+``` 
+however if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[".*CalciteContextException.*"]` then the query error response will be transformed to


Suggested change

however if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[".*CalciteContextException.*"]` then the query error response will be transformed to

On the other hand, if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[".*CalciteContextException.*"]` then Druid transforms the query error response to the following:

…es from error response (apache#11755) * add doc * add doc * address comments * fix typo * address comments

add doc

5fcb7bb

maytasm changed the title ~~Add documentations for https://github.com/apache/druid/pull/11711~~ Add documentations for config to filter internal Druid-related messages from error response (https://github.com/apache/druid/pull/11711) Sep 29, 2021

add doc

d78c0b8

clintropolis added the Area - Documentation label Sep 29, 2021

techdocsmith requested changes Sep 30, 2021

View reviewed changes

clintropolis changed the title ~~Add documentations for config to filter internal Druid-related messages from error response (https://github.com/apache/druid/pull/11711)~~ Add documentations for config to filter internal Druid-related messages from error response Sep 30, 2021

clintropolis added the Release Notes label Sep 30, 2021

maytasm added 2 commits September 30, 2021 20:47

address comments

44d174f

fix typo

c596c2b

techdocsmith reviewed Sep 30, 2021

View reviewed changes

techdocsmith approved these changes Sep 30, 2021

View reviewed changes

address comments

b2874d3

maytasm merged commit 129911a into apache:master Oct 1, 2021

maytasm deleted the IMPLY-10674-2 branch October 1, 2021 10:49

maytasm mentioned this pull request Oct 5, 2021

Fix incorrect property key for error response transform config #11772

Merged

1 of 9 tasks

abhishekagarwal87 added this to the 0.23.0 milestone May 11, 2022

riovic918data pushed a commit to riovic918data/druid that referenced this pull request Jun 12, 2026

Add documentations for config to filter internal Druid-related messag…

2abb5cb

…es from error response (apache#11755) * add doc * add doc * address comments * fix typo * address comments


		### API Error Response

		Druid API error responses can be configured to hide internal information (such as Druid class name, stack trace, thread name, servlet name, code, line/column number, and host/ip).

	Druid API error responses can be configured to hide internal information (such as Druid class name, stack trace, thread name, servlet name, code, line/column number, and host/ip).
	You can configure Druid API error responses to hide internal information like the Druid class name, stack trace, thread name, servlet name, code, line/column number, or host/ip.

	\|`druid.server.http.showDetailedJettyErrors`\|When set to true, any error resulted from the Jetty layer / Jetty filter will have `servlet`, `message`, `url`, `status` and (if exist) `cause` fields in the JSON response. When set to false, the JSON response will only contains `message`, `url`, and `status`. Note that the value of the fields remain unchanged.\|true\|
	\|`druid.server.http.showDetailedJettyErrors`\|When set to true, any error from the Jetty layer / Jetty filter includes `servlet`, `message`, `url`, `status`, and `cause`, if it exists, fields in the JSON response. When set to false, the JSON response only includes `message`, `url`, and `status`. The field values remain unchanged.\|true\|

	\|`druid.server.http.errorResponseTransformStrategy.strategy`\|Error response transform strategy to use to transform error response produced within Druid services.\|`none`\|
	\|`druid.server.http.errorResponseTransformStrategy.strategy`\|Error response transform strategy. The strategy controls how Druid transforms error responses from Druid services. When empty, Druid leaves error responses unchanged.\|`none`\|


		##### Error response transform strategy

		Error response transform strategy can be use to transform error response produced within Druid services.

	Error response transform strategy can be use to transform error response produced within Druid services.
	You can use an error response transform strategy to transform error responses from within Druid services to hide internal information.

	- Any SQL qeury API (`/druid/v2/sql/` POST) that failed. The fields `errorClass` and `host` will always be null while the field `errorMessage` will be transformed based on the set strategy.
	- For Any SQL query API, for example `POST /druid/v2/sql/...` that fails, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.


		###### No error response transform strategy

		In this mode, error response produced within Druid services are left unchanged and returned to the API client as is.

	In this mode, error response produced within Druid services are left unchanged and returned to the API client as is.
	In this mode, Druid leaves error responses from underlying services unchanged and returns the unchanged errors to the API client.

	This is the default Druid error response mode. Enable this strategy explicitly by setting druid.server.http.errorResponseTransformStrategy.strategy to none.
	This is the default Druid error response mode. To explicitly enable this strategy, set `druid.server.http.errorResponseTransformStrategy.strategy` to "none".

	- For Any SQL query API, for example `POST /druid/v2/sql/...` that fails, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.
	- For Any SQL query API that fails, for example `POST /druid/v2/sql/...`, Druid sets the fields `errorClass` and `host` to null. Druid applies the transformation strategy to the `errorMessage` field.

	For example, if the original query error response is:
	For example, consider the following error response:

	if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[]` then the query error response will be transformed to
	If `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[]`, Druid transforms the query error response to the following:

	however if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[".CalciteContextException."]` then the query error response will be transformed to
	On the other hand, if `druid.server.http.errorResponseTransformStrategy.allowedRegex` is set to `[".CalciteContextException."]` then Druid transforms the query error response to the following:

Conversation

maytasm commented Sep 29, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

techdocsmith Sep 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

techdocsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

techdocsmith Sep 30, 2021 •

edited

Loading