Repository navigation
Better URI namespaced lookups behavior with single files #2669
Description
We had a lookup like this:
{
"namespace" : "channel_lookup",
"pollPeriod" : "P1D",
"namespaceParseSpec" : {
"keyFieldName" : "key",
"valueFieldName" : "value",
"format" : "customJson"
},
"uri" : "file:/opt/data/druid/channel-lookup.json",
"type" : "uri"
}In this situation I'd have expected it to match exactly the file /opt/data/druid/channel-lookup.json, but it actually matched a different file named "init" in the same directory. Reading through the code it looks like the file name of a uri is intentionally ignored in favor of searching the parent directory for things matching the versionRegex.
IMO this is pretty surprising behavior (even though the docs do suggest that this can happen).
Could we improve it in one (or all) of these ways?
- don't do the regex-matching-in-parent-dir thing when a specific file is provided and versionRegex = null
- when doing the regex-matching-in-parent-dir thing, only match files if they match the versionRegex and have the provided uri as a prefix (instead of only matching the versionRegex)
- include an example in the docs of how to use a specific file for a lookup
- describe the regex-matching-in-dir process front and center under "URI namespace update" rather than only under the section for versionRegex.
Activity
drcrallen commentedon Mar 16, 2016
after #1576 goes in I'll update the extension to be able to use the dynamic configuration stuff. I'll see if I can get this fixed at the same time. I agree making it smarter and clearer about the uri is the right way to go.
My suggestion is having 3 fields:
uri- An absolute URI to grab data fromXOR
uriPrefix- A prefix for doing pattern matching.nodeRegex- An optional regex for matching the node part of the uri path. something like"channel-lookup[\-0-9]*\.json"in your example would match any of the following:channel-lookup.json,channel-lookup-0000.jsonbut NOT includechannel-lookup.json.backupvogievetsky commentedon Mar 16, 2016
from an API perspective I think @drcrallen 's suggestion is the way to go.
drcrallen commentedon Mar 16, 2016
@vogievetsky I amended the comment just a bit at the end to clarify regex behavior, just FYI
[QTL] Make URI Exctraction Namespace take more sane arguments (#2738)
apache#2669 Support parquet formatter