This help question uncovered a severe performance issue in out meta; when frequently called in a foreach loop:
[bbox:{{bbox}}];
node;
foreach->.a(
node(around.a:0)->.b;
(.b; - .a);
out meta;
);
http://overpass-turbo.eu/s/6C8
Although this query only returns very few nodes as a result, it runs for many minutes. I tracked this down to an expensive Meta_Collector initialization for user names, which is called from print:cc:896 (inside Print_Statement::tags_by_id).
// formulate meta query if meta data shall be printed
Meta_Collector< TIndex, typename TObject::Id_Type > meta_printer(items, transaction, meta_file_prop);
On each foreach iteration, an array of 350k++ user names is loaded from disk and then discarded again, to provide the necessary details for out meta;. In terms of throughput, the out; version outperforms out meta; by at least a factor of 50 according to my measurements.
Caching that information across foreach-iterations seems to be a reasonable solution for this issue. There's even a User_Data_Cache available, but unfortunately it's not being used in Meta_Collector.
#0 0x00007f3d04cb95c0 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00000000004d351c in pair (__b="", __a=<optimized out>,
this=0x7fff4e02b380) at /usr/include/c++/4.8/bits/stl_pair.h:113
#2 operator[] (__k=<optimized out>, this=0x7fff4e02b890)
at /usr/include/c++/4.8/bits/stl_map.h:469
#3 Meta_Collector<Uint31_Index, Uint32_Index>::Meta_Collector<Way_Skeleton> (
this=0x7fff4e02b7e0, items=..., transaction=...,
meta_file_prop=<optimized out>, user_data=<optimized out>)
at ../src/overpass_api/statements/meta_collector.h:123
Python Exception <class 'ValueError'> Cannot find type const std::map<Uint31_Index, std::vector<Way_Skeleton, std::allocator<Way_Skeleton> >, std::less<Uint31_Index>, std::allocator<std::pair<Uint31_Index const, std::vector<Way_Skeleton, std::allocator<Way_Skeleton> > > > >::_Rep_type:
#4 0x00000000004e41ac in Print_Statement::tags_by_id<Uint31_Index, Way_Skeleton> (this=this@entry=0x8d0480, items=std::map with 0 elements, file_prop=...,
FLUSH_SIZE=FLUSH_SIZE@entry=524288, target=..., rman=..., transaction=...,
meta_file_prop=meta_file_prop@entry=0xd1e850,
element_count=@0x7fff4e02b980: 0)
at ../src/overpass_api/statements/print.cc:896
#5 0x00000000004c8999 in Print_Statement::execute (this=0x8d0480, rman=...)
at ../src/overpass_api/statements/print.cc:2370
#6 0x0000000000487d8d in Foreach_Statement::execute (this=0x8d0760, rman=...)
at ../src/overpass_api/statements/foreach.cc:89
#7 0x00000000004a8d61 in Osm_Script_Statement::execute (this=0x8cf370,
rman=...) at ../src/overpass_api/statements/osm_script.cc:397
This help question uncovered a severe performance issue in
out meta;when frequently called in aforeachloop:http://overpass-turbo.eu/s/6C8
Although this query only returns very few nodes as a result, it runs for many minutes. I tracked this down to an expensive Meta_Collector initialization for user names, which is called from print:cc:896 (inside Print_Statement::tags_by_id).
On each
foreachiteration, an array of 350k++ user names is loaded from disk and then discarded again, to provide the necessary details forout meta;. In terms of throughput, theout;version outperformsout meta;by at least a factor of 50 according to my measurements.Caching that information across foreach-iterations seems to be a reasonable solution for this issue. There's even a User_Data_Cache available, but unfortunately it's not being used in Meta_Collector.