Skip to content

perf: out meta in foreach loop #165

Description

@mmd-osm

This help question uncovered a severe performance issue in out meta; when frequently called in a foreach loop:

[bbox:{{bbox}}];

node;
foreach->.a(
node(around.a:0)->.b;
(.b; - .a);
out meta;
);  

http://overpass-turbo.eu/s/6C8

Although this query only returns very few nodes as a result, it runs for many minutes. I tracked this down to an expensive Meta_Collector initialization for user names, which is called from print:cc:896 (inside Print_Statement::tags_by_id).

  // formulate meta query if meta data shall be printed
  Meta_Collector< TIndex, typename TObject::Id_Type > meta_printer(items, transaction, meta_file_prop);

On each foreach iteration, an array of 350k++ user names is loaded from disk and then discarded again, to provide the necessary details for out meta;. In terms of throughput, the out; version outperforms out meta; by at least a factor of 50 according to my measurements.

Caching that information across foreach-iterations seems to be a reasonable solution for this issue. There's even a User_Data_Cache available, but unfortunately it's not being used in Meta_Collector.

#0  0x00007f3d04cb95c0 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00000000004d351c in pair (__b="", __a=<optimized out>, 
    this=0x7fff4e02b380) at /usr/include/c++/4.8/bits/stl_pair.h:113
#2  operator[] (__k=<optimized out>, this=0x7fff4e02b890)
    at /usr/include/c++/4.8/bits/stl_map.h:469
#3  Meta_Collector<Uint31_Index, Uint32_Index>::Meta_Collector<Way_Skeleton> (
    this=0x7fff4e02b7e0, items=..., transaction=..., 
    meta_file_prop=<optimized out>, user_data=<optimized out>)
    at ../src/overpass_api/statements/meta_collector.h:123
Python Exception <class 'ValueError'> Cannot find type const std::map<Uint31_Index, std::vector<Way_Skeleton, std::allocator<Way_Skeleton> >, std::less<Uint31_Index>, std::allocator<std::pair<Uint31_Index const, std::vector<Way_Skeleton, std::allocator<Way_Skeleton> > > > >::_Rep_type: 
#4  0x00000000004e41ac in Print_Statement::tags_by_id<Uint31_Index, Way_Skeleton> (this=this@entry=0x8d0480, items=std::map with 0 elements, file_prop=..., 
    FLUSH_SIZE=FLUSH_SIZE@entry=524288, target=..., rman=..., transaction=..., 
    meta_file_prop=meta_file_prop@entry=0xd1e850, 
    element_count=@0x7fff4e02b980: 0)
    at ../src/overpass_api/statements/print.cc:896
#5  0x00000000004c8999 in Print_Statement::execute (this=0x8d0480, rman=...)
    at ../src/overpass_api/statements/print.cc:2370
#6  0x0000000000487d8d in Foreach_Statement::execute (this=0x8d0760, rman=...)
    at ../src/overpass_api/statements/foreach.cc:89
#7  0x00000000004a8d61 in Osm_Script_Statement::execute (this=0x8cf370, 
    rman=...) at ../src/overpass_api/statements/osm_script.cc:397

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions