Skip to content

Perf: Add a parallel FindMany method to speed up queries for a key #21

Description

@meling

Right now lookup using the Find(key) method for a single key isn't necessarily a significant performance bottleneck. However, when checking all keys, when len(keys) is large, it can be quite slow.

We should add a parallel FindMany method; still need to think more about the signature of the method to make it flexible for several uses, but here is a first sketch:

func (bb BBHash) FindMany(keys []uint64) []uint64 {
	var wg sync.WaitGroup
	keyChunks := slices.Chunk(keys, 1000)
	a := make([][]uint64, 0, len(keys)/1000)
	for chunk := range keyChunks {
		wg.Add(1)
		idx := make([]uint64, len(chunk))
		a = append(a, idx)
		go func() {
			defer wg.Done()
			for i, key := range chunk {
				idx[i] = bb.Find(key)
			}
		}()
	}
	wg.Wait()
	return slices.Flatten(a) // Flatten does not exist
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceCan possibly improve performance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions