Digest List - PhotoDNA

[gleithoernchen]

Hi,
Is there a way to get the PhotoDNA hashes from a given DigestList? Similiar like superutilities.misc.DigestList works for MD5 hashes.
I have troubles importing a .txt file containing PhotoDNA hashes into Nuix as DigestList.
I can export a DigestList containing PhotoDNA hashes from Nuix. But importing does not working...

[JuicyDragon]

Sorry, the notification email about your message was put in a funny spot so I only just now noticed it.
I believe digest lists still only support MD5 (at least I am pretty sure that was the case a while back). Are you able to elaborate a bit on what you are trying to accomplish? I might be able to assist you.

[gleithoernchen]

No problem :-)
I also found nothing than item.getPhotoDNARobustHash from the Nuix Engine Docs related to this.
I have a list of PhotoDNAHashes that cannot be imported as a Digest List, probably because the format of the hashes is different that the format of the PhotoDNAHashes that Nuix calculates.
I wanted to import that hashes using that library...

[JuicyDragon]

Yeah sorry, as far as I know, digest lists only support MD5. Interestingly, the binary format has a small header before the digests in the file, and the header includes "MD5", so I suspect they intended to support more types at some point.

Not wanting to leave you hanging, this is the next best thing I could think of. It takes an input file that can have 1 or 2 columns (CSV format), headers row must be present.

• PhotoDNAHash
• Score (Optional)

It will read in the PhotoDNA hashes and tag matches in the case by running them in search batches. If the Score column is present in the file, will use that score in the search for that hash. Hits have the specified tag applied to them. Using this you should be able to tag the PhotoDNA hits then if you want functionality similar to including a digest list in a search, instead include that tag in the search.

Note: I ran some tests and had success, but you will want to test this yourself as well. This script is provided with Apache 2.0 license (as-is)!

=begin
Copyright 2025 Nuix

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
=end

input_file = "C:/Temp/photodna_hashes.txt"
tag_to_apply = "PhotoDNA|Match"

require 'csv'

# Reads PhotoDNA hashes from a CSV file and builds Nuix search queries.
# The CSV should have headers "PhotoDNAHash" and optionally "Score".
# 
# @param file_path [String] Path to the CSV file containing PhotoDNA hashes
# @param hashes_per_query [Integer] Number of hashes per query. 0 = all hashes in one query
# @param default_score [Integer] Default score threshold if not specified in CSV (0-100, nil for no score)
# @return [Array<String>] Array of Nuix query strings
def build_photodna_queries(file_path, hashes_per_query, default_score = nil)
  # Validate file exists
  raise "File not found: #{file_path}" unless File.exist?(file_path)
  
  # Read and parse CSV file
  hashes = []
  line_num = 0
  
  CSV.foreach(file_path, headers: true) do |row|
    line_num += 1
    
    # Skip empty lines
    next if row['PhotoDNAHash'].nil? || row['PhotoDNAHash'].strip.empty?
    
    hash_value = row['PhotoDNAHash'].strip
    
    # Validate hash is base64-like (basic check)
    unless hash_value.match?(/^[A-Za-z0-9+\/=]+$/)
      puts "WARNING: Line #{line_num}: Invalid PhotoDNA hash format: #{hash_value}"
      next
    end
    
    # Get score from CSV or use default
    score = row['Score']
    score = score.strip.to_i if score && !score.strip.empty?
    score = default_score if score.nil? || score == 0
    
    # Validate score range if present
    if score && (score < 0 || score > 100)
      puts "WARNING: Line #{line_num}: Invalid score #{score} (must be 0-100), skipping"
      next
    end
    
    # Escape special characters for Nuix queries (backslash and quote)
    escaped_hash = hash_value.gsub('\\', '\\\\\\\\').gsub('"', '\\"')
    
    # Build the photo-dna field value with optional score
    if score
      hashes << "\"#{escaped_hash};#{score}\""
    else
      hashes << "\"#{escaped_hash}\""
    end
  end
  
  raise "No valid PhotoDNA hashes found in file: #{file_path}" if hashes.empty?
  
  puts "Loaded #{hashes.size} PhotoDNA hash(es) from #{file_path}"
  
  # Build queries based on hashes_per_query setting
  queries = []
  
  if hashes_per_query == 0
    # Single query with all hashes
    query = "photo-dna:(" + hashes.join(" OR ") + ")"
    queries << query
  else
    # Split into multiple queries
    hashes.each_slice(hashes_per_query) do |hash_batch|
      query = "photo-dna:(" + hash_batch.join(" OR ") + ")"
      queries << query
    end
  end
  
  puts "Built #{queries.size} query/queries (#{hashes_per_query == 0 ? 'all' : hashes_per_query} hashes per query)"
  
  return queries
end

# Tags items matching PhotoDNA hashes from a CSV file.
# Searches the current Nuix case and applies the specified tag to all matching items.
# Uses BulkAnnotater for efficient tagging in batches.
#
# @param nuix_case [Case] The current Nuix case object
# @param file_path [String] Path to the CSV file containing PhotoDNA hashes
# @param tag_name [String] Tag to apply to matching items
# @param hashes_per_query [Integer] Number of hashes per search query (0 = all in one query)
# @param default_score [Integer] Default score threshold if not in CSV (nil for no score)
# @param bulk_tag_batch_size [Integer] Number of items to tag per BulkAnnotater batch (default: 10000)
def tag_photodna_matches(nuix_case, file_path, tag_name, hashes_per_query, default_score = nil, bulk_tag_batch_size = 10000)
  puts "=" * 80
  puts "PhotoDNA Tagging Operation"
  puts "=" * 80
  puts "File: #{file_path}"
  puts "Tag: #{tag_name}"
  puts "Hashes per query: #{hashes_per_query == 0 ? 'ALL' : hashes_per_query}"
  puts "Default score: #{default_score || 'None'}"
  puts "Bulk tag batch size: #{bulk_tag_batch_size}"
  puts "=" * 80
  puts ""
  
  start_time = Time.now
  
  # Build queries from the CSV file
  puts "Building queries..."
  queries = build_photodna_queries(file_path, hashes_per_query, default_score)
  puts ""
  
  # Track overall statistics
  total_items_found = 0
  total_items_tagged = 0
  query_num = 0
  
  # Get BulkAnnotater for efficient tagging
  bulk_annotater = $utilities.getBulkAnnotater
  
  # Process each query
  queries.each do |query|
    query_num += 1
    query_start = Time.now
    
    puts "-" * 80
    puts "Query #{query_num}/#{queries.size}"
    puts "-" * 80
    
    # Truncate query display if too long
    display_query = query.length > 512 ? query[0..509] + "..." : query
    puts "Query: #{display_query}"
    
    begin
      # Search the case
      puts "Searching..."
      items = nuix_case.searchUnsorted(query)
      item_count = items.size
      total_items_found += item_count
      
      puts "Found: #{item_count} item(s)"
      
      if item_count > 0
        # Tag items in batches using BulkAnnotater
        items_array = items.to_a
        batch_num = 0
        total_batches = (item_count.to_f / bulk_tag_batch_size).ceil
        
        items_array.each_slice(bulk_tag_batch_size) do |batch|
          batch_num += 1
          batch_start = Time.now
          
          # Tag the batch
          bulk_annotater.addTag(tag_name, batch)
          total_items_tagged += batch.size
          
          batch_elapsed = Time.now - batch_start
          
          # Progress update for this batch
          puts "  Tagged batch #{batch_num}/#{total_batches}: #{batch.size} item(s) in %.2f seconds" % batch_elapsed
        end
        
        query_elapsed = Time.now - query_start
        puts "Query completed in %.2f seconds" % query_elapsed
      else
        puts "No items to tag"
      end
      
    rescue => e
      puts "ERROR: Query failed - #{e.message}"
      puts e.backtrace.first(5).join("\n")
    end
    
    puts ""
  end
  
  # Final summary
  total_elapsed = Time.now - start_time
  
  puts "=" * 80
  puts "SUMMARY"
  puts "=" * 80
  puts "Total queries executed: #{queries.size}"
  puts "Total items found: #{total_items_found}"
  puts "Total items tagged: #{total_items_tagged}"
  puts "Tag applied: #{tag_name}"
  puts "Total time: %.2f seconds" % total_elapsed
  puts "=" * 80
end

# Example usage:
# Assuming $current_case is available in the Nuix scripting context
#
# Tag all matches with 250 hashes per query, default score of 75
# tag_photodna_matches($current_case, "C:/path/to/photodna_hashes.csv", "PhotoDNA|Match", 250, 75)
#
# Tag all matches in a single query, no default score
# tag_photodna_matches($current_case, "C:/path/to/photodna_hashes.csv", "PhotoDNA|Match", 0, nil)
#
# Tag with custom bulk tagging batch size
# tag_photodna_matches($current_case, "C:/path/to/photodna_hashes.csv", "PhotoDNA|Match", 250, 75, 5000)


# Tag all matches with 250 hashes per query, default score of 75
tag_photodna_matches($current_case, input_file, tag_to_apply, 250, 75)

[gleithoernchen]

Thanks a lot for providing the script.
Unfortunately I got an error, because of the wrong format of the photoDNA values.
They have that format:
0807011E1607581B4049DD0C4933FF091CADBA1909291B281205055BD002088B77101B964547298A11FF19CA01560184051943338809106E9E140D79346112B009E12B7901260E1F30A3B52F5B583217A75D0F3C6B81108E2697324603201C000E7D08A838340D25835F0447265F05345B37273805492006022004141F06080C35350E2A071C240B3F0D20260F40140B
And that format can be imported in other tools.
But Nuix PhotoDNA hashes have a format like:
NWcoByT/PBFHZ3NP/zYkET9lQRheSVweAmkUBxT/uicrWP9P/xWuFzeNMy+GUkpYDTgMGDRqkSVCQ/9/YCaMGB4vMgM4QYMDHTQFHEduHSvTvXyDWj4UJB4SAiUsEQ0YIyMGIR5BEGFIMElsPSwoXSE5IztbDQVAJBUBLw4qFhQbCyISNQMnGBc1FBA6ERsO
So are you aware of different formats for photoDNA hash values? Can they converted somehow?

[JuicyDragon]

Unfortunately, I do not know the answer to that. I did a quick web search but didn't really find anything defining the correct format. You might want to submit a question about this to our support portal as they are more likely to know the answer to that.

[gleithoernchen]

Neither have I. This whole PhotoDNA topic is a bit 'under the hood'.
I requested this in the support portal two weeks ago and have not yet received a solution.
But thank you very much for your help.

[gleithoernchen]

Hi @JuicyDragon,
Just to give you an update.
The values need to be converted to base64 strings then it works:

require 'base64'

hex = "0807011E1607581B4049DD0C4933FF091CADBA1909291B281205055BD002088B77101B964547298A11FF19CA01560184051943338809106E9E140D79346112B009E12B7901260E1F30A3B52F5B583217A75D0F3C6B81108E2697324603201C000E7D08A838340D25835F0447265F05345B37273805492006022004141F06080C35350E2A071C240B3F0D20260F40140B"
bytes = [hex].pack('H*')
base64 = Base64.strict_encode64(bytes)

puts base64