File sink throughput bottleneck: HTTP source → remap transform → file sink maxes out at ~12k EPS (target: 50+)

[mrSpencer91]

Question

Setup: Logstash forwards Windows event logs via HTTP (json_batch) to Vector on a dedicated archive server. Vector remaps the events (hostname extraction, timestamp parsing) and writes them to per-hostname log files.

Logstash (32 workers, json_batch) → HTTP → Vector (remap) → file sink (~3000 unique files)

Environment: Vector 0.44 in Docker, 20 CPUs, 32GB RAM, Linux. The archive server handles no other workload.

Problem: Logstash delivers ~25k EPS to Vector's HTTP source, but the file sink only sustains ~12k EPS. Component utilization is at 100%. We've been iterating on this for a week across multiple test runs and have eliminated several suspects:

What we've ruled out:

• Logstash HTTP output is not the bottleneck — delivers 25k+ EPS consistently with 32 workers and json_batch
• Disk I/O on the archive server looks normal, no write saturation visible in node_exporter metrics
• Network latency is negligible (0.47ms RTT)
• format => "json" vs "json_batch" — confirmed that json_batch works correctly with Vector (events are properly split), using "json" sends one HTTP request per event and halves throughput
• LevelDB disk buffer was a confirmed bottleneck in earlier tests (97.6% of events waited 1–2s in buffer_send_duration). Switched to memory buffer, which eliminated that specific latency — but the overall sink throughput didn't improve as expected
• Acknowledgements caused cascading timeouts (Logstash HTTP timeout → "Source send cancelled" → event drops). Disabled with acknowledgements: false

What we're seeing now (memory buffer, ack disabled):

• buffer_events builds up to ~1.3M over ~2 minutes, then drops sharply to ~160k — repeating sawtooth pattern
• vector_open_files{component_id="file_archive"} shows the same sawtooth: climbs to ~3000, drops to 0, climbs back
• The mass close/reopen cycle happens every ~2 min, well before idle_timeout_secs: 300 would trigger
• component_utilization stays at 100% throughout

Current config:

  sources:
    http_in:
      type: http_server
      address: "0.0.0.0:8080"
      framing:
        method: newline_delimited
      decoding:
        codec: json

  transforms:
    extract_routing:
      type: remap
      inputs: ["http_in"]
      source: |
        # hostname extraction chain + timestamp parsing

  sinks:
    file_archive:
      type: file
      inputs: ["extract_routing"]
      path: "/var/log/remote/hosts/{{ hostname }}"
      buffer:
        type: memory
        max_events: 500000
        when_full: block
      idle_timeout_secs: 300
      acknowledgements:
        enabled: false

Questions:

1. What limits file sink throughput when writing to ~3000 unique files concurrently? Is there an internal file handle pool, flush interval, or max_open_files default that could explain the periodic mass close?
2. Are there recommended tuning options for high-fanout file sinks at this scale?
3. Could the remap transform be contributing to the bottleneck at 100% utilization, or does the utilization metric reflect only the sink?

Any guidance appreciated — happy to share Prometheus metrics exports or the full vector.yaml if helpful.

Vector Config

# =============================================================================
# Vector Log Archive Configuration
# =============================================================================
# Receives JSON logs via HTTP from Logstash and syslog-ng.
# Stores logs in a structured layout under:
#   /archive/{MM}/{DD}/{hostname}/{hostname}/{hostname}-{YY}-{MM}-{DD}-{HH}.log
#
# NOTE: No parsing or content transformation of logs.
#       Only routing fields are derived from existing log fields.
# =============================================================================

api:
  enabled: true
  address: "0.0.0.0:8686"

# -----------------------------------------------------------------------------
# DATA DIR (for internal Vector metadata, e.g. cursor state)
# -----------------------------------------------------------------------------
data_dir: /var/lib/vector

# -----------------------------------------------------------------------------
# SOURCE: HTTP endpoint for incoming JSON logs
# -----------------------------------------------------------------------------
sources:
  http_in:
    type: http_server
    address: "0.0.0.0:8080"

    # Logstash and syslog-ng send newline-delimited JSON or single JSON objects
    framing:
      method: newline_delimited
    decoding:
      codec: json

    # Strict mode: rejects requests with incorrect Content-Type
    strict_path: false

    # Authentication (optional, recommended for production)
    # auth:
    #   strategy: bearer
    #   token: "YOUR_SECRET_TOKEN"

  internal_metrics:
    type: internal_metrics

# -----------------------------------------------------------------------------
# TRANSFORM: Extract routing fields from timestamp and hostname
# -----------------------------------------------------------------------------
# Used exclusively for file path routing. Log content is NOT modified.
# Routing fields (_r_*) are stripped in the sink configuration.
# -----------------------------------------------------------------------------
transforms:
  extract_routing:
    type: remap
    inputs:
      - http_in
    source: |
      # -----------------------------------------------------------------------
      # Read timestamp
      # Supports the following fields (priority: @timestamp > timestamp > date):
      #   - Logstash:   "@timestamp" (ISO 8601)
      #   - syslog-ng:  "timestamp" or "date"
      # -----------------------------------------------------------------------
      ts_raw = if .@timestamp != null { .@timestamp } else if .timestamp != null { .timestamp } else if .date != null { .date } else { null }

      ts = now()
      if ts_raw != null {
        ts_parsed, err = parse_timestamp(to_string!(ts_raw), format: "%+")
        if err == null {
          ts = ts_parsed
        } else {
          log("Could not parse timestamp, falling back to now(). Value: " + to_string!(ts_raw), level: "warn")
        }
      } else {
        log("No timestamp field found, falling back to now().", level: "warn")
      }

      # -----------------------------------------------------------------------
      # Read hostname
      # Supports the following fields (priority: hostname > host > logsource):
      #   - Logstash:   "host" (often an object: host.name)
      #   - syslog-ng:  "hostname" or "HOST"
      # -----------------------------------------------------------------------
      _r_host = string(.hostname) ?? string(.host.name) ?? string(.host) ?? string(."HOST") ?? string(.logsource) ?? "unknown-host"

      # Sanitize hostname (allow only alphanumeric, hyphen, dot)
      _r_host = replace(_r_host, r'[^a-zA-Z0-9.\-]', "_")

      # Routing metadata for file path (stripped from output)
      ._r_host  = _r_host
      ._r_month = format_timestamp!(ts, format: "%m")
      ._r_day   = format_timestamp!(ts, format: "%d")
      ._r_year  = format_timestamp!(ts, format: "%y")
      ._r_hour  = format_timestamp!(ts, format: "%H")

# -----------------------------------------------------------------------------
# SINK: Structured file storage
# Path: /archive/{MM}/{DD}/{hostname}/{hostname}/{hostname}-{YY}-{MM}-{DD}-{HH}.log
# -----------------------------------------------------------------------------
sinks:
  file_archive:
    type: file
    idle_timeout_secs: 300
    inputs:
      - extract_routing

    # File path template using the extracted routing fields
    path: "/archive/{{ ._r_month }}/{{ ._r_day }}/{{ ._r_host }}/{{ ._r_host }}-{{ ._r_year }}-{{ ._r_month }}-{{ ._r_day }}-{{ ._r_hour }}.log"

    # Encoding: JSON, routing fields are NOT written to output
    encoding:
      codec: json
      except_fields:
        - "_r_host"
        - "_r_month"
        - "_r_day"
        - "_r_year"
        - "_r_hour"

    # Buffer: protects against data loss during short backpressure spikes
    buffer:
      type: memory
      max_events: 3000000
      when_full: block

    # Acknowledgements: when enabled, HTTP source only sends 200 after
    # the event has been successfully written. Currently disabled.
    acknowledgements:
      enabled: false

  # Prometheus metrics exporter
  prometheus_exporter:
    type: prometheus_exporter
    inputs:
      - internal_metrics
    address: "0.0.0.0:9598"

Vector Logs

No response