* Integrate fastify-postgres and fastify-kafka * Explicitly add node-rdkafka for its types * Port UUIDT from Python * Consume events from Kafka * Update yarn.lock * Port handle_timestamp * Enhance startFastifyInstance with config * Prettier * Handle dates with Luxon for similarity to Python's datetime * Fix UUIDT stringification * Port _capture_ee * Create EventsProcessor with plugins server access * Rework ingestion to separate from Fastify, port over more Django * Update utils.test.ts * Remove fastify-kafka and fastify-postgres * Add protobuf * Add types * Update yarn.lock * Improve info * Update server.ts * Port element functions * Use Kafka producer stream and make overall optimizations * Add session recording * Add plugin processing * Fix castTimestampOrNow * Update yarn.lock * Improve typing and consume event messages in batches of 100 * Improve code clarity * Add timing with StatsD * Update style * Format * Merge and alias * Are last stuff * Reimplement Protobuf with protos compiled to JS+TS * Fix things * Fix UUID#toString implementation * Update UUID number handling * Unify * Prettier * Unify utils * Add newline * Prettier but correctly * Fix types * Improve types * Bump version to 0.5.0 * Update worker.test.ts * Update types.ts * Update yarn.lock * Update compile:protobuf with eslint --fix * Fix typing * Don't bump version * Fix style * Use @posthog/node-rdkafka for Worker thread support * Improve logging * Unify Redis and Kafka approach to queuing events * Fix consuming from Kafka * Make some optimizations * Don't introduce KAFKA_EVENTS_HANDOFF * Consume 1 kiloevent / 50 ms = 20 kiloevents / s * Don't start web server by default We don't need it at all yet. * Improve Redis logging * Fix connecting to dockerized Redis No idea why this fixes the issue, but it does. * Update yarn.lock * Fix merge * Clear Kafka consumption interval on graceful exit * Improve logging and error handling * Smooth out ingestion errors * Fix StatsD prefixing * Move @types/luxon to devDependencies * Use Kafka.js Producer instead of node-rdkafka * Use EventProto#encodeDelimited instead of #encode * Start UUIDT series from 0 instead of 1 * Use event UUID from Django server * Make some fixes and improvements * Remove console.logs * Use plugin-scaffold 0.2.7 * Fix style camelCase FTW * Simplify compile:typescript * await startQueue * Clean up castTimestampOrNow * Don't use pesky ||= * Change consumer group to 'group1' from main repo * Revert "await startQueue" This reverts commit 1ca29ba2c6db22dceef4fb02dac5136846d4dfa1. * Exit if queue failed to start * Clean up elements handling * Don't commit compiled protos * Rename consumer group to clickhouse-ingestion * Consume from topic events_ingestion_handoff * ee dev script * Backport set_once and event name sanitization * Add direct ClickHouse support base * Upgrade plugin-scaffold to 0.2.8 * Add clickhouse to server object * Support ClickHouse persons and person distinct IDs * Update README with ClickHouse settings * Fix sanitizeEventName * Await queue start * Prettier README * Try ClickHouse CI skeleton * Update yarn.lock * Possibly fix CI * Revert basic tests job to old name * Cache pip install * Fix ClickHouse teardown and add Yarn installation * Await startQueue in tests * Try existing test in new workflow * Don't cache pip requirements * Add Redis to CH CI * Make some env var changes * Try test_default as Django test DB name * Try test_posthog as Django test DB name * Don't tear down ClickHouse * Make env vars common to all tests-clickhouse steps * Debug Postgres * Debug Postgres actually * Revert "Make env vars common to all tests-clickhouse steps" This reverts commit 07bfd2917fc18b6674649fdd196432ed5c8e6a37. * Try fixing DB name discrepancy * Make some env vars common * Debug Postgres better * Remove Postgres debug code * Fix inserting * Fix posthog_team.app_urls * Catch insertRow errors * Add plugin.organization_id in base tests * DELETE FROM posthog_organization * Rework CI to use Django everywhere * Reorder tests for readability * Fix missing KAFKA_HOSTS in CI * Optimize createPosthog * Start separating Postgres- and ClickHouse-based suites * Fix regexes possibly * Fix regexes actually possibly * Improve fatal error handling * Fix type * Fix import * Debug Kafka queue crash * Debug consumer * Fix organizationId * Make some more suites Postgres-specific * Check out master branch of main repo of Django server in CI * Start schedule before queue * Clean up logging * Update Dockerfile * Refactor Postgres processing to DB class * Use more of DB methods * Quit Redis * Update test database name * Debug consumer * Don't mock KafkaJS! * Add new Kafka testing system incomplete * Clean up new testing system * Make prettier compatible with ESLint * Add compatibility with Plugin.latest_tag in test * Fix minor issues * Try not terminating program on Kafka consumer crash * Change KAFKA_ADVERTISED_HOST_NAME to localhost * Address feedback * Separate Postgres and DB classes * Consume from kafka_events topic in KafkaObserver * Fix process-event test file name * Check for UUID in handed off events * Return from EventsProcessor.processEvent * use `db.postgresQuery` instead of vague `db.query` * Run the plugins suite universally * Fix closeServer * javaScriptVariableCase * Capture handleTimestamp error with Sentry * Hand off message in test * Fix plugins suite imports * Run the vm suite universally * more javaScriptVariableCase * disconnect without waiting for pending replies (.diconnect() instead of .quit()) * dot quit seems to work better? * Add sanitizeSqlIdentifier * Fix updating person properties * Fix problems * Don't use status in piscina.js * Revert "Run the vm suite universally" This reverts commit 966056fe8b1479c85cdcd0494df5ce58e50f103a. * Fix sanitizeSqlIdentifier test * Use waitForProcessedMessages * Optimize updatePersonProperties * Rework DummyPostHog to capturing internal server events efficiently * Fix nested test files not being ran * Run prettier over Markdown * Fix mock paths * Fix issues * Fix some * Don't truncate Kafka tables * Start kafkaObserver inside test * Support Team.is_demo * Increase process-event tests timeout to 60 s * Cache Python location in ClickHouse CI * Debug KafkaObserver * Cache pip in all CI jobs * Add bash script for running tests locally * Try out a different way of watching Kafka messages * Simplify DummyPostHog back again * Use posthog-js-lite in EventsProcessor * Sanitize SQL identifiers more strictly Remove all characters in identifiers that are neither letter, digit or underscore. * Don't resolve KafkaObserver start before connection * Add ee.tasks.webhooks_ee.post_event_to_webhook_ee * Expect Client to have been called twice * Adjust resetTestDatabase for Organization.personalization * Update Cliient call asserts * Postgres ingestion + process event tests (#116) * start with the postgres event ingestion process event tests * get first test to work * remove siteUrl * pass partial event to test that it still works and retain parity with the django test * another test * refactor * add more tests * opt out of posthog in test mode * add first alias test * always use UTC times when talking to postgres * prevent a crash * bit of clarity to help debug * fix bug with table name * fix bug with passing object instead of id * add some alias tests (all green now) * save merged properties * few more tests * more missing tests * fix test * team event properties test * fix bug * another test (partial) * clarify postgres magic * different timestamp format for creating event in clickhouse & postgresql * make element tests fail * capture first team event test * insert session recording events * generate element hashes * create elements and element groups * "key in object" only works with objects, not arrays * test an extra thing * add missing awaits that caused things to be done out of order * few extra tests * another test * await for things to happen * fix to work with latest master * client is called twice - it's initialized for sending celery tasks on webhooks as well * check webhook celery client queue * split into postgres & shared process event test * add query counter * clickhouse process event tests v0.1 * fix vm test * fix "Your test suite must contain at least one test." error for shared tests * also run non-ingestion tests under test:postgres * Clean up utils.ts * Clean up TimestampFormat * Put get* type Postgres functions in DB class Co-authored-by: Michael Matloka <dev@twixes.com> * postgres e2e ingestion test (#120) * Kafka e2e tests (#121) * kafka e2e test * kafka e2e test * kafka host from env * get some ch process event tests working * more working clickhouse tests * wait a bit longer * add chainToElements, fix elementsToString bug * remove quotes from inside sanitizeSqlIdentifier to also work with clickhouse * split dev setup command * fetch elements from clickhouse * more elements * bugfix * refactor and reuse wait function * ingest kafka events with the right structure * remove leftover * fix clickhouse timestamp * simplify process event creation and call the methods directly * fix uuid test * refactor delayed event fetching to support session recording events * catch bad uuids * wait for session recording events in test * use right timestamp for session recording events * use the same database as posthog (the app) * use local db * deserialize clickhouse session recording events * split dev scripts * try to make tests work by specifying db * increase kafka log level * cleanup * pass idl protos to clickhouse in github actions * start the clickhouse container in another step * let's try like this * WIP * WIP * sudo * also alias zookeeper * export zookeeper * use docker-compose.ch.yml * detached * element group test * create tests * debug test * remove some redundancy * reduce some noise * try to make topics * compatible with posthog migration 0122 * hide error * try to close e2e open handles * reuse kafkaProducer on server * Add DB.clickhouseQuery * Put isUUIDFormat on the UUID class Co-authored-by: Michael Matloka <dev@twixes.com> * Update README.md with PLUGIN_SERVER_INGESTION * rename CLICKHOUSE_USERNAME --> CLICKHOUSE_USER for consistency with the rest of the app * Delete run-tests-locally.sh * Resolve all kafka offsets (#124) * resolve all kafka offsets * remove a few lines * clean up * sort events according to offsets * add sort info * simplify offset line * add comment * Type pluginEvents as PluginEvent[] * Commit last offset of batch manually Co-authored-by: Michael Matloka <dev@twixes.com> * Add KAFKA_INCOMING_TOPIC and clean up unused code (#125) * ingest to clickhouse only if PLUGIN_SERVER_INGESTION is enabled * KAFKA_INCOMING_TOPIC * test clickhouse connection * remove some unused code * Replace _INCOMING_ with _CONSUMPTION_ * Add KAFKA_CONSUMPTION_TOPIC to README.md * add // Co-authored-by: Michael Matloka <dev@twixes.com> * Postgres parity tests (#126) * create postgres parity tests, fix some bugs * create all topics * person and distinct id parity tests * add TODOs * fetch distinct ids from clickhouse * create a specialised function for moving distinct ids and fix postgres/clickhouse person_id difference (number vs string) * test for updating distinct ids * createPerson, updatePerson and deletePerson * add a debug line to help debug flaky github action * remove falsehood * Remove "_handoff" from Kafka topic (#127) * remove _HANDOFF from topic * add plugin_ to plugin server ingestion topic * User merge test (#129) * merge test and postgres query simplification * postgres fix * remove "_HANDOFF" * small comment * Switch ClickHouse driver (#128) * Use driver @posthog/clickhouse instead of clickhouse * Update CH querying * Fix clickhouseQuery usage * Don't quote 64-bit ints in JSON from CH Co-authored-by: Marius Andra <marius.andra@gmail.com>

Uh oh!

Add documentation for the deployment #34

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions