Migrate to PostHog Cloud

Prior to starting a historical data migration, ensure you do the following:

1. Create a project on our US or EU Cloud.
2. Sign up to a paid product analytics plan on the billing page (historic imports are free but this unlocks the necessary features).
3. Raise an in-app support request (Target Area: Data Management) detailing where you are sending events from, how, the total volume, and the speed. For example, "we are migrating 30M events from a self-hosted instance to EU Cloud using the migration scripts at 10k events per minute."
4. Wait for the OK from our team before starting the migration process to ensure that it completes successfully and is not rate limited.
5. Set the historical_migration option to true when capturing events in the migration.

Requirements

• An existing project, either:
  • on PostHog cloud.
  • on a self-hosted PostHog instance running at least 1.30.0. For upgrade instructions, take a look at this guide.
• A new project on PostHog Cloud.

Approach

This migration has 3 steps:

1. Migrate your metadata (projects, dashboards, insights, actions, cohorts, feature flags, experiments, annotations).

2. Migrate your events, this also creates the necessary person, person distinct ID, and related records.

3. Switch tracking in your product to set-up replication from the old project if needed and to start sending events to the new project.

Migrate your metadata

To migrate metadata like projects, dashboards, insights, actions, feature flags, and more, use the PostHog migrate metadata script. This requires:

1. Installing TypeScript and ts-node. You can do this by running npm install -g typescript ts-node in your terminal.
2. Your old instance project API key.
3. Your new cloud instance project API key, which you can get from your project settings.

Note: This process has the following caveats:

1. Every object's "created by" information will appear as if it was created by you.
2. Every object's "created at" information will appear as if it was created at the time you ran this script.

1. Clone the repo and cd into it
   Terminal

   git clone https://github.com/PostHog/posthog-migrate-meta
   cd posthog-migrate-meta

2. Install the dependencies by running yarn
3. Run the script
   Terminal

   ts-node index.ts --source [posthog instance you want to migrate from] --sourcekey [personal api key for that instance] --destination [posthog instance you want to migrate to.] --destinationkey [personal api key for destination instance]

For more information on the options see the repo's readme

Migrate your events to Cloud

Before you start, disable all the apps in the destination cloud project (e.g. GeoIP). Keeping these enabled may change the events you are migrating.

For more details about historical migrations, see our migration docs.

From self-hosted to Cloud

To migrate your events, you can read data directly from your ClickHouse cluster and ingest the data with the Python library using our self-hosted migration tool.

First, clone the repo and install the requirements.

git clone https://github.com/PostHog/posthog-migration-tools
cd posthog-migration-tools
pip3 install -r requirements.txt

Next, run the migration script with your ClickHouse details, PostHog details, start date, end date, and fetch limit.

python3 ./migrate.py \
   --clickhouse-url https://some.clickhouse.cluster:8443 \
   --clickhouse-user default \
   --clickhouse-password some-password \
   --clickhouse-database posthog \
   --team-id 1234 \
   --posthog-url https://app.posthog.com \
   --posthog-api-token "abx123" \
   --start-date 2023-06-18T13:00:00Z \
   --end-date 2023-06-18T13:10:00 \
   --fetch-limit 10000

This script prints a "cursor" in the case the migration fails. It can be used to resume from where it got to with by adding the --cursor argument to the command above.

python3 ./migrate.py \
   --clickhouse-url https://some.clickhouse.cluster:8443 \
   --clickhouse-user default \
   --clickhouse-password some-password \
   --clickhouse-database posthog \
   --team-id 1234 \
   --posthog-url https://app.posthog.com \
   --posthog-api-token "abx123" \
   --start-date 2023-06-18T13:00:00Z \
   --end-date 2023-06-18T13:10:00 \
   --fetch-limit 10000 \
   --cursor the-cursor-value-from-the-output

Notes:

• This script adds a $lib property of posthog-python, overriding any $lib property already set.
• If the script fails for some reason, just run it again with the latest cursor. There are some transient issues that can happen that are solved by re-running the script.

Between Cloud instances (e.g. US -> EU Cloud)

You must raise a support ticket in-app for the PostHog team to do this migration for you. This option is only available to customers on the team or enterprise plan as it requires significant engineering time.

Switching tracking in your product

To make sure your person properties get the latest values, don't switch over tracking until historical events have been migrated (we'll keep a daily export in the old project for the incoming events, so you don't need to worry about missing any events).

Now that we've migrated our events, the next step is to switch over tracking within your product to direct any new events to your new PostHog cloud instance.

1. Re-enable any apps that you disabled earlier (e.g. GeoIP).

2. Begin swapping out your Project API key and instance address within all the areas of your product that you track. Once done, events using the new API key will go directly to your Cloud instance.

Migrating your custom apps

If the app was filtering or transforming events before ingestion:

1. Check if we already have an app that does what you need (fastest option). You can see the list of apps here, look for ingestion filtering and event transformation.

2. Move this logic your app before you send the event (also fast).

3. If you can make your app generalizable enough that others can benefit, submit your app to the store. To do this, convert anything specific to your configuration into a plugin.json config value.

If the app is used to send events to a custom destination:

1. Check to see if we already have an app or batch export that does what you need (fastest option). You can see the list of apps here, look for batch exports and destinations.

2. Convert your app to work as a webhook (also fast). We are soon releasing a webhook destination. You can subscribe for updates on the roadmap

3. If you can make your app generalizable enough that others can benefit then submit your app to the store. To do this, you must convert anything specific to your configuration into a plugin.json config value

If the options above don't work and you were previously paying a substantial amount self-hosting, then email us at sales@posthog.com with a link to the public GitHub repo and we can see if it's appropriate as a private cloud app.