Make sure the Judgment CLI is installed before proceeding. Please refer to Installation for more information.

During the setup process, .tfstate files will be generated by Terraform. These files keep track of the state of the infrastructure deployed by Terraform. DO NOT DELETE THESE FILES.

​
Introduction

The self-host command is used to deploy and manage your own self-hosted instance of Judgment. self-host main is used to deploy a Supabase project along with the Judgment AWS infrastructure. self-host https-listener is used to configure HTTPS for the Judgment instance, and is only required to run if you don’t proceed with the listener setup as part of the self-host main command.

​
Usage

To see usage information, run any of the following:

judgment self-host --help
judgment self-host main --help
judgment self-host https-listener --help

​
1. Prerequisites

Before proceeding, ensure you have:

  1. An empty AWS account that has been registered with us for self-hosting (for hosting the self-hosted Judgment instance)
  2. An Osiris API key from Judgment Labs (if you plan to use Osiris for evaluations)
  3. An available email address and the corresponding app password (learn about app passwords for Gmail, Outlook, Yahoo, Zoho, or Fastmail) for the email address (for sending email invitations to users on the self-hosted instance)
  4. A Supabase account and organization that you have admin access to (for data storage)

Refer to the Getting Started with Self-Hosting guide for more information about the prerequisites. Make sure you have followed all steps in Part 1 and 2 of the guide before proceeding.

​
AWS CLI Setup

You’ll need to install and configure AWS CLI with the AWS account mentioned in the above warning.

brew install awscli

After installation, configure your local environment with the relevant AWS credentials:

aws configure

​
Terraform CLI Setup

Terraform CLI is required for deploying the AWS infrastructure.

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

​
2. Deploying

To deploy a self-hosted instance of Judgment:

  1. Create a credentials file (e.g., creds.json) with the following format:
{
    "supabase_token": "your_supabase_personal_access_token_here",
    "org_id": "your_supabase_organization_id_here",
    "db_password": "your_desired_supabase_database_password_here",
    "invitation_sender_email": "email_address_to_send_org_invitations_from",
    "invitation_sender_app_password": "app_password_for_invitation_sender_email",
    "osiris_api_key": "your_osiris_api_key_here (optional)",
    "openai_api_key": "your_openai_api_key_here (optional)",
    "togetherai_api_key": "your_togetherai_api_key_here (optional)",
    "anthropic_api_key": "your_anthropic_api_key_here (optional)"
}

For supabase_token, to retrieve your Supabase personal access token, you can either use an existing one or generate a new one here.

For org_id, you can retrieve it from the URL of your Supabase dashboard (make sure you have the correct organization selected in the top left corner, such as Test Org in the image below). For example, if your organization URL is https://supabase.com/dashboard/org/uwqswwrmmkxgrkfjkdex, then your org_id is uwqswwrmmkxgrkfjkdex.

db_password can be any password of your choice. It is necessary for creating the Supabase project and can be used later to directly connect to the project database.

invitation_sender_email and invitation_sender_app_password are required because the only way to add users to the self-hosted Judgment instance is via email invitations.

The four LLM API keys are optional. If you are not planning to run evaluations with the models that require any of these API keys, you do not need to specify them.

  1. Run the main self-host command. The command syntax is:
judgment self-host main [OPTIONS]

Required options:

  • --root-judgment-email or -e: Email address for the root Judgment user
  • --root-judgment-password or -p: Password for the root Judgment user
  • --domain-name or -d: Domain name to request SSL certificate for (make sure you own this domain)

Optional options:

For --supabase-compute-size, only “nano” is available on the free tier of Supabase. If you want to use a larger size, you will need to upgrade your organization to a paid plan.

  • --creds-file or -c: Path to credentials file (default: creds.json)
  • --supabase-compute-size or -s: Size of the Supabase compute instance (default: small)
    • Available sizes: nano, micro, small, medium, large, xlarge, 2xlarge, 4xlarge, 8xlarge, 12xlarge, 16xlarge
  • --invitation-email-service or -i: Email service for sending organization invitations (default: gmail)
    • Available services: gmail, outlook, yahoo, zoho, fastmail

Example usage:

judgment self-host main \
--root-judgment-email root@example.com \
--root-judgment-password password \
--domain-name api.example.com \
--creds-file creds.json \
--supabase-compute-size nano \
--invitation-email-service gmail

This command will:

  1. Create a new Supabase project
  2. Create a root Judgment user in the self-hosted environment with the email and password provided
  3. Deploy the Judgment AWS infrastructure using Terraform
  4. Configure the AWS infrastructure to communicate with the new Supabase database
  5. * Request an SSL certificate from AWS Certificate Manager for the domain name provided
  6. ** Optionally wait for the certificate to be issued and set up the HTTPS listener

* For the certificate to be issued, this command will return two DNS records that must be manually added to your DNS registrar/service.

** You will be prompted to either continue with the HTTPS listener setup now or to come back later. If you choose to proceed with the setup now, the program will wait for the certificate to be issued before continuing.

​
3. Setting up the HTTPS listener

This step is optional; you can choose to have the HTTPS listener setup done as part of the main self-host command.

This command will only work after judgment self-host main has already been run AND the ACM certificate has been issued. To accomplish this:

  1. Add the two DNS records returned by the main self-host command to your DNS registrar/service
  2. Monitor the ACM console here until the certificate has status ‘Issued’

To set up the HTTPS listener, run:

judgment self-host https-listener

This command will:

  1. Set up the HTTPS listener with the certificate issued by AWS Certificate Manager
  2. Return the url to the HTTPS-enabled domain which now points to your self-hosted Judgment server