Interact with Databricks foundation models • chattr

Intro

Databricks customers have access to foundation model APIs like DBRX, Meta Llama 3 70B, and Mixtral 8x7B. Databricks also provides the ability to train and deploy custom models.

chattr supports the following models on Databricks by default:

Supported Databricks pay-per-token foundation models
Model	Databricks Model Name	`chattr` Name
DBRX Instruct	`databricks-dbrx-instruct`	`databricks-dbrx`
Meta-Llama-3-70B-Instruct	`databricks-meta-llama-3-70b-instruct`	`databricks-meta-llama3-70b`
Mixtral-8x7B Instruct	`databricks-mixtral-8x7b-instruct`	`databricks-mixtral8x7b`

Authentication

Databricks requires a host (workspace URL) and token to authenticate. Both are required for any non-Databricks application, such as chattr, to interact with the models in the Databricks workspace.

The token can be generated by the user in the workspace under the developer settings (docs) and the host can be found in the workspaces URL (docs).

By default, chattr will look for the credentials in environment variables:

DATABRICKS_HOST
DATABRICKS_TOKEN

Use Sys.setenv() to set the variable. The downside of using this method is that the variable will only be available during the current R session:

Sys.setenv("DATABRICKS_HOST" = "https://xxxxx.cloud.databricks.com")
Sys.setenv("DATABRICKS_TOKEN" = "####################")

A preferred method is to save the secret key to the .Renviron file. This way, there is no need to load the environment variable every time you start a new R session. The .Renviron file is available in your home directory. Here is an example of the entry:

DATABRICKS_HOST = https://xxxxx.cloud.databricks.com
DATABRICKS_TOKEN = ####################

Change the model

Supported Models

By default, chattr is setup to interact with GPT 4 (gpt-4). To switch to Meta Llama 3 70B you can run:

library(chattr)

chattr_use("databricks-meta-llama3-70b")
#> 
#> ── chattr
#> • Provider: Databricks
#> • Path/URL: serving-endpoints
#> • Model: databricks-meta-llama-3-70b-instruct
#> • Label: Meta Llama 3 70B (Databricks)

Custom Models

If a model doesn’t appear in the supported table but is deployed on Databricks model serving as OpenAI-compatible (configured with llm/v1/chat in mlflow) then you can specify the model name explicitly with chattr_use()

For example if you have deployed a fine-tuned version LLM with an endpoint name of "CustomLLM":

library(chattr)

# use any existing databricks foundation model name (e.g. datarbicks-dbrx)
# then adjust the default model name to 'CustomMixtral'
chattr_use(x = "databricks-dbrx", model = "CustomLLM")
#> 
#> ── chattr
#> • Provider: Databricks
#> • Path/URL: serving-endpoints
#> • Model: CustomLLM
#> • Label: DBRX (Databricks)

Data files and data frames

Because it is information about your environment and work space, by default chattr avoids sending any data files, and data frame information to Databricks. Sending this information is convenient because it creates a shorthand for your requests. If you wish to submit this information as part of your prompts, use chattr_defaults(), for example:

chattr_defaults(max_data_files = 10)
chattr_defaults(max_data_frames = 10)

These two commands will send 10 data frames, and 10 data files as part of your prompt. You can decide the number to limit this by. The more you send, the larger your prompt.

If any of these is set to anything but 0, a warning will show up every time you start the Shiny app:

• Provider: Databricks
• Path/URL: serving-endpoints
• Model: databricks-dbrx-instruct
• Label: DBRX (Databricks)
! A list of the top 10 files will be sent externally to Databricks with every request
To avoid this, set the number of files to be sent to 0 using chattr::chattr_defaults(max_data_files = 0)Î