1. Abstract
This guide explains how to integrate the Board platform with Databricks across different deployment scenarios. It focuses on two primary connectivity approaches: JDBC-based integration via Hybrid data pipeline for Board Cloud environments, and ODBC-based integration for Board On-Premises installations. The article outlines the required configurations, authentication methods, prerequisites, and key considerations to establish a reliable and performant connection to Databricks SQL warehouses.
2. Context
Modern data architectures frequently rely on platforms like Databricks to manage and process large-scale data using distributed compute engines. To enable planning, analytics, and reporting in Board, this data must be made accessible through supported connectivity methods.
Board supports different integration patterns depending on the deployment model:
- In Board Cloud environments, connectivity to Databricks is typically established using JDBC via Data Pipeline, which acts as the central integration layer.
- In Board On-Premises environments, connectivity is achieved using ODBC drivers installed locally on the Board server.
In more advanced or secure network configurations (for example, when Databricks is deployed behind private endpoints or restricted networks), the On-Premises Connector (OPC) can be introduced as part of the Data Pipeline architecture to bridge connectivity. While OPC is not covered in detail in this guide, it may be relevant in scenarios where direct outbound connectivity from Data Pipeline is not possible.
Because Databricks is built on a Spark-based engine, there are some important considerations—such as metadata handling and SQL behavior—that differ from traditional relational databases. These should be taken into account when designing and configuring the integration.
This guide is intended for technical users responsible for implementing and managing Board integrations with Databricks across both cloud and on-premise environments.
3. Content
3.1 Authentication Methods
Databricks supports the following authentication methods for both JDBC and ODBC:
Method | Description | Recommendation |
|---|
Personal Access Token (PAT) | Token generated in Databricks workspace | Simplest and most commonly used |
OAuth 2.0 (Service Principal) | Machine-to-machine authentication using Azure AD / service principal | Enterprise / secure environments |
3.2 JDBC Connection (Board Cloud via Data pipeline)
Step 1- Create a new JDBC connection in data pipeline
The Databricks JDBC (Simba) driver is already available in Data pipeline by default, so no manual driver upload is required.
To begin, simply create a new JDBC connection
Step 2 - Configure JDBC Connection in Data pipeline
- Driver Class – com.databricks.client.jdbc.Driver
- Server Hostname (e.g. adb-xxxx.azuredatabricks.net or dbc-xxxx.cloud.databricks.com)
- Port: 443
- HTTP Path (from SQL Warehouse- e.g. /sql/1.0/warehouses/bbdd757bde2899ee)
- Catalog/Schema (optional but recommended)
JDBC Configuration - PAT Authentication
Connection URL:
- jdbc:databricks://<server-hostname>:443;
- transportMode=http;
- ssl=1;
- httpPath=<http-path>;
- AuthMech=3;
- UID=token;
- PWD=<personal-access-token>;
- ConnCatalog=<catalog>;
- ConnSchema=<schema>;
In this configuration:
- AuthMech=3 indicates PAT authentication
- UID=token is required
- PWD contains the personal access token
Databricks identifies AuthMech=3 as PAT authentication for the JDBC driver.
Example:
jdbc:databricks://adb-2872436780791176.16.azuredatabricks.net:443;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/bbdd757bde2899ee;ConnCatalog=samples;ConnSchema=nyctaxi;socketTimeout=0;
JDBC Configuration – OAuth 2.0 (Service Principal)
Connection URL:
- jdbc:databricks://<server-hostname>:443;
- transportMode=http;
- ssl=1;
- httpPath=<http-path>;
- AuthMech=11;
- Auth_Flow=1;
- OAuth2ClientId=<Client-ID>;
- OAuth2Secret=<Client-Secret>;
- ConnCatalog=<catalog>;
- ConnSchema=<schema>;
In this configuration:
- AuthMech=11 indicates OAuth 2.0 authentication
- Auth_Flow=1 indicates the use of the OAuth 2.0 Client Credentials flow (M2M)
- OAuth2ClientId – The application ID of the service principal
- OAuth2Secret – The client secret for the service principal
Example:
jdbc:databricks://adb-2872436780791176.16.azuredatabricks.net:443/default;transportMode=http;ssl=1;httpPath=/sql/1.0/warehouses/bbdd757bde2899ee;AuthMech=11;Auth_Flow=1;OAuth2ClientId=********************;OAuth2Secret=**************************;
Important: Avoid exposing full catalogs or schemas in Databricks when configuring the connection.
Always define:
- ConnCatalog=<catalog>
- ConnSchema=<schema>
This helps:
- Improve metadata discovery performance
- Prevent timeouts during schema browsing
- Reduce unnecessary load on the Databricks environment
3.3 ODBC Connection
For on-premise Board installations, connectivity to Databricks is established using the Databricks ODBC driver installed on the Board server.
Step 1 - Install ODBC Driver
- Install the Databricks ODBC Driver
Step 2 - Configuration
ODBC Configuration PAT Authentication
In the HTTP Properties, setup the HTTP Path which is the unique path for the specific Databricks SQL Warehouse or cluster
Advanced Options: Server Side Properties
The catalog and schema are set using Server Side Properties (databricks.catalog, databricks.schema) to limit metadata exposure.
Set databricks.catalog and databricks.schema to limit metadata exposure and improve performance.
ODBC Configuration OAuth 2.0 (Service Principal)
Except for the authentication the rest of the configuration remains the same (SSL options, HTTP Properties, Server Side Properties)
3.4 Board Configuration and Schema Discovery
Data Source Configuration and SQL Dialect
Schema Discovery and Browse in Data Reader
Note – Metadata Performance:
When using Databricks foreign catalogs, metadata discovery and browsing may be slower and can result in timeouts. Where possible, limit the scope of the catalog or schema exposed to Board to improve metadata browsing performance.
3.5 Writeback and Bulk Insert
Board supports writeback to Databricks using Bulk Insert for both Cloud and On-Premises deployments. The required configuration depends on whether the connection is established through JDBC or ODBC.
Bulk Insert enables Board to efficiently write multiple records to a Databricks table as part of a Dataflow or other writeback process.
3.5.1 Board On-Premises via ODBC
For Board On-Premises environments, writeback to Databricks can be performed using the Databricks ODBC driver.
To enable Bulk Insert, configure the Databricks ODBC connection with:
This enables the driver to correctly process the parameterized SQL statements generated by Board during Bulk Insert operations.
Once the ODBC connection is configured, a Board Dataflow can use the Databricks Data Source as a target and write records directly to the Databricks table.
Note: The Board service account must have the required permissions on the target Databricks catalog, schema, and table.
3.5.1 Board Cloud via Data Pipeline - JDBC
For Board Cloud environments, writeback to Databricks is supported through Data Pipeline using the Databricks JDBC driver.
Bulk Insert operations use parameterized SQL statements. To ensure these statements are handled correctly by the Databricks JDBC driver, the Data Pipeline environment requires a driver-specific configuration that disables the use of JDBC national character APIs for this driver.
The Databricks JDBC driver must be configured with:
This configuration is applied through a driver-specific Data Pipeline datastore profile associated with:
com.databricks.client.jdbc.Driver
Because the profile is associated specifically with the Databricks JDBC driver class, it does not affect other JDBC drivers or Data Sources configured in Data Pipeline.
Once this configuration is applied, Board can use the Databricks Data Source as the target of a Dataflow and perform parameterized Bulk Insert operations directly against Databricks tables.
Note: The required Databricks JDBC driver and Data Pipeline configuration must be available in the Board Cloud environment. If writeback is required for an existing environment, contact Board Support to confirm that the required configuration is enabled.
3.6 Best Practices
Limit Metadata Scope
- JDBC
- ConnCatalog=<catalog>
- ConnSchema=<schema>
- ODBC:
- databricks.catalog=<catalog>
- databricks.schema=<schema>
These parameters serve the same purpose in JDBC and ODBC, but use different naming conventions.
Use SQL Warehouse
Prefer Databricks SQL Warehouse (Serverless)
- Faster startup/cluster spin-up- reduces connection delays and overhead
- Better Performance and concurrency
- More stable connectivity
4. Conclusion
Integrating Board with Databricks provides a flexible and scalable approach for accessing and working with data stored in the Databricks platform.
Board can connect to Databricks through JDBC in Board Cloud environments using Data Pipeline, and through ODBC in Board On-Premises environments. These connections support both data retrieval and writeback scenarios.
With the appropriate driver configuration in place, Board can perform direct writeback to Databricks using Bulk Insert, allowing Databricks to be used not only as a source for reporting and analysis, but also as a target for Board data integration processes.
As with any integration, the final configuration should take into account authentication, network connectivity, driver settings, permissions, and the expected data volumes to ensure reliable operation.