Connecting to Microsoft SharePoint

For information on the available WHL files for supported environments, and how to install the appropriate file for your Python distribution, see Package Installation.

For information on the module to import, and how to configure the necessary connection properties in a connection string, see Establishing a Connection.

Other available connection properties can be used to configure other aspects of the connector capabilities.

Python Version Support

The CData Python Connector for Microsoft SharePoint can be installed and used in Python 3.10 or newer.

Microsoft SharePoint Version Support

The connector supports all versions of Microsoft SharePoint that support the SOAP API. This includes: Windows SharePoint Services 3.0, SharePoint Server 2007+ (2010, 2013, etc.), and SharePoint Online. The connector models the custom lists of your SharePoint site as bidirectional tables; when you connect, the connector retrieves the metadata for these tables by calling SharePoint Web services. Supported authentication schemes are NTLM, Basic, Digest, Forms, Kerberos, SSO, STS (security token services), and SharePoint authentication cookies.

See Also

• Using the Connector: Establish connections and query Microsoft SharePoint through Python code.
• From SQLAlchemy: Use SQLAlchemy to establish a connection with dialect URL, and interact with Microsoft SharePoint data using mapped classes and Sessions.

Dependencies

The Python connectors require that Python 3.10 or newer be installed.

Installation

The CData Python Connector for Microsoft SharePoint is available as a WHL file for Windows, Linux, and Mac. Each connector is built using the Python 3.10 Stable ABI (indicated by the abi3 tag in the filename), so a single wheel supports any Python 3.10 or newer installation — there is no need to match your exact Python minor version. Use the "pip install" command with the appropriate WHL file for your platform.

Windows:

pip install cdata_sharepoint_connector-26.0.9655-cp310-abi3-win_amd64.whl

Linux:

pip install cdata_sharepoint_connector-26.0.9655-cp310-abi3-linux_x86_64.whl

macOS:

pip install cdata_sharepoint_connector-26.0.9655-cp310-abi3-macosx_12_0_arm64.whl

The macOS wheel supports arm64 (Apple Silicon) architectures only on macOS 12 and newer.

Regardless of the environment, certain distributions might require that the "pip3 install" command be used instead, to differentiate from a Python 2 distribution that might exist already. After installation, confirm whether the connector is successfully installed by running the "pip list" command. If "cdata_sharepoint_connector" is present in the list output by the command, then the installation was successful.

Upgrading

When upgrading, "pip install" does not automatically clean up old JRE files. To avoid leftover files that could cause JVM errors, uninstall the previous version before installing the new one.

Licensing

After the installation is complete, a separate step is needed to activate a license for the connector. Among the CData assets in the distribution's site packages, there is an install-license tool that activates this license. From within the distribution's site-packages folder, after navigating to the "cdata/installlic_sharepoint" folder, simply use a command like the below to activate the license. Omitting the <key> argument activates a trial license:

• Windows:
  

  ./install-license.exe <key>

• Linux / Mac:
  

  ./install-license.sh <key>

Sometimes, file access issues may cause pip to install the connector in a fallback file path that is not the python distribution's main or primary site-packages location. This can make it difficult to find where the connector was installed, and from there, the license activator. In that event, this python script below will print out the full file path of the connector's native file. This file will be stored in the mentioned cdata folder, from which the installlic_sharepoint folder is trivial to find:

import os
import cdata.sharepoint
path = os.path.abspath(cdata.sharepoint.__file__)
print(path)

Uninstallation

If the connector needs to be uninstalled for any reason, do so by running the pip uninstall command, as in the example below:

pip uninstall cdata-sharepoint-connector

The objects available within our connector are accessible from the "cdata.sharepoint" module. To use the module's objects directly:

1. Import the module as follows:
   

   import cdata.sharepoint as mod

2. To establish a connection string, call the connect() method from the connector object using an appropriate connection string, such as:
   

   mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

Connecting to Microsoft SharePoint

Regardless of whether you will connect online or on-premises, what architecture will be used, and which Lists and Documents will be accessed, connecting to Microsoft SharePoint requires exactly two things:

• Set the URL connection property.
• Set the appropriate authentication properties for your working environment.

Setting URL:

Microsoft SharePoint works with all Lists and Documents in the global Microsoft SharePoint site, or all Lists and Documents at individual sites.

To work with all Lists and Documents in the global Microsoft SharePoint site, set the URL connection property to your Site Collection URL. For example:

https://teams.contoso.com

To work with all Lists and Documents at an individual site, set the URL connection property to your individual site URL. For example:

 https://teams.contoso.com/TeamA

The following sections describe how to set the appropriate authentication properties for your working environment. For information about how to create a custom OAuth application (required for use with AzureAD in a Web application; optional for AzureAD access via a Desktop application or a Headless Server), see Creating a Custom Entra ID (Azure AD) Application.

Microsoft SharePoint Online

Set SharePointEdition to "SharePoint Online" and set the User and Password to the credentials you use to log onto SharePoint, for example, the credentials to your Microsoft Online Services account.

Microsoft SharePoint online supports a number of cloud-based architectures, each of which supports a different set of authentication schemes:

• Microsoft Entra ID (Azure AD)
• Single sign-on (SSO) via the ADFS, Okta, OneLogin, or PingFederate SSO identity provider
• Azure MSI
• Azure Password
• OAuthJWT

If the user account domain is different from the domain configured with the identity provider, set SSODomain to the latter. This property may be required for any SSO.

Microsoft Entra ID (Azure AD)

Note: Microsoft has rebranded Azure AD as Entra ID. In topics that require the user to interact with the Entra ID Admin site, we use the same names Microsoft does. However, there are still CData connection properties whose names or values reference "Azure AD".

Microsoft Entra ID (Azure AD) is a connection type that leverages OAuth to authenticate. OAuth requires the authenticating user to interact with Microsoft SharePoint using an internet browser. The driver facilitates this in several ways as described below.

Your organization may require Admin Consent when authorizing a new AzureAD application for your Azure Tenant. In all AzureAD flows, any initial installation and use of an AzureAD application requires that an administrator approve the application for their Azure Tenant. For details, see Creating a Custom Entra ID (Azure AD) Application.

Desktop Applications

CData provides an embedded OAuth application that simplifies OAuth desktop authentication. Alternatively, you can create a custom AzureAD application. See Creating a Custom Entra ID (Azure AD) Application for information about creating custom applications and reasons for doing so.

For authentication, the only difference between the two methods is that you must set two additional connection properties when using custom AzureAD applications.

After setting the following connection properties, you are ready to connect:

• InitiateOAuth: GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
• CallbackURL: The Redirect URL in your application settings.
• Custom applications only:
  • OAuthClientId: The client Id in your application settings.
  • OAuthClientSecret: The client secret in your application settings.

When you connect the connector opens the OAuth endpoint in your default browser. Log in and grant permissions to the application. The connector then completes the OAuth process:

1. Extracts the access token from the callback URL and authenticates requests.
2. Obtains a new access token when the old one expires.
3. Saves OAuth values in OAuthSettingsLocation. These stored values persist across connections.

Web Applications

When connecting via a Web application, you must create and register a custom AzureAD application with Microsoft SharePoint. See Creating a Custom Entra ID (Azure AD) Application for more information about custom applications. You can then use the connector to acquire and manage the OAuth token values.

Get an OAuth access token:

Set the following connection properties to obtain the OAuthAccessToken:

• OAuthClientId: The client Id in your application settings.
• OAuthClientSecret: The client secret in your application settings.

Call stored procedures to complete the OAuth exchange:

1. Call the GetOAuthAuthorizationURL stored procedure. Set the CallbackURL input to the callback URL you specified in your application settings. If necessary, set the Scope parameter to request custom permissions.

   The stored procedure returns the URL to the OAuth endpoint.

2. Open the URL, log in, and authorize the application. You are redirected back to the callback URL.
3. Call the GetOAuthAccessToken stored procedure. Set the AuthMode input to WEB. Set the Verifier input to the "code" parameter in the query string of the callback URL. If necessary, set the Scope parameter to request custom permissions.

Once you have obtained the access and refresh tokens, you can connect to data and refresh the OAuth access token either automatically or manually.

Automatic refresh of the OAuth access token:

To have the driver automatically refresh the OAuth access token, set the following on the first data connection:

• InitiateOAuth: REFRESH.
• OAuthClientId: The client Id in your application settings.
• OAuthClientSecret: The client secret in your application settings.
• OAuthAccessToken: The access token returned by GetOAuthAccessToken.
• OAuthRefreshToken: The refresh token returned by GetOAuthAccessToken.
• OAuthSettingsLocation: The location where the connector saves the OAuth token values, which persist across connections.

On subsequent data connections, the values for OAuthAccessToken and OAuthRefreshToken are taken from OAuthSettingsLocation.

Manual refresh of the OAuth access token:

The only value needed to manually refresh the OAuth access token when connecting to data is the OAuth refresh token.

Use the RefreshOAuthAccessToken stored procedure to manually refresh the OAuthAccessToken after the ExpiresIn parameter value returned by GetOAuthAccessToken has elapsed, then set the following connection properties:

• OAuthClientId: The client Id in your application settings.
• OAuthClientSecret: The client secret in your application settings.

Now call RefreshOAuthAccessToken with OAuthRefreshToken set to the OAuth refresh token returned by GetOAuthAccessToken. After the new tokens have been retrieved, open a new connection by setting the OAuthAccessToken property to the value returned by RefreshOAuthAccessToken.

Finally, store the OAuth refresh token so that you can use it to manually refresh the OAuth access token after it has expired.

Headless Machines

To configure the driver to use OAuth with a user account on a headless machine, you must authenticate on another device that has an internet browser.

1. Choose one of these two options:
   • Option 1: Obtain the OAuthVerifier value as described in "Obtain and Exchange a Verifier Code" below.
   • Option 2: Install the connector on another machine and transfer the OAuth authentication values after you authenticate through the usual browser-based flow, as described in "Transfer OAuth Settings" below.

2. Now configure the connector to automatically refresh the access token from the headless machine.

Option 1: Obtain and Exchange a Verifier Code

To obtain a verifier code, you must authenticate at the OAuth authorization URL.

Follow the steps below to authenticate from the machine with an internet browser and obtain the OAuthVerifier connection property.

1. Choose one of these options:
   • If you are using the Embedded OAuth Application click Microsoft SharePoint OAuth endpoint to open the endpoint in your browser.
   • If you are using a custom OAuth application, create the Authorization URL by setting the following properties:
     • InitiateOAuth: OFF.
     • OAuthClientId: The client Id assigned when you registered your application.
     • OAuthClientSecret: The client secret assigned when you registered your application.
     Now call the GetOAuthAuthorizationURL stored procedure with the appropriate CallbackURL. Open the URL returned by the stored procedure in a browser.

2. Log in and grant permissions to the connector. You are redirected to the callback URL, which contains the verifier code.
3. Save the value of the verifier code. Later you will set this in the OAuthVerifier connection property.

Next, you must exchange the OAuth verifier code for OAuth refresh and access tokens. Set the following properties:

On the headless machine, set the following connection properties to obtain the OAuth authentication values:

• InitiateOAuth: REFRESH.
• OAuthVerifier: The verifier code.
• OAuthSettingsLocation: The location of the file where the driver saves the OAuth token values that persist across connections.
• Custom applications only:
  • OAuthClientId: The client Id in your custom OAuth application settings.
  • OAuthClientSecret: The client secret in the custom OAuth application settings.

After the OAuth settings file is generated, you must re-set the following properties to connect:

• InitiateOAuth: REFRESH.
• OAuthSettingsLocation: The location containing the encrypted OAuth authentication values. Make sure this location grants read and write permissions to the connector to enable the automatic refreshing of the access token.
• Custom applications only:
  • OAuthClientId: The client Id assigned when you registered your application.
  • OAuthClientSecret: The client secret assigned when you registered your application.

Option 2: Transfer OAuth Settings

Prior to connecting on a headless machine, you must create and install a connection with the driver on a device that supports an internet browser. Set the connection properties as described in "Desktop Applications" above.

After completing the instructions in "Desktop Applications", the resulting authentication values are encrypted and written to the location specified by OAuthSettingsLocation. The default filename is OAuthSettings.txt.

Once you have successfully tested the connection, copy the OAuth settings file to your headless machine.

On the headless machine, set the following connection properties to connect to data:

• InitiateOAuth: REFRESH.
• OAuthSettingsLocation: The location of your OAuth settings file. Make sure this location gives read and write permissions to the connector to enable the automatic refreshing of the access token.
• Custom applications only:
  • OAuthClientId: The client Id assigned when you registered your application.
  • OAuthClientSecret: The client secret assigned when you registered your application.

Single Sign-On Identity Providers

ADFS

Set the AuthScheme to ADFS. You must set the following connection properties:

• User: The ADFS user.
• Password: The user's ADFS password.
• SSODomain (optional): The domain configured with the ADFS identity provider.

Example connection string:

AuthScheme=ADFS;User=ADFSUserName;Password=ADFSPassword;URL='http://sharepointserver/mysite';

Okta

Set the AuthScheme to Okta. The following connection properties are used to connect to Okta:

• User: The Okta user.
• Password: The user's Okta password.
• SSODomain (optional): The domain configured with the OKTA identity provider.

Example connection string:

AuthScheme=Okta;User=oktaUserName;Password=oktaPassword;URL='http://sharepointserver/mysite';

OneLogin

Set the AuthScheme to OneLogin. The following connection properties are used to connect to OneLogin:

• User: The OneLogin user.
• Password: The user's OneLogin password.
• SSODomain (optional): The domain configured with the OneLogin identity provider.

Example connection string:

AuthScheme=OneLogin;User=OneLoginUserName;Password=OneLoginPassword;URL='http://sharepointserver/mysite';

PingFederate

Set the AuthScheme to PingFederate. The following connection properties are used to connect to PingFederate:

• User: The PingFederate user.
• Password: PingFederate password for the user.
• SSODomain (optional): The domain configured with the PingFederate identity provider.

Example connection string:

AuthScheme=PingFederate;User=PingFederateUserName;Password=PingFederatePassword;URL='http://sharepointserver/mysite';

Azure MSI

If you are running Microsoft SharePoint on an Azure VM, you can leverage Azure Managed Service Identity (MSI) credentials to connect:

• AuthScheme: AzureMSI.

The MSI credentials are automatically obtained for authentication.

Azure Password

To connect using your Azure dredentials directly, specify the following connection properties:

• AuthScheme: AzurePassword
• User: The user account used to connect to Azure
• Password: The password used to connect to Azure
• AzureTenant: Directory (tenant) ID, found on the Overview page of the OAuth application used to authenticate to Microsoft SharePoint on Azure.

OAuthJWT Certificate

Set the AuthScheme to OAUTHJWT. The following connection properties are used to connect to Microsoft SharePoint:

• AzureTenant: The tenant you wish to connect to.
• OAuthJWTCert: The JWT certificate store.
• OAuthJWTCertType: The type of key store containing the JWT certificate.
• OAuthJWTIssuer: The issuer for the JWT assertion.
• OAuthJWTCertPassword: The password associated with the JWT certificate. Set this is your certificate type requires a password.

Microsoft SharePoint On-Premises

Microsoft SharePoint On-Premises supports a number of premise-based architectures:

• Windows (NTLM)
• Kerberos
• ADFS
• Anonymous Access

Set SharePointEdition to "SharePoint On-Premises" to use the following authentication types.

Windows (NTLM)

This is the most common authentication type. As such, the connector is preconfigured to use NTLM as the default; simply set the Windows User and Password to connect.

Kerberos

Set the AuthScheme to NEGOTIATE, and then set the following Kerberos connection properties:

• KerberosKDC: The host name or IP Address of your Kerberos KDC machine.
• KerberosSPN: The service and host of the Microsoft SharePoint Kerberos Principal. This is the value prior to the '@' symbol (for instance, MyService/MyHost) of the principal value (for instance, MyService/MyHost@EXAMPLE.COM).

For details on how to authenticate with Kerberos, see Using Kerberos.

ADFS

Set the AuthScheme to ADFS, and then set the following connection properties:

• User: The ADFS user.
• Password: ADFS password for the user.
• SSOLoginURL: The WS-trust endpoint of the ADFS server.

You also need the to set SSOProperties to authenticate to ADFS. Specify the value of the RelyingParty parameter; it is located on the ADFS server for SharePoint. Example connection string:

AuthScheme=ADFS;User=ADFSUserName;Password=ADFSPassword;SSOLoginURL='https://<authority>/adfs/services/trust/2005/usernamemixed';SSO Properties ='RelyingParty=urn:sharepoint:sp2016;';

Anonymous Access

Set the AuthScheme to NONE along with the URL.

Java Native Interface (JNI) is a standard programming interface for writing Java native methods and embedding the Java virtual machine into native applications.

The connector leverages the JNI for improved performance on Mac and Linux.

Configure the Config INI File

The Linux and Mac editions of the Microsoft SharePoint python connector are configured with an ini file. This file is used to set several parameters, including JNI behavior. This file is to be located in:

{path_to_distribution_site-packages}/cdata/config.ini

Ensure that any configuration properties you set in the ini file fall under the following section name (adjust the 311 number if you are using an different python version from 3.11):

• For Linux:
  

  [sharepoint.cpython-311-x86_64-linux-gnu.so]

• For Mac:
  

  [sharepoint.cpython-311-darwin.so]

Configure the JNI connector's behavior by editing the properties in the connector's config.ini file. The connector can be configured as follows:

• LOGFILE: Set this the same way as the CDATA_LOGFILE envrionment variable below.
• JAVA_HOME: Configure the path to the JVM library location used to launch the JVM.
• CLASS_PATH: Use a colon-separated list to configure the paths to the third-party jar libraries.

Configure Environment Variables

Additionally, set the following environment variables:

• CDATA_JAVA_HOME: Configure the path to the JVM library location used to launch the JVM.
• CDATA_JVM_OPTIONS: Place JVM options here.
• CDATA_LOGFILE: Set this in the following scheme: <SCHEME>://<TAG>[|<LEVEL>]
  • SCHEME: The options are STDOUT, FILE.
    • STDOUT: Both the native wrapper and odbc core log into stdout. The Logfile and Verbosity properties can override the behavior of ODBC core.
    • FILE: The native wrapper logs into <FILENAME> while the odbc core logs into <FILENAME>.driver.log. The Logfile and Verbosity properties can override the behavior of ODBC core.
  • TAG
    • For STDOUT, set this to 1. For FILE, set this to the filename.
  • LEVEL
    • Set to one of: FATAL | ERROR | WARNING | INFO | DEBUG

The following are some examples of this syntax:

• STDOUT://1|DEBUG
• FILE:///tmp/my_py.log|DEBUG

Custom Logger

The Python connector supports a custom logging mechanism for redirecting log output to any destination, such as a cloud storage service or logging framework. Use setCustomLoggerFactory() to register a factory function that creates a logger instance for each connection.

The factory function receives the context string from the Logfile connection property (the portion after CUSTOM://) and must return an object with a writeLog(verbosity, message) method.

To enable custom logging:

1. Call setCustomLoggerFactory() with your factory function before opening connections.
2. Set Logfile to CUSTOM:// followed by a context string to identify the connection.
3. Set Verbosity to the desired log level.

The following example demonstrates a custom logger factory that creates a separate logger instance per connection:

import cdata.sharepoint as mod
import time

class MyLogger:
    def __init__(self, loggerId):
        self.loggerId = loggerId
    def writeLog(self, verbosity, message):
        print("[MyLogger " + self.loggerId + "] " + message)

def createLogger(context):
    return MyLogger(context[len("MyLoggerId="):])

mod.setCustomLoggerFactory(createLogger)

conn1 = mod.connect("...;Logfile=CUSTOM://MyLoggerId=1;Verbosity=2;")
# do something with conn1
time.sleep(1)  # Wait for logs to flush from conn1

conn2 = mod.connect("...;Logfile=CUSTOM://MyLoggerId=2;Verbosity=2;")
# do something with conn2
time.sleep(1)  # Wait for logs to flush from conn2

Creating a Custom Entra ID (Azure AD) Application

Note: Microsoft has rebranded Azure AD as Entra ID. In topics that require the user to interact with the Entra ID Admin site, we use the same names Microsoft does. However, there are still CData connection properties whose names or values reference "Azure AD".

CData embeds OAuth Application Credentials with CData branding that can be used when using Azure to connect via either a Desktop Application or a Headless Machine. However, in all cases, connecting to Azure via a Web application requires creating a custom OAuth application. You might also want to create a custom OAuth application to:

• control branding of the Authentication Dialog,
• control the redirect URI that the application redirects the user to after the user authenticates, or
• customize the permissions that you are requesting from the user.

The following sections describe how to create a custom OAuth application using Azure Portal.

Azure Portal

To obtain OAuth values for your app, the OAuthClientId and OAuthClientSecret, and register a custom OAuth application:

1. Log in to the Azure Portal.
2. In the left-hand navigation pane, navigate to Azure Active Directory > App Registrations.
3. Click Add.
4. Enter an application name.
5. Select Any Azure AD Directory - Multi Tenant.
6. Set the redirect url to http://localhost:33333 (the connector's default) or set a different port of your choice.
7. Set CallbackURL to the exact reply URL you defined.
   The Portal creates the new application.
8. Navigate to the "Certificates & Secrets" section.
9. Create a client secret for the application, and select a duration.
10. After you save the key, the key value is displayed once. Immediately, set OAuthClientSecret to the displayed value. Set OAuthClientId to the Application Id.
11. Select API Permissions.
12. Click Add. If your application authenticates on behalf of a signed-in user, choose Delegated Permissions.
13. In the API Permissions section, click on Add a permission and select SharePoint. Choose the permissions you want your app to have. To view and edit lists, you have to select (at least) the AllSites.Manage permission.
14. Save your changes.

Note: If you have selected to use permissions that require admin consent, you can grant them from the current tenant on the API Permissions page. Otherwise, see "OAuth: Admin Consent", below.

OAuth: Admin Consent

Admin consent refers to when the Admin for an Azure Active Directory tenant grants permissions to an application that requires an administrator in your organization to consent to the use case. The embedded application within the CData Python Connector for Microsoft SharePoint, contains no permissions that require administrator consent. Therefore, this information applies only to custom applications.

When creating a new OAuth application in the Azure Portal, you must specify which permissions the application requires. Some permissions may be marked with "Admin Consent Required". For example, all Groups permissions require Admin Consent. If your application requires admin consent, there are two ways you can do this.

The easiest way to grant admin consent is to have an administrator log into the Azure Portal and navigate to the application you have created in App Registrations. Under API Permissions, click Grant Consent, which grants permissions on the tenant under which it was created.

If your organization has multiple tenants or you need to grant application permissions for other tenants outside your organization, use the GetAdminConsentURL stored procedure to generate the Admin Authorization URL. After the OAuth application is successfully authorized, it returns a Boolean indicating that permissions have been granted.

After the administrator has approved the OAuth Application, you can continue to authenticate.

The connector supports App-Only authentication using the AzureServicePrincipalCert authentication scheme and Azure AD application permissions. This setup enables applications to securely access Microsoft SharePoint data without requiring user credentials, making it ideal for automated tasks, background services, and integrations that need seamless access to SharePoint sites.

Important: The Sites.Selected permission is only supported when using SharePoint Online and the REST schema. This setup is not compatible with on-premises SharePoint deployments or other schemas such as CSOM or SOAP.

Microsoft SharePoint permissions for Azure AD applications vary based on the level of access required. Some permissions, like Sites.FullControl.All, grant broad administrative access across all Microsoft SharePoint sites, while others, like Sites.Selected, provide a more restricted, site-specific approach. The appropriate permission depends on your organization's security requirements and how much control should be given to the application.

The following information walks through configuring the connector with an Azure AD App to authenticate using certificate-based App-Only access. While this documentation focuses on using the Sites.Selected permission for granular site access, the same steps can apply to other permissions if broader or different levels of access are needed.

Registering an Azure AD App

To authenticate with Microsoft SharePoint using App-Only permissions, you must first create an Azure AD application in Microsoft Entra ID (formerly Azure AD). An Azure AD application acts as a secure identity that allows external applications—like the connector—to interact with Microsoft SharePoint using OAuth authentication. This eliminates the need for storing user credentials and enables secure, certificate-based authentication.

Once the application is registered, it is assigned a Client ID and associated with an Azure Tenant. These values are required later when configuring authentication.

Important: This setup requires two Azure AD applications, the main application that connects to Microsoft SharePoint (App 1) and a temporary Admin App (App 2) which is used to grant the main application access to specific Microsoft SharePoint sites. The Admin App must have Sites.FullControl.All permission. If you don't already have an Admin App, you must create one before continuing to grant site access. If you intend to grant access to all sites using Sites.FullControl.All, you only need one application, as it will have tenant-wide permissions without requiring a second app for site assignment. However, if you are restricting access to specific sites, you must create the Admin App before continuing.

Use the following steps to register an application:

1. Log in to the Azure Portal.
2. In the left-hand navigation pane, navigate to Microsoft Entra ID (formerly Azure Active Directory) > App registrations.
3. Click New Registration.
4. Enter an Application Name.
5. Set Supported Account Types to: Accounts in this organizational directory only.
6. Click Register.
7. (Optional): If you don't already have an Admin App, repeat the steps above to create a second app.

Copy and save these values for later as they are required when configuring your connection properties:

• Application (Client) ID – This uniquely identifies the application and is used as the OAuthJWTIssuer in your connection settings.
• Directory (Tenant) ID – This identifies the tenant where the app is registered and is used as the AzureTenant in your connection settings.

Generating and Uploading a Certificate

To use App-Only authentication with the AzureServicePrincipalCert scheme, you must generate and upload a self-signed certificate to your application. This certificate allows secure authentication and replaces traditional password-based authentication.

Note: If you created an Admin App (App 2) to grant permissions to specific sites, it does not require a certificate. Instead, you must generate a client secret for App 2. You can do this in the Azure Portal by navigating to Certificates & Secrets and creating a new client secret.

Use the following steps to generate and upload a certificate:

1. Run the following script in PowerShell (Run as Administrator):
   

   .\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate YYYY-MM-DD -EndDate YYYY-MM-DD

2. Export the public (.CER) and private (.PFX) keys.
3. Return to the Azure Portal and open the application you created in Step 1.
4. Navigate to Certificates & Secrets in the left-hand menu.
5. Click Upload Certificate.
6. Select the .CER file (public key) you exported earlier.
7. Click Add.

Assigning Permissions to the App

By default, newly created applications do not have access to Microsoft SharePoint data. To allow the application to interact with Microsoft SharePoint, you must assign the necessary API permissions. Microsoft provides different levels of permissions for Microsoft SharePoint, ranging from full access to all sites (Sites.FullControl.All) to limited access to specific sites (Sites.Selected). The permission model you choose determines how much control the application has over Microsoft SharePoint data.

The following steps focus on Sites.Selected, a permission that allows administrators to grant access only to specific Microsoft SharePoint sites. Unlike Sites.FullControl.All, which gives the application full access across the entire Microsoft SharePoint tenant, Sites.Selected ensures that the application can only interact with approved sites.

When assigning the Sites.Selected permission in the Azure Portal, you must select it under the SharePoint API, not Microsoft Graph. This is because the connector uses the SharePoint REST API for connecting to Microsoft SharePoint, and assigning the permission under the wrong API results in authorization failures.

Use the following steps to assign permissions:

1. Open the application you previously created.
2. Navigate to API Permissions in the left-hand navigation menu.
3. Click Add a Permission.
4. Under What type of permissions does your application require?, choose Application Permissions.
5. In the search bar, type Sites.Selected, then check the box next to it.
6. Click Add permissions.
7. To apply these changes, click Grant admin consent for [Your Tenant Name] and confirm.

Granting SharePoint Site Access

By default, the Sites.Selected permission does not automatically grant access to any Microsoft SharePoint site. You must explicitly assign site permissions to the application using PowerShell. This ensures that the application can only interact with approved sites, maintaining security and control over Microsoft SharePoint data.

Note: The connector does not support assigning site permissions via Microsoft Graph API. All site-level permission assignments must be completed using PowerShell with the PnP.PowerShell module.

Before proceeding, ensure you have:

• Two Azure AD applications:
  • App 1 (The SharePoint App) with Sites.Selected permission
  • App 2 (An Admin App) with Sites.FullControl.All permission to grant access
• PowerShell installed with the PnP.PowerShell module
• The SharePoint site URL where you want to grant access

Step 1: Install the PnP PowerShell Module

1. Open PowerShell as an administrator.
2. Run the following command:
   

   Install-Module -Name PnP.PowerShell

3. If prompted, confirm the installation by pressing "Y".

Step 2: Connect to the SharePoint Site

1. Run the following command to connect to your SharePoint site:
   

   Connect-PnPOnline -Url "https://yourtenant.sharepoint.com/sites/yourSite" -Interactive

2. Replace https://yourtenant.sharepoint.com/sites/yourSite with the URL of your target SharePoint site.
3. Authenticate using your credentials.

Step 3: Grant Access to the SharePoint App

1. Run the following command to grant write access to the app:
   

          Grant-PnPAzureADAppSitePermission
         -AppId "{sharepoint_app_client_id}"
         -DisplayName "CData SharePoint App"
         -Site "https://yourtenant.sharepoint.com/sites/yourSite"
         -Permissions Write
       

   Note: If you need to grant read-only access instead of write access, update the -Permissions parameter to -Permissions Read.
2. Replace {sharepoint_app_client_id} with your SharePoint application's Client ID.
3. Replace https://yourtenant.sharepoint.com/sites/yourSite with your SharePoint site URL.
4. Press Enter to execute the command.

Note: After granting the application access to a SharePoint site, it may take some time for the permissions to fully propagate. If you do not have immediate access to the site, wait a few hours and try again.

Configuring Connection Properties

Once you've registered your application, uploaded the certificate, and assigned SharePoint site permissions, configure connector by specifying these exact connection properties in your application's connection string. These settings instruct connector how to authenticate to your SharePoint site using App-Only authentication with the AzureServicePrincipalCert scheme:

Note: Although the AuthScheme has been updated to AzureServicePrincipalCert, the certificate-related connection properties (OAuthJWTCert, OAuthJWTCertType, and OAuthJWTCertPassword) remain unchanged. These properties are still required to supply the certificate used in the underlying JWT-based authentication flow.

URL=https://{name}.sharepoint.com/sites/{site}/;
SharePointEdition=SharePoint Online;
Schema=REST;
AzureTenant={tenant id};
OAuthJWTIssuer={client id};
AuthScheme=AzureServicePrincipalCert;
OAuthJWTCert={file path to certificate (.PFX)};
OAuthJWTCertType=PFXFile;
OAuthJWTCertPassword={certificate password};

Replace the placeholders with the specific values from your setup:

• URL: The SharePoint site URL that your app was granted access to via the Sites.Selected permission.
• AzureTenant: Your Azure Active Directory tenant ID.
• OAuthJWTIssuer: The Client ID for the Azure AD application. This replaces OAuthClientId for the AzureServicePrincipalCert authentication scheme.
• OAuthJWTCert: The absolute file path to your .PFX certificate file.
• OAuthJWTCertType: Set this explicitly to PFXFile as required. If you're using a binary representation of the certificate instead of a file, convert the certificate to a Base64-encoded blob and set this value to PFXBLOB.
• OAuthJWTCertPassword: Password you assigned to the .PFX certificate when exporting it.

Microsoft SharePoint REST API is supported both on SharePoint OnPremise and on SharePoint Online. To connect using the REST API set Schema to REST.

The property SharePointEdition may be used to define the edition of SharePoint.

SharePoint Online

SharePoint Online uses OAuth standard to authenticate. Follow the steps under "Authenticating to SharePoint Online" in Establishing a Connection for more information.

SharePoint OnPremise

Follow the steps under "Authenticating to SharePoint On Premises" in Establishing a Connection for more information.

Kerberos

To authenticate to Microsoft SharePoint with Kerberos, set AuthScheme to NEGOTIATE.

Authenticating to Microsoft SharePoint via Kerberos requires you to define authentication properties and to choose how Kerberos should retrieve authentication tickets.

Retrieve Kerberos Tickets

Kerberos tickets are used to authenticate the requester's identity. The use of tickets instead of formal logins/passwords eliminates the need to store passwords locally or send them over a network. Users are reauthenticated (tickets are refreshed) whenever they log in at their local computer or enter kinit USER at the command prompt.

The connector provides three ways to retrieve the required Kerberos ticket, depending on whether or not the KRB5CCNAME and/or KerberosKeytabFile variables exist in your environment.

MIT Kerberos Credential Cache File

This option enables you to use the MIT Kerberos Ticket Manager or kinit command to get tickets. With this option there is no need to set the User or Password connection properties.

This option requires that KRB5CCNAME has been created in your system.

To enable ticket retrieval via MIT Kerberos Credential Cache Files:

1. Ensure that the KRB5CCNAME variable is present in your environment.
2. Set KRB5CCNAME to a path that points to your credential cache file. (For example, C:\krb_cache\krb5cc_0 or /tmp/krb5cc_0.) The credential cache file is created when you use the MIT Kerberos Ticket Manager to generate your ticket.
3. To obtain a ticket:
   1. Open the MIT Kerberos Ticket Manager application.
   2. Click Get Ticket.
   3. Enter your principal name and password.
   4. Click OK.

   If the ticket is successfully obtained, the ticket information appears in Kerberos Ticket Manager and is stored in the credential cache file.

The connector uses the cache file to obtain the Kerberos ticket to connect to Microsoft SharePoint.

Note: If you would prefer not to edit KRB5CCNAME, you can use the KerberosTicketCache property to set the file path manually. After this is set, the connector uses the specified cache file to obtain the Kerberos ticket to connect to Microsoft SharePoint.

Keytab File

If your environment lacks the KRB5CCNAME environment variable, you can retrieve a Kerberos ticket using a Keytab File.

To use this method, set the User property to the desired username, and set the KerberosKeytabFile property to a file path pointing to the keytab file associated with the user.

User and Password

If your environment lacks the KRB5CCNAME environment variable and the KerberosKeytabFile property has not been set, you can retrieve a ticket using a user and password combination.

To use this method, set the User and Password properties to the user/password combination that you use to authenticate with Microsoft SharePoint.

Enabling Cross-Realm Authentication

More complex Kerberos environments can require cross-realm authentication where multiple realms and KDC servers are used. For example, they might use one realm/KDC for user authentication, and another realm/KDC for obtaining the service ticket.

To enable this kind of cross-realm authentication, set the KerberosRealm and KerberosKDC properties to the values required for user authentication. Also, set the KerberosServiceRealm and KerberosServiceKDC properties to the values required to obtain the service ticket.

Fine Tuning the Microsoft SharePoint Connection

To make it easier to access data in advanced integrations, use the following connection properties to control column name identifiers and other aspects of data access:

• UseDisplayNames: Set this to true to return column names that match field names in the underlying API.
  By default, the connector uses column names that match the field names defined in SharePoint.
• UseSimpleNames: Set this to true to perform substitutions on special characters in column names that SharePoint allows but that many databases typically do not.
• ShowPredefinedColumns: Set this to false to exclude fields derived from fields in the list; for example, Author and CreatedAt.
  This setting excludes the predefined fields from being returned in SELECT * statements and schema discovery.
• ShowHiddenColumns: When true, columns marked as hidden in SharePoint will be displayed by the connector.

General Changes

Date	Version	Source	Category	Type	Description
2026-06-04	26.0.9651	Microsoft SharePoint	Data Model	Added	Added the ListFilesFromFolder stored procedure to the REST schema.
2026-05-27	26.0.9643	General	Connection	Removed	Removed the deprecated ReplaceInvalidTypesWithNull connection property. Use the ReplaceInvalidValuesWithNull property instead.
2026-05-26	26.0.9642	Microsoft SharePoint	Connection	Added	Added the ExposedTableTypes connection property, which controls how lists and views are exposed as tables for the REST schema.
2026-05-22	26.0.9638	Python		Removed	Remove support for Intel x64 architecture on macOS
2026-05-15	26.0.9631	Microsoft SharePoint	Connection	Removed	Removed support for the SOAP schema with the SharePoint Online edition. Setting Schema=SOAP with SharePointEdition=SharePoint Online now throws an exception. Use the REST schema for SharePoint Online. The SharePoint On-Premises edition continues to support the SOAP schema.
2026-05-11	26.0.9627	Microsoft SharePoint	Data Model	Added	Added the UpdateAttachment stored procedure to the REST schema.
2026-05-07	26.0.9623	General	Data Model	Added	Added the ColumnCapabilities column to the sys_tablecolumns system table. This column is a bit mask denoting the column's write capabilities.
2026-05-07	26.0.9623	Python		Changed	Updated embedded JRE to jre-17.0.19+10 (Linux x64 / MacOs x64).
2026-05-04	26.0.9620	Microsoft SharePoint	Connection	Removed	Removed the deprecated SharePointOAuth option from the AuthScheme connection property.
2026-05-04	26.0.9620	Microsoft SharePoint	Data Model	Removed	In the REST schema, removed the AccessRequestListUrl, RequestAccessEmail, UseAccessRequestDefault, DescriptionTranslations, and TitleTranslations columns from the Subsites view.
2026-04-29	26.0.9615	Microsoft SharePoint	Connection	Removed	Removed the NONE enum value from the Scope connection property.
2026-04-15	26.0.9601	General	Query Exec	Changed	String comparisons using GREATER, LESS, and CONTAINS operators are now case-insensitive by default.
2026-04-07	26.0.9593	Microsoft SharePoint	Compatibility	Changed	The value for the URI connection property must match the internal name (Sharepoint URL) of the drive being accessed, rather than the display name.
2026-04-01	26.0.9587	Microsoft SharePoint	Data Model	Changed	When IncludeLinkedColumns is enabled, additional "Linked" columns are exposed. Previously, these were described as foreign key references, but they represent entire related rows. These columns no longer return reference values.
2026-02-23	25.0.9550	Microsoft SharePoint		Added	Added the GetFileSensitivityLabel stored procedure to the REST schema.
2026-01-13	25.0.9509	General		Added	Added support for the REGEXP_REPLACE() string function.
2025-12-21	25.0.9486	Python		Added	Added support for custom loggers in Python connectors on Linux and macOS.
2025-12-05	25.0.9470	General		Added	Added support for the INSERT INTO SELECT statement, with driver-side execution for providers that do not support the operation natively.
2025-12-04	25.0.9469	Microsoft SharePoint		Deprecated	Microsoft will discontinue support for the NTLM, Basic, and SharePointOAuth authentication schemes in 2026. This change affects the SharePoint Online edition. NTLM and Basic authentication schemes remain supported for SharePoint On-Premises.
2025-11-14	25.0.9449	Microsoft SharePoint		Changed	The data type of currency fields in SharePoint has been changed from float to decimal. This also applies when ResolveCalculatedTypes is set to true.
2025-11-06	25.0.9441	Microsoft SharePoint		Added	Added the TableListTypes connection property. This property specifies which SharePoint list templates are exposed as tables.
2025-10-30	25.0.9434	Python		Changed	Updated embedded JRE to jre-17.0.17+10 (Linux x64 / MacOs x64).
2025-10-06	25.0.9410	General		Added	Support for parsing datetime formats using ".S" and ",S" for milliseconds and nanoseconds.
2025-10-06	25.0.9410	Microsoft SharePoint		Changed	REST: Exposed different operations related to files, folders, list items, and permissions, as follows: Added three new stored procedures that support asynchronous folder operations: CopyFolderJob, MoveFolderJob, and GetJobStatus. Implemented an internal polling mechanism for monitoring job completion when WaitJobToFinish=True. This mechanism supports up to 6 retries, starting at two-second intervals. By default, two of the new stored procedures (CopyFolderJob and MoveFolderJob) return job attributes required to query status using GetJobStatus.
2025-09-12	25.0.9386	General		Added	Added the IsInsertable, IsUpdateable, and IsDeleteable columns to the sys_tables table.
2025-09-10	25.0.9384	General		Changed	All columns in statically defined Views are now reported as read-only.
2025-09-09	25.0.9383	Microsoft SharePoint		Changed	Changed the IsNullable attribute to true in the REST Data Model for dynamic list columns that SharePoint was previously reporting as non-nullable.
2025-09-03	25.0.9377	General		Changed	Corrected the behavior when IN criteria with NULL values are used in the projection part. It now returns NULL instead of 0. For example, "NULL IN (1,2)" returns "NULL".
2025-09-01	25.0.9375	General		Added	Added support for using the CAST function with infinity values. This function can cast "inf" and "-inf" to DOUBLE, FLOAT, or REAL.
2025-09-01	25.0.9375	Microsoft SharePoint		Added	Added the ResolveCalculatedTypes property. This property controls whether SharePoint calculated columns use their actual data types instead of being treated as varchar (string). This applies to both the SOAP and REST schemas.
2025-08-29	25.0.9372	Microsoft SharePoint		Changed	When using the REST schema, the Scope connection property now automatically uses the ".default" scope when the property is set to "None", or when you have not set a value for the property.
2025-08-21	25.0.9364	General		Changed	Report behavior change: Fixed inconsistent string value comparisons in non-table queries. For example, "SELECT 'A' = 'a'" previously returned false, but it now returns true.
2025-08-13	25.0.9356	General		Changed	Changed the maximum number of pages held in memory from 15 to 5 for the page providers to decrease heap usage.
2025-08-07	25.0.9350	Microsoft SharePoint		Added	Added the FileExtension field to the Attachments table.
2025-07-18	25.0.9330	Microsoft SharePoint		Removed	Removed the OAuthGrantType property. The grant type is now set implicitly through the 'AuthScheme' property. For example, you can use the 'OAuthPassword' AuthScheme instead of AuthScheme=OAuth with OAuthGrantType=Password.
2025-07-18	25.0.9330	Microsoft SharePoint		Changed	The Lists view was updated from a dynamic view to a static view to account for a SharePoint inaccuracy that excludes dynamically returned fields from being used in queries. Any Lists columns that were previously returned dynamically, but are not included in the current static data model, are no longer returned.
2025-07-12	25.0.9324	Microsoft SharePoint		Changed	The Table column in the CreateSchema stored procedure (SOAP schema) has been renamed to TableName.
2025-07-12	25.0.9324	Microsoft SharePoint		Removed	The TableDescription, WriteToFile, SchemaFile, and Columns columns have been removed from the CreateSchema stored procedure (SOAP schema).
2025-07-12	25.0.9324	Microsoft SharePoint		Added	The FileName column has been added to the CreateSchema stored procedure (SOAP schema).
2025-07-11	25.0.9323	Microsoft SharePoint		Added	Added the AddUserToGroup and RemoveUserFromGroup stored procedures to the REST schema.
2025-07-09	25.0.9321	Microsoft SharePoint		Added	Added the AddPage stored procedure to the REST schema. Added the Id column to the Groups and Roles tables. Added the OwnerId column and OwnerName and DefaultUserLoginName pseudocolumns to the Groups table. Added the RoleName mirror column to Groups table.
2025-07-09	25.0.9321	Microsoft SharePoint		Changed	Changed the ID column to Id in the Users table. Changed the Group and Role filter pseudocolumns in the Users table to GroupName and RoleName mirror columns. Changed the UserName and GroupName filter pseudocolumns in the Roles table to UserLoginName and GroupName mirror columns. Changed the UserName filter pseudocolumn in the Groups table to UserLoginName mirror column. Replaced the UPDATE statements in the Roles and Groups tables with the UpdateRole and UpdateGroup stored procedures.
2025-07-09	25.0.9321	Microsoft SharePoint		Removed	Removed DefaultLogin and OwnerLogin columns from the Groups table.
2025-07-07	25.0.9319	Python		Removed	Removed the 32-bit version of Windows Python.
2025-07-04	25.0.9316	Microsoft SharePoint		Removed	Removed the UseIdURL connection property because it has been deprecated.
2025-07-02	25.0.9314	Python		Removed	Removed support for Python 3.9.
2025-06-25	25.0.9307	General		Removed	Removed the "ADLS Gen 1" value from the ConnectionType property.
2025-06-25	25.0.9307	Python		Added	Added support for Python 3.13 in Windows, Linux, and Mac editions.
2025-06-25	25.0.9307	Python		Removed	Removed support for Python 3.8 as it is no longer supported.
2025-06-20	25.0.9302	General		Added	Created the following functions: TEXT_ENCODE: encodes a string into a different charset (UTF8 → UTF7 and returns a binary array as the result). TEXT_DECODE: takes a binary array and decodes it back into a string when provided the charset. BASE64_ENCODE: takes a binary array and encodes it as a base64 string (varchar). BASE64_DECODE: takes a base 64-encoded string and decodes it into a binary array.
2025-06-18	25.0.9300	General		Changed	The internal code for exception handling has been refactored. Exception messages returned during certain error conditions may now have different wording or formatting.
2025-06-10	25.0.9292	Microsoft SharePoint		Changed	Changed MaxLength input type from string to integer in the AddListColumn and UpdateListColumn stored procedures in the SharePoint SOAP schema.
2025-05-29	25.0.9280	Microsoft SharePoint		Added	Added the CheckPermissions stored procedure to the SharePoint REST schema, which verifies the effective permissions of a specific user or group on a SharePoint list or list item.
2025-05-27	25.0.9278	General		Removed	Removed the "Proprietary" enum option from ProxyAuthscheme.
2025-05-12	25.0.9263	Python		Changed	Updated embedded JRE to jre-17.0.15+6 (Linux x64 / MacOS x64) and jre-17.0.15+6 (MacOS aarch64).
2025-03-27	25.0.9217	Microsoft SharePoint		Added	In tables or views whose corresponding OData entity is marked with hasStream:true in its API metadata responses, the MediaReadLink column is added to the table or view metadata. When present, this column displays the link to the OData entity's media stream.
2025-03-25	25.0.9215	Microsoft SharePoint		Added	Added the AllPages and AllEvents views.
2025-02-28	25.0.9190	Microsoft SharePoint		Added	Added the following views to the REST schema: AllFiles, AllLists, and Sites. Added the ChunkSize column to the UploadDocument stored procedure in the REST schema.
2025-02-15	24.0.9177	General		Added	Added support for converting unsigned integer types to the nearest signed data type that has enough precision to hold the unsigned value.This is done for JDBC only because it does not have support for unsigned data types.
2025-01-27	24.0.9158	Microsoft SharePoint		Added	Added a new stored procedure, AddImage, to the REST schema.
2025-01-06	24.0.9137	Microsoft SharePoint		Added	Added the SiteURL input parameter to the CreateFolder stored procedure in the REST schema.
2024-12-05	24.0.9105	Microsoft SharePoint		Added	Added support for chunked file uploads in the SOAP Data Model UploadDocument stored procedure.
2024-12-02	24.0.9102	Microsoft SharePoint		Added	Added the Comments view to the REST schema.
2024-11-27	24.0.9097	General		Added	Added ThreadId to LogModule output. Logfile lines now include the Thread ID associated with the action being performed.
2024-11-21	24.0.9091	Microsoft SharePoint		Added	Added a new view, ListItems, to the SharePoint REST schema. It includes the most common fields found across various SharePoint lists.
2024-11-08	24.0.9078	Microsoft SharePoint		Added	Added a new column, ItemURL, to the Attachments view in the SharePoint REST schema. This column provides a link that allows users to open the attachment directly in their browser.
2024-10-31	24.0.9070	Microsoft SharePoint		Added	Added a new column, PrincipalType, to the Users view in the Rest schema. It defines the type of principal using bitwise values: None=0, User=1, Distribution List=2, Security Group=4, SharePoint Group=8, and All=15.
2024-10-25	24.0.9064	Microsoft SharePoint		Added	Added the following support group name fields to the Groups view in the REST schema based on document reference: LoginName, Title, IsHiddenInUI, and PrincipalType.
2024-10-10	24.0.9049	Microsoft SharePoint		Added	Added a new connection property, GetColumnsMetadata, to control how column metadata is retrieved in the REST schema.
2024-10-03	24.0.9042	Microsoft SharePoint		Added	Added a new connection property, IncludeLookupDisplayValueColumns, which is only applicable for the REST schema.
2024-08-19	24.0.8997	Microsoft SharePoint		Changed	In the Roles table (SOAP schema), changed the data type of the Permissions field from long to string.
2024-08-12	24.0.8990	Microsoft SharePoint		Added	Added the following stored procedures to the REST schema: AddRoleAssignment, BreakRoleInheritance, and RemoveRoleAssingment.
2024-07-16	24.0.8963	Microsoft SharePoint		Changed	The RoleDefinitionBindings view now has a composite key consisting of both Id and PrincipleID. Formerly it had a primary key.
2024-06-14	24.0.8931	Microsoft SharePoint		Changed	Changed AzureAD to the default AuthScheme.
2024-06-14	24.0.8931	Microsoft SharePoint		Deprecated	Deprecated the OAuth AuthScheme. AzureAD, AzurePassword, or AzureServicePrincipalCert should be used instead.
2024-06-13	24.0.8930	Microsoft SharePoint		Added	Added the PrincipalType column to the RoleAssignmentMember view in the REST data model.
2024-06-13	24.0.8930	Microsoft SharePoint		Changed	In the Attachments view in the REST data model, changed the Updated column from varchar to datetime. In the Attachments view in the REST data model, changed the ItemId column from varchar to int. In the Files view in the REST data model, changed the Size column from varchar to long. In the Files view in the REST data model, changed the TimeCreated and TimeLastModified columns from varchar to datetime. In the RoleAssignments view in the REST data model, changed the ItemId column from varchar to int. In the RoleAssignmentMember view in the REST data model, changed various columns from varchar to int, datetime, or boolean. In the RoleDefinitionBindings view in the REST data model, changed various columns from varchar to int, long, or boolean.
2024-06-12	24.0.8929	Microsoft SharePoint		Deprecated	Deprecated the OAuthJWT AuthScheme.
2024-06-12	24.0.8929	Microsoft SharePoint		Added	Added AzureServicePrincipalCert to the supported AuthSchemes.
2024-06-11	24.0.8928	Microsoft SharePoint		Removed	Removed the KeepFieldUserResources field from the SubSites view in the REST schema.
2024-06-08	24.0.8925	Microsoft SharePoint		Changed	Changed the data type of the Updated column in the RoleAssignments view from varchar to timestamp in the REST schema.
2024-06-08	24.0.8925	Microsoft SharePoint		Removed	Removed the duplicate PrincipalId column with the varchar data type from the RoleAssignments view in the REST schema.
2024-06-05	24.0.8922	Python		Added	Added support for Python 3.12.
2024-05-24	24.0.8910	Microsoft SharePoint		Changed	Changed data type of Users.Id, Users.GroupId, and Groups.Id from varchar to integer.
2024-05-09	24.0.8895	General		Changed	The ROUND function previously did not accept negative precision values. That feature has now been restored.
2024-04-09	23.0.8865	Microsoft SharePoint		Added	Added the AddList, AddListColumn, DeleteList, and DeleteListColumn stored procedures to the REST Schema.
2024-03-27	23.0.8852	Microsoft SharePoint		Added	Added the MaxSelectLength hidden connection property to control the maximum number of characters allowed in the OData $select query option. If the limit is exceeded we will make multiple requests to the endpoint for the corresponding entity where each request contains a subset of the columns and we will perform a self join client-side as a final step to get the full data. This is useful in cases when the user is reading data from lists, libraries, or other entities exposed in our data model that have many columns and where the generated URL might hit an API limit (too long) or may not work correctly.
2024-03-15	23.0.8840	General		Added	Created a new SQL function called STRING_COMPARE that provides java's String.compare() ability to SQL queries. Returns a number representative of the compared value of two strings
2024-03-11	23.0.8836	Microsoft SharePoint		Added	Enabled the ShowPredefinedColumns connection property for the REST schema. The property behaves similarly to the SOAP schema. If set to false, all columns derived from a base type will be removed from the column listing.
2024-02-23	23.0.8819	Microsoft SharePoint		Changed	Renamed the UploadDocument and CopyDocument outputs using standard names. _dlc_DocId and _dlc_DocIdUrl have been renamed to DocumentId and DocumentIdUrl respectively.
2024-02-23	23.0.8819	Microsoft SharePoint		Removed	Removed the ReturnID input for the UploadDocument and CopyDocument stored procedures. The document metadata will be returned by default. Removed the MetadataName# and MetadataValue# inputs. The operations initiated by these inputs can be called by using UPDATE and SELECT queries with the outputted Id of the document. Removed the vti_author and vti_etag outputs.
2024-02-19	23.0.8815	Microsoft SharePoint		Changed	Changed the endpoint for file retrieval. The ID column in the Files view is updated to show the document GUID.
2024-01-22	23.0.8787	Microsoft SharePoint		Changed	Changed the default behavior of including linked columns. Linked columns are used to facilitate deep inserts which is not valuable for SharePoint lists, so linked columns are not included by default.
2024-01-19	23.0.8784	Microsoft SharePoint		Changed	Reimplemented the UseDisplayNames feature for the REST schema due to the many differences between the OData metadata and the API that is used to retrieve the display names. Column data types may change to match the OData metadata since the OData standard is used for all list operations. Enabled the ShowHiddenColumns connection property for the REST schema. Hidden columns are not shown by default, to match the behavior of the SOAP schema. An extra call is needed to achieve this which may affect performance. Set ShowHiddenColumns to true to switch to the old behavior.
2024-01-17	23.0.8782	Microsoft SharePoint		Added	Added the Permanently parameter to the DeleteDocument stored procedure (REST schema) for triggering different behaviors. Enabling it deletes the document specified permanently, whereas disabling it moves the document to the recycle bin instead (default behavior).
2024-01-17	23.0.8782	Microsoft SharePoint		Removed	Removed the RelativeURL and DocumentName parameters from the DeleteDocument stored procedure (REST schema).
2024-01-17	23.0.8782	Microsoft SharePoint		Replaced	Replaced the RelativeURL and DocumentName parameters of the DeleteDocument stored procedure (REST schema) with the Path parameter.
2023-12-12	23.0.8746	Microsoft SharePoint		Changed	Added Success as a standard output for all stored procedures to indicate whether the execution was successful or not. The output is a boolean and replaces the Result and Status outputs.
2023-11-29	23.0.8733	General		Changed	The ROUND function doesn't accept the negative precision values anymore.
2023-11-29	23.0.8733	General		Changed	The returning types of the FDMonth, FDQuarter, FDWeek, LDMonth, LDQuarter, LDWeek functions are changed from Timestamp to Date. The return type of the ABS function will be consistent with the parameter value type.
2023-11-28	23.0.8732	General		Added	Added the HMACSHA256 formatter to allow for secrets to be decoded if it is in base64 format
2023-10-24	23.0.8697	Microsoft SharePoint		Changed	The default value for the hidden connection property IncludeReferenceColumn for the REST data model has changed to false and the ParentReference columns is no longer listed by default.
2023-08-29	23.0.8641	Python		Added	Added support for SQLAlchemy 2.0.
2023-06-30	23.0.8581	Microsoft SharePoint		Changed	Enabled the UseDisplayNames connection property for the REST schema. Enabling this property returns the display name as the column name and may affect performance and change the column data type.
2023-06-20	23.0.8571	General		Added	Added the new sys_lastresultinfo system table.
2023-05-19	23.0.8539	Python		Added	Added support for Python 3.11 on Windows, Linux and Mac.
2023-05-16	23.0.8536	Python		Removed	Removed support for Python 3.7 on Windows and Linux
2023-04-25	23.0.8515	General		Removed	Removed support for the SELECT INTO CSV statement. The core code doesn't support it anymore.
2023-03-01	22.0.8460	Microsoft SharePoint		Added	Added the ServerRelativeUrl column to the Lists view.
2023-01-19	22.0.8419	Microsoft SharePoint		Added	Added the links on the Connecting to REST API page under Sharepoint Online and Sharepoint OnPremise sections.
2022-12-22	22.0.8391	Microsoft SharePoint		Added	Added the RenameAttachmentOrDocument and MoveAttachmentOrDocument stored procedures to the SOAP and REST schemas.
2022-12-21	22.0.8390	Microsoft SharePoint		Changed	Changed the data type of the PertainingToTerm column from boolean to string in the GetValidTerms view.
2022-12-19	22.0.8388	Microsoft SharePoint		Added	Added support for the ReadTimeout option to the download stored procedures in the REST schema. ReadTimeout can be used to force a download to fail after a certain time, unlike Timeout which only triggers if the download stalls for that amount of time.
2022-12-14	22.0.8383	General		Changed	Added the Default column to the sys_procedureparameters table.
2022-12-08	22.0.8377	Microsoft SharePoint		Added	Added the WriteToFile parameter for the CreateSchema stored procedure in the SOAP schema. This defaults to true and must be disabled to write the schema to FileStream or FileData.
2022-12-08	22.0.8377	Microsoft SharePoint		Removed	Removed the SchemaDirectory parameter from CreateSchema stored procedure in the SOAP schema. Instead, the Location connection property path is used to create the schema.
2022-11-30	22.0.8369	Microsoft SharePoint		Added	Added the UseEntityTypeName property to determine if the table name should be EntityTypeName instead of the title in the REST schema.
2022-11-15	22.0.8354	Python		Changed	Updated embedded JRE to jre8u345-b01(Linux x64 / MacOS x64) and jre-17.0.5+8(MacOS aarch64).
2022-10-25	22.0.8333	Microsoft SharePoint		Added	Added the FileStream input attribute to add outputstream and FileData output attribute to print the response in DownloadAttachment, DownloadDocument stored procedures in SOAP and REST schema. Added the FileStream input attribute to add outputstream and FileData output attribute to print the response in CreateSchema stored procedure in SOAP schema. Added the Content input attribute to add inputstream in AddAttachment, UploadDocument stored procedures in SOAP and REST schema.
2022-09-30	22.0.8308	General		Changed	Added the IsPath column to the sys_procedureparameters table.
2022-07-29	22.0.8245	Microsoft SharePoint		Removed	Removed the Germany enum value from AzureEnvironments, as Microsoft has retired its Germany-based cloud.
2022-05-24	22.0.8179	Microsoft SharePoint		Changed	Changed provider name to Microsoft SharePoint.
2022-05-19	22.0.8174	Microsoft SharePoint		Deprecated	OAuthGrantType has been deprecated. Use the AuthScheme connection property instead.
2022-05-18	22.0.8173	Python		Added	Added support for Python 3.10 on Windows, Linux, and Mac Added support for Python 3.9 on Mac Added support for Mac M1
2022-05-18	22.0.8173	Python		Removed	Removed support for Python 3.6 on Windows and Linux
2022-03-15	21.0.8109	Microsoft SharePoint		Added	Added a new Id column for the Users view in the REST Schema.
2022-03-14	21.0.8108	Microsoft SharePoint		Added	Added the ItemId column to get RoleAssignments, RoleAssignmentMember, RoleDefinitionBindings.
2021-09-02	21.0.7915	General		Added	Added support for the STRING_SPLIT table-valued function in the CROSS APPLY clause.
2021-08-23	21.0.7905	Microsoft SharePoint		Added	Added the CreateFolder, UploadDocument, DeleteDocument, CopyDocument, CheckInDocument, CheckOutDocument, DiscardCheckOutDocument, AddAttachment, DeleteAttachment stored procedures to the REST Schema.
2021-08-10	21.0.7892	Microsoft SharePoint		Added	Added new AuthScheme "SharePointOAuth" to support client credentials flow using SharePoint App.
2021-08-07	21.0.7889	General		Changed	Added the KeySeq column to the sys_foreignkeys table.
2021-08-06	21.0.7888	General		Changed	Added the new sys_primarykeys system table.
2021-08-05	21.0.7887	Microsoft SharePoint		Removed	Removed the Permissions view from the REST schema.
2021-08-05	21.0.7887	Microsoft SharePoint		Replacements	Replaced Permissions with the RoleAssignments, RoleAssignmentMember, and RoleDefinitionBindings views in the REST schema.
2021-07-29	21.0.7880	Microsoft SharePoint		Added	Added the LoginName, Title, and IsHiddenInUI columns to the Users view in the REST schema. Added the Id column to the Groups view in the REST schema.
2021-07-28	21.0.7879	Microsoft SharePoint		Added	Added support for the Attachments, Permissions, and SubSites views in the REST schema.
2021-07-27	21.0.7878	Microsoft SharePoint		Added	Added support for OAuth Authentication without JWT cert for Client Credentials by when using a SharePoint App.
2021-07-23	21.0.7874	General		Changed	Updated the Literal Function Names for relative date/datetime functions. Previously, relative date/datetime functions resolved to a different value when used in the projection as opposed to the predicate. For example: SELECT LAST_MONTH() AS lm, Col FROM Table WHERE Col > LAST_MONTH(). Formerly, the two LAST_MONTH() methods would resolve to different datetimes. Now, they will match. As a replacement for the previous behavior, the relative date/datetime functions in the criteria may have an 'L' appended to them. For example: WHERE col > L_LAST_MONTH(). This will continue to resolve to the same values that were previously calculated in the criteria. Note that the "L_" prefix will only work in the predicate - it not available for the projection.
2021-06-16	21.0.7837	Microsoft SharePoint		Added	Added support for the PingFederate identity provider in the Sharepoint Online SOAP schema.
2021-06-05	21.0.7826	Microsoft SharePoint		Added	Added support for the AzureServicePrinciple authentication scheme only using a JWT certs instead of the OAuthClientSecret. Added support to authenticate submitting JWT certs instead of the OAuthClientSecret for the AzureServicePrinciple, OAuth, and AzureAD authentication schemes.
2021-04-25	21.0.7785	General		Added	Added support for handling client side formulas during insert / update. For example: UPDATE Table SET Col1 = CONCAT(Col1, " - ", Col2) WHERE Col2 LIKE 'A%'
2021-04-23	21.0.7783	General		Changed	Updated how display sizes are determined for varchar primary key and foreign key columns so they will match the reported length of the column.
2021-04-16	21.0.7776	General		Added	Non-conditional updates between two columns is now available to all drivers. For example: UPDATE Table SET Col1=Col2
2021-04-16	21.0.7776	General		Changed	Reduced the length to 255 for varchar primary key and foreign key columns.
2021-04-16	21.0.7776	General		Changed	Updated implicit and metadata caching to improve performance and support for multiple connections. Old metadata caches are not compatible - you need to generate new metadata caches if you are currently using CacheMetadata.
2021-04-16	21.0.7776	General		Changed	Updated index naming convention to avoid duplicates.
2021-04-15	21.0.7775	General		Changed	Kerberos authentication is updated to use TCP by default, but will fall back to UDP if a TCP connection cannot be established.
2021-03-31	21.0.7760	Microsoft SharePoint		Deprecated	The UseSSO connection property is deprecated. Select the preferred SSO scheme directly from the AuthScheme property instead. The URNAddress is deprecated. This property is used inside the SSOProperties and it should be specified when authenticating to ADFS on Sharepoint On-Premise. Instead RelyingParty should be used, as in other drivers.
2020-11-03	20.0.7612	Microsoft SharePoint		Added	Added support for retrieving information regarding the current logged in user via the GetCurrentUser stored procedure.

This section provides a walk-through for writing Microsoft SharePoint data access code in Python script.

For more information on the available data source entities and how to query them with SQL, see Data Model. For the SQL syntax, see SQL Compliance.

Connecting from Code

For information on how to deploy the connector and configure the connection to Microsoft SharePoint, see Package Installation and Establishing a Connection.

For information on how to connect with the sharepoint.connector module and its related classes, see Connecting.

Executing SQL

The connection's cursor object is used to directly execute SQL queries. For information on how to execute SELECT statements and process the returned result sets, see Querying Data. For information on to modify the data in Microsoft SharePoint with INSERT, UPDATE, and DELETE statements, see Modifying Data .

Executing Stored Procedures

You can call stored procedures by using the EXECUTE statement. For further information, see Calling Stored Procedures.

Batch Processing

For information about how to modify several rows of Microsoft SharePoint data at once using parameterized INSERT, UPDATE, and DELETE statements, see Batch Processing.

Connecting with the cdata.sharepoint Module:

The connector's module is used directly to establish a connection with the data source. It does this by using a connection string as its argument. For example:

import cdata.sharepoint as mod
conn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

Once the connection is created, you can use it to execute subsequent SQL queries.

After connecting as described in Connecting, you can use the open connection to execute SQL statements.

Executing Queries

To execute SQL statements that return data, use the execute() method. Once a query is executed, the result set is fetched from the cursor. This result set can then be iterated over to process the records individually.

For example:

cur = conn.execute("SELECT Id, Location FROM Calendar")
rs = cur.fetchall()
for row in rs:
	print(row)

Parameterized Queries

Various Python collections, such as arrays and tuples, can act as additional arguments for the execute() method. This enables you to parameterize the queries executed and help to prevent SQL Injection.

For example:

cmd = "SELECT Id, Location FROM Calendar WHERE Location <> ?"
params = ["Chapel Hill"]
cur = conn.execute(cmd, params)
rs = cur.fetchall()
for row in rs:
	print(row)

The connection is also used to issue INSERT, UPDATE, and DELETE commands to the data source. Parameters can be used with these statements if desired.

Note that the connector does not support transactions. As with normal write operations, all SQL statements executed by this connector affect the data source immediately. Call the connection's commit() method following the execution.

Insert

The following example adds a new record to the table:

cmd = "INSERT INTO Calendar (Id, Location) VALUES (?, ?)"
params = ["France", "U.S.A."]
cur = conn.execute(cmd, params)
print("Records affected: ", cur.rowcount)

Update

The following example modifies an existing record in the table:

cmd = "UPDATE Calendar SET Location = ? WHERE Id = ?"
params = ["U.S.A.", "1"]
cur = conn.execute(cmd, params)
print("Records affected: ", cur.rowcount)

Delete

The following example removes an existing record from the table:

cmd = "DELETE FROM Calendar WHERE Id = ?"
params = ["1"]
cur = conn.execute(cmd, params)
print("Records affected: ", cur.rowcount)

You can execute stored procedures using either the execute() or callproc() method of the connection.

Calling Stored Procedures Using Execute()

When you call stored procedures by issuing EXECUTE commands, the stored procedure arguments are parameterized. For example:

cmd = "EXECUTE ListItems List = ?"
params = ["Calendar"]
conn.execute(cmd, params)

Calling Stored Procedures Using Callproc()

When you call stored procedured by issuing the callproc() method, the stored procedure arguments are a procedure name and a list of parameters. For example:

cur = conn.cursor()
params = ["Calendar"]
cur.callproc("ListItems", params)

This Python connector also supports writing to the data source via batch processing, using the cursor object's executemany() method. This requires both a SQL statement string and a data frame of values that act as a series of parameters for executing the SQL statement.

Note that the connector does not support transactions. As with normal write operations, all SQL statements executed by this connector affect the data source immediately. Call the connection's commit() method following the execution.

Insert

The following example adds new records to the table:

cur = conn.cursor()
cmd = "INSERT INTO Calendar (Id, Location) VALUES (?, ?)"
params = [["France", "U.S.A."], ["France", "U.S.A."]]
cur.executemany(cmd, params)
print("Records affected: ", cur.rowcount)

Update

The following example modifies existing records in the table:

cur = conn.cursor()
cmd = "UPDATE Calendar SET Location = ? WHERE Id = ?"
params = [["U.S.A.", "1"], ["U.S.A.", "1"]]
cur.executemany(cmd, params)
print("Records affected: ", cur.rowcount)

Delete

The following example removes existing records from the table:

cur = conn.cursor()
cmd = "DELETE FROM Calendar WHERE Id = ?"
params = [["1"], ["1"]]
cur.executemany(cmd, params)
print("Records affected: ", cur.rowcount)

The connector is integrated with other tools and packages within Python.

Python Integration Guides

The following sections show how to create and use connections with the connector in common packages in Python:

• From SQLAlchemy
• From Pandas
• From Matplotlib
• From Petl

Complete List of Microsoft SharePoint Integration Quickstarts

For information on connecting from other applications, see Microsoft SharePoint integration guides.

The CData Python Connector for Microsoft SharePoint includes a Dialect class that enables integration with SQLAlchemy. Bear in mind that several aspects of connector functionality are not currently supported in SQLAlchemy 2.0 or above. If necessary, downgrade SQLAlchemy to version 1.4 or 1.3 before using this connector.

The following sections detail various aspects of this integration:

Connecting From SQLAlchemy

To construct a URL with which SQLAlchemy loads and uses the appropriate connector automatically, see Connecting

Reflecting Metadata With SQLAlchemy

To learn how to model Microsoft SharePoint tables with mapped classes, see Reflecting Metadata.

Querying Data From SQLAlchemy

To learn how to use mapped classes to query the associated tables, see Querying Data.

Modifying Data From SQLAlchemy

The connector provides INSERT/UPDATE/DELETE functionality in SQLAlchemy. To learn how to call the session's execute() method to affect the data in the data source, see Modifying Data.

Connecting With a Dialect URL

Establishing a connection using SQLAlchemy requires a specific URL format. For this connector, you can create the engine using either of the following URL formats:

Format 1

from sqlalchemy import create_engine
engine = create_engine("sharepoint:///?User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

Format 2

from sqlalchemy import create_engine
engine = create_engine("sharepoint://User:Password@/?Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

For SQLAlchemy 2.0, the dialect name is sharepoint_2. To establish a connection, use the following URL format:

from sqlalchemy import create_engine
engine = create_engine("sharepoint_2:///?User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

SQLAlchemy can act as an Object-relational Map (ORM). This enables you to treat records of a database table as instantiable records. To leverage this functionality, you must reflect the underlying metadata in one of the following ways.

Note: The following examples employ SQLAlchemy 1.4.

Modeling Data Using a Mapping Class

Use "sqlalchemy.ext.declarative.declarative_base" to declare a mapping class for the table you wish to model in the ORM. A known table in the data model is modeled either partially or completely, as shown in the following example:

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Calendar(Base):
	__tablename__ = "Calendar"
	Id = Column(String, primary_key=True)
	Id = Column(String)
	Location = Column(String)

Automatically Reflecting Metadata

Rather than mapping tables manually, SQLAlchemy can discover the metadata for one or more tables automatically. To accomplish this across the entire data model, use automap_base:

from sqlalchemy import MetaData
from sqlalchemy.ext.automap import automap_base
meta = MetaData()
abase = automap_base(metadata=meta)
abase.prepare(autoload_with=engine)
Calendar = abase.classes.Calendar

You can also reflect a single table with an inspector. When reflecting this way, providing a list of specific columns to map is optional:

from sqlalchemy import MetaData, Table
from sqlalchemy import inspect
meta = MetaData()
insp = inspect(engine)
Calendar_table = Table("Calendar", meta)
insp.reflect_table(Calendar_table, ["Id","Location"])

After you use the steps in Connecting to connect, and use one of the methods in Reflecting Metadata to reflect some of the metadata, you can use a session object to query data.

Querying Data Using the Query Method

If the mapping class has been prepared, use it with a session object to query the data source. After binding the engine to the session, provide the mapping class to the session's query method.

For example:

engine = create_engine("sharepoint:///?User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")
factory = sessionmaker(bind=engine)
session = factory()
for instance in session.query(Calendar).filter_by(Location="Chapel Hill"):
	print("Id: ", instance.Id)
	print("Id: ", instance.Id)
	print("Location: ", instance.Location)
	print("---------")

Querying Data Using the Execute Method

The session object can also run the query with the execute() method alongside the appropriate Table object. Assuming you have an active session, the following is just as viable:

Calendar_table = Calendar.metadata.tables["Calendar"]
for instance in session.execute(Calendar_table.select().where(Calendar_table.c.Location == "Chapel Hill")):
	print("Id: ", instance.Id)
	print("FullName: ", instance.Name)
	print("City: ", instance.BillingCity)
	print("---------")

Implicit Joining

If mapped classes of related Microsoft SharePoint objects have a singular foreign key relationship, the classes are implicitly joined. After importing the necessary objects, a relationship is established between your two mapped classes, as in the example below:

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, String, Integer, DateTime, ForeignKey
from sqlalchemy.orm import sessionmaker, relationship

Base = declarative_base()
class Contact(Base):
	__tablename__ = "Contact"
	Id = Column(Integer, primary_key=True)
	Name = Column(String)
	Email = Column(String)
	BirthDate = Column(DateTime)
	AccountId = Column(String, ForeignKey("Account.Id"))
	Account_Link = relationship("Account", back_populates="Contact_Link")

class Account(Base):
	__tablename__ = "Account"
	Id = Column(String, primary_key=True)
	Name = Column(String)
	BillingCity = Column(String)
	NumberOfEmployees = Column(Integer)
	Contact_Link = relationship("Contact", order_by=Contact.Id, back_populates="Account_Link")

Once the relationship is established, the tables are queried simultaneously using the session's query() method. For example:

rs = session.query(Account, Contact).filter(Account.Id == Contact.AccountId)
for Ac, Ct in rs:
  print("AccountId: ", Ac.Id)
  print("AccountName: ", Ac.Name)
  print("ContactId: ", Ct.Id)
  print("ContactName: ", Ct.Name)

Other Join Forms

In situations where mapped classes have either no foreign keys or multiple foreign keys, you may need different forms of the JOIN query to accommodate them. Using the earlier classes as examples, the following JOIN queries are possible as well:

• Explicit condition (necessary if there are no foreign keys in your mapped classes):
  

  rs = session.query(Account, Contact).join(Contact, Account.Id == Contact.AccountId)
  for Ac, Ct in rs:

• Left-to-right relationship:
  

  rs = session.query(Account, Contact).join(Account.Contact_Link)
  for Ac, Ct in rs:

• Left-to-right relationship with explicit target:
  

  rs = session.query(Account, Contact).join(Contact, Account.Contact_Link)
  for Ac, Ct in rs:

• String form of a left-to-right relationship:
  

  rs = session.query(Account, Contact).join("Contact_Link")
  for Ac, Ct in rs:

SQLAlchemy ORM also exposes support for other clauses in SQL, such as ORDER BY, GROUP BY, LIMIT, and OFFSET. All of these are supported by this connector:

ORDER BY

The following example sorts by a specified column using the session object's query() method:

rs = session.query(Calendar).order_by(Calendar.AnnualRevenue)
for instance in rs:
	print("Id: ", instance.Id)
	print("Id: ", instance.Id)
	print("Location: ", instance.Location)
	print("---------")

You can also use the session object's execute() method perform an ORDER BY. For example:

rs = session.execute(Calendar_table.select().order_by(Calendar_table.c.AnnualRevenue))
for instance in rs:

GROUP BY

The following example uses the session object's query() method to group records with a specified column:

rs = session.query(func.count(Calendar.Id).label("CustomCount"), Calendar.Id).group_by(Calendar.Id)
for instance in rs:
	print("Count: ", instance.CustomCount)
	print("Id: ", instance.Id)
	print("---------")

You can also use the session object's execute() method to perform a GROUP BY:

rs = session.execute(Calendar_table.select().with_only_columns([func.count(Calendar_table.c.Id).label("CustomCount"), Calendar_table.c.Id]).group_by(Calendar_table.c.Id))
for instance in rs:

LIMIT and OFFSET

The following example uses the session object's query() method to skip the first 100 records and fetch the following 25:

rs = session.query(Calendar).limit(25).offset(100)
for instance in rs:
	print("Id: ", instance.Id)
	print("Id: ", instance.Id)
	print("Location: ", instance.Location)
	print("---------")

You can also use the session object's execute() method to set a LIMIT or OFFSET:

rs = session.execute(Calendar_table.select().limit(25).offset(100))
for instance in rs:

Certain aggregate functions can also be used within SQLAlchemy by using the func module.

To import this module, execute:

from sqlalchemy.sql import func

Once func is imported, the following aggregate functions are available:

COUNT

The following example counts the number of records in a set of groups using the session object's query() method.

rs = session.query(func.count(Calendar.Id).label("CustomCount"), Calendar.Id).group_by(Calendar.Id)
for instance in rs:
	print("Count: ", instance.CustomCount)
	print("Id: ", instance.Id)
	print("---------")

You can also execute COUNT using the session object's execute() method:

rs = session.execute(Calendar_table.select().with_only_columns([func.count(Calendar_table.c.Id).label("CustomCount"), Calendar_table.c.Id])group_by(Calendar_table.c.Id))
for instance in rs:

SUM

This example calculates the cumulative amount of a numeric column in a set of groups.

rs = session.query(func.sum(Calendar.AnnualRevenue).label("CustomSum"), Calendar.Id).group_by(Calendar.Id)
for instance in rs:
	print("Sum: ", instance.CustomSum)
	print("Id: ", instance.Id)
	print("---------")

You can also invoke SUM using the session object's execute() method.

rs = session.execute(Calendar_table.select().with_only_columns([func.sum(Calendar_table.c.AnnualRevenue).label("CustomSum"), Calendar_table.c.Id]).group_by(Calendar_table.c.Id))
for instance in rs:

AVG

This example uses the session object's query() method to calculate the average amount of a numeric column in a set of groups:

rs = session.query(func.avg(Calendar.AnnualRevenue).label("CustomAvg"), Calendar.Id).group_by(Calendar.Id)
for instance in rs:
	print("Avg: ", instance.CustomAvg)
	print("Id: ", instance.Id)
	print("---------")

You can also use the session object's execute() method to invoke AVG:

rs = session.execute(Calendar_table.select().with_only_columns([func.avg(Calendar_table.c.AnnualRevenue).label("CustomAvg"), Calendar_table.c.Id]).group_by(Calendar_table.c.Id))
for instance in rs:

MAX and MIN

This example finds the maximum value and minimum value of a numeric column in a set of groups.

rs = session.query(func.max(Calendar.AnnualRevenue).label("CustomMax"), func.min(Calendar.AnnualRevenue).label("CustomMin"), Calendar.Id).group_by(Calendar.Id)
for instance in rs:
	print("Max: ", instance.CustomMax)
	print("Min: ", instance.CustomMin)
	print("Id: ", instance.Id)
	print("---------")

You can also use the session object's execute() method to invoke MAX and MIN:

rs = session.execute(Calendar_table.select().with_only_columns([func.max(Calendar_table.c.AnnualRevenue).label("CustomMax"), func.min(Calendar_table.c.AnnualRevenue).label("CustomMin"), Calendar_table.c.Id]).group_by(Calendar_table.c.Id))
for instance in rs:

Commands can be executed individually by the session with a call to "execute()".

Obtaining the Table Object

The query supplied to this method is constructed using the associated Table object of a mapped class. This Table object is obtained from the mapped class's metadata field, as below:

Calendar_table = Calendar.metadata.tables["Calendar"]

Once the table object is obtained, the write operations are executed in the following ways. The queries are executed immediately without the need for a call to "commit()":

Insert

The following example adds a new record to the table:

session.execute(Calendar_table.insert(), {"Id": "France", "Location": "U.S.A."})

Update

The following example modifies an existing record in the table:

session.execute(Calendar_table.update().where(Calendar_table.c.Id == "1").values(Id="France", Location="U.S.A."))

Delete

The following example removes an existing record from the table:

session.execute(Calendar_table.delete().where(Calendar_table.c.Id == "1"))

When combined with the connector, Pandas can be used to generate data frames that contain your Microsoft SharePoint data. Once created, a data frame can be passed to various other Python packages.

Connecting

Pandas relies on an SQLAlchemy engine to execute queries. Before you can use Pandas you must import it:

import pandas as pd
from sqlalchemy import create_engine
engine = create_engine("sharepoint:///?User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

Querying Data

In Pandas, SELECT queries are provided in a call to the read_sql() method, alongside a relevant connection object. Pandas executes the query on that connection, and returns the results in the form of a data frame, which can be used for a variety of purposes.

df = pd.read_sql("""
	SELECT
	   Id,
	   Location,
     $exNumericCol;
	FROM Calendar;""", engine)
print(df)

Modifying Data

To insert new records into a table, create a new data frame, and define its fields accordingly. When that is done, call to_sql() on the data frame to perform the INSERT operation with the connector, as shown in the example below. You must set the "if _exists" argument to "append" to prevent Pandas from attempting building the table from scratch. To prevent Pandas from writing the data frame index as a column, set index=False.

df = pd.DataFrame({"Id": ["France"], "Location": ["U.S.A."]})
df.to_sql("Calendar", con=engine, if_exists="append", index=False)

Matplotlib contains a number of tools that can graphically model Microsoft SharePoint data after being fed a data frame From Pandas.

Using PyPlot

Before any Matplotlib tool, such as pyplot, can be used, it must be imported:

from matplotlib import pyplot as plt

Once a Pandas data frame is obtained, it can be used to create a plot visualizing Microsoft SharePoint data. For example, the following plot generates and displays a bar graph relating Id and AnnualRevenue values:

df.plot(kind="bar", x="Id", y=["AnnualRevenue"])
plt.show()

The connector can be used to create ETL applications and pipelines for CSV data in Python using Petl.

Install Required Modules

Install the Petl modules using the pip utility.

pip install petl

Connecting

After you import the modules, including the CData Python Connector for Microsoft SharePoint, you can use the connector's connect function to create a connection using a valid Microsoft SharePoint connection string. If you prefer not to use a direct connection, you can use a SQLAlchemy engine.

import petl as etl
import cdata.sharepoint as mod
cnxn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")

Extract, Transform, and Load the Microsoft SharePoint Data

Create a SQL query string and store the query results in a DataFrame.

sql = "SELECT	Id, Location FROM Calendar "
table1 = etl.fromdb(cnxn,sql)

Loading Data

With the query results stored in a DataFrame, you can load your data into any supported Petl destination. The following example loads the data into a CSV file.

etl.tocsv(table1,'output.csv')

Modifying Data

Insert new rows into Microsoft SharePoint tables using Petl's appenddb function.

table1 = [['Id','Location'],['France','U.S.A.']]
etl.appenddb(table1,cnxn,'Calendar')

The extension supports schema discovery by using SQL queries to available System Tables.

Using SQL

The following sections describe the discovery of metadata through several System Tables:

• Tables and Views
• Columns
• Procedures

The connector possesses system tables that are used to discover the tables and views available in the data model. Of these system tables, "sys_tables" and "sys_views" are used to fetch information about the available tables and views respectively:

Tables

import cdata.sharepoint as mod
conn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_tables"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

Views

import cdata.sharepoint as mod
conn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_views"
cur.execute(cmd, params)
rs = cur.fetchall()
for row in rs:
	print(row)

The available columns for any given table are fetched from a system table called "sys_tablecolumns". A specific table name is provided in the WHERE criteria to restrict the table from which the column information is fetched:

import cdata.sharepoint as mod
conn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_tablecolumns WHERE TableName = 'Calendar'"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

Procedures

A system table called "sys_procedures" is queried to obtain the available stored procedures that are executed:

import cdata.sharepoint as mod
conn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_procedures"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

Parameters

The input parameters of any stored procedure are similarly obtained from the "sys_procedureparameters" system table:

import cdata.sharepoint as mod
conn = mod.connect("User=MyUserAccount;Password=MyPassword;Auth Scheme=NTLM;URL=http://sharepointserver/mysite;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_procedureparameters WHERE ProcedureName = 'ListItems'"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

This section details a selection of advanced features of the Microsoft SharePoint connector.

User Defined Views

The connector supports the use of user defined views, virtual tables whose contents are decided by a pre-configured user defined query. These views are useful when you cannot directly control queries being issued to the drivers. For an overview of creating and configuring custom views, see User Defined Views .

SSL Configuration

Use SSL Configuration to adjust how connector handles TLS/SSL certificate negotiations. You can choose from various certificate formats;. For further information, see the SSLServerCert property under "Connection String Options" .

Firewall and Proxy

Configure the connector for compliance with Firewall and Proxy, including Windows proxies and HTTP proxies. You can also set up tunnel connections.

Caching Data

Caching Data enables faster access to data and reduces the number of API calls, improving performance. The connector supports a simple caching model where multiple connections can also share the cache over time. When configuring the cache connection, you can specify automatic or explicit data caching.

Query Processing

The connector offloads as much of the SELECT statement processing as possible to Microsoft SharePoint and then processes the rest of the query in memory (client-side).

For further information, see Query Processing.

Logging

For an overview of configuration settings that can be used to refine CData logging, see Logging. Only two connection properties are required for basic logging, but there are numerous features that support more refined logging, which enables you to use the LogModules connection property to specify subsets of information to be logged.

Exception Handling

For an overview of how exceptions are reported and the components of an exception, see Exception Handling.

The CData Python Connector for Microsoft SharePoint supports the use of user defined views: user-defined virtual tables whose contents are decided by a preconfigured query. User defined views are useful in situations where you cannot directly control the query being issued to the driver; for example, when using the driver from a tool.

Use a user defined view to define predicates that are always applied. If you specify additional predicates in the query to the view, they are combined with the query already defined as part of the view.

There are two ways to create user defined views:

• Create a JSON-formatted configuration file defining the views you want.
• DDL statements.

Defining Views Using a Configuration File

User defined views are defined in a JSON-formatted configuration file called UserDefinedViews.json. The connector automatically detects the views specified in this file.

You can also have multiple view definitions and control them using the UserDefinedViews connection property. When you use this property, only the specified views are seen by the connector.

This user defined view configuration file is formatted so that each root element defines the name of a view, and includes a child element, called query, which contains the custom SQL query for the view.

For example:

{
	"MyView": {
		"query": "SELECT * FROM Calendar WHERE MyColumn = 'value'"
	},
	"MyView2": {
		"query": "SELECT * FROM MyTable WHERE Id IN (1,2,3)"
	}
}

Use the UserDefinedViews connection property to specify the location of your JSON configuration file. For example:

"UserDefinedViews", "C:\\Users\\yourusername\\Desktop\\tmp\\UserDefinedViews.json"

Defining Views Using DDL Statements

The connector is also capable of creating and altering the schema via DDL Statements such as CREATE LOCAL VIEW, ALTER LOCAL VIEW, and DROP LOCAL VIEW.

Create a View

To create a new view using DDL statements, provide the view name and query as follows:

CREATE LOCAL VIEW [MyViewName] AS SELECT * FROM Customers LIMIT 20;

If no JSON file exists, the above code creates one. The view is then created in the JSON configuration file and is now discoverable. The JSON file location is specified by the UserDefinedViews connection property.

Alter a View

To alter an existing view, provide the name of an existing view alongside the new query you would like to use instead:

ALTER LOCAL VIEW [MyViewName] AS SELECT * FROM Customers WHERE TimeModified > '3/1/2020';

The view is then updated in the JSON configuration file.

Drop a View

To drop an existing view, provide the name of an existing schema alongside the new query you would like to use instead.

DROP LOCAL VIEW [MyViewName]

This removes the view from the JSON configuration file. It can no longer be queried.

Schema for User Defined Views

In order to avoid a view's name clashing with an actual entity in the data model, user defined views are exposed in the UserViews schema by default. To change the name of the schema used for UserViews, reset the UserViewsSchemaName property.

Working with User Defined Views

For example, a SQL statement with a user defined view called UserViews.RCustomers only lists customers in Raleigh:

SELECT * FROM Customers WHERE City = 'Raleigh';

An example of a query to the driver:

SELECT * FROM UserViews.RCustomers WHERE Status = 'Active';

Resulting in the effective query to the source:

SELECT * FROM Customers WHERE City = 'Raleigh' AND Status = 'Active';

That is a very simple example of a query to a user defined view that is effectively a combination of the view query and the view definition. It is possible to compose these queries in much more complex patterns. All SQL operations are allowed in both queries and are combined when appropriate.

Customizing the SSL Configuration

To enable TLS, set the following:

• URL: Prefix the connection string with https://

With this configuration, the connector attempts to negotiate TLS with the server. The server certificate is validated against the default system trusted certificate store. You can override how the certificate gets validated using the SSLServerCert connection property.

To specify another certificate, see the SSLServerCert connection property.

Client SSL Certificates

The Microsoft SharePoint connector also supports setting client certificates. Set the following to connect using a client certificate.

• SSLClientCert: The name of the certificate store for the client certificate.
• SSLClientCertType: The type of key store containing the TLS/SSL client certificate.
• SSLClientCertPassword: The password for the TLS/SSL client certificate.
• SSLClientCertSubject: The subject of the TLS/SSL client certificate.

Connecting Through a Firewall or Proxy

HTTP Proxies

Note: The connector uses the system proxy settings by default, without further configuration needed. If you want to connect to other proxies, set ProxyAutoDetect to False and read further.

To authenticate to an HTTP proxy, set the following:

• ProxyServer: the hostname or IP address of the proxy server that you want to route HTTP traffic through.
• ProxyPort: the TCP port that the proxy server is running on.
• ProxyAuthScheme: the authentication method the connector uses when authenticating to the proxy server.
• ProxyUser: the username of a user account registered with the proxy server.
• ProxyPassword: the password associated with the ProxyUser.

Other Proxies

Set the following properties:

• To use a proxy-based firewall, set FirewallType, FirewallServer, and FirewallPort.
• To tunnel the connection, set FirewallType to TUNNEL.
• To authenticate, specify FirewallUser and FirewallPassword.
• To authenticate to a SOCKS proxy, additionally set FirewallType to SOCKS5.

Caching Data

Caching data provides several benefits, including faster access to data and reducing the number of API calls, which improve performance. The connector supports a simple caching model where multiple connections can also share the cache over time. You can enable and configure caching features by setting the necessary connection properties.

Contents

The sections in this chapter detail the connector's caching functionality and link to the corresponding connection properties, as well as SQL statements.

Configuring the Cache Connection

Configuring the Cache Connection describes the properties that you can set when configuring the cache database.

Caching Metadata

Caching Metadata describes the CacheMetadata property. This property determines whether or not to cache the table metadata to a file store.

Automatically Caching Data

Automatically Caching Data describes how the connector automatically refreshes the cache when the AutoCache property is set.

Explicitly Caching Data

Explicitly Caching Data describes how you can decide what data is stored in the cache and when it is updated.

Data Type Mapping

Data Type Mapping shows the mappings between the data types configured in the schema and the data types in the database.

Configuring the Caching Database

This section describes the properties for caching data to the persistent store of your choice.

CacheLocation

The CacheLocation property species the path to a file-system-based database. When caching is enabled, a file-system-based database is used by default. If CacheLocation is not specified, this database is stored at the path in Location. If neither of these connection properties are specified, the connector uses a platform-dependent default location.

CacheConnection

The CacheConnection property specifies a database driver and the connection string to the caching database.

CacheDriver and CacheProvider

Both the CacheDriver and CacheProvider properties are supported. Each specifies a database driver and the connection string to the caching database. CacheDriver is designed for Linux and MacOS; CacheProvider is Windows-based.

This section describes how to enable caching metadata and how to update the metadata cache.

Before being able to query data, the connector requires relevant metadata to be retrieved. By default, metadata is cached in memory and shared across connections. But if you want to persist across processes, or if metadata requests are expensive, the solution is to cache the metadata to disk.

Enable Caching Metadata

To enable caching of metadata, set CacheMetadata = true and see Configuring the Cache Connection for instructions on how to configure your connection string. The connector caches the metadata the first time it is needed and uses the metadata cache for subsequent requests.

Update the Metadata Cache

Because metadata is cached, changes to metadata on the live source, for example, adding or removing a column or attribute, are not automatically reflected in the metadata cache. To get updates to the live metadata, you need to delete or drop the cached data.

Automatically caching data is useful when you do not want to rebuild the cache for each query. When you query data for the first time, the connector automatically initializes and builds a cache in the background. When AutoCache = true, the connector uses the cache for subsequent query executions, resulting in faster response times.

If replication is enabled, the data is generated once and then copied to local and cloud data stores. With incremental updates, the connector achieves a performance advantage over dropping the cached tables and retrieving the entire table again on every refresh. With iterative updates, the connector only performs the query from the last time that the date was refreshed. If replication is not enabled, updates to the cache require downloading the entire data set.

Configuring Automatic Caching

To automatically update the cache and return results from the local cache, set the following connection string properties:

• AutoCache: This property automatically updates the cache when the value is set to true.
• CacheTolerance: This property ensures that the data retrieved from the database is the most current version. The default value is 600 seconds (10 minutes). The connector checks with the data source for newer records after the tolerance interval has expired. Otherwise, it returns the data directly from the cache.

Caching the Calendar Table

The following example caches the Calendar table in the file specified by the CacheLocation property of the connection string.

SELECT Id, Location FROM Calendar WHERE Location <> 'Chapel Hill'

Common Use Case

A common use for automatically caching data is to improve driver performance when making repeated requests to a live data source, such as building a report or creating a visualization. With auto caching enabled, repeated requests to the same data may be executed in a short period of time, but within an allowable tolerance (CacheTolerance) of what is considered "live" data.

With explicit caching (AutoCache = false), you decide exactly what data is cached and when to query the cache instead of the live data. Explicit caching gives you full control over the cache contents by using CACHE Statements. This section describes some strategies to use the caching features offered by the connector.

Creating the Cache

To load data in the cache, issue the following statement.

CACHE SELECT * FROM tableName WHERE ...

Once the statement is issued, any matching data in tableName is loaded into the corresponding table.

Updating the Cache

This section describes two ways to update the cache.

Updating with the SELECT Statement

The following example shows a statement that can update modified rows and add missing rows in the cached table. However, this statement does not delete extra rows that are already in the cache. This statement only merges the new rows or updates the existing rows.

CACHE SELECT * FROM Calendar WHERE Location <> 'Chapel Hill'

Updating with the TRUNCATE Statement

The following example shows a statement that can update modified rows and add missing rows in the cached table. This statement can also delete rows in the cache table that are not present in the live data source.

  CACHE WITH TRUNCATE SELECT * FROM Calendar WHERE Location <> 'Chapel Hill'
  

Query the Data in Online or Offline Mode

This section describes how to query the data in online or offline mode.

Online: Select Cached Tables

You can use the tableName#CACHE syntax to explicitly execute queries to the cache while still online, as shown in the following example.

SELECT * FROM Calendar#CACHE

Offline: Select Cached Tables

With Offline = true, SELECT statements always execute against the local cache database, regardless of whether you explicitly specify the cached table or not. Modification of the cache is disabled in Offline mode to prevent accidentally updating only the cached data. Executing a DELETE/UPDATE/INSERT statement while in Offline mode results in an exception.

The following example selects from the local cache but not the live data source because Offline = true.

SELECT * FROM Calendar WHERE Location='Chapel Hill' ORDER BY Location ASC

Delete Data from the Cache

You can delete data from the cache by building a direct connection to the database. Note that the connector does not support manually deleting data from the cache.

Common Use Case

A common use for caching is to have an application always query the cached data and only update the cache at set intervals, such as once every day or every two hours. There are two ways in which this can be implemented:

• AutoCache = false and Offline = false. All queries issued by the application explicitly reference the tableName#CACHE table. When the cache needs to be updated, the application executes a tableName#CACHE ... statement to bring the cached data up to date.
• Offline = true. Caching is transparent to the application. All queries are executed against the table as normal, so most application code does not need to be aware that caching is done. To update the cached data, simply create a separate connection with Offline = false and execute a tableName#CACHE ... statement.

The connector maps types from the data source to the corresponding data type available in the chosen cache database. The following table shows the mappings between the data types configured in the schema and the data types in the database. Some schema types have synonyms which are all listed in the Schema column.

Data Type Mapping

Note: String columns can map to different data types depending on their length.

Schema	.NET	JDBC	SQL Server	Derby	MySQL	Oracle	SQLite	Access
int, integer, int32	Int32	int	int	INTEGER	INT	NUMBER	integer	LONG
smallint, short, int16	Int16	short	smallint	SMALLINT	SMALLINT	NUMBER	integer	SHORT
double, float, real	Double	double	float	DOUBLE	DOUBLE	NUMBER	double	DOUBLE
date	DateTime	java.sql.Date	date	DATE	DATE	DATE	date	DATETIME
datetime, timestamp	DateTime	java.sql.Date	datetime	TIMESTAMP	DATETIME	TIMESTAMP	datetime	DATETIME
time, timespan	TimeSpan	java.sql.Time	time	TIME	TIME	TIMESTAMP	datetime	DATETIME
string, varchar	String	java.lang.String	If length > 4000: nvarchar(max), Otherwise: nvarchar(length)	If length > 32672: LONG VARCHAR, Otherwise VARCHAR(length)	If length > 255: LONGTEXT, Otherwise: VARCHAR(length)	If length > 4000: CLOB, Otherwise: VARCHAR2(length)	nvarchar(length)	If length > 255: LONGTEXT, Otherwise: VARCHAR(length)
long, int64, bigint	Int64	long	bigint	BIGINT	BIGINT	NUMBER	bigint	LONG
boolean, bool	Boolean	boolean	tinyint	SMALLINT	BIT	NUMBER	tinyint	BIT
decimal, numeric	Decimal	java.math.BigDecimal	decimal	DECIMAL	DECIMAL	DECIMAL	decimal	CURRENCY
uuid	Guid	java.util.UUID	nvarchar(length)	VARCHAR(length)	VARCHAR(length)	VARCHAR2(length)	nvarchar(length)	VARCHAR(length)
binary, varbinary, longvarbinary	byte[]	byte[]	binary(1000) or varbinary(max) after SQL Server 2000, image otherwise	BLOB	LONGBLOB	BLOB	BLOB	LONGBINARY

Query Processing

CData has a client-side SQL engine built into the connector library. This enables support for the full capabilities that SQL-92 offers, including filters, aggregations, functions, etc.

For sources that do not support SQL-92, the connector offloads as much of SQL statement processing as possible to Microsoft SharePoint and then processes the rest of the query in memory (client-side). This results in optimal performance.

For data sources with limited query capabilities, the connector handles transformations of the SQL query to make it simpler for the connector. The goal is to make smart decisions based on the query capabilities of the data source to push down as much of the computation as possible. The Microsoft SharePoint Query Evaluation component examines SQL queries and returns information indicating what parts of the query the connector is not capable of executing natively.

The Microsoft SharePoint Query Slicer component is used in more specific cases to separate a single query into multiple independent queries. The client-side Query Engine makes decisions about simplifying queries, breaking queries into multiple queries, and pushing down or computing aggregations on the client-side while minimizing the size of the result set.

There's a significant trade-off in evaluating queries, even partially, client-side. There are always queries that are impossible to execute efficiently in this model, and some can be particularly expensive to compute in this manner. CData always pushes down as much of the query as is feasible for the data source to generate the most efficient query possible and provide the most flexible query capabilities.

More Information

For a full discussion of how CData handles query processing, see CData Architecture: Query Execution.

Logging

Capturing connector logging can be very helpful when diagnosing error messages or other unexpected behavior.

Basic Logging

To begin capturing connector logging, set these properties:

• Logfile: A filepath that designates the name and location of the log file.
• Verbosity: A numerical value (1-5) that determines the amount of detail in the log. See the page in the Connection Properties section for an explanation of the five levels.
• MaxLogFileSize: When the limit is hit, a new log is created in the same folder with the date and time appended to the end. The default limit is 100 MB. Values lower than 100 kB will use 100 kB as the value instead.
• MaxLogFileCount: A string specifying the maximum file count of log files. When the limit is hit, a new log is created in the same folder with the date and time appended to the end and the oldest log file will be deleted. Minimum supported value is 2. A value of 0 or a negative value indicates no limit on the count.

Once these properties are set, the connector populates the log file as it carries out various tasks, such as when authentication is performed or queries are executed. If the specified file doesn't already exist, it is created.

Log Verbosity

The verbosity level determines the amount of detail that the connector reports to the Logfile. Supported Verbosity levels range from 1 to 5.

The following list describes each level:

1	Setting Verbosity to 1 logs the query, the number of rows returned by it, the start of execution and the time taken, and any errors.
2	Setting Verbosity to 2 logs everything included in Verbosity 1, cache queries, and additional information about the request.
3	Setting Verbosity to 3 also logs HTTP headers, as well as the body of the request and the response.
4	Setting Verbosity to 4 also logs transport-level communication with the data source. This includes SSL negotiation.
5	Setting Verbosity to 5 also logs communication with the data source and additional details that may be helpful in troubleshooting problems. This includes interface commands.

For normal operations, Verbosity should not be set to greater than 1. At higher verbosities you can log substantial amounts of data, which can delay execution times.

To refine the logged content further by showing/hiding specific categories of information, see LogModules.

Sensitive Data

Verbosity levels of 3 and higher may capture information that you do not want shared outside of your organization. The following lists information of concern for each level:

• Verbosity 3: The full body of the request and the response, which includes all the data returned by the connector
• Verbosity 4: SSL certificates
• Verbosity 5: Any extra transfer data not included at Verbosity 3, such as non human-readable binary transfer data

Note: Although we mask sensitive values, such as passwords, in the connection string and any request in the log, it is always best practice to review the logs for any sensitive information before sharing outside your organization.

Advanced Logging

You may want to refine the exact information that is recorded to the log file. This can be accomplished using the LogModules property. This property allows you to filter the logging using a semicolon-separated list of logging modules.

Example property value:

LogModules=INFO;EXEC;SSL;SQL;META;

Note that the logfile filtering triggered by the Verbosity connection property takes precedence over the filtering imposed by this connection property. This means that operations of a higher verbosity level than the level specified in the Verbosity connection property are not printed in the logfile, even if they belong to one of the modules specified in this connection property.

The available modules and submodules are:

Module Name	Module Description	Submodules
INFO	General Information. Includes the connection string, product version (build number), and initial connection messages.	Connec – Information related to creating or destroying connections. Messag – Generic label for messages pertaining to connections, the connection string, and product version. These messages are typically specific to the connector, rather than being received and passed along directly from the service.
EXEC	Query Execution. Includes execution messages for user-written SQL queries, parsed SQL queries, and normalized SQL queries. Success/failure messages for queries and query pages appear here as well.	Messag – Messages pertaining to query execution. These messages are typically specific to the connector, rather than being received and passed along directly from the service. Normlz – Query normalization steps. Query normalization is when the product takes the user-submitted query and rewrites the query to get the same results with optimal performance. Origin – This label applies to any messages recording a user's original query (the exact, unaltered, non-normalized query executed by the user). Page – Messages related to query paging. Parsed – Query parsing steps. Parsing is the process of converting the user-submitted query into a standardized format for easier processing.
HTTP	HTTP protocol messages. Includes HTTP requests/responses (including POST messages), as well as Kerberos related messages.	KERB – HTTP requests related to Kerberos. Messag – Messages pertaining to HTTP protocols. These messages are typically specific to the connector, rather than being received and passed along directly from the service. Unpack – This label applies to messages about zipped data being returned from the service API and unpacked by the product. Res – Messages containing HTTP responses. Req – Messages containing HTTP requests.
WSDL	Messages pertaining to the generation of WSDL/XSD files.	—
SSL	SSL certificate messages.	Certif – Messages pertaining to SSL certificates.
AUTH	Authentication related failure/success messages.	Messag – Messages pertaining to authentication. These messages are typically specific to the connector, rather than being received and passed along directly from the service. OAuth – Messages related to OAuth authentication. Krbros – Kerberos-related authentication messages.
SQL	Includes SQL transactions, SQL bulk transfer messages, and SQL result set messages.	Bulk – Messages pertaining to bulk query execution. Cache – Messages related to reading row data from and writing row data to the product's cache for better performance. Messag – Messages pertaining to SQL transactions. These messages are typically specific to the connector, rather than being received and passed along directly from the service. ResSet – Query resultsets. Transc – Messages related to handling transactions, including information about the number of jobs executed and backup table handling.
META	Metadata cache and schema messages.	Cache – Messages related to reading from and modifying column and table definitions in the product's cache for better performance. Schema – Messages related to retrieving metadata from or modifying the service schema. MemSto – Messages related to writing to or reading from in-memory metadata cache. Storag – Messages relating to storing metadata on disk or in an external data store, rather than in memory.
FUNC	Information related to executing SQL functions.	Errmsg – Error messages related to executing SQL functions.
TCP	Incoming and outgoing raw bytes on TCP transport layer messages.	Send – Raw data sent via the TCP protocol. Receiv – Raw data received via the TCP protocol.
FTP	Messages pertaining to the File Transfer Protocol.	Info – Status messages related to communication in the FTP protocol. Client – Messages related to actions taken by the FTP client (the product) during FTP communication. Server – Messages related to actions taken by the FTP server during FTP communication.
SFTP	Messages pertaining to the Secure File Transfer Protocol.	Info – Status messages related to communication in the SFTP protocol. To_Server – Messages related to actions taken by the SFTP client (the product) during SFTP communication. From_Server – Messages related to actions taken by the SFTP server during SFTP communication.
POP	Messages pertaining to data transferred via the Post Office Protocol.	Client – Messages related to actions taken by the POP client (the product) during POP communication. Server – Messages related to actions taken by the POP server during POP communication. Status – Status messages related to communication in the POP protocol.
SMTP	Messages pertaining to data transferred via the Simple Mail Transfer Protocol.	Client – Messages related to actions taken by the SMTP client (the product) during SMTP communication. Server – Messages related to actions taken by the SMTP server during SMTP communication. Status – Status messages related to communication in the SMTP protocol.
CORE	Messages relating to various internal product operations not covered by other modules.	—
DEMN	Messages related to SQL remoting.	—
CLJB	Messages about bulk data uploads (cloud job).	Commit – Submissions for bulk data uploads.
SRCE	Miscellaneous messages produced by the product that don't belong in any other module.	—
TRANCE	Advanced messages concerning low-level product operations.	—

Exception Handling

Exceptions can be surfaced from either the API or the CData Python Connector for Microsoft SharePoint. Each exception will have an error code, an error message, and a SQL state.

Error Codes

The error code classifies the type of error.

0	NONE	Used for unclassified errors and internally handled errors. This code also covers data source-specific errors that do not fit in any specific category.
65537	TCP_UNKNOWN_HOST	Unable to resolve a hostname (DNS failure).
65538	TCP_CONNECTION_REFUSED	Could not connect to the remote port.
65539	TCP_AUTH_FAILED	Login failed when using a binary authentication protocol. Use this for auth errors when the protocol is not HTTP (LDAP, SASL, Kerberos, ...).
65540	TCP_TIMEOUT	Did not receive a response after sending a request to the server.
65541	TCP_PROTOCOL	For wire protocol drivers. Either the server sent a bad packet that we are unable to process, or we cannot construct a packet to send.
131073	TLS_SERVER_UNTRUSTED	Could not verify SSL server certificate.
131074	TLS_CLIENT_UNTRUSTED	Server did not accept the client certificate we sent.
196609	OAUTH_DECRYPT_FAILED	OAuthEncryptKey did not decrypt the OAuthSettings file.
196610	OAUTH_MISSING_CLIENT_INFO	OAuthClientId / OAuthClientSecret / OAuthJWTCert is missing.
196611	OAUTH_MISSING_PROP	General OAuth property missing. OAUTH_MISSING_CLIENT_INFO is used for missing client ID/secret and JWT cert.
196612	OAUTH_NO_ACCESS_TOKEN	Unable to retrieve access token. Only use this when getting a token in GetOAuthAccessToken / RefreshOAuthAccessToken.
196613	OAUTH_TOKEN_EXPIRED	The access token expired. Normally used with a RefreshOAuth/OAuthException behavior.
196614	OAUTH_INVALID_PROP	OAuth property has an invalid value. OAUTH_MISSING_CLIENT_INFO / OAUTH_MISSING_PROP is used if the value is not set.
262145	HTTP_REQUEST_TIMEOUT	Did not receive a response from the HTTP server.
262146	HTTP_CLIENT_ERROR	Generic HTTP 4xx error. Only use for 4xx errors not covered by other codes.
262147	HTTP_AUTH_FAILED	HTTP 401 error.
262148	HTTP_LIMIT_EXCEEDED	HTTP 429 error.
262149	HTTP_SERVER_ERROR	HTTP 5xx error.
262150	HTTP_NOT_FOUND_ERROR	HTTP 404 error.
327681	CORE_TIMEOUT	General timeout. Not related to a specific network request.
327682	CORE_OP_NOT_ALLOWED	Operation blocked by provider permissions.
327683	CORE_CONNECTION_CONFIG	Connection configuration is not valid.
327684	CORE_SERIALIZE	Failed to encode data into a specific format (XML, JSON, CSV, ...).
327685	CORE_DESERIALIZE	Failed to decode data from a specific format (XML, JSON, CSV, ...).
393217	SQL_SYNTAX_ERROR	Unable to parse a SQL query.
393218	SQL_MISSING_COLUMNS	Query did not include required columns.
393219	SQL_MISSING_PARAMS	Stored procedure call did not include required parameters.
393220	SQL_QUERY_NOT_SUPPORTED	A part of the query is not allowed in the current context.
458753	SSH_SERVER_UNTRUSTED	Could not verify SSH server.
524289	STORAGE_LIST_EXCEPTION	Issue listing storage resources.
524290	STORAGE_RESOURCE_NOT_FOUND	Issue finding storage resources.
524291	STORAGE_ROOT_RESOURCE_NOT_FOUND	The root resource (bucket/share/drive) was not found; cannot create it in flat file drivers.
524292	STORAGE_RESOURCE_NOT_A_DIRECTORY	Storage resource is not a directory.
524293	STORAGE_RESOURCE_NOT_A_FILE	Storage resource is not a file.
524294	STORAGE_PERMISSIONS_DENIED	Storage permissions denied.

SQL State

The SQL state is used when throwing generic provider errors to the wrapper and indicates the success or failure of a call.

Some of the common SQL states are listed below:

07007	REQUIRED_CLAUSE	Class Code 07: Dynamic SQL Error.
08001	OPEN_CONNECTION	Class Code 08: Connection Exception. The connection was unable to be established to the application server or other server.
08004	REJECT_CONNECTION	The application server rejected establishment of the connection.
42501	PRIVILEGE_IDENTIFIED_OBJECT	Class Code 42: Syntax Error or Access Rule Violation. The authorization ID does not have the privilege to perform the specified operation on the identified object.
42506	AUTH_FAILED	Owner authorization failure occurred.
42601	SQL_SYNTAX	A character, token, or clause is invalid or missing.

Error Message

The error message provides more detailed reasoning about why the error occurred. It provides an explanation of the issue, and may include steps on how to resolve it.

The CData Python Connector for Microsoft SharePoint supports several operations on data, including querying, deleting, modifying, and inserting.

SELECT Statements

See SELECT Statements for a syntax reference and examples.

See Data Model for information on the capabilities of the Microsoft SharePoint API.

INSERT Statements

See INSERT Statements for a syntax reference and examples, as well as retrieving the new records' Ids.

UPDATE Statements

The primary key Id is required to update a record. See UPDATE Statements for a syntax reference and examples.

DELETE Statements

The primary key Id is required to delete a record. See DELETE Statements for a syntax reference and examples.

CACHE Statements

CACHE statements allow granular control over the connector's caching functionality. For a syntax reference and examples, see CACHE Statements.

For more information on the caching feature, see Caching Data.

EXECUTE Statements

Use EXECUTE or EXEC statements to execute stored procedures. See EXECUTE Statements for a syntax reference and examples.

Names and Quoting

• Table and column names are considered identifier names; as such, they are restricted to the following characters: [A-Z, a-z, 0-9, _:@].
• To use a table or column name with characters not listed above, the name must be quoted using square brackets ([name]) in any SQL statement.
• Parameter names can optionally start with the @ symbol (e.g., @p1 or @CustomerName) and cannot be quoted.
• Strings must be quoted using single quotes (e.g., 'John Doe').

The connector provides functions that are similar to those that are available with most standard databases. These functions are implemented in the CData provider engine and thus are available across all data sources with the same consistent API. Three categories of functions are available: string, date, and math.

The connector interprets all SQL function inputs as either strings or column identifiers, so you need to escape all literals as strings, with single quotes. For example, contrast the SQL Server syntax and connector syntax for the DATENAME function:

• SQL Server:
  

  SELECT DATENAME(yy,GETDATE())

• connector:
  

  SELECT DATENAME('yy',GETDATE())

String Functions

These functions perform string manipulations and return a string value. See STRING Functions for more details.

Date Functions

These functions perform date and date time manipulations. See DATE Functions for more details.

Math Functions

These functions provide mathematical operations. See MATH Functions for more details.

ASCII(character_expression)

Returns the ASCII code value of the left-most character of the character expression.

• character_expression: The character expression.

                      SELECT ASCII('0');
                      --  Result: 48
                    

BASE64_ENCODE(input_binary)

Returns the Base64-encoded string form of a binary input.

• input_binary: The binary value to encode.

                        SELECT BASE64_ENCODE(BinaryData);
                    -- Result: 'QmFzZTY0RW5jb2RlZA=='
                    

BASE64_DECODE(input_string)

Returns the binary result of decoding a Base64-encoded string.

• input_string: The Base64-encoded string.

                        SELECT BASE64_DECODE('QmFzZTY0RW5jb2RlZA==');
                    -- Result: (binary output)
                    

CHAR(integer_expression)

Converts the integer ASCII code to the corresponding character.

• integer_expression: The integer from 0 through 255.

                      SELECT CHAR(48);
                      -- Result: '0'
                    

CHARINDEX(expressionToFind ,expressionToSearch [,start_location ])

Returns the starting position of the specified expression in the character string.

• expressionToFind: The character expression to find.
• expressionToSearch: The character expression, typically a column, to search.
• start_location: An optional character position to start searching for expressionToFind in expressionToSearch.

                      SELECT CHARINDEX('456', '0123456');
                      -- Result: 4

                      SELECT CHARINDEX('456', '0123456', 5);
                      -- Result: -1
                    

CHAR_LENGTH(character_expression),

Returns the number of UTF-8 characters present in the expression.

• character_expression: The set of characters to be evaluated for length.

				 SELECT CHAR_LENGTH('sample text') FROM Account LIMIT 1
				 -- Result: 11			
				

CONCAT(string_value1, string_value2, ..., string_valueN)

Returns the string that is the concatenation of two or more string values.

• string_value1: The first string to be concatenated.
• string_value2: The second string to be concatenated.
• string_valueN: (optional) Any additional strings to be concatenated.

                      SELECT CONCAT('Hello, ', 'world!');
                      -- Result: 'Hello, world!'
                    

CONTAINS(expressionToSearch, expressionToFind)

Returns 1 if expressionToFind is found within expressionToSearch; otherwise, 0.

• expressionToSearch: The character expression, typically a column, to search.
• expressionToFind: The character expression to find.

                      SELECT CONTAINS('0123456', '456');
                      -- Result: 1

                      SELECT CONTAINS('0123456', 'Not a number');
                      -- Result: 0
                    

ENDSWITH(character_expression, character_suffix)

Returns 1 if character_expression ends with character_suffix; otherwise, 0.

• character_expression: The character expression.
• character_suffix: The character suffix to search for.

                      SELECT ENDSWITH('0123456', '456');
                      -- Result: 1

                      SELECT ENDSWITH('0123456', '012');
                      -- Result: 0
                    

FILESIZE(uri)

Returns the number of bytes present in the file at the specified file path.

• uri: The path of the file from which to read the size.

				SELECT FILESIZE('C:/Users/User1/Desktop/myfile.txt');
				-- Result: 23684
				

FORMAT(value [, parseFormat], format )

Returns the value formatted with the specified format.

• value: The string to format.
• format: The string specifying the output syntax of the date or numeric format.
• parseFormat: The string specifying the input syntax of the date value. Not applicable to numeric types.

                      SELECT FORMAT(12.34, '#');
                      -- Result: 12

                      SELECT FORMAT(12.34, '#.###');
                      -- Result: 12.34

                      SELECT FORMAT(1234, '0.000E0');
                      -- Result: 1.234E3
                      
                      SELECT FORMAT('2019/01/01', 'yyyy-MM-dd');
                      -- Result: 2019-01-01
                      
                      SELECT FORMAT('20190101', 'yyyyMMdd', 'yyyy-MM-dd');
                      -- Result: '2019-01-01'
                    

HASHBYTES(algorithm, value)

Returns the hash of the input value as a byte array using the given algorithm. The supported algorithms are MD5, SHA1, SHA2_256, SHA2_512, SHA3_224, SHA3_256, SHA3_384, and SHA3_512.

• algorithm: The algorithm to use for hashing. Must be one of MD5, SHA1, SHA2_256, SHA2_512, SHA3_224, SHA3_256, SHA3_384, or SHA3_512.
• value: The value to hash. Must be either a string or byte array.

                      SELECT HASHBYTES('MD5', 'Test');
                      -- Result (byte array): 0x0CBC6611F5540BD0809A388DC95A615B
                    

INDEXOF(expressionToSearch, expressionToFind [,start_location ])

Returns the starting position of the specified expression in the character string.

• expressionToSearch: The character expression, typically a column, to search.
• expressionToFind: The character expression to find.
• start_location: An optional character position to start searching for expressionToFind in expressionToSearch.

                      SELECT INDEXOF('0123456', '456');
                      -- Result: 4

                      SELECT INDEXOF('0123456', '456', 5);
                      -- Result: -1
                    

ISALPHABETIC(character_expression)

Returns 1 if the character expression consists only of alphabetic characters; otherwise, 0.

• character_expression: The string expression to evaluate.

                      SELECT ISALPHABETIC('Hello');
                      -- Result: 1

                      SELECT ISALPHABETIC('Hello123');
                      -- Result: 0

                      SELECT ISALPHABETIC('Hello!');
                      -- Result: 0
                    

ISALPHANUMERIC(character_expression)

Returns 1 if the character expression consists only of alphabetic and numeric characters; otherwise, 0.

• character_expression: The string expression to evaluate.

                      SELECT ISALPHANUMERIC('Hello123');
                      -- Result: 1

                      SELECT ISALPHANUMERIC('123');
                      -- Result: 1

                      SELECT ISALPHANUMERIC('Hello.123');
                      -- Result: 0
                    

ISNUMERIC(character_expression)

Returns 1 if the character expression consists only of numeric digits and up to one decimal point; otherwise, 0.

• character_expression: The string expression to evaluate.

                      SELECT ISNUMERIC('123');
                      -- Result: 1

                      SELECT ISNUMERIC('123.45');
                      -- Result: 1

                      SELECT ISNUMERIC('123.45.67');
                      -- Result: 0

                      SELECT ISNUMERIC('12a3');
                      -- Result: 0
                    

JSON_EXTRACT(json, jsonpath)

Selects any value in a JSON array or object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to extract.
• jsonpath: The XPath used to select the nodes. The JSONPath must be a string constant. The values of the nodes selected will be returned in a token-separated list.

                      SELECT JSON_EXTRACT('{"test": {"data": 1}}', '$.test');
                      -- Result: '{"data":1}'

                      SELECT JSON_EXTRACT('{"test": {"data": 1}}', '$.test.data');
                      -- Result: 1

                      SELECT JSON_EXTRACT('{"test": {"data": [1, 2, 3]}}', '$.test.data[1]');
                      -- Result: 2
                    

LEFT ( character_expression , integer_expression )

Returns the specified number of characters counting from the left of the specified string.

• character_expression: The character expression.
• integer_expression: The positive integer that specifies how many characters will be returned counting from the left of character_expression.

                      SELECT LEFT('1234567890', 3);
                      -- Result: '123'
                    

LEN(string_expression)

Returns the number of characters of the specified string expression.

• string_expression: The string expression.

                      SELECT LEN('12345');
                      -- Result: 5
                    

LOCATE(substring,string)

Returns an integer representing how many characters into the string the substring appears.

• substring: The substring to find inside larger string.
• string: The larger string that is searched for the substring.
• start locations: An optional integer that sets the character position (offset) from which to start searching.

				SELECT LOCATE('sample','XXXXXsampleXXXXX');
				-- Result: 6

                SELECT LOCATE('sample', 'XXXXXsampleXXXXX', 7)
                -- Result: 0
				

LOWER ( character_expression )

Returns the character expression with the uppercase character data converted to lowercase.

• character_expression: The character expression.

                      SELECT LOWER('MIXED case');
                      -- Result: 'mixed case'
                    

LTRIM(character_expression)

Returns the character expression with leading blanks removed.

• character_expression: The character expression.

                      SELECT LTRIM('     trimmed');
                      -- Result: 'trimmed'
                    

MASK(string_expression, mask_character [, start_index [, end_index ]])

Replaces the characters between start_index and end_index with the mask_character within the string.

• string_expression: The string expression to be searched.
• mask_character: The character to mask with.
• start_index: The optional number of characters to leave unmasked at beginning of string. Defaults to 0.
• end_index: The optional number of characters to leave unmasked at end of string. Defaults to 0.

                        SELECT MASK('1234567890','*',);
                        -- Result: '**********'
                        SELECT MASK('1234567890','*', 4);
                        -- Result: '1234******'
                        SELECT MASK('1234567890','*', 4, 2);
                        -- Result: '1234****90'  
                    

NCHAR(integer_expression)

Returns the Unicode character with the specified integer code as defined by the Unicode standard.

• integer_expression: The integer from 0 through 65535 (0 through xFFFF).

OCTET_LENGTH(character_expression),

Returns the number of bytes present in the expression.

• character_expression: The set of characters to be be evaluated.

				 SELECT OCTET_LENGTH('text') FROM Account LIMIT 1
				 -- Result: 4
				

PATINDEX(pattern, expression)

Returns the starting position of the first occurrence of the pattern in the expression. Returns 0 if the pattern is not found.

• pattern: The character expression that contains the sequence to be found. The wild-card character % can be used only at the start or end of the expression.
• expression: The expression, typically a column, to search for the pattern.

                      SELECT PATINDEX('123%', '1234567890');
                      -- Result: 1

                      SELECT PATINDEX('%890', '1234567890');
                      -- Result: 8

                      SELECT PATINDEX('%456%', '1234567890');
                      -- Result: 4
                    

POSITION(expressionToFind IN expressionToSearch)

Returns the starting position of the specified expression in the character string.

• expressionToFind: The character expression to find.
• expressionToSearch: The character expression, typically a column, to search.

                      SELECT POSITION('456' IN '123456');
                      -- Result: 4

                      SELECT POSITION('x' IN '123456');
                      -- Result: 0
                    

QUOTENAME(character_string [, quote_character])

Returns a valid SQL Server-delimited identifier by adding the necessary delimiters to the specified Unicode string.

• character_string: The string of Unicode character data. The string is limited to 128 characters. Inputs greater than 128 characters return null.
• quote_character: An optional single character to be used as the delimiter. These include:
  • a single quotation mark (')
  • a left or right bracket ([])
  • a double quotation mark (")
  • a left or right parenthesis ( () )
  • a greater or less than sign (><)
  • a left or right brace ({})
  • a backtick (`)

  If quote_character is not specified brackets are used. If an unacceptable character is supplied, it returns NULL.

                      SELECT QUOTENAME('table_name');
                      -- Result: '[table_name]'

                      SELECT QUOTENAME('table_name', '"');
                      -- Result: '"table_name"'

                      SELECT QUOTENAME('table_name', '[');
                      -- Result: '[table_name]'
                    

REGEXP_REPLACE(expr, pattern [, replacement [, position [, occurrence [, match_type]]]])

Replaces occurrences of a regular expression pattern in the input string with a specified value and returns the resulting string.

• expr: The string expression to be searched.
• pattern: The regular expression pattern to match.
• replacement: (optional) The string to replace each matched occurrence of pattern with. Supports backreferences \1 through \9 and escape sequences \n, \r, \t, and \\. By default, this argument is an empty string, meaning matched portions are removed from the output string.
• position: (optional) The 1-based starting position used when searching for regular expression matches in expr. The default is 1. All characters prior to the starting position are included in the output string unaltered. Skipped characters are ignored when calculating regular expression matches, even if they match pattern.
• occurrence: (optional) Specifies whether all occurrences of pattern,, or only a specific occurrence of pattern are replaced. The default is 0, which means all occurrences of pattern are replaced with replacement. Set to 1 to only replace the first instance of the pattern; 2 to replace the second; etc.
• match_type: (optional) Modifiers used to customize matching behavior. Supported values are: 'c' (case-sensitive, default), 'i' (case-insensitive), 'm' (multiline), 'n' (dot matches newline), 'x' (extended mode). These can be freely combined by including the letters back to back. For example, 'im' applies the functionality of both 'i' and 'm'. The regular expression syntax used is that of the Extended mode ('x') ignores whitespace and allows inline comments. If the pattern needs to match a literal space, it must be explicitly escaped.

                      SELECT REGEXP_REPLACE('abc123def456', '\d+', 'NUM');
                      -- Result: 'abcNUMdefNUM'

                      SELECT REGEXP_REPLACE('Hello\nHELLO\nhello', '^hello', 'X', 1, 0, 'im');
                      -- Result: 'X\nX\nX'
                    

REPLACE(string_expression, string_pattern, string_replacement)

Replaces all occurrences of a string with another string.

• string_expression: The string expression to be searched. This can be a character or binary data type.
• string_pattern: The substring to be found. Cannot be an empty string.
• string_replacement: The replacement string.

                      SELECT REPLACE('1234567890', '456', '|');
                      -- Result: '123|7890'

                      SELECT REPLACE('123123123', '123', '.');
                      -- Result: '...'

                      SELECT REPLACE('1234567890', 'a', 'b');
                      -- Result: '1234567890'
                    

REPLICATE ( string_expression ,integer_expression )

Repeats the string value the specified number of times.

• string_expression: The string to replicate.
• integer_expression: The repeat count.

                      SELECT REPLACE('x', 5);
                      -- Result: 'xxxxx'
                    

REVERSE ( string_expression )

Returns the reverse order of the string expression.

• string_expression: The string.

                      SELECT REVERSE('1234567890');
                      -- Result: '0987654321'
                    

RIGHT ( character_expression , integer_expression )

Returns the right part of the string with the specified number of characters.

• character_expression: The character expression.
• integer_expression: The positive integer that specifies how many characters of the character expression will be returned.

                      SELECT RIGHT('1234567890', 3);
                      -- Result: '890'
                    

RTRIM(character_expression)

Returns the character expression after it removes trailing blanks.

• character_expression: The character expression.

                      SELECT RTRIM('trimmed     ');
                      -- Result: 'trimmed'
                    

SOUNDEX(character_expression)

Returns the four-character Soundex code, based on how the string sounds when spoken.

• character_expression: The alphanumeric expression of character data.

                      SELECT SOUNDEX('smith');
                      -- Result: 'S530'
                    

SPACE(repeatcount)

Returns the string that consists of repeated spaces.

• repeatcount: The number of spaces.

                      SELECT SPACE(5);
                      -- Result: '     '
                    

SPLIT(string, delimiter, offset)

Returns a section of the string between to delimiters.

• string: The string to split.
• delimiter: The character to split the string with.
• offset: The number of the split to return. Positive numbers are treated as offsets from the left, and negative numbers are treated as offsets from the right.

                      SELECT SPLIT('a/b/c/d', '/', 1);
                      -- Result: 'a'
                      SELECT SPLIT('a/b/c/d', '/', -2);
                      -- Result: 'c'
                    

STARTSWITH(character_expression, character_prefix)

Returns 1 if character_expression starts with character_prefix; otherwise, 0.

• character_expression: The character expression.
• character_prefix: The character prefix to search for.

                      SELECT STARTSWITH('0123456', '012');
                      -- Result: 1

                      SELECT STARTSWITH('0123456', '456');
                      -- Result: 0
                    

STR ( float_expression [ , integer_length [ , integer_decimal ] ] )

Returns the character data converted from the numeric data. For example, STR(123.45, 6, 1) returns 123.5.

• float_expression: The float expression.
• length: The optional total length to return. This includes decimal point, sign, digits, and spaces. The default is 10.
• decimal: The optional number of places to the right of the decimal point. The decimal must be less than or equal to 16.

                      SELECT STR('123.456');
                      -- Result: '123'

                      SELECT STR('123.456', 2);
                      -- Result: '**'

                      SELECT STR('123.456', 10, 2);
                      -- Result: '123.46'
                    

STUFF(character_expression , integer_start , integer_length , replaceWith_expression)

Inserts a string into another string. It deletes the specified length of characters in the first string at the start position and then inserts the second string into the first string at the start position.

• character_expression: The string expression.
• start: The integer value that specifies the location to start deletion and insertion. If start or length is negative, null is returned. If start is longer than the string to be modified, character_expression, null is returned.
• length: The integer that specifies the number of characters to delete. If length is longer than character_expression, deletion occurs up to the last character in replaceWith_expression.
• replaceWith_expression: The expression of character data that will replace length characters of character_expression beginning at the start value.

                      SELECT STUFF('1234567890', 3, 2, 'xx');
                      -- Result: '12xx567890'
                    

SUBSTRING(string_value FROM start FOR length)

Returns the part of the string with the specified length; starts at the specified index.

• string_value: The character string.
• start: The positive integer that specifies the start index of characters to return.
• length: Optional. The positive integer that specifies how many characters will be returned.

                      SELECT SUBSTRING('1234567890' FROM 3 FOR 2);
                      -- Result: '34'

                      SELECT SUBSTRING('1234567890' FROM 3);
                      -- Result: '34567890'
                    

You can also drop the FROM and FOR clauses:

                    SELECT SUBSTRING('1234567890', 3, 2)
                    --Result: '34'
                    SELECT SUBSTRING('1234567890', 3)
                    --Result: '34567890'
                    

TEXT_ENCODE(input_string, charset)

Returns binary output by encoding a string using the specified character set.

• input_string: The plain text string.
• charset: The character set to use, such as 'UTF-8', 'ISO-8859-1'.

                    SELECT TEXT_ENCODE('Café', 'UTF-8');
                    -- Result: (binary output)
                    

TEXT_DECODE(input_binary, charset)

Returns a string decoded from binary data using the specified character set.

• input_binary: The binary value to decode.
• charset: The character set used for decoding.

                    SELECT TEXT_DECODE(BinaryData, 'UTF-8');
                    -- Result: 'Café'
                    

TOSTRING(string_value1)

Converts the value of this instance to its equivalent string representation.

• string_value1: The string to be converted.

                      SELECT TOSTRING(123);
                      -- Result: '123'

                      SELECT TOSTRING(123.456);
                      -- Result: '123.456'

                      SELECT TOSTRING(null);
                      -- Result: ''
                    

TRIM(trimspec trimchar FROM string_value)

Returns the character expression with leading and/or trailing blanks removed.

• trimspec: Optional. If included must be one of the keywords BOTH, LEADING or TRAILING.
• trimchar: Optional. If included should be a one-character string value.
• string_value: The string value to trim.

                      SELECT TRIM('     trimmed     ');
                      -- Result: 'trimmed'

                      SELECT TRIM(LEADING FROM '     trimmed     ');
                      -- Result: 'trimmed     '

                      SELECT TRIM('-' FROM '-----trimmed-----');
                      -- Result: 'trimmed'

                      SELECT TRIM(BOTH '-' FROM '-----trimmed-----');
                      -- Result: 'trimmed'

                      SELECT TRIM(TRAILING '-' FROM '-----trimmed-----');
                      -- Result: '-----trimmed'
                    

UNICODE(ncharacter_expression)

Returns the integer value defined by the Unicode standard of the first character of the input expression.

• ncharacter_expression: The Unicode character expression.

UPPER ( character_expression )

Returns the character expression with lowercase character data converted to uppercase.

• character_expression: The character expression.

                      SELECT UPPER('MIXED case');
                      -- Result: 'MIXED CASE'
                    

XML_EXTRACT(xml, xpath [, separator])

Extracts an XML document using the specified XPath to flatten the XML. A comma is used to separate the outputs by default, but this can be changed by specifying the third parameter.

• xml: The XML document to extract.
• xpath: The XPath used to select the nodes. The nodes selected will be returned in a token-separated list.
• separator: The optional token used to separate the items in the flattened response. If this is not specified, the separator will be a comma.

                      SELECT XML_EXTRACT('<vowels><ch>a</ch><ch>e</ch><ch>i</ch><ch>o</ch><ch>u</ch></vowels>', '/vowels/ch');
                      -- Result: 'a,e,i,o,u'

                      SELECT XML_EXTRACT('<vowels><ch>a</ch><ch>e</ch><ch>i</ch><ch>o</ch><ch>u</ch></vowels>', '/vowels/ch', ';');
                      -- Result: 'a;e;i;o;u'
                    

ABS ( numeric_expression )

Returns the absolute (positive) value of the specified numeric expression.

• numeric_expression: The expression of an indeterminate numeric data type except for the bit data type.

                      SELECT ABS(15);
                      -- Result: 15

                      SELECT ABS(-15);
                      -- Result: 15
                    

ACOS ( float_expression )

Returns the arc cosine, the angle in radians whose cosine is the specified float expression.

• float_expression: The float expression that specifies the cosine of the angle to be returned. Values outside the range from -1 to 1 return null.

                      SELECT ACOS(0.5);
                      -- Result: 1.0471975511966
                    

ASIN ( float_expression )

Returns the arc sine, the angle in radians whose sine is the specified float expression.

• float_expression: The float expression that specifies the sine of the angle to be returned. Values outside the range from -1 to 1 return null.

                      SELECT ASIN(0.5);
                      -- Result: 0.523598775598299
                    

ATAN ( float_expression )

Returns the arc tangent, the angle in radians whose tangent is the specified float expression.

• float_expression: The float expression that specifies the tangent of the angle to be returned.

                      SELECT ATAN(10);
                      -- Result: 1.47112767430373
                    

ATN2 ( float_expression1 , float_expression2 )

Returns the angle in radians between the positive x-axis and the ray from the origin to the point (y, x) where x and y are the values of the two specified float expressions.

• float_expression1: The float expression that is the y-coordinate.
• float_expression2: The float expression that is the x-coordinate.

                      SELECT ATN2(1, 1);
                      -- Result: 0.785398163397448
                    

CEILING ( numeric_expression ) or CEIL( numeric_expression )

Returns the smallest integer greater than or equal to the specified numeric expression.

• numeric_expression: The expression of an indeterminate numeric data type except for the bit data type.

                      SELECT CEILING(1.3);
                      -- Result: 2

                      SELECT CEILING(1.5);
                      -- Result: 2

                      SELECT CEILING(1.7);
                      -- Result: 2
                    

COS ( float_expression )

Returns the trigonometric cosine of the specified angle in radians in the specified expression.

• float_expression: The float expression of the specified angle in radians.

                      SELECT COS(1);
                      -- Result: 0.54030230586814
                    

COT ( float_expression )

Returns the trigonometric cotangent of the angle in radians specified by float_expression.

• float_expression: The float expression of the angle in radians.

                      SELECT COT(1);
                      -- Result: 0.642092615934331
                    

DEGREES ( numeric_expression )

Returns the angle in degrees for the angle specified in radians.

• numeric_expression: The angle in radians, an expression of an indeterminate numeric data type except for the bit data type.

                      SELECT DEGREES(3.1415926);
                      -- Result: 179.999996929531
                    

EXP ( float_expression )

Returns the exponential value of the specified float expression. For example, EXP(LOG(20)) is 20.

• float_expression: The float expression.

                      SELECT EXP(2);
                      -- Result: 7.38905609893065
                    

EXPR ( expression )

Evaluates the expression.

• expression: The expression. Operators allowed are +, -, *, /, ==, !=, >, <, >=, and <=.

                      SELECT EXPR('1 + 2 * 3');
                      -- Result: 7

                      SELECT EXPR('1 + 2 * 3 == 7');
                      -- Result: true
                    

FLOOR ( numeric_expression )

Returns the largest integer less than or equal to the numeric expression.

• numeric_expression: The expression of an indeterminate numeric data type except for the bit data type.

                      SELECT FLOOR(1.3);
                      -- Result: 1

                      SELECT FLOOR(1.5);
                      -- Result: 1

                      SELECT FLOOR(1.7);
                      -- Result: 1
                    

GREATEST(int1,int2,....)

Returns the greatest of the supplied integers.

				SELECT GREATEST(3,5,8,10,1)
				-- Result: 10			
				

HEX(value)

Returns a the equivalent hex for the input value.

• value: A string or numerical value to be converted into hex.

				SELECT HEX(866849198);
				-- Result: 33AB11AE
				
				SELECT HEX('Sample Text');
				-- Result: 53616D706C652054657874
				

JSON_AVG(json, jsonpath)

Computes the average value of a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_AVG('[1,2,3,4,5]', '$[x]');
                      -- Result: 3

                      SELECT JSON_AVG('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 3

                      SELECT JSON_AVG('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 4.5
                    

JSON_COUNT(json, jsonpath)

Returns the number of elements in a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_COUNT('[1,2,3,4,5]', '$[x]');
                      -- Result: 5

                      SELECT JSON_COUNT('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 5

                      SELECT JSON_COUNT('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 2
                    

JSON_MAX(json, jsonpath)

Gets the maximum value in a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_MAX('[1,2,3,4,5]', '$[x]');
                      -- Result: 5

                      SELECT JSON_MAX('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 5

                      SELECT JSON_MAX('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[..3]');
                      -- Result: 4
                    

JSON_MIN(json, jsonpath)

Gets the minimum value in a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_MIN('[1,2,3,4,5]', '$[x]');
                      -- Result: 1

                      SELECT JSON_MIN('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 1

                      SELECT JSON_MIN('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 4
                    

JSON_SUM(json, jsonpath)

Computes the summary value in JSON according to the JSONPath expression. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_SUM('[1,2,3,4,5]', '$[x]');
                      -- Result: 15

                      SELECT JSON_SUM('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 15

                      SELECT JSON_SUM('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 9
                    

LEAST(int1,int2,....)

Returns the least of the supplied integers.

				SELECT LEAST(3,5,8,10,1)
				-- Result: 1			
				

LOG ( float_expression [, base ] )

Returns the natural logarithm of the specified float expression.

• float_expression: The float expression.
• base: The optional integer argument that sets the base for the logarithm.

                      SELECT LOG(7.3890560);
                      -- Result: 1.99999998661119
                    

LOG10 ( float_expression )

Returns the base-10 logarithm of the specified float expression.

• float_expression: The expression of type float.

                      SELECT LOG10(10000);
                      -- Result: 4
                    

MOD(dividend,divisor)

Returns the integer value associated with the remainder when dividing the dividend by the divisor.

• dividend: The number to take the modulus of.
• divisor: The number to divide the dividend by when determining the modulus.

				SELECT MOD(10,3);
				-- Result: 1
				

NEGATE(real_number)

Returns the opposite to the real number input.

• real_number: The real number to find the opposite of.

				SELECT NEGATE(10);
				-- Result: -10
				
				SELECT NEGATE(-12.4)
				--Result: 12.4
				

PI ( )

Returns the constant value of pi.

                  SELECT PI()
                  -- Result: 3.14159265358979 
                

POWER ( float_expression , y )

Returns the value of the specified expression raised to the specified power.

• float_expression: The float expression.
• y: The power to raise float_expression to.

                      SELECT POWER(2, 10);
                      -- Result: 1024

                      SELECT POWER(2, -2);
                      -- Result: 0.25
                    

RADIANS ( float_expression )

Returns the angle in radians of the angle in degrees.

• float_expression: The degrees of the angle as a float expression.

                      SELECT RADIANS(180);
                      -- Result: 3.14159265358979
                    

RAND ( [ integer_seed ] )

Returns a pseudorandom float value from 0 through 1, exclusive.

• seed: The optional integer expression that specifies the seed value. If seed is not specified, a seed value at random will be assigned.

                      SELECT RAND();
                      -- This result may be different, since the seed is randomized
                      -- Result: 0.873159630165044

                      SELECT RAND(1);
                      -- This result will always be the same, since the seed is constant
                      -- Result: 0.248668584157093
                    

ROUND ( numeric_expression [ ,integer_length] [ ,function ] )

Returns the numeric value rounded to the specified length or precision.

• numeric_expression: The expression of a numeric data type.
• length: The optional precision to round the numeric expression to. When this is omitted, the default behavior will be to round to the nearest whole number.
• function: The optional type of operation to perform. When the function parameter is omitted or has a value of 0 (default), numeric_expression is rounded. When a value other than 0 is specified, numeric_expression is truncated.

                      SELECT ROUND(1.3, 0);
                      -- Result: 1

                      SELECT ROUND(1.55, 1);
                      -- Result: 1.6

                      SELECT ROUND(1.7, 0, 0);
                      -- Result: 2

                      SELECT ROUND(1.7, 0, 1);
                      -- Result: 1
                      
                      SELECT ROUND (1.24);
                      -- Result: 1.0
                    

SIGN ( numeric_expression )

Returns the positive sign (1), 0, or negative sign (-1) of the specified expression.

• numeric_expression: The expression of an indeterminate data type except for the bit data type.

                      SELECT SIGN(0);
                      -- Result: 0

                      SELECT SIGN(10);
                      -- Result: 1

                      SELECT SIGN(-10);
                      -- Result: -1
                    

SIN ( float_expression )

Returns the trigonometric sine of the angle in radians.

• float_expression: The float expression specifying the angle in radians.

                     SELECT SIN(1);
                     -- Result: 0.841470984807897
                    

SQRT ( float_expression )

Returns the square root of the specified float value.

• float_expression: The expression of type float.

                      SELECT SQRT(100);
                      -- Result: 10
                    

SQUARE ( float_expression )

Returns the square of the specified float value.

• float_expression: The expression of type float.

                      SELECT SQUARE(10);
                      -- Result: 100

                      SELECT SQUARE(-10);
                      -- Result: 100
                    

TAN ( float_expression )

Returns the tangent of the input expression.

• float_expression: The expression of type float.

                      SELECT TAN(1);
                      -- Result: 1.5574077246549
                    

TRUNC(decimal_number,precision)

Returns the supplied decimal number truncated to have the supplied decimal precision.

• decimal_number: The decimal value to truncate.
• precision: The number of decimal places to truncate the decimal number to.

				SELECT TRUNC(10.3423,2);
				-- Result: 10.34
				

_ROW_NUMBER_()

Returns a row index as an additional column.

				SELECT ColumnName, _ROW_NUMBER_() FROM TableName
				-- Result: ColumnData, 0
				ColumnData2, 1
				ColumnData3, 2
				

CURRENT_DATE()

Returns the current date value.

                  SELECT CURRENT_DATE();
                  -- Result: 2018-02-01
                

CURRENT_TIMESTAMP()

Returns the current time stamp of the database system as a datetime value. This value is equal to GETDATE and SYSDATETIME, and is always in the local timezone.

                  SELECT CURRENT_TIMESTAMP();
                  -- Result: 2018-02-01 03:04:05
                

DATEADD (datepart , integer_number , date [, dateformat])

Returns the datetime value that results from adding the specified number (a signed integer) to the specified date part of the date.

• datepart: The part of the date to add the specified number to. The valid values and abbreviations are
  • year (yy, yyyy)
  • quarter (qq, q)
  • month (mm, m)
  • week (wk, ww)
  • weekday (dw)
  • dayofyear (dy, y)
  • day (dd, d)
  • hour (hh)
  • minute (mi, n)
  • second (ss, s)
  • millisecond (ms)
• number: The number to be added.
• date: The expression of the datetime data type.
• dateformat: The optional output date format.

                  SELECT DATEADD('d', 5, '2018-02-01');
                  -- Result: 2018-02-06

                  SELECT DATEADD('hh', 5, '2018-02-01 00:00:00');
                  -- Result: 2018-02-01 05:00:00
                

DATEDIFF ( datepart , startdate , enddate )

Returns the difference (a signed integer) of the specified time interval between the specified start date and end date.

• datepart: The part of the date that is the time interval of the difference between the start date and end date. The valid values and abbreviations are:
  • Year (year, yyyy, yy)
  • Quarter (quarter, qq, q)
  • Month (month, mm, m)
  • Week (week, wk, ww)
  • Weekday (weekday, dw)
  • Dayofyear (dayofyear, dy, y)
  • Day (day, dd, d)
  • Hour (hour, hh)
  • Minute (minute, mi, n)
  • Second (second, ss, s)
  • Millisecond (millisecond, ms)
• startdate: The datetime expression of the start date.
• enddate: The datetime expression of the end date.

                  SELECT DATEDIFF('d', '2018-02-01', '2018-02-10');
                  -- Result: 9

                  SELECT DATEDIFF('hh', '2018-02-01 00:00:00', '2018-02-01 12:00:00');
                  -- Result: 12
                

DATE_FORMAT(date,format)

Returns the date or timestamp in the format specified. This function mirrors the MySQL DATE_FORMAT function.

• date: A date or timestamp string.
• format: The specifier string of the desired output format. The list of supported format specifiers comes from the MySQL DATE_FORMAT function (see link to MySQL documentation above).

					SELECT DATE_FORMAT('9/4/2021 3:11:53 AM','%h')
					-- Result: 03
				  

DATEFROMPARTS(integer_year, integer_month, integer_day)

Returns the datetime value for the specified year, month, and day.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.

                    SELECT DATEFROMPARTS(2018, 2, 1);
                    -- Result: 2018-02-01
                  

DATENAME(datepart , date)

Returns the character string that represents the specified date part of the specified date.

• datepart: The part of the date to return. The valid values and abbreviations are year (yy, yyyy), quarter (qq, q), month (mm, m), dayofyear (dy, y), day (dd, d), week (wk, ww), weekday (dw), hour (hh), minute (mi, n), second (ss, s), millisecond (ms), microsecond (mcs), and nanosecond (ns).
• date: The datetime expression.

                     SELECT DATENAME('yy', '2018-02-01');
                     -- Result: '2018'

                     SELECT DATENAME('dw', '2018-02-01');
                     -- Result: 'Thursday'
                   

DATEPART(datepart, date [,integer_datefirst])

Returns a character string that represents the specified date part of the specified date.

• datepart: The part of the date to return. The valid values and abbreviations are year (yy, yyyy), quarter (qq, q), month (mm, m), dayofyear (dy, y), day (dd, d), week (wk, ww), weekday (dw), hour (hh), minute (mi, n), second (ss, s), millisecond (ms), microsecond (mcs), nanosecond (ns), ISODOW, ISO_WEEK (isoweek, isowk,isoww), and ISOYEAR.
• date: The datetime string.
• datefirst: The optional integer representing the first day of the week. The default is 7, Sunday.

                    SELECT DATEPART('yy', '2018-02-01');
                    -- Result: 2018

                    SELECT DATEPART('dw', '2018-02-01');
                    -- Result: 5
                  

DATETIMEFROMPARTS(integer_year, integer_month, integer_day, integer_hour, integer_minute, integer_seconds, integer_milliseconds)

Returns the datetime value for the specified date parts.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.
• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.
• seconds: The integer expression specifying the seconds.
• milliseconds: The integer expression specifying the milliseconds.

                    SELECT DATETIMEFROMPARTS(2018, 2, 1, 1, 2, 3, 456);
                    -- Result: 2018-02-01 01:02:03.456
                  

DATETIME2FROMPARTS(integer_year, integer_month, integer_day, integer_hour, integer_minute, integer_seconds, integer_fractions, integer_precision)

Returns the datetime value for the specified date parts.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.
• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.
• seconds: The integer expression specifying the seconds.
• fractions: The integer expression specifying the fractions of the second.
• precision: The integer expression specifying the precision of the fraction.

				    SELECT DATETIME2FROMPARTS(2018, 2, 1, 1, 2, 3, 456, 3);
                    -- Result: 2018-02-01 01:02:03.456
                  

DATE_TRUNC(date, datepart)

Truncates the date to the precision of the given date part. Modeled after the Oracle TRUNC function.

• date: The datetime string that specifies the date.
• datepart: Refer to the Oracle documentation for valid datepart syntax.

				    SELECT DATE_TRUNC('05-04-2005', 'YY');
                    -- Result: '1/1/2005'
					
                    SELECT DATE_TRUNC('05-04-2005', 'MM');
                    -- Result: '5/1/2005'                    
                  

DATE_TRUNC2(datepart, date, [weekday])

Truncates the date to the precision of the given date part. Modeled after the PostgreSQL date_trunc function.

• datepart: One of 'millennium', 'century', 'decade', 'year', 'quarter', 'month', 'week', 'day', 'hour', 'minute' or 'second'.
• date: The datetime string that specifies the date.
• weekday: The optional day of the week to use as the first day for 'week'. One of 'sunday', 'monday', etc.

                    SELECT DATE_TRUNC2('year', '2020-02-04');
                    -- Result: '2020-01-01'

                    SELECT DATE_TRUNC2('week', '2020-02-04', 'monday');
                    -- Result: '2020-02-02', which is the previous Monday
                  

DAY(date)

Returns the integer that specifies the day component of the specified date.

• date: The datetime string that specifies the date.

                    SELECT DAY('2018-02-01');
                    -- Result: 1
                  

DAYNAME(date)

Returns the name of the day of the week of the specified date.

• date: The datetime string that specifies the date.

                    SELECT DAYNAME('8/18/2021');
                    -- Result: Wednesday
                  

DAYOFMONTH(date)

Returns the day of the month of the given date part.

• date: The datetime string that specifies the date.

				  SELECT DAYOFMONTH('04/15/2000');
				  -- Result: 15
				  

DAYOFWEEK(date)

Returns the day of the week of the given date part.

• date: The datetime string that specifies the date.

				  SELECT DAYOFWEEK('04/15/2000');
				  -- Result: 7
				  

DAYOFYEAR(date)

Returns the day of the year of the given date part.

• date: The datetime string that specifies the date.

				  SELECT DAYOFYEAR('04/15/2000');
				  -- Result: 106
				  

EOMONTH(date [, integer_month_to_add ]) or LAST_DAY(date)

Returns the last day of the month that contains the specified date with an optional offset.

• date: The datetime expression specifying the date for which to return the last day of the month.
• integer_month_to_add: The optional integer expression specifying the number of months to add to the date before calculating the end of the month.

                  SELECT EOMONTH('2018-02-01');
                  -- Result: 2018-02-28
                  
                  SELECT LAST_DAY('2018-02-01');
                  -- Result: 2018-02-28

                  SELECT EOMONTH('2018-02-01', 2);
                  -- Result: 2018-04-30
                

EXTRACT(date_part FROM date_column_name)

Returns the last day of the month that contains the specified date with an optional offset.

• date_part: One of the following date components: YEAR, MONTH, DAY, HOUR, MINUTE, SECOND.
• date_column_name: The name of a date column in a table.

                  SELECT EXTRACT(YEAR FROM DateColumn)
                  -- Result: 2021
                

FDWEEK(date)

Returns the first day of the week of the given date part.

• date: The datetime string that specifies the date.
• weeks to add: An optional integer expression specifying the number of months to add to the date before calculating the first day of the week.

				  SELECT FDWEEK('02-08-2018');
				  -- Result: 2/4/2018

          SELECT FDWEEK('02-08-2018', 1)
          --Result: 02/11/2018
				  

FDMONTH(date)

Returns the first day of the month of the given date part.

• date: The datetime string that specifies the date.
• month to add: An optional integer expression specifying the number of months to add to the date before calculating the first day of the month.

				  SELECT FDMONTH('02-08-2018');
				  -- Result: 2/1/2018

          SELECT FDMONTH('02-08-2018', 1) 
          --Result: 03/01/2018
				  

FDQUARTER(date)

Returns the first day of the quarter of the given date part.

• date: The datetime string that specifies the date.
• quarters to add: An optional integer expression specifying the number of months to add to the date before calculating the first day of the quarter.

				  SELECT FDQUARTER('05-08-2018');
				  -- Result: 4/1/2018

          SELECT FDQUARTER('05-08-2018',1)
          --Result: 07/01/2018
				  

FILEMODIFIEDTIME(uri)

Returns the time stamp associated with the Date Modified of the relevant file.

• uri: An absolute path pointing to a file on the local file system.

				 SELECT FILEMODIFIEDTIME('C:/Documents/myfile.txt');
				 -- Result: 6/25/2019 10:06:58 AM
				 

FROM_DAYS(datevalue)

Returns a date derived from the number of days after 1582-10-15 (based upon the Gregorian calendar). This will be equivalent to the MYSQL FROM_DAYS function.

• datevalue: A integer value representing the number of days since 1582-10-15.

				SELECT FROM_DAYS(736000);
				-- Result: 2/6/2015
				

FROM_UNIXTIME(time, issecond)

Returns a representation of the unix_timestamp argument as a value in YYYY-MM-DD HH:MM:SS expressed in the current time zone.

• time: The time stamp value from epoch time. Milliseconds are accepted.
• issecond: Indicates the time stamp value is milliseconds to epoch time.

                      SELECT FROM_UNIXTIME(1540495231, 1);
                      -- Result: 2018-10-25 19:20:31

                      SELECT FROM_UNIXTIME(1540495357385, 0);
                      -- Result: 2018-10-25 19:22:37
                    

GETDATE()

Returns the current time stamp of the database system as a datetime value. This value is equal to CURRENT_TIMESTAMP and SYSDATETIME, and is always in the local timezone.

                  SELECT GETDATE();
                  -- Result: 2018-02-01 03:04:05
                

GETUTCDATE()

Returns the current time stamp of the database system formatted as a UTC datetime value. This value is equal to SYSUTCDATETIME.

In addition, GETUTCDATE can take an optional second parameter, a date and time that are converted to UTC.

                  SELECT GETUTCDATE();
                  -- For example, if the local timezone is Eastern European Time (GMT+2)
                  -- Result: 2018-02-01 05:04:05

                  SELECT GETUTCDATE('2020/08/31 13:56:00')
                  --Result: '2020-08-31 17:56:00'
                

HOUR(date)

Returns the hour component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT HOUR('02-02-2020 11:30:00');
				-- Result: 11
				

ISDATE(date, [date_format])

Returns 1 if the value is a valid date, time, or datetime value; otherwise, 0.

• date: The datetime string.
• date_format: The optional datetime format.

                      SELECT ISDATE('2018-02-01', 'yyyy-MM-dd');
                      -- Result: 1

                      SELECT ISDATE('Not a date');
                      -- Result: 0
                    

LAST_WEEK()

Returns a time stamp equivalent to exactly one week before the current date.

				SELECT LAST_WEEK();	//Assume the date is 3/17/2020	
			 -- Result: 3/10/2020 00:00:00
				

LAST_MONTH()

Returns a time stamp equivalent to exactly one month before the current date.

	
				SELECT LAST_MONTH(); //Assume the date is 3/17/2020
				-- Result: 2/17/2020 00:00:00
				

LAST_YEAR()

Returns a time stamp equivalent to exactly one year before the current date.

				SELECT LAST_YEAR();	//Assume the date is 3/17/2020	
				-- Result: 3/10/2019 00:00:00
				

LDWEEK(date)

Returns the last day of the provided week.

• date: The datetime string.
• weeks to add: An optional integer expression specifying the number of months to add to the date before calculating the last day of the week.

				SELECT LDWEEK('02-02-2020');
				-- Result: 2/8/2020
				

LDMONTH(date)

Returns the last day of the provided month.

• date: The datetime string.
• months to add: An optional integer expression specifying the number of months to add to the date before calculating the last day of the month.

				SELECT LDMONTH('02-02-2020');
				-- Result: 2/29/2020

        SELECT LDMONTH('02-08-2020', 1)
        --Result: 03/31/2020
				

LDQUARTER(date)

Returns the last day of the provided quarter.

• date: The datetime string.
• quarters to add: An optional integer expression specifying the number of months to add to the date before calculating the last day of the quarter.

				SELECT LDQUARTER('02-02-2020');
				-- Result: 3/31/2020

        SELECT LDQUARTER('02-02-2020',1)
        --Result: 06/30/2020
				

MAKEDATE(year, days)

Returns a date value from a year and a number of days.

• year: The year
• days: The number of days into the year. Value must be greater than 0.

          SELECT MAKEDATE(2020, 1);
          -- Result: 2020-01-01
        

MINUTE(date)

Returns the minute component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT MINUTE('02-02-2020 11:15:00');
				-- Result: 15
				

MONTH(date)

Returns the month component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT MONTH('02-02-2020');
				-- Result: 2
				

QUARTER(date)

Returns the quarter associated with the provided datetime.

• date: The datetime string that specifies the date.

				SELECT QUARTER('02-02-2020');
				-- Result: 1
				

SECOND(date)

Returns the second component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT SECOND('02-02-2020 11:15:23');
				-- Result: 23
				

SMALLDATETIMEFROMPARTS(integer_year, integer_month, integer_day, integer_hour, integer_minute)

Returns the datetime value for the specified date and time.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.
• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.

                      SELECT SMALLDATETIMEFROMPARTS(2018, 2, 1, 1, 2);
                      -- Result: 2018-02-01 01:02:00
                    

STRTODATE(string,format)

Parses the provided string value and returns the corresponding datetime.

• string: The string value to be converted to datetime format.
• format: A format string which describes how to interpret the first string input. A few special formats are available as well, including UNIX, UNIXMILIS, TICKS, and FILETICKS.

				SELECT STRTODATE('03*04*2020','dd*MM*yyyy');
				-- Result: 4/3/2020
				

SYSDATETIME()

Returns the current time stamp as a datetime value of the database system. It is equal to GETDATE and CURRENT_TIMESTAMP, and is always in the local timezone.

                  SELECT SYSDATETIME();
                  -- Result: 2018-02-01 03:04:05
                

SYSUTCDATETIME()

Returns the current system date and time as a UTC datetime value. It is equal to GETUTCDATE.

                  SELECT SYSUTCDATETIME();
                  -- For example, if the local timezone is Eastern European Time (GMT+2)
                  -- Result: 2018-02-01 05:04:05
                

TIMEFROMPARTS(integer_hour, integer_minute, integer_seconds, integer_fractions, integer_precision)

Returns the time value for the specified time and with the specified precision.

• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.
• seconds: The integer expression specifying the seconds.
• fractions: The integer expression specifying the fractions of the second.
• precision : The integer expression specifying the precision of the fraction.

                      SELECT TIMEFROMPARTS(1, 2, 3, 456, 3);
                      -- Result: 01:02:03.456
                    

TO_DAYS(date)

Returns the number of days since 0000-00-01. This will only return a value for dates on or after 1582-10-15 (based upon the Gregorian calendar). This will be equivalent to the MYSQL TO_DAYS function.

• date: The datetime string that specifies the date.

				SELECT TO_DAYS('02-06-2015');
				-- Result: 736000
				

WEEK(date)

Returns the week (of the year) associated with the provided datetime.

• date: The datetime string that specifies the date.

				SELECT WEEK('02-17-2020 11:15:23');
				-- Result: 8
				

YEAR(date)

Returns the integer that specifies the year of the specified date.

• date: The datetime string.

                      SELECT YEAR('2018-02-01');
                      -- Result: 2018
                    

The following date literal functions can be used to filter date fields using relative intervals. Note that while the <, >, and = operators are supported for these functions, <= and >= are not.

L_TODAY()

The current day.

  SELECT * FROM MyTable WHERE MyDateField = L_TODAY()

L_YESTERDAY()

The previous day.

  SELECT * FROM MyTable WHERE MyDateField = L_YESTERDAY()

L_TOMORROW()

The following day.

  SELECT * FROM MyTable WHERE MyDateField = L_TOMORROW()

L_LAST_WEEK()

Every day in the preceding week.

  SELECT * FROM MyTable WHERE MyDateField = L_LAST_WEEK()

L_THIS_WEEK()

Every day in the current week.

  SELECT * FROM MyTable WHERE MyDateField = L_THIS_WEEK()

L_NEXT_WEEK()

Every day in the following week.

  SELECT * FROM MyTable WHERE MyDateField = L_NEXT_WEEK()

Also available:

• L_LAST/L_THIS/L_NEXT MONTH
• L_LAST/L_THIS/L_NEXT QUARTER
• L_LAST/L_THIS/L_NEXT YEAR

L_LAST_N_DAYS(n)

The previous n days, excluding the current day.

  SELECT * FROM MyTable WHERE MyDateField = L_LAST_N_DAYS(3)

L_NEXT_N_DAYS(n)

The following n days, including the current day.

  SELECT * FROM MyTable WHERE MyDateField = L_NEXT_N_DAYS(3)

Also available:

• L_LAST/L_NEXT_90_DAYS

L_LAST_N_WEEKS(n)

Every day in every week, starting n weeks before current week, and ending in the previous week.

  SELECT * FROM MyTable WHERE MyDateField = L_LAST_N_WEEKS(3)

L_NEXT_N_WEEKS(n)

Every day in every week, starting the following week, and ending n weeks in the future.

  SELECT * FROM MyTable WHERE MyDateField = L_NEXT_N_WEEKS(3)

Also available:

• L_LAST/L_NEXT_N_MONTHS(n)
• L_LAST/L_NEXT_N_QUARTERS(n)
• L_LAST/L_NEXT_N_YEARS(n)

A SELECT statement can consist of the following basic clauses.

• SELECT
• INTO
• FROM
• JOIN
• WHERE
• GROUP BY
• HAVING
• UNION
• ORDER BY
• LIMIT

SELECT Syntax

The following syntax diagram outlines the syntax supported by the SQL engine of the connector:

SELECT {
  [ TOP <numeric_literal> | DISTINCT ]
  { 
    * 
    | { 
        <expression> [ [ AS ] <column_reference> ] 
        | { <table_name> | <correlation_name> } .* 
      } [ , ... ] 
  }
  { 
    FROM <table_reference> [ [ AS ] <identifier> ] 
  } [ , ... ]
  [ [  
      INNER | { { LEFT | RIGHT | FULL } [ OUTER ] } 
    ] JOIN <table_reference> [ ON <search_condition> ] [ [ AS ] <identifier> ] 
  ] [ ... ] 
  [ WHERE <search_condition> ]
  [ GROUP BY <column_reference> [ , ... ]
  [ HAVING <search_condition> ]
  [ UNION [ ALL ] <select_statement> ]
  [ 
    ORDER BY 
    <column_reference> [ ASC | DESC ] [ NULLS FIRST | NULLS LAST ]
  ]
  [ 
    LIMIT <expression>
    [ 
      { OFFSET | , }
      <expression> 
    ]
  ] 
} | SCOPE_IDENTITY() 

<expression> ::=
  | <column_reference>
  | @ <parameter> 
  | ?
  | COUNT( * | { [ DISTINCT ] <expression> } )
  | { AVG | MAX | MIN | SUM | COUNT } ( <expression> ) 
  | NULLIF ( <expression> , <expression> ) 
  | COALESCE ( <expression> , ... ) 
  | CASE <expression>
      WHEN { <expression> | <search_condition> } THEN { <expression> | NULL } [ ... ]
    [ ELSE { <expression> | NULL } ]
    END 
  | {RANK() | DENSE_RANK()} OVER ([PARTITION BY <column_reference>] {ORDER BY <column_reference>})
  | <literal>
  | <sql_function> 

<search_condition> ::= 
  {
    <expression> { = | > | < | >= | <= | <> | != | LIKE | NOT LIKE | IN | NOT IN | IS NULL | IS NOT NULL | AND | OR | CONTAINS | BETWEEN | IS DISTINCT FROM | IS NOT DISTINCT FROM } [ <expression> ]
  } [ { AND | OR } ... ] 

Examples

1. Return all columns:
   

   SELECT * FROM Calendar

2. Rename a column:
   

   SELECT [Location] AS MY_Location FROM Calendar

3. Cast a column's data as a different data type:
   

   SELECT CAST(AnnualRevenue AS VARCHAR) AS Str_AnnualRevenue FROM Calendar

4. Search data:
   

   SELECT * FROM Calendar WHERE Location <> 'Chapel Hill'

5. Return the number of items matching the query criteria:
   

   SELECT COUNT(*) AS MyCount FROM Calendar 

6. Return the number of unique items matching the query criteria:
   

   SELECT COUNT(DISTINCT Location) FROM Calendar 

7. Return the unique items matching the query criteria:
   

   SELECT DISTINCT Location FROM Calendar 

8. Sort a result set in ascending order:
   

   SELECT Id, Location FROM Calendar  ORDER BY Location ASC

9. Restrict a result set to the specified number of rows:
   

   SELECT Id, Location FROM Calendar LIMIT 10 

10. Parameterize a query to pass in inputs at execution time. This enables you to create prepared statements and mitigate SQL injection attacks.
    

    SELECT * FROM Calendar WHERE Location = @param

See Explicitly Caching Data for information on using the SELECT statement in offline mode.

Pseudo Columns

Some input-only fields are available in SELECT statements. These fields, called pseudo columns, do not appear as regular columns in the results, yet may be specified as part of the WHERE clause. You can use pseudo columns to access additional features from Microsoft SharePoint.

    SELECT * FROM Calendar WHERE CAMLQuery = '<Query><Where><Or><Gt><FieldRef Name="Balance" /><Value Type="Number">10</Value></Gt></Or></Where></Query>'
    

Aggregate Functions

For SELECT examples using aggregate functions, see Aggregate Functions.

JOIN Queries

See JOIN Queries for SELECT query examples using JOINs.

Date Literal Functions

Date Literal Functions contains SELECT examples with date literal functions.

Window Functions

See Window Functions for SELECT examples containing window functions.

Table-Valued Functions

See Table-Valued Functions for SELECT examples with table-valued functions.

COUNT

Returns the number of rows matching the query criteria.

SELECT COUNT(*) FROM Calendar WHERE Location = 'Chapel Hill'

COUNT(DISTINCT)

Returns the number of distinct, non-null field values matching the query criteria.

SELECT COUNT(DISTINCT Id) AS DistinctValues FROM Calendar WHERE Location <> 'Chapel Hill'

AVG

Returns the average of the column values.

SELECT Location, AVG(AnnualRevenue) FROM Calendar WHERE Location <> 'Chapel Hill'  GROUP BY Location

MIN

Returns the minimum column value.

SELECT MIN(AnnualRevenue), Location FROM Calendar WHERE Location <> 'Chapel Hill' GROUP BY Location

MAX

Returns the maximum column value.

SELECT Location, MAX(AnnualRevenue) FROM Calendar WHERE Location <> 'Chapel Hill' GROUP BY Location

SUM

Returns the total sum of the column values.

SELECT SUM(AnnualRevenue) FROM Calendar WHERE Location = 'Chapel Hill'

The CData Python Connector for Microsoft SharePoint supports standard SQL joins like the following examples.

Inner Join

An inner join selects only rows from both tables that match the join condition:

SELECT d.LinkFileName, u.Name FROM Documents d, Users u WHERE d.CheckOutUser = u.Name

Left Join

A left join selects all rows in the FROM table and only matching rows in the JOIN table:

SELECT d.LinkFileName, u.Name FROM Users u LEFT JOIN Documents d ON d.CheckOutUser = u.Name

Window functions allow you to create computed fields from a group of rows (a window) that return a result for each row, as opposed to one computed result for a set of rows, as is the case with aggregate functions. The connector supports the following window function syntax.

Note: Window function support is an experimental feature of the connector. This functionality extends beyond the connector's core scope of being SQL-92 compliant. As such, performance with window functions may not be optimal.

Window Function Clauses

OVER

The OVER clause defines the window over which window functions are performed.

SELECT A, B, <window function> OVER (<window frame>) FROM TableName

The <window function> refers to any supported window function clause, and the <window frame> refers to one or more clauses that specify the logic by which the window is defined.

PARTITION BY

The PARTITION BY clause subdivides a window into sub-windows called partitions. For each unique value in the column specified in the PARTITION BY clause, every record with that value collectively forms an individual partition.

SELECT A, B, <window function> OVER (PARTITION BY A ORDER BY B) From Calendar

The <window function> refers to any supported window function clause.

Window Functions

The connector supports math, ranking, and analytic window functions.

Math

These window functions perform mathematical operations on the records within the window.

COUNT()

Calculates the number of records in each partition. The calculated column is of the data type "int".

In each partition, every record will display the total number of records in that partition.

SELECT Name, Role, Earnings, COUNT() OVER (PARTITION BY Role) FROM Employees

COUNT_BIG()

Calculates the number of records in each partition. The calculated column is of the data type "bigint".

In each partition, every record will display the total number of records in that partition.

SELECT Name, Role, Earnings, COUNT_BIG() OVER (PARTITION BY Role) FROM Employees

MIN(numeric_column)

Calculates the minimum value of a numerical column per partition.

In each partition, every record will display the minimum value of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, MIN(Earnings) OVER (PARTITION BY Role) FROM Employees

MAX(numeric_column)

Calculates the maximum value of a numerical column per partition.

In each partition, every record will display the maximum value of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, MAX(Earnings) OVER (PARTITION BY Role) FROM Employees

SUM(numeric_column)

Calculates the sum of a numerical column per partition.

In each partition, every record will display the sum of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, SUM(Earnings) OVER (PARTITION BY Role) FROM Employees

AVG(numeric_column)

Calculates the average value of a numerical column per partition.

In each partition, every record will display the average value of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, AVG(Earnings) OVER (PARTITION BY Role) FROM Employees

MEDIAN(numeric_column)

Calculates the median value of a numerical column per partition.

In each partition, every record will display the median value of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, MEDIAN(Earnings) OVER (PARTITION BY Role) FROM Employees

STDEV(numeric_column)

Calculates the standard deviation of a numerical column per partition.

In each partition, every record will display the standard deviation of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, STDEV(Earnings) OVER (PARTITION BY Role) FROM Employees

STDEVP(numeric_column)

Calculates the population standard deviation of a numerical column per partition.

In each partition, every record will display the population standard deviation of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, STDEVP(Earnings) OVER (PARTITION BY Role) FROM Employees

VAR(numeric_column)

Calculates the statistical standard variance of a numerical column per partition.

In each partition, every record will display the statistical standard variance of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, VAR(Earnings) OVER (PARTITION BY Role) FROM Employees

VARP(numeric_column)

Calculates the variance population of a numerical column per partition.

In each partition, every record will display the variance population of numeric_column across the records in that partition.

SELECT Name, Role, Earnings, VARP(Earnings) OVER (PARTITION BY Role) FROM Employees

Ranking

These window functions rank records that fall within the window and its partitions.

RANK()

Assigns a rank number to each record in a window based on the value of the column specified in the required ORDER BY clause.

If two or more records have an equal value in the in ranked column, they all receive the same rank number and the rank count increments internally, skipping ahead one rank number for each record with a duplicate value in the ORDER BY column.

SELECT Id, Location, RANK() OVER (ORDER BY Location) AS Rank FROM Calendar

If you add a PARTITION BY clause, a separate set of ranks is calculated for each partition.

SELECT Id, Location, RANK() OVER (PARTITION BY Id ORDER BY Location) AS Rank FROM Calendar

DENSE_RANK()

Operates like the RANK() function, but it doesn't increment the internal rank counter for each record with a duplicate value in the ranked column.

This means that, while records with identical values in the ORDER BY column still share a rank number, the function never skips a rank number.

SELECT Id, Location, DENSE_RANK() OVER (PARTITION BY Id ORDER BY Location) AS Rank FROM Calendar

If you add a PARTITION BY clause, a separate set of ranks is calculated for each partition.

SELECT Id, Location, DENSE_RANK() OVER (PARTITION BY Id ORDER BY Location) AS Rank FROM Calendar

ROW_NUMBER()

Calculates a row number for each record. An ORDER BY clause in the OVER clause is required.

SELECT Name, Role, Earnings, ROW_NUMBER() OVER (ORDER BY Role) FROM Employees

If you define multiple partitions with PARTITION BY, a new set of row numbers are calculated for each partition.

SELECT Name, Role, Earnings, ROW_NUMBER() OVER (PARTITION BY Role ORDER BY Earnings) FROM Employees

NTILE()

Distributes rows of an ordered partition into a specified number of approximately equal groups, or buckets. It assigns each group a bucket number starting from one. For each row in a group, the NTILE() function assigns a bucket number representing the group to which the row belongs.

The syntax of NTILE() is:

NTILE(buckets) OVER (
    [PARTITION BY partition_expression, ... ]
    ORDER BY sort_expression [ASC | DESC], ...
)

The following are paramaters that NTILE() supports:

• buckets: The number of buckets into which the rows are divided. The buckets can be an expression or subquery that evaluates to a positive integer. It cannot be a window function.
• PARTITION BY: distributes rows of a result set into partitions to which the NTILE() function is applied.
• ORDER BY is clause that specifies the logical order of rows in each partition to which the NTILE() is applied.

If the number of rows is not divisible by the buckets, the NTILE() function returns groups of two sizes with the difference by one. The larger groups always precede the smaller group in the order set by ORDER BY in the OVER() clause.

If the total of rows is divisible by the number of buckets, the function divides the rows evenly among buckets. The following statement creates a new table named ntile_demo that stores 10 integers:

CREATE TABLE sales.ntile_demo (
	v INT NOT NULL
);
	
INSERT INTO sales.ntile_demo(v) 
VALUES(1),(2),(3),(4),(5),(6),(7),(8),(9),(10);	
	
SELECT * FROM sales.ntile_demo;

This statement uses the NTILE() function to divide ten rows into three groups:

SELECT 
	v, 
	NTILE (3) OVER (
		ORDER BY v
	) buckets
FROM 
	sales.ntile_demo;

Analytical

These window functions perform analytical operations on the records within the window.

PERCENT_RANK()

Calculates the relative rank SQL Percentile of each row. It returns values greater than zero, but the maximum value is one. It does not count any NULL values. This function is nondeterministic.

The syntax of PERCENT_RANK() is:

PERCENT_RANK() OVER (
      [PARTITION BY partition_expression, ... ]
      ORDER BY sort_expression [ASC | DESC], ...
  )
  

This syntax uses the following parameters.

• PARTITION BY: By default, SQL Server treats the whole data set as a single set. You can specify the PARTITION BY clause to divide data into multiple sets. The Percent_Rank function performs the analytical calculations on each set. This parameter is optional.
• ORDER BY: Sorts the data in either ascending or descending order. This parameter is required.

Table-valued functions are functions that return a table (rowset).

Note: Table-valued function support is an experimental feature of the connector. This functionality extends beyond the connector's core scope of being SQL-92 compliant. As such, performance with these functions may not be optimal.

Table-Valued Function Clauses

CROSS APPLY

The CROSS APPLY operator is used to perform a subquery on each row of a table or resultset produced by a preceding table expression.

<table_expression_1> CROSS APPLY <table_expression_2>

The second table expression can reference results from the first table expression to create derived columns or an altered recordset via a table-valued function.

Each resulting record is an instance of the record it's splitting, with all the same column values, except for the column(s) containing the value split by the function.

WITH

The WITH clause is used alongside certain table-valued functions to match against constructs within the structure being split (keys, element names, attribute names, etc.) and/or to specify metadata for the columns generated from the function.

SELECT A.ColumnName, X.DerivedColumnName FROM TableName A CROSS APPLY <table-valued function> WITH (DerivedColumnName varchar(255)) AS X

Table-Valued Functions

STRING_SPLIT(input_text,delimiter)

Takes each record in the recordset of the preceding table expression, splits the column containing delimiters (input_text) into substrings separated by the delimiter, and returns one record per substring.

• input_text: A column whose value you want to parse.
• delimiter: The character used to split the value of the column specified in input_text.

Suppose there is a column called "SplitColumn" with the following content:

One-Two-Three

To unpack this value across multiple records:

SELECT A.ID, X.Value FROM [TableWithDelimitedStringField] A CROSS APPLY STRING_SPLIT(A.SplitColumn,'-') WITH (Value VARCHAR(255)) AS X

-- Results:
-----------
|ID|Value|
|1|One|
|1|Two|
|1|Three|

JSONTABLE(json_content,[jsonpath])

For each record in the recordset of the preceding table expression, returns one record for each instance of a key in a JSON array (json_content) that matches the key(s) specified in the WITH clause, at the scope specified by the "jsonpath" input.

• json_content: A JSON "table" (array of objects). The contents can nest, but this must be a single JSON array, not any other JSON structure, at the root level.
  • The values of every instance of the key(s) provided in the WITH clause are retrievable only for substructures which are immediate children of the root-level JSON array.
• jsonpath: An optional JSONPath query defining the scope, within the json_content array, that you want to pull content from. The JSON key(s) identified in the WITH clause must exist at the scope defined in this parameter. This defaults to the JSON root ($).

Consider a sample table with a single record, including an ID column and column with JSON content called "JSONColumn" with the following content:

[
	{
		"name": "Samuel",
		"email": "sam@gmail.com",
		"extrainfo": {
			"city": "Seattle"
		}
	},
	{
		"name": "Katherine",
		"email": "kat@gmail.com",
	},
	{
		"name": "George",
		"email": "george23@gmail.com",
	},
	{
		"name": "Carlos",
		"email": "carlos32@gmail.com",
	}
]

To extract all values for a certain key, specify the scope in the JSONTABLE function and provide the desired key(s) in the WITH clause.

SELECT A.ID, X.name FROM [TableWithJSONField] A CROSS APPLY JSONTABLE(A.JSONColumn) WITH (name VARCHAR(255)) AS X

-- Results: 
|ID|name|
---------
|1 |Samuel|
|1 |Katherine|
|1 |George|
|1 |Carlos|

XMLTABLE(xml_content,[xpath,child_type])

For each record in the resultset of the preceding table expression, returns one record for each of the elements and/or attributes in an XML structure (xml_content) that match the tag name(s) and/or attribute name(s) specified in the WITH clause, at the scope specified in the "xpath" input.

• xml_content: A column containing an XML structure.
• xpath: An optional XPath that specifies the scope within the XML structure at which the connector extracts content matching the tag/attribute name(s) specified in the WITH clause.
  • When extracting the content of sub-elements, the connector can retrieve all content from tags at the root level, (depth 0) immediate children of the root (depth 1), and children of those children (depth 2).
  • When extracting element attribute content, the connector can retrieve all content from tags containing the specified attribute at the root level (depth 0) and from immediate children of root-level elements (depth 1).
• child_type: An optional parameter that specifies the part(s) of the parent element (specified in the xpath input) that the column(s) provided in the WITH clause are checked against to identify content.
  • You can supply the following values:
    • 0: The column(s) in the WITH clause are checked for matches against the parent element's attribute names and sub-element tag names.
    • 1: The column(s) in the WITH clause are checked for matches against the parent element's attribute names.
    • 2: The column(s) in the WITH clause are checked for matches against the parent element's sub-element tag names.
  • When not supplied, this defaults to 0.

Extracting Sub-Element Values

Consider a sample table with a single record, including an ID column and a column with XML content called "XMLContent" with the following content:

<shoppingList>
    <item>
        <name>Apples</name>
        <quantity>3</quantity>
        <unit>Kg</unit>
    </item>
    <item>
        <name>Bread</name>
        <quantity>2</quantity>
        <unit>Loaf</unit>
		<extrainfo>
			<Type>Whole-Grain</Type>
		</extrainfo>
    </item>
    <item>
        <name>Milk</name>
        <quantity>1</quantity>
        <unit>Carton</unit>
    </item>
    <item>
        <name>Eggs</name>
        <quantity>12</quantity>
        <unit></unit>
    </item>
</shoppingList>

To extract sub-element content, specify the scope in the XMLTABLE function and provide the desired element name(s) in the WITH clause. Note that this will not work if the XMLTABLE function's child_type input is set to 1.

SELECT A.ID, X.name FROM [TableWithXMLField] A CROSS APPLY XMLTABLE(A.XMLContent,'//*/item') WITH (name VARCHAR(255)) AS X

-- Results: 
|ID|name|
---------
|1|Apples|
|1|Bread|
|1|Milk|
|1|Eggs|

Extracting Values Using Element Tag Attributes

Suppose you have this sample table with a single record, including an ID column and a column with XML content called "XMLContent" with the following content:

<restaurant>
  <dish type="appetizer">
    <name lang="en">Caprese Salad</name>
    <chef>Chef Giovanni</chef>
    <price currency="USD">9.99</price>
  </dish>
  <dish type="main-course">
    <name lang="fr">Boeuf Bourguignon</name>
    <chef>Chef Marie</chef>
    <price currency="EUR">19.99</price>
  </dish>
  <dish type="dessert">
    <name lang="es">Tres Leches Cake</name>
    <chef>Chef Alejandro</chef>
    <price currency="MXN">89.99</price>
  </dish>
</restaurant>

To extract attribute content, specify the scope in the XMLTABLE function and provide the desired attribute name(s) in the WITH clause. Note that this will not work if the XMLTABLE function's child_type input is set to 2.

SELECT A.ID, X.type FROM [TableWithXMLField] A CROSS APPLY XMLTABLE(A.XMLContent,'//*/dish') WITH (type VARCHAR(255)) AS X

-- Results: 
|ID|type|
---------
|1|appetizer|
|1|main-course|
|1|dessert|

CSVTABLE(csv_content,[delimiter])

For each record in the resultset of the preceding table expression, reads from a column that contains a CSV table (csv_content) and for each record in that CSV table, returns one record containing the value of the CSV column(s) specified in the WITH clause.

• csv_content: A column containing a CSV table.
• delimiter: An optional custom delimiter (instead of a comma) which splits the CSV content contained in the csv_content input.

Consider a sample table with a single record, including an ID column and a column containing CSV table called "CSVContent" with the following content:

Name;Category;Price
Apple;Fruit;0.99
Spaghetti;Pasta;5.49
Chicken Breast;Meat;8.99
Broccoli;Vegetable;2.49

To select every value in the "Name" column and account for the custom delimiter (;):

SELECT A.ID, X.Name FROM [TableWithCSVField] A CROSS APPLY CSVTABLE(A.CSVContent,';') WITH (Name VARCHAR(255)) AS X

-- Results:
|ID|Name|
-----------
|1|Apple|
|1|Spaghetti|
|1|Chicken Breast|
|1|Broccoli|

To create new records, use INSERT statements.

INSERT Syntax

The INSERT statement specifies the columns to be inserted and the new column values. You can specify the column values in a comma-separated list in the VALUES clause, as shown in the following example:

INSERT INTO <table_name> 
( <column_reference> [ , ... ] )
VALUES 
( { <expression> | NULL } [ , ... ] ) 
  

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

The following is an example query:

INSERT INTO Calendar (Location) VALUES ('U.S.A.')

To modify existing records, use UPDATE statements.

Update Syntax

The UPDATE statement takes as input a comma-separated list of columns and new column values as name-value pairs in the SET clause, as shown in the following example:

UPDATE <table_name> SET <select_statement> | {<column_reference> = <expression> [ , ... ]} WHERE { Id = <expression>  } [ { AND | OR } ... ] 

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

The following is an example query:

UPDATE Calendar SET Location='U.S.A.' WHERE Id = @myId

To delete information from a table, use DELETE statements.

DELETE Syntax

The DELETE statement requires the table name in the FROM clause and the row's primary key in the WHERE clause, as shown in the following example:

<delete_statement> ::= DELETE FROM <table_name> WHERE { Id = <expression> } [ { AND | OR } ... ]

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

The following is an example query:

DELETE FROM Calendar WHERE Id = @myId

When caching is enabled, CACHE statements provide complete control over the data that is cached and the table to which it is cached. The CACHE statement executes the SELECT statement specified and caches its results to a table with the same name in the cache database or to table specified in <cached_table_name>. The connector updates or inserts rows to the cache depending on whether or not they already exist in the cache, so the primary key, which is used to identify existing rows, must be included in the selected columns.

See Caching Data for more information on different caching strategies.

CACHE Statement Syntax

The cache statement may include the following options that alter its behavior:

CACHE [ <cached_table_name> ] [ WITH TRUNCATE | AUTOCOMMIT | SCHEMA ONLY | DROP EXISTING | ALTER SCHEMA ] <select_statement> 

WITH TRUNCATE

If this option is set, the connector removes existing rows in the cache table before adding the selected rows. Use this option if you want to refresh the entire cache table but keep its existing schema.

AUTOCOMMIT

If this option is set, the connector commits each row individually. Use this option if you want to ignore the rows that could not be cached due to some reason. By default, the entire result set is cached as a single transaction.

DROP EXISTING

If this option is set, the connector drops the existing cache table before caching the new results. Use this option if you want to refresh the entire cache table, including its schema.

SCHEMA ONLY

If this option is set, the connector creates the cache table based on the SELECT statement without executing the query.

ALTER SCHEMA

If this option is set, the connector alters the schema of the existing table in the cache if it does not match the schema of the SELECT statement. This option results in new columns or dropped columns, if the schema of the SELECT statement does not match the cached table.

Common Queries

Use the following cache statement to cache all rows of a table:

CACHE SELECT * FROM Calendar

Use the following cache statement to cache all rows of a table into the cache table CachedCalendar:

CACHE CachedCalendar SELECT * FROM Calendar

Use the following cache statement for incremental caching. The DateModified column may not exist in all tables. The cache statement shows how incremental caching would work if there were such a column. Also, notice that, in this case, the WITH TRUNCATE and DROP EXISTING options are specifically omitted, which would have deleted all existing rows.

CACHE CachedCalendar SELECT * FROM Calendar WHERE DateModified > '2013-04-04'

Use the following cache statements to create a table with all available columns that will then cache only a few of them. The sequence of statements cache only Id and Location even though the cache table CachedCalendar has all the columns in Calendar.

CACHE CachedCalendar SCHEMA ONLY SELECT * FROM Calendar
CACHE CachedCalendar SELECT Id, Location FROM Calendar

To execute stored procedures, you can use EXECUTE or EXEC statements.

EXEC and EXECUTE assign stored procedure inputs, referenced by name, to values or parameter names.

Stored Procedure Syntax

To execute a stored procedure as an SQL statement, use the following syntax:

 
{ EXECUTE | EXEC } <stored_proc_name> 
{
  [ @ ] <input_name> = <expression>
} [ , ... ]

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

Example Statements

Reference stored procedure inputs by name:

EXECUTE my_proc @second = 2, @first = 1, @third = 3;

Execute a parameterized stored procedure statement:

EXECUTE my_proc second = @p1, first = @p2, third = @p3; 

PIVOT and UNPIVOT can be used to change a table-valued expression into another table.

PIVOT

PIVOT rotates a table-value expression by turning unique values from one column into multiple columns in the output. PIVOT can run aggregations where required on any column value.

PIVOT Synax

 
"SELECT 'AverageCost' AS Cost_Sorted_By_Production_Days, [0], [1], [2], [3], [4]
FROM
(
SELECT DaysToManufacture, StandardCost
FROM Production.Product
) AS SourceTable
PIVOT
(
AVG(StandardCost)
FOR DaysToManufacture IN ([0], [1], [2], [3], [4])
) AS PivotTable;"

UNPIVOT

UNPIVOT carries out nearly the opposite to PIVOT by rotating columns of a table-valued expressions into column values.

UNPIVOT Sytax

 
"SELECT VendorID, Employee, Orders
FROM
(SELECT VendorID, Emp1, Emp2, Emp3, Emp4, Emp5
FROM pvt) p
UNPIVOT
(Orders FOR Employee IN
(Emp1, Emp2, Emp3, Emp4, Emp5)
)AS unpvt;"

For further information on PIVOT and UNPIVOT, see FROM clause plus JOIN, APPLY, PIVOT (Transact-SQL)

Use INSERT INTO SELECT queries to select a list of records from one table and insert those same records into another table as a group. Inserting batches of records in this way may result in improved query performance compared to using many individual INSERT INTO queries.

The table whose records are selected for insertion into another table can be either a real table or a user-defined temporary table.

Inserting Records from Real Tables

To insert a group of records from one real, non-temporary, source table into another destination table, you can use an INSERT INTO SELECT query. This type of query is formatted similarly to a standard INSERT INTO query, except the VALUES clause is substituted with a SELECT query targeting the source table. All records matched by the embedded SELECT query are inserted into the destination table.

If the source table and destination tables have different column names, you must map columns from the source table to the corresponding columns in the destination table you want to insert them into. Perform this mapping by specifying the destination table columns in the same order as the source table columns you want to match them with. For example:

INSERT INTO DestinationTable (A,B,C,D) SELECT Q,R,S,T FROM SourceTable

In this example, the first source column (Q) is inserted into the first destination column (A), the second source column (R) is inserted into the second destination column (B), and so on.

If the source table and destination table both have the same column list with the same names, you can use a streamlined query.

INSERT INTO DestinationTableWithSameColumns SELECT * FROM SourceTable

In this example, there is no need to specify a list of columns for either the source or destination table, because their metadata already matches.

Inserting Records from Temporary Tables

You can manually define and populate temporary tables to hold a list of records for later bulk insertion.

Populate the Temporary Table

To create a temporary table, you must give it a name ending in "#TEMP" and execute an INSERT INTO query using that name, as if that table already existed in the database. After executing the first INSERT INTO, the temporary table exists and can receive subsequent INSERTs. For example:

INSERT INTO Calendar#TEMP (Location, MyCustomField__c) VALUES ('New Calendar', '9000');
INSERT INTO Calendar#TEMP (Location, MyCustomField__c) VALUES ('New Calendar 2', '9001');
INSERT INTO Calendar#TEMP (Location, MyCustomField__c) VALUES ('New Calendar 3', '9002');

This creates a temporary table called Calendar#TEMP with two columns and three rows of data. Since type cannot be determined on the temporary table itself, all values are considered strings and later converted to the proper type when they are inserted together into the real (non-temporary) table of interest.

Insert Temporary Table Contents into Real Tables

Once your temporary table is populated, execute an INSERT INTO SELECT query targeting the real (non-temporary) table you want to insert the temporary table's records into. This is formatted similarly to a standard INSERT INTO query, except the VALUES clause is substituted with a SELECT query targeting the matching columns in the temporary table. For example:

INSERT INTO Calendar (Location, MyCustomField__c) SELECT Location, MyCustomField__c FROM Calendar#TEMP

In this example, the full contents of Calendar#TEMP are inserted into the Calendar.

Results

The LastResultInfo#TEMP temporary table contains details about the most recently executed query that uses the contents of a temporary table in an embedded SELECT clause, as is the case for INSERT INTO SELECT queries that use a temporary table as the source of records. This table is cleared and repopulated each time such a query is executed. LastResultInfo#TEMP includes information such as whether the query in question succeeded, and how many rows were affected by the query.

Temporary Table Lifespan

Temporary tables only last as long as the connection remains open. When the connection to Microsoft SharePoint is closed, all temporary tables are cleared, including the LastResultInfo#TEMP table.

To perform multiple updates in a single request to Microsoft SharePoint,first use the INSERT INTO syntax to insert a temporary table of data into Microsoft SharePoint. This works by first populating a temporary table with the data you are going to submit to Microsoft SharePoint. Once you have all of the data you want to update, use UPDATE SELECT FROM to pass the temporary table data into the table in Microsoft SharePoint.

Populate the Temporary Table

The temporary table you are populating is dynamic and is created at run time the first time you insert to it. Temporary tables are denoted by a # appearing in their name. When using a temporary table to update, the temporary table must be named in the format [TableName]#TEMP, where TableName is the name of the table you are inserting to. For example:

INSERT INTO Calendar#TEMP (Id, Name, MyCustomField__c) VALUES ('AX1000001', 'New Calendar', '9000');
INSERT INTO Calendar#TEMP (Id, Name, MyCustomField__c) VALUES ('AX1000002', 'New Calendar 2', '9001');
INSERT INTO Calendar#TEMP (Id, Name, MyCustomField__c) VALUES ('AX1000003', 'New Calendar 3', '9002');

This creates a temporary table called Calendar#TEMP with three columns and three rows of data. Since type cannot be determined on the temporary table itself, all values are stored in memory as strings. The values are later converted to the proper type when they are submitted to the Calendar table.

Update the Actual Table

Once your temporary table is populated, it is now time to update the actual table in Microsoft SharePoint. You can do this by performing an UPDATE to the actual table and selecting the input data from the temporary table. For example:

UPDATE Calendar (Id, Location, MyCustomField__c) SELECT Id, Location, MyCustomField__c FROM Calendar#TEMP

In this example, the full contents of the Calendar#TEMP table are passed into the Calendar table. This results in fewer requests being submitted to Microsoft SharePoint since multiple updates may be submitted with each request, which is much better for performance if you have many records to update.

Results

The results of the query are stored in the LastResultInfo#TEMP temporary table. This table is cleared and repopulated the next time data is modified by passing in a temporary table. Please be aware that the LastResultInfo#TEMP table has no predefined schema. You need to check its metadata at run time before reading data.

Temporary Table Life Span

Temporary tables only last as long as the connection remains open. When the connection to Microsoft SharePoint is closed, all temporary tables are cleared, including the LastResultInfo#TEMP table.

To perform multiple deletes in a single request to Microsoft SharePoint, first use the INSERT INTO syntax to create an in-memory temporary table of data to be deleted. Once you have all of the data you want to delete added to temporary table, use DELETE FROM syntax to delete data from the live table in Microsoft SharePoint. This functionality is also available via the standard Batch Processing API available in JDBC.

Populate the Temporary Table

The temporary table you are populating is dynamic and is created at run time the first time you insert to it. Temporary tables are denoted by a # appearing in their name. When using a temporary table to delete, the temporary table must be named in the format [TableName]#TEMP, where TableName is the name of the table you are inserting to. For example:

INSERT INTO Calendar#TEMP (Id) VALUES ('AX1000001');
INSERT INTO Calendar#TEMP (Id) VALUES ('AX1000002');
INSERT INTO Calendar#TEMP (Id) VALUES ('AX1000003');

This creates a temporary table called Calendar#TEMP with one column and three rows of data. Since type cannot be determined on the temporary table itself, all values are stored in memory as strings. They are later converted to the proper type when they are submitted to the Calendar table.

Delete from the Actual Table

Once your temporary table is populated, it is now time to insert to the actual table in Microsoft SharePoint. You can do this by performing a DELETE from the actual table and selecting the input data from the temporary table. For example:

DELETE FROM Calendar WHERE EXISTS SELECT Id FROM Calendar#TEMP

In this example, the full contents of the Calendar#TEMP table are passed into the Calendar table. This results in fewer requests being submitted to Microsoft SharePoint since multiple deletes may be submitted with each request, which is much better for performance if you have many records to delete.

Results

The results of the query are stored in the LastResultInfo#TEMP temporary table. This table is cleared and repopulated the next time data is modified by passing in a temporary table. Please be aware that the LastResultInfo#TEMP table has no predefined schema. You need to check its metadata at run time before reading data.

Temporary Table Life Span

Temporary tables only last as long as the connection remains opened. When the connection to Microsoft SharePoint is closed, all temporary tables are cleared, including the LastResultInfo#TEMP table.

The CData Python Connector for Microsoft SharePoint models Microsoft SharePoint data as an easy-to-use SQL database with tables, views, and stored procedures. Live connectivity to these objects means that any changes to your Microsoft SharePoint account are immediately reflected in the connector.

Tables

The connector can expose custom lists from Microsoft SharePoint that are not mentioned in the Tables. The data model illustrates a sample of what your SharePoint site might look like. The actual data model will be obtained dynamically based on your user credentials and SharePoint site.

Common tables include:

Table	Description
Attachments	Manages attachments for SharePoint list items, allowing retrieval and deletion. Essential for users who frequently handle file attachments within SharePoint.
Groups	Allows the creation, modification, deletion, and retrieval of SharePoint security groups. Essential for managing user permissions and access control.
Roles	Allows the creation, modification, deletion, and retrieval of SharePoint roles and permission assignments. Useful for customizing access control.
Users	Manages SharePoint users, allowing updates, deletions, and retrieval of user details. Important for keeping SharePoint user management up-to-date.
Views	Lists all subsites within a SharePoint site, including hierarchy details. Helps with site navigation and organization.

Views

Typically, entities that cannot be modified are represented as Views, or read-only tables.

Common views include:

Table	Description
FileVersions	Lists all available versions of a document stored in SharePoint, including version history details. Useful for tracking changes and restoring previous document versions.
GetValidTerms	Retrieves a list of valid managed metadata terms associated with a specific column in a SharePoint list. Helps enforce consistent categorization and tagging of SharePoint content.
Lists	Retrieves metadata about all SharePoint lists available on the site, including properties and settings. Useful for understanding the structure and usage of SharePoint lists.
Permissions	The permissions for a site or list. Note: If ItemId is empty, set the ObjectType to List or Web (an ObjectName must be specified when the ObjectType is List). If not, you must specify the ObjectName along with the ItemID.
Subsites	Allows the creation, modification, deletion, and retrieval of SharePoint roles and permission assignments. Useful for customizing access control.

You can also access custom views of a list as relational views. To get data from a custom view of a list, you can set the ViewID pseudo column in the WHERE clause.

SELECT * FROM ListName WHERE ViewID='ID of the view'

You can get the ID of the view from the Views list. You must specify the List pseudo column to get a list of views for that list. For instance:

SELECT * FROM Views WHERE List ='ListName'

Stored Procedures

Stored Procedures are actions that are invoked via SQL queries. They perform tasks beyond standard CRUD operations, including managing users, documents, and attachments.

The connector sets defaults to facilitate the maximum number of integrations; however, the following connection properties allow a greater granularity of customization, which is useful in advanced integrations:

• ResolveCalculatedTypes: Controls whether SharePoint calculated columns are assigned a SQL data type corresponding to the result type of their formula. When enabled, this property automatically determines the data type of each calculated column by reading the result type of its formula (such as Number, Currency, DateTime, or Yes/No) and mapping that result type to the closest native SQL type. When disabled, all calculated columns are treated as strings.
• CreateIDColumns: Indicates whether or not to create supplemental Id columns for SharePoint columns that use values from information stored in other Lists.
• FolderOption: An option to determine how to display folders in results. Enter either FilesOnly, FilesAndFolders, Recursive, or RecursiveAll.
• PseudoColumns: Indicates whether to report pseudo columns as columns in the table metadata.

The connector maps types from the data source to the corresponding data type available in the schema. The table below documents these mappings.

Microsoft SharePoint	CData Schema
Choice (menu)	string
Currency	decimal
Date and Time	datetime
Hyperlink or Picture	string
Lookup	string
Multiple lines of text	string
Number	float
Person or Group	string
Single line of text	string
Task Outcome	string
Yes/No	bool

The connector models the data in Microsoft SharePoint as a list of tables in a relational database that can be queried using standard SQL statements.

CData Python Connector for Microsoft SharePoint Tables

Name	Description
Attachments	Manages attachments for SharePoint list items, allowing retrieval and deletion. Essential for users who frequently handle file attachments within SharePoint.
Groups	Allows the creation, deletion and retrieval of SharePoint security groups. Essential for managing user permissions and access control.
Roles	Allows the creation, deletion and retrieval of SharePoint roles. Useful for customizing access control.
Users	Manage SharePoint users, allowing updates, deletions, and retrieval of user details. Important for keeping SharePoint user management up-to-date.
Views	Retrieves, creates, updates, or deletes views in SharePoint lists, allowing customization of displayed data. Useful for tailoring list views to specific business needs.

Manages attachments for SharePoint list items, allowing retrieval and deletion. Essential for users who frequently handle file attachments within SharePoint.

Table Specific Information

Select

The List and ItemId columns are required to return Attachments.

Sample Query

The following query retrieves attachments from the list named MyAttachmentList for the item with ItemId = 4:

SELECT * FROM Attachments WHERE List = 'MyAttachmentList' AND ItemId = 4;

Insert

Call the AddAttachment stored procedure to add new attachments to a list item.

Columns

Name	Type	ReadOnly	Description
Url [KEY]	String	True	The URL path to the attachment file. Useful for accessing or downloading the attachment.
List	String	True	The internal name of the SharePoint list containing the attachment. Helps identify the source list for the attachment.
ListDisplayName	String	True	The display name of the SharePoint list containing the attachment. Useful for user-friendly identification of the list.
ItemID	String	True	The unique identifier of the item in the list to which the attachment is linked. Helps track associated files.
Name	String	True	The name of the attachment file. Useful for displaying file names and managing attachments.

Allows the creation, deletion and retrieval of SharePoint security groups. Essential for managing user permissions and access control.

Table Specific Information

SELECT

Retrieves all groups created in the SharePoint Account:

SELECT * FROM Groups

Retrieve all groups with the specified names in your SharePoint Account:

SELECT * FROM Groups WHERE [Name] = 'Group1'
SELECT * FROM Groups WHERE [Name] IN ('Group1', 'Group2')

Retrieve the groups in which a specific user belongs.

SELECT * FROM Groups WHERE [UserLoginName] = "LoginName"

Retrieve the groups which have a specific role assigned to them.

SELECT * FROM Groups WHERE [RoleName] = "RoleName"

INSERT

You can create groups by specifying writable (ReadOnly=false) columns in the INSERT statement as shown in the query example below. Note that some columns are always required, while other columns can be optionally specified.

INSERT INTO Groups(Name, Description, DefaultUserLoginName, OwnerName, OwnerType) VALUES('Testing Group 5', 'Testing Group 5.', 'RIDDLERSP2013\\administrator', 'Testing Group 4', 'group')

DELETE

You can delete a group by specifying the Name column in the criteria as shown in the query example below:

DELETE FROM Groups WHERE Name = 'Group1'

Columns

Name	Type	ReadOnly	Description
Name [KEY]	String	False	The name of the group. Helps identify the group within SharePoint.
Id	String	True	The unique identifier of the group.
Description	String	False	A brief description of the group. Useful for understanding its purpose and membership.
OwnerId	String	True	The unique identifier of the group owner.
OwnerType	String	False	Specifies whether the owner is a user or another group. Helps define group management hierarchy. The allowed values are user, group.
UserLoginName	String	True	A filter for reading the groups in which a specific user belongs. If this column is not specified in the criteria, it will have null values.
RoleName	String	True	A filter for reading the groups which have a specific role assigned to them. If this column is not specified in the criteria, it will have null values.

Pseudo-Columns

Pseudo column fields are used in the WHERE clause of SELECT statements and offer more granular control over the data returned from the data source.

Name	Type	Description
OwnerName	String	The name of the user or group who should be the owner of the group to create. This is a write-only column which should be used only in 'INSERT' statements.
DefaultUserLoginName	String	The user name of the default user for the group. This value should be in the format DOMAIN\\username. This is a write-only column which should be used only in 'INSERT' statements.

Allows the creation, deletion and retrieval of SharePoint roles. Useful for customizing access control.

Table Specific Information

SELECT

Retrieves all roles created in the SharePoint Account:

SELECT * FROM Roles

Retrieve all roles with the specified names in your SharePoint Account:

SELECT * FROM Roles WHERE [Name] = 'Role1'
SELECT * FROM Roles WHERE [Name] IN ('Role1', 'Role2')

Retrieve the roles assigned to a specific group.

SELECT * FROM Roles WHERE [GroupName] = "GroupName"

Retrieve the roles assigned to a specific user.

SELECT * FROM Roles WHERE [UserLoginName] = "LoginName"

INSERT

You can create roles by specifying writable (ReadOnly=false) columns in the INSERT statement as shown in the query example below. Note that some columns are always required, while other columns can be optionally specified.

INSERT INTO Roles (Name, Description, Permissions) VALUES ('Testing Role 3', 'Role for testing.', '1073741826')

DELETE

You can delete a role by specifying the Name column in the criteria as shown in the query example below:

DELETE FROM Roles WHERE Name = 'ReadOnly'

Columns

Name	Type	ReadOnly	Description
Name [KEY]	String	False	The name of the role. Helps identify the permission level assigned to users or groups.
Id	String	True	The unique identifier of the role.
Description	String	False	A brief description of the role. Useful for understanding its purpose and the permissions it grants.
Permissions	String	False	The mask of permissions granted to the role. Helps define access control levels. To learn more about permission masks, check out the 'Permission Masks' section in Permissions.
RoleType	String	True	Specifies the type of role. Useful for differentiating between built-in roles and custom roles.
IsHidden	Boolean	True	Indicates whether the role is hidden from the user interface. Helps manage roles that are system-defined or restricted.
UserLoginName	String	True	A filter for reading the roles assigned to a specific user. If this column is not specified in the criteria, it will have null values.
GroupName	String	True	A filter for reading the roles assigned to a specific group. If this column is not specified in the criteria, it will have null values.

Manage SharePoint users, allowing updates, deletions, and retrieval of user details. Important for keeping SharePoint user management up-to-date.

Table Specific Information

SELECT

Retrieves all users created for the SharePoint Account:

SELECT * FROM Users

Retrieve all users with the specified login names in your SharePoint Account:

SELECT * FROM Users WHERE [LoginName] = 'DOMAIN\\User1'
SELECT * FROM Users WHERE [LoginName] IN ('DOMAIN\\User1', 'DOMAIN\\User2')

Retrieve users that belong to a specific group:

SELECT * FROM Users WHERE [GroupName] = "GroupName"

Retrieve users that have a specific role assigned to them:

SELECT * FROM Users WHERE [RoleName] = "RoleName"

UPDATE

You can update user data by specifying the LoginName column in the criteria as shown in the query example below:

UPDATE Users SET Notes = 'User 1 notes.' WHERE LoginName = 'DOMAIN\\User1'

DELETE

You can delete a user by specifying the LoginName column in the criteria as shown in the query example below:

DELETE FROM Users WHERE LoginName = 'DOMAIN\\User1'

Columns

Name	Type	ReadOnly	Description
LoginName [KEY]	String	True	The login name of the user, typically in DOMAIN\\username format. Helps authenticate and identify users within SharePoint.
Id	String	True	A unique identifier assigned to the user. Useful for referencing users in queries and permission management.
Name	String	False	The display name of the user. Useful for showing user-friendly names in SharePoint interfaces.
Email	String	False	The primary email address associated with the user. Used for communication and notifications.
IsInDomainGroup	Boolean	True	Indicates whether the user is a member of a domain group. Helps manage group-based access control.
IsSiteAdmin	Boolean	True	Indicates whether the user has administrative privileges for the SharePoint site. Helps identify high-level access users.
Notes	String	False	Optional notes or additional information related to the user. Useful for internal documentation and tracking.
SecurityId	String	True	The security identifier (SID) assigned to the user. Helps in managing and tracking user permissions.
GroupName	String	False	A filter for reading the users in a specific group. If this column is not specified in the criteria, it will have null values.
RoleName	String	False	A filter for reading the users which are assigned a specific role. If this column is not specified in the criteria, it will have null values.

Retrieves, creates, updates, or deletes views in SharePoint lists, allowing customization of displayed data. Useful for tailoring list views to specific business needs.

Table Specific Information

Views is a special table. It may be used to get, update, insert, and delete views from a specified List.

Select

To return results from Views, you must specify either the ID or List column in the SELECT statement.

Sample Queries

Using the List column:

SELECT * FROM Views WHERE List = 'MyListName';

Using the ID column:

SELECT * FROM Views WHERE ID = 'list1|{24676099-47E8-4C07-BABE-47EB9BEBA2F9}';

Insert

The List, Name, and Fields columns are required to insert to this table.

Columns

Name	Type	ReadOnly	Description
ID [KEY]	String	True	A unique identifier for the view. Used to reference and manage specific views in SharePoint.
List	String	True	The name of the list associated with the view. A list must be specified in SELECT statements if the view ID is not provided.
ViewID	String	True	The unique identifier of the view within a specific list. Useful for managing multiple views within a list.
Name	String	False	The display name of the view. Helps users easily identify and select views.
Type	String	False	The type of view, such as Standard, Calendar, or Datasheet. This value is required for inserts and updates. The allowed values are CALENDAR, GRID, HTML. The default value is HTML.
Fields	String	False	A comma-separated list of fields included in the view. Space-sensitive; ensure proper formatting for queries.
IsDefault	Boolean	False	Indicates whether the view is the default view for the list. Helps determine the primary view for users.
Query	String	False	The query used to filter or sort data in the view. Helps customize list display based on specific conditions.

Views are similar to tables in the way that data is represented; however, views are read-only.

Queries can be executed against a view as if it were a normal table.

CData Python Connector for Microsoft SharePoint Views

Name	Description
FileVersions	Lists all available versions of a document stored in SharePoint, including version history details. Useful for tracking changes and restoring previous document versions.
GetValidTerms	Retrieves a list of valid managed metadata terms associated with a specific column in a SharePoint list. Helps enforce consistent categorization and tagging of SharePoint content.
Lists	Retrieves metadata about all SharePoint lists available on the site, including properties and settings. Useful for understanding the structure and usage of SharePoint lists.
Permissions	Retrieves permission information for a SharePoint site, list, or item.
Subsites	Lists all subsites within a SharePoint site, including hierarchy details. Helps with site navigation and organization.

Lists all available versions of a document stored in SharePoint, including version history details. Useful for tracking changes and restoring previous document versions.

View Specific Information

To return results from this view, you must specify both the Library and File columns.

Sample Query

The following query retrieves all versions of the file MyExcelFile.xlsx located in the Documents/testd library:

SELECT * FROM FileVersions WHERE Library = 'Documents/testd' AND File = 'MyExcelFile.xlsx';

Columns

Name	Type	Description
ID [KEY]	String	A unique identifier for the file version. Useful for tracking and managing version history.
Comments	String	User-provided comments about the particular file version. Helps document changes or provide context for modifications.
CreateBy	String	The username of the SharePoint user who modified this version of the file. Useful for tracking authorship and accountability.
Date	Datetime	The date and time when this version of the file was created. Helps monitor file updates and changes over time.
Size	String	The size of this specific version of the file. Useful for storage management and version comparison.
Url	String	The URL path to access this specific version of the file. Helps users retrieve or download older versions.
Library	String	The name of the SharePoint document library where the file is stored. A library must be specified to retrieve file versions. The default value is Shared Documents.
File	String	The name of the file for which versions are being listed. A file must be specified to retrieve its version history.

Retrieves a list of valid managed metadata terms associated with a specific column in a SharePoint list. Helps enforce consistent categorization and tagging of SharePoint content.

Table Specific Information

GetValidTerms is a special view used to retrieve valid terms for a taxonomy or managed metadata column in a given Microsoft SharePoint list. To query this view, you must provide both the List and ColumnName columns.

Sample Query

The following query retrieves valid terms for the TermsC column in the list1 list:

SELECT * FROM GetValidTerms WHERE List = 'list1' AND ColumnName = 'TermsC';

Columns

Name	Type	Description
ID [KEY]	String	A unique identifier for the term. Useful for referencing and managing taxonomy terms in SharePoint.
TermLabelValue	String	The label assigned to the term. Helps users identify and apply terms to content.
Description	String	A brief description of the term set. Useful for understanding the purpose and usage of the term set.
NameInRequestedLang	String	The name of the term set in the language requested by the client. Helps with multilingual support.
IsOpen	Boolean	Indicates whether the term set is open for adding new terms. Useful for managing controlled vocabularies.
Deprecated	Boolean	Indicates whether the term is deprecated. Helps prevent usage of outdated or obsolete terms.
InternalId	String	An internal identifier for the term. Useful for system-level term management.
TermSetContact	String	The contact person or group responsible for managing the term set. Useful for governance and support.
ContainerDesc	String	A container node that holds metadata descriptions. Helps structure taxonomy information.
SingleTermLabelDesc	String	A detailed description of a single term label. Useful for providing additional context.
IsDefaultLabel	Boolean	Indicates whether the term label is the default for the term. Helps standardize term usage.
BelongsTo	String	The term set to which this term belongs. Useful for managing hierarchical taxonomies.
IsTaggingAvailable	Boolean	Indicates whether the term set is available for tagging. Helps control content classification.
TermPath	String	The hierarchical path of the term with term labels. Useful for navigating term relationships.
TermpathoftermwithIds	String	The hierarchical path of the term with its unique identifiers. Helps track term lineage.
ChildTerms	String	A custom sort order for child terms within the term hierarchy. Useful for organizing terms.
HasChildTerms	Boolean	Indicates whether the term has child terms. Helps manage nested taxonomy structures.
PertainingToTerm	String	The identifier of the term that this term set information pertains to. Useful for hierarchical organization.

Pseudo-Columns

Pseudo column fields are used in the WHERE clause of SELECT statements and offer more granular control over the data returned from the data source.

Name	Type	Description
List	String	The name of the SharePoint list for which valid terms are being retrieved. Helps with taxonomy integration.
ColumnName	String	The column in the list for which valid terms are being retrieved. Useful for metadata enforcement.
LocaleId	String	The locale ID for the term. Defaults to 1033 (English). Helps support multilingual taxonomy.

Retrieves metadata about all SharePoint lists available on the site, including properties and settings. Useful for understanding the structure and usage of SharePoint lists.

View Specific Information

Lists can be used to list the tables in SharePoint. This will only return actual lists in SharePoint and not any special tables associated with the connector.

The following columns can be used in the WHERE clause: Title and BaseTemplate.

Columns

Name	Type	Description
ID [KEY]	String	A unique identifier for the SharePoint list. Useful for referencing lists in queries and workflows.
Title	String	The display name of the list. Can be used in the WHERE clause with wildcard (*) for filtering.
Description	String	A brief summary of the list’s purpose and contents. Helps users understand the list's function.
BaseTemplate	String	Indicates the template type used to create the list. Can be used in WHERE clause for filtering list types.
Version	Double	The current version number of the list. Useful for tracking updates and changes.
Url	String	The default URL of the list. Helps users navigate to the list directly.
EmailAlias	String	The email alias assigned to the list. Useful for enabling email-based list interactions.
ImageUrl	String	The URL of the image associated with the list. Helps visually identify lists.
ItemCount	Integer	The total number of items currently stored in the list. Useful for reporting and analytics.
Item_Deleted	Datetime	The date and time when the last item was deleted from the list. Helps track data modifications.
Item_Modified	Datetime	The date and time when the last item was modified in the list. Useful for monitoring recent activity.
SendToUrl	String	The URL where list items are sent when using Send To functionality. Useful for document routing.
Created	Datetime	The date and time when the list was originally created. Useful for tracking list history.
AllowDeletion	String	Indicates whether items in the list can be deleted. Helps manage data retention policies.
AllowMultiResponses	Boolean	Indicates whether multiple responses are allowed for surveys. Useful for feedback collection.
Direction	String	Specifies text reading order: 'LTR' for left-to-right, 'RTL' for right-to-left, or 'None' for no directionality.
EnableAssignedToEmail	Boolean	Indicates whether automatic email notifications are sent to assigned users. Applicable to issue tracking lists.
EnableAttachments	Boolean	Indicates whether attachments are allowed on list items. Not applicable to document libraries.
EnableModeration	Boolean	Indicates whether content approval is enabled for the list. Helps enforce content review policies.
EnableVersioning	Boolean	Indicates whether versioning is enabled for the list. Useful for tracking changes to list items.
Hidden	Boolean	Indicates whether the list is hidden from the Documents and Lists page, Quick Launch bar, and other navigation menus.
MultipleDataList	Boolean	Indicates whether a meeting workspace site contains data for multiple meeting instances within the site.
Ordered	Boolean	Indicates whether list items can be manually ordered on the Edit View page. Useful for prioritized lists.
Showuser	Boolean	Indicates whether user names are displayed in survey results. Helps identify respondents.