Connecting to Elasticsearch

For information on the module to import, and how to configure the necessary connection properties in a connection string, see Establishing a Connection.

Other available connection properties can be used to configure other aspects of the connector capabilities.

Python Version Support

Elasticsearch Version Support

See Also

• Using the Connector: Establish connections and query Elasticsearch through Python code.
• From SQLAlchemy: Use SQLAlchemy to establish a connection with dialect URL, and interact with Elasticsearch data using mapped classes and Sessions.

Dependencies

Installation

Windows:

pip install cdata_elasticsearch_connector-26.0.9655-cp310-abi3-win_amd64.whl

Linux:

pip install cdata_elasticsearch_connector-26.0.9655-cp310-abi3-linux_x86_64.whl

macOS:

pip install cdata_elasticsearch_connector-26.0.9655-cp310-abi3-macosx_12_0_arm64.whl

The macOS wheel supports arm64 (Apple Silicon) architectures only on macOS 12 and newer.

Regardless of the environment, certain distributions might require that the "pip3 install" command be used instead, to differentiate from a Python 2 distribution that might exist already. After installation, confirm whether the connector is successfully installed by running the "pip list" command. If "cdata_elasticsearch_connector" is present in the list output by the command, then the installation was successful.

Upgrading

When upgrading, "pip install" does not automatically clean up old JRE files. To avoid leftover files that could cause JVM errors, uninstall the previous version before installing the new one.

Licensing

After the installation is complete, a separate step is needed to activate a license for the connector. Among the CData assets in the distribution's site packages, there is an install-license tool that activates this license. From within the distribution's site-packages folder, after navigating to the "cdata/installlic_elasticsearch" folder, simply use a command like the below to activate the license. Omitting the <key> argument activates a trial license:

• Windows:
  

  ./install-license.exe <key>

• Linux / Mac:
  

  ./install-license.sh <key>

Sometimes, file access issues may cause pip to install the connector in a fallback file path that is not the python distribution's main or primary site-packages location. This can make it difficult to find where the connector was installed, and from there, the license activator. In that event, this python script below will print out the full file path of the connector's native file. This file will be stored in the mentioned cdata folder, from which the installlic_elasticsearch folder is trivial to find:

import os
import cdata.elasticsearch
path = os.path.abspath(cdata.elasticsearch.__file__)
print(path)

Uninstallation

If the connector needs to be uninstalled for any reason, do so by running the pip uninstall command, as in the example below:

pip uninstall cdata-elasticsearch-connector

The objects available within our connector are accessible from the "cdata.elasticsearch" module. To use the module's objects directly:

1. Import the module as follows:
   

   import cdata.elasticsearch as mod

2. To establish a connection string, call the connect() method from the connector object using an appropriate connection string, such as:
   

   mod.connect("Server=127.0.0.1;Port=9200;")

Connecting to Elasticsearch Service

Set the following to connect to data:

• Server should be set to the IP Address or domain of the Elasticsearch instance. The Server could also be set to a comma-delimited list of node addresses or hostnames from a single cluster.
  

  Server=01.02.03.04 
    OR 
  Server=01.01.01.01:1234,02.02.02.02:5678

• Port should be set to the configured port for the Elasticsearch instance. If you include a port in a node specification for the Server property, that included port will take precedence over the specification for Port for that node only.

The connector uses X-Pack Security for authentication and TLS/SSL encryption. You can prefix the server value with "https://" to connect using TLS/SSL.

Connecting to Amazon OpenSearch Service

Set the following to connect to data:

• Server should be set to the Endpoint URL for the Amazon ES instance.
• Port should be set to 443.
• AWSRegion should be set to the Amazon AWS region where the Elasticsearch instance is being hosted (the connector will attempt to automatically identify the region based on the Server value).

The connector uses X-Pack Security for authentication and TLS/SSL encryption.

Note: Requests are signed using AWS Signature Version 4.

Authenticating to Elasticsearch

Obtain AWS Keys

To obtain the credentials for an IAM user:

1. Sign into the IAM console.
2. In the navigation pane, select Users.
3. To create or manage the access keys for a user, select the user and then navigate to the Security Credentials tab.

To obtain the credentials for your AWS root account:

1. Sign into the AWS Management console with the credentials for your root account.
2. Select your account name or number.
3. In the menu that displays, select My Security Credentials.
4. To manage or create root account access keys, click Continue to Security Credentials and expand the "Access Keys" section.

Standard Authentication

Set the AuthScheme to Basic, and set User and Password properties and/or use PKI (public key infrastructure) to authenticate. Once the connector is connected, X-Pack performs user authentication and grants role permissions based on the realms you have configured.

To use PKI, set the SSLClientCert, SSLClientCertType, SSLClientCertSubject, and SSLClientCertPassword properties.

Note: TLS/SSL and client authentication must be enabled on X-Pack to use PKI.

Securing Elasticsearch Connections

To enable TLS/SSL in the connector, set UseSSL to true;.

Root Credentials

To authenticate using account root credentials, set these parameters:

• AuthScheme: AwsRootKeys.
• AWSAccessKey: The access key associated with the AWS root account.
• AWSSecretKey: The secret key associated with the AWS root account.

Note: Amazon discourages using root credentials for anything beyond simple testing. The account root credentials have the full permissions of the user, posing a security risk and making this the least secure authentication method.

If multi-factor authentication is required, specify the following:

• CredentialsLocation: The location of the settings file where MFA credentials are saved.
• MFASerialNumber: The serial number of the MFA device if one is being used.
• MFAToken: The temporary token available from your MFA device.

Note: If you want to control the duration of the temporary credentials, set the TemporaryTokenDuration property (default: 3600 seconds).

Temporary Credentials

To authenticate using temporary credentials, specify the following:

• AuthScheme: TemporaryCredentials.
• AWSAccessKey: The access key of the IAM user who will assume the role.
• AWSSecretKey: The secret key of the IAM user who will assume the role.
• AWSSessionToken: Your AWS session token, provided with your temporary credentials. For details, see AWS Identity and Access Management User Guide.

The connector can now request resources using the same permissions provided by long-term credentials (such as IAM user credentials) for the lifespan of the temporary credentials.

To authenticate using both temporary credentials and an IAM role, set all the parameters described above, and specify these additional parameters:

• AWSRoleARN: The Role ARN for the role you'd like to authenticate with. This prompts the connector to retrieve credentials for the specified role.
• AWSExternalId (optional): Only required if you are assuming a role in another AWS account.

If multi-factor authentication is required, specify the following:

• CredentialsLocation: The location of the settings file where MFA credentials are saved.
• MFASerialNumber: The serial number of the MFA device if one is being used.
• MFAToken: The temporary token available from your MFA device.

Note: If you want to control the duration of the temporary credentials, set the TemporaryTokenDuration property (default: 3600 seconds).

Using AWS From an EC2 Instance

Set AuthScheme to AwsEC2Roles.

If you are using the connector from an EC2 Instance and have an IAM Role assigned to the instance, you can use the IAM Role to authenticate. Since the connector automatically obtains your IAM Role credentials and authenticates with them, it is not necessary to specify AWSAccessKey and AWSSecretKey.

If you are also using an IAM role to authenticate, you must additionally specify the following:

• AWSRoleARN: Specify the Role ARN for the role you'd like to authenticate with. This causes the connector to attempt to retrieve credentials for the specified role.
• AWSExternalId (optional): Only required if you are assuming a role in another AWS account.

IMDSv2 Support

The Elasticsearch connector now supports IMDSv2. Unlike IMDSv1, the new version requires an authentication token. Endpoints and response are the same in both versions.

In IMDSv2, the Elasticsearch connector first attempts to retrieve the IMDSv2 metadata token and then uses it to call AWS metadata endpoints. If it is unable to retrieve the token, the connector reverts to IMDSv1.

Note that this method of authentication is only possible with Opensearch Service, and not with Elasticsearch.

AWS IAM Roles

To authenticate as an AWS role, set these properties:

• AWSAccessKey: The access key of the IAM user to assume the role for.
• AWSSecretKey: The secret key of the IAM user to assume the role for.
• AWSRoleARN: Specify the Role ARN for the role you'd like to authenticate with. This will cause the connector to attempt to retrieve credentials for the specified role.
• AWSExternalId (optional): Only required if you are assuming a role in another AWS account.

If multi-factor authentication is required, specify the following:

• CredentialsLocation: The location of the settings file where MFA credentials are saved.
• MFASerialNumber: The serial number of the MFA device if one is being used.
• MFAToken: The temporary token available from your MFA device.

Note: If you want to control the duration of the temporary credentials, set the TemporaryTokenDuration property (default: 3600 seconds).

Note: In some circumstances it might be preferable to use an IAM role for authentication, rather than the direct security credentials of an AWS root user. If you are specifying the AWSAccessKey and AWSSecretKey of an AWS root user, you cannot use roles.

Kerberos

Please see Using Kerberos for details on how to authenticate with Kerberos.

API Key

To authenticate using APIKey set the following:

• AuthScheme: Set this to APIKey.
• APIKey: Set this to APIKey returned from Elasticsearch.
• APIKeyId: Set this to the Id returned alongside APIKey.

Java Native Interface (JNI) is a standard programming interface for writing Java native methods and embedding the Java virtual machine into native applications.

The connector leverages the JNI for improved performance on Mac and Linux.

Configure the Config INI File

The Linux and Mac editions of the Elasticsearch python connector are configured with an ini file. This file is used to set several parameters, including JNI behavior. This file is to be located in:

{path_to_distribution_site-packages}/cdata/config.ini

Ensure that any configuration properties you set in the ini file fall under the following section name (adjust the 311 number if you are using an different python version from 3.11):

• For Linux:
  

  [elasticsearch.cpython-311-x86_64-linux-gnu.so]

• For Mac:
  

  [elasticsearch.cpython-311-darwin.so]

Configure the JNI connector's behavior by editing the properties in the connector's config.ini file. The connector can be configured as follows:

• LOGFILE: Set this the same way as the CDATA_LOGFILE envrionment variable below.
• JAVA_HOME: Configure the path to the JVM library location used to launch the JVM.
• CLASS_PATH: Use a colon-separated list to configure the paths to the third-party jar libraries.

Configure Environment Variables

Additionally, set the following environment variables:

• CDATA_JAVA_HOME: Configure the path to the JVM library location used to launch the JVM.
• CDATA_JVM_OPTIONS: Place JVM options here.
• CDATA_LOGFILE: Set this in the following scheme: <SCHEME>://<TAG>[|<LEVEL>]
  • SCHEME: The options are STDOUT, FILE.
    • STDOUT: Both the native wrapper and odbc core log into stdout. The Logfile and Verbosity properties can override the behavior of ODBC core.
    • FILE: The native wrapper logs into <FILENAME> while the odbc core logs into <FILENAME>.driver.log. The Logfile and Verbosity properties can override the behavior of ODBC core.
  • TAG
    • For STDOUT, set this to 1. For FILE, set this to the filename.
  • LEVEL
    • Set to one of: FATAL | ERROR | WARNING | INFO | DEBUG

The following are some examples of this syntax:

• STDOUT://1|DEBUG
• FILE:///tmp/my_py.log|DEBUG

Custom Logger

The Python connector supports a custom logging mechanism for redirecting log output to any destination, such as a cloud storage service or logging framework. Use setCustomLoggerFactory() to register a factory function that creates a logger instance for each connection.

The factory function receives the context string from the Logfile connection property (the portion after CUSTOM://) and must return an object with a writeLog(verbosity, message) method.

To enable custom logging:

1. Call setCustomLoggerFactory() with your factory function before opening connections.
2. Set Logfile to CUSTOM:// followed by a context string to identify the connection.
3. Set Verbosity to the desired log level.

The following example demonstrates a custom logger factory that creates a separate logger instance per connection:

import cdata.elasticsearch as mod
import time

class MyLogger:
    def __init__(self, loggerId):
        self.loggerId = loggerId
    def writeLog(self, verbosity, message):
        print("[MyLogger " + self.loggerId + "] " + message)

def createLogger(context):
    return MyLogger(context[len("MyLoggerId="):])

mod.setCustomLoggerFactory(createLogger)

conn1 = mod.connect("...;Logfile=CUSTOM://MyLoggerId=1;Verbosity=2;")
# do something with conn1
time.sleep(1)  # Wait for logs to flush from conn1

conn2 = mod.connect("...;Logfile=CUSTOM://MyLoggerId=2;Verbosity=2;")
# do something with conn2
time.sleep(1)  # Wait for logs to flush from conn2

Kerberos

To authenticate to Elasticsearch with Kerberos, set AuthScheme to NEGOTIATE.

Authenticating to Elasticsearch via Kerberos requires you to define authentication properties and to choose how Kerberos should retrieve authentication tickets.

Retrieve Kerberos Tickets

The connector provides three ways to retrieve the required Kerberos ticket, depending on whether or not the KRB5CCNAME and/or KerberosKeytabFile variables exist in your environment.

MIT Kerberos Credential Cache File

This option enables you to use the MIT Kerberos Ticket Manager or kinit command to get tickets. With this option there is no need to set the User or Password connection properties.

This option requires that KRB5CCNAME has been created in your system.

To enable ticket retrieval via MIT Kerberos Credential Cache Files:

1. Ensure that the KRB5CCNAME variable is present in your environment.
2. Set KRB5CCNAME to a path that points to your credential cache file. (For example, C:\krb_cache\krb5cc_0 or /tmp/krb5cc_0.) The credential cache file is created when you use the MIT Kerberos Ticket Manager to generate your ticket.
3. To obtain a ticket:
   1. Open the MIT Kerberos Ticket Manager application.
   2. Click Get Ticket.
   3. Enter your principal name and password.
   4. Click OK.

   If the ticket is successfully obtained, the ticket information appears in Kerberos Ticket Manager and is stored in the credential cache file.

The connector uses the cache file to obtain the Kerberos ticket to connect to Elasticsearch.

Note: If you would prefer not to edit KRB5CCNAME, you can use the KerberosTicketCache property to set the file path manually. After this is set, the connector uses the specified cache file to obtain the Kerberos ticket to connect to Elasticsearch.

Keytab File

If your environment lacks the KRB5CCNAME environment variable, you can retrieve a Kerberos ticket using a Keytab File.

To use this method, set the User property to the desired username, and set the KerberosKeytabFile property to a file path pointing to the keytab file associated with the user.

User and Password

If your environment lacks the KRB5CCNAME environment variable and the KerberosKeytabFile property has not been set, you can retrieve a ticket using a user and password combination.

To use this method, set the User and Password properties to the user/password combination that you use to authenticate with Elasticsearch.

Enabling Cross-Realm Authentication

To enable this kind of cross-realm authentication, set the KerberosRealm and KerberosKDC properties to the values required for user authentication. Also, set the KerberosServiceRealm and KerberosServiceKDC properties to the values required to obtain the service ticket.

Fine Tuning Data Access

You can use the following properties to gain greater control over Elasticsearch API features and the strategies the connector uses to surface them:

• GenerateSchemaFiles: This property enables you to persist table metadata in static schema files that are easy to customize, to persist your changes to column data types, for example. You can set this property to "OnStart" to generate schema files for all tables in your database at connection. The resulting schemas are based on the connection properties you use to configure Automatic Schema Discovery.
  Or, you can set this property to "OnUse" to generate schemas based on a query.
  To use the resulting schema files, set the Location property to the folder containing the schemas.
• QueryPassthrough: This property enables you to use Elasticsearch's Search DSL language instead of SQL.
• RowScanDepth: This property determines the number of rows that will be scanned to detect column data types when generating table metadata. This property applies if you are working with the dynamic schemas generated from Automatic Schema Discovery or if you are using QueryPassthrough.

Custom URLs

If a custom URL is required, using the form [Server]:[Port]/[URLPathPrefix], the 'URLPathPrefix' value can be specified via the Other property. For example: URLPathPrefix=myprefix

The connector will use the specified path prefix to build the URL required for connecting to the Elasticsearch API endpoints.

Querying Multiple Indices

Multiple indices can be queried by executing a query using one of the following formats:

• Query all indices via the _all view: SELECT * FROM [_all]

• Query a list of indices: SELECT * FROM [index1,index2,index3]

• Query indices matching a wildcard pattern: SELECT * FROM [index*]

Note, index lists can contain wildcards and indices can be excluded by prefixing an index with '-'. For example: SELECT * FROM [index*,-index3]

Fine Tuning Performance

• PageSize: This property enables you to optimize performance based on your resource provisioning.
  Paging has an impact on sorting performance in a distributed system, as each shard must first sort results before submitting them to the coordinating server.
  By default, the connector requests a page size of 10,000. This is the default index.max_result_window setting in Elasticsearch.
• MaxResults: This property sets a limit on the results for queries at connection time, without requiring that you specify a LIMIT clause.
  By default, this is the same value as the index.max_result_window setting in Elasticsearch.

  If you are using the Scroll API, set ScrollDuration instead.

• ScrollDuration: This property specifies how long the server should keep the search context alive. Setting this property to a nonzero value and time unit enables the Scroll API.

General Changes

Elasticsearch is a document-oriented database that provides high performance searching, flexibility, and scalability. These features are not necessarily incompatible with a standards-compliant query language like SQL-92. In this section we will show various schemes that the connector offers to bridge the gap with relational SQL and an Elasticsearch database.

The connector models Elasticsearch objects into relational tables and translates SQL queries into Elasticsearch queries to get the requested data. See Schema Mapping for more details on how Elasticsearch objects are mapped to tables to generate schemas. See Query Mapping for more details on how various Elasticsearch operations are represented as SQL.

The Automatic Schema Discovery scheme automatically finds the data types by retrieving the mapping for the Elasticsearch type. You can use RowScanDepth, FlattenArrays, and FlattenObjects to control the relational representation of the collections in Elasticsearch.

Optionally, you can use Custom Schema Definitions to project your chosen relational structure on top of a Elasticsearch object. This allows you choose your own column names, their data types, and the location of their values in the collection.

When GenerateSchemaFiles is set, you can persist schemas for all collections in the database or for the results of SELECT queries.

The CData Python Connector for Elasticsearch models the Elasticsearch REST APIs as relational tables and stored procedures that can be accessed with standard SQL. This enables access from standards-based tools.

The table definitions are dynamically retrieved. When you connect, the connector connects to Elasticsearch and retrieves the schemas, list of tables, and the metadata for the tables by querying the Elasticsearch REST server. Any changes to the remote data are immediately reflected in your queries.

The following table maps Elasticsearch concepts to relational ones:

Elasticsearch Versions 6 and Above:

Note: Starting in Elasticsearch 6, indices are limited to a single type. Therefore the type is no longer treated as a table, since an index and type have a one-to-one relation. Types are hidden and used internally where necessary to issue the proper request to Elasticsearch.

Elasticsearch Versions Prior to Version 6:

Elasticsearch contains the ability to establish parent-child relationships. This relationship maps closely to SQL JOIN functionality. The connector models these parent-child relationships in a way to enable the ability to perform JOIN queries.

Elasticsearch Versions 6 and Above:

In version 6 and above of Elasticsearch, relationships are established by using the join datatype. Included in this functionality is the ability to define multiple children for a single parent and to create multiple levels of relations.

The connector supports all of these relationships and will generate a separate table for each relation in Elasticsearch. The table name will be in the form: [index]_[relation].

All child tables will have an additional column containing the parent table id. The column name will be in the form: _[parent_table]_id. This column is a foreign key to the _id column of the parent table and can be used to perform SQL JOIN queries.

When querying these tables individually, filtering logic is pushed to the server to improve performance by only returning the data relevant to the table selected.

Elasticsearch Versions Prior to Version 6:

In versions prior to 6, a relationship is established between two types via a _parent field. This creates a single parent-child relationship.

The tables identified in this parent-child relationship do not change (they are still based on the Elasticsearch type). However the child table will have an additional column containing the parent id. The column name will be in the form: _[parent_table]_id. This column is a foreign key to the _id column of the parent table and can be used to perform SQL JOIN queries.

Below is the raw data used throughout this chapter. Following is the mapping for the "insured" table (index):

{
  "insured": {
    "mappings": {
      "properties": {
        "name": { "type":"string" },
        "address": {
          "street": { "type":"string" },
          "city": { "type":"string" },
          "state": { "type":"string" }
        },
        "insured_ages": { "type": "integer" },
        "vehicles": {
          "type": "nested",
          "properties": {
            "year": { "type":"integer" },
            "make": { "type":"string" },
            "model": { "type":"string" },
            "body_style" { "type": "string" }
          }
        }
      }
    }
  }
}

The following is the sample data set for the "insured" table (index):

{
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "insured",
        "_type": "_doc",
        "_id": "1",
        "_score": 1,
        "_source": {
          "name": "John Smith",
          "address": {
            "street": "Main Street",
            "city": "Chapel Hill",
            "state": "NC"
          },
          "insured_ages": [ 17, 43, 45 ],
          "vehicles": [
            {
              "year": 2015,
              "make": "Dodge",
              "model": "RAM 1500",
              "body_style": "TK"
            },
            {
              "year": 2015,
              "make": "Suzuki",
              "model": "V-Strom 650 XT",
              "body_style": "MC"
            },
            {
              "year": 1992,
              "make": "Harley Davidson",
              "model": "FXR",
              "body_style": "MC"
            }
          ]
        }
      },
      {
        "_index": "insured",
        "_type": "_doc",
        "_id": "2",
        "_score": 1,
        "_source": {
          "name": "Joseph Newman",
          "address": {
            "street": "Oak Street",
            "city": "Raleigh",
            "state": "NC"
          },
          "insured_ages": [ 23, 25 ],
          "vehicles": [
            {
              "year": 2010,
              "make": "Honda",
              "model": "Accord",
              "body_style": "SD"
            },
            {
              "year": 2008,
              "make": "Honda",
              "model": "Civic",
              "body_style": "CP"
            }
          ]
        }
      }
    ]
  }
}

The connector automatically infers a relational schema by retrieving the mapping of the Elasticsearch type. The columns and data types are generated from the retrieved mapping.

Detecting Arrays

Any field within Elasticsearch can be an array of values, but this is not explicitly defined within the mapping. To account for this, the connector will query the data to detect if any fields contain arrays. The number of Elasticsearch documents retrieved during this array scanning is based on the RowScanDepth property.

Elasticsearch nested types are special types that denote an array of objects and thus will always be treated as such when generating the metadata.

Detecting Columns

The columns identified during the discovery process depend on the FlattenArrays and FlattenObjects properties.

Example Data Set

To provide an example of how these options work, consider the following mapping (where 'insured' is the name of the table):

{
  "insured": {
    "properties": {
      "name": { "type":"string" },
      "address": {
        "street": { "type":"string" },
        "city": { "type":"string" },
        "state": { "type":"string" }
      },
      "insured_ages": { "type": "integer" },
      "vehicles": {
        "type": "nested",
        "properties": {
          "year": { "type":"integer" },
          "make": { "type":"string" },
          "model": { "type":"string" },
          "body_style" { "type": "string" }
        }
      }
    }
  }
}

Also consider the following example data for the above mapping:

{
  "_source": {
    "name": "John Smith",
    "address": {
      "street": "Main Street",
      "city": "Chapel Hill",
      "state": "NC"
    },
    "insured_ages": [ 17, 43, 45 ], 
    "vehicles": [
      {
        "year": 2015,
        "make": "Dodge",
        "model": "RAM 1500",
        "body_style": "TK"
      },
      {
        "year": 2015,
        "make": "Suzuki",
        "model": "V-Strom 650 XT",
        "body_style": "MC"
      },
      {
        "year": 2012,
        "make": "Honda",
        "model": "Accord",
        "body_style": "4D"
      }
    ]
  }
}

Using FlattenObjects

If FlattenObjects is set, all nested objects will be flattened into a series of columns. The above example will be represented by the following columns:

If FlattenObjects is not set, then the address.street, address.city, and address.state columns will not be broken apart. The address column of type string will instead represent the entire object. Its value would be the following:

{street: "Main Street", city: "Chapel Hill", state: "NC"}

Using FlattenArrays

The FlattenArrays property can be used to flatten array values into columns of their own. This is only recommended for arrays that are expected to be short. It is best to leave unbounded arrays as they are and piece out the data for them as needed using JSON Functions.

Note: Only the top-most array will be flattened. Any subarrays will be represented as the entire array.

The FlattenArrays property can be set to 3 to represent the arrays in the example above as follows (this example is with FlattenObjects not set):

Using Both FlattenObjects and FlattenArrays

If FlattenObjects is set along with FlattenArrays (set to 1 for brevity), the vehicles field will be represented as follows:

The connector offers three basic configurations to model documents as tables, described in the following sections. The connector will parse the Elasticsearch document and identify the nested documents.

• Flattened Documents Model: Implicitly join nested documents into a single table.
• Relational Model: Model nested documents as individual tables containing a primary key and a foreign key that links to the parent document.
• Top-Level Document Model: Model a top-level view of an Elasticsearch document. Nested documents are returned as JSON strings.

For users who need access to the entirety of their nested Elasticsearch data, flattening the data into a single table is the best option. The connector will use streaming and only parses the Elasticsearch data once per query in this mode.

Joining Object Arrays into a Single Table

With DataModel set to "FlattenedDocuments", nested documents will behave as separate tables and act in the same manner as a SQL JOIN. Any nested documents, at the same height (e.g. sibling documents), will be treated as a SQL CROSS JOIN.

Example

Below is a sample query and the results, based on the sample document in Raw Data. This implicitly JOINs the insured document with the nested vehicles document.

Query

The following query drills into the nested documents in each insured document.

SELECT
  [_id],
  [name],
  [address.street] AS address_street,
  [address.city.first] AS address_city,
  [address.state.last] AS address_state,
  [insured_ages],
  [year],
  [make],
  [model],
  [body_style],
  [_insured_id],
  [_vehicles_c_id]
FROM
  [insured]

Results

See Also

• Automatic Schema Discovery: Configure the columns reported in the table schemas.
• FreeForm;: Use dot notation to select nested data.
• VerticalFlattening;: Access nested object arrays as separate tables.
• JSON Functions: Manipulate the data returned to perform client-side aggregation and transformations.

Using a top-level document view of the Elasticsearch data provides ready access to top-level elements. The connector returns nested elements in aggregate, as single columns.

One aspect to consider is performance. You forego the time and resources to process and parse nested elements -- the connector parses the returned data once, using streaming to read the JSON data. Another consideration is your need to access any data stored in nested parent elements, and the ability of your tool or application to process JSON.

Modeling a Top-Level Document View

With DataModel set to "Document" (the default), the connector scans only the top-level object by default. The top-level object elements are available as columns due to the default object flattening. Nested objects are returned as aggregated JSON.

Example

Below is a sample query and the results, based on the sample document in Raw Data. The query results in a single "insured" table.

Query

The following query pulls the top-level object elements and the vehicles array into the results.

SELECT
  [_id],
  [name],
  [address.street] AS address_street,
  [address.city] AS address_city,
  [address.state] AS address_state,
  [insured_ages],
  [vehicles]
FROM
  [insured]
  

Results

With a document view of the data, the address object is flattened into 3 columns (when FlattenObjects set to true) and the _id, name, insured_ages, and vehicles elements are returned as individual columns, resulting in a table with 7 columns.

[{"year":2015,"make":"Dodge","model":"RAM 1500","body_style":"TK"},{"year":2015,"make":"Suzuki","model":"V-Strom 650 XT","body_style":"MC"},{"year":1992,"make":"Harley Davidson","model":"FXR","body_style":"MC"}]

[{"year":2010,"make":"Honda","model":"Accord","body_style":"SD"},{"year":2008,"make":"Honda","model":"Civic","body_style":"CP"}]

See Also

• Automatic Schema Discovery: Configure column discovery with horizontal flattening.
• FreeForm;: Use dot notation to select nested data.
• VerticalFlattening;: Access nested object arrays as separate tables.
• JSON Functions: Manipulate the data returned to perform client-side aggregation and transformations.

The CData Python Connector for Elasticsearch can be configured to create a relational model of the data, treating nested documents as individual tables containing a primary key and a foreign key that links to the parent document. This is particularly useful if you need to work with your Elasticsearch data in existing BI, reporting, and ETL tools that expect a relational data model.

Joining Nested Arrays as Tables

With DataModel set to "Relational", any JOINs are controlled by the query. Any time you perform a JOIN query, the Elasticsearch index will be queried once for each table (nested document) included in the query.

Example

Below is a sample query against the sample document in Raw Data, using a relational model.

Query

The following query explicitly JOINs the insured and vehiclestables.

SELECT 
  [insured].[_id], 
  [insured].[name], 
  [insured].[address.street] AS address_street, 
  [insured].[address.city.first] AS address_city, 
  [insured].[address.state.last] AS address_state, 
  [insured].[insured_ages], 
  [vehicles].[year], 
  [vehicles].[make], 
  [vehicles].[model], 
  [vehicles].[body_style],
  [vehicles].[_insured_id],
  [vehicles].[_c_id]
FROM 
  [insured]
JOIN 
  [vehicles] 
ON 
  [insured].[_id] = [vehicles].[_insured_id]

Results

In the example query, each vehicle document is JOINed to its parent insured object to produce a table with 5 rows.

See Also

• Automatic Schema Discovery: Configure the columns reported in the table schemas.
• FreeForm;: Use dot notation to select nested data.
• VerticalFlattening;: Access nested object arrays as separate tables.
• JSON Functions: Manipulate the data returned to perform client-side aggregation and transformations.

The connector can return JSON structures as column values. The connector enables you to use standard SQL functions to work with these JSON structures. The examples in this section use the following array:

[
     { "grade": "A", "score": 2 },
     { "grade": "A", "score": 6 },
     { "grade": "A", "score": 10 },
     { "grade": "A", "score": 9 },
     { "grade": "B", "score": 14 }
]

JSON_EXTRACT

SELECT Name, JSON_EXTRACT(grades,'[0].grade') AS Grade, JSON_EXTRACT(grades,'[0].score') AS Score FROM Students;

JSON_COUNT

SELECT Name, JSON_COUNT(grades,'[x]') AS NumberOfGrades FROM Students;

JSON_SUM

SELECT Name, JSON_SUM(score,'[x].score') AS TotalScore FROM Students;

JSON_MIN

SELECT Name, JSON_MIN(score,'[x].score') AS LowestScore FROM Students;

JSON_MAX

SELECT Name, JSON_MAX(score,'[x].score') AS HighestScore FROM Students;

DOCUMENT

The DOCUMENT function can be used to retrieve the entire document as a JSON string. See the following query and its result as an example:

SELECT DOCUMENT(*) FROM Employee;

 
{
  "_index": "megacorp",
  "_type": "employee",
  "_id": "2",
  "_score": 1,
  "_source": {
    "first_name": "Jane",
    "last_name": "Smith",
    "age": 32,
    "about": "I like to collect rock albums",
    "interests": [
      "music"
    ]
  }
} 

This section describes how SQL statements are interpreted and translated into Elasticsearch queries. Examples are also provided to explain the behavior of various queries.

Query/Filter Context and Scoring

Text Matching and Search

To demonstrate this point, an analyzed field in Elasticsearch was created with a value of 'Bike'. After being analyzed, the value will be stored in the inverted index (using the default analyzer) as 'bike'. A non-analyzed field, on the other hand, would not analyze the search value and thus would be stored as 'Bike'.

When performing searches, some Elasticsearch query types run the search value through an analyzer (which will make the search case insensitive) and some do not (making the search case sensitive). Additionally, the default analyzer breaks up fields containing multiple words into separate terms. When performing searches on these fields, Elasticsearch may return records that contain the same words but in a different order. For example, a search is performed using a value of 'blue sky' but a record with 'sky blue' is returned.

To work around these case-sensitivity and ordering issues, the CData Python Connector for Elasticsearch will identify the column as analyzed or non-analyzed and will issue the appropriate Elasticsearch query based on the specified operator (such as =) and the search value.

Equals and Not Equals

Analyzed Columns
Analyzed columns are stored after being run through an analyzer. As a result of that, the search values specified will be run through an analyzer on the Elasticsearch server prior to the search. This makes the searches case-insensitive (provided the analyzer used handles casing).

Non-Analyzed Columns
Non-analyzed columns are stored without being run through an analyzer. Thus, non-analyzed columns are case sensitive and thus search values specified for these columns are case sensitive. If the search value is a single word, the connector will check the filter with the original casing specified along with three common forms: uppercase, lowercase, and capitalized. If the search value contains multiple words, the search value will be sent as-is and thus is case sensitive.

IN and NOT IN

LIKE and NOT LIKE

Aggregate Filtering

JSON objects and arrays of objects will be treated as raw strings and all filtering will be performed by the connector. Therefore an equals operation must match the entire JSON aggregate to return a result, unless a CONTAINS or LIKE operation is used.

If JSON objects are flattened into individual columns (via FlattenObjects and FlattenArrays), the column for the specific JSON field will be treated as individual columns. Thus the data type will be that as contained in the Elasticsearch mapping and all filters will be pushed to the server (where applicable).

JSON primitive array aggregates will also be treated as raw strings by default and filters will be performed by the connector. To filter data based on whether a primitive array contains a single value, the INARRAY function can be used (e.g. INARRAY(column) = 'value'). When performing a search on array fields, Elasticsearch looks at each value individually within an array. Thus when the INARRAY function is specified in a WHERE clause, the filter will be pushed to the server which performs a search within an array.

Primitive arrays may consist of different data types, such as strings or ints. Therefore the INARRAY function supports comparison operators applicable to the data type within the Elasticsearch mapping for the field. For example, INARRAY(int_array) > 5, will return all rows of data in which the int_array contains a value greater than 5. Supported comparison operators include the use of the LIKE operator for string arrays.

View schemas persist the relational structure the connector infers for Elasticsearch types and queries. To provide an example of how custom schemas work, we will use the below mapping (where 'insured' is the name of the table).

{
  "insured": {
    "properties": {
      "name": { "type":"string" },
      "address": {
        "street": { "type":"string" },
        "city": { "type":"string" },
        "state": { "type":"string" }
      },
      "insured_ages": { "type": "integer" },
      "vehicles": {
        "type": "nested",
        "properties": {
          "year": { "type":"integer" },
          "make": { "type":"string" },
          "model": { "type":"string" },
          "body_style" { "type": "string" }
        }
      }
    }
  }
}

Also, consider the following example data for the above mapping:

{
  "_source": {
    "name": "John Smith",
    "address": {
      "street": "Main Street",
      "city": "Chapel Hill",
      "state": "NC"
    },
    "insured_ages": [ 17, 43, 45 ], 
    "vehicles": [
      {
        "year": 2015,
        "make": "Dodge",
        "model": "RAM 1500",
        "body_style": "TK"
      },
      {
        "year": 2015,
        "make": "Suzuki",
        "model": "V-Strom 650 XT",
        "body_style": "MC"
      },
      {
        "year": 2012,
        "make": "Honda",
        "model": "Accord",
        "body_style": "4D"
      }
    ]
  }
}

Defining a Custom Schema

Schemas persisted when GenerateSchemaFiles is set are placed into the folder specified by the Location property. For example, set GenerateSchemaFiles to "OnUse" and execute a SELECT query:

SELECT * FROM insured

You can then change column behavior in the resulting schema. The following schema uses the other:xPath property to define where the data for a particular column should be retrieved from. Using this model you can flatten arbitrary levels of hierarchy.

The es_index and es_type attributes specify the Elasticsearch index and type to retrieve. The es_index and es_type attributes give you the flexibility to use multiple schemas for the same type. If es_type is not specified, the filename determines the collection that is parsed.

Below is an example is an example of the column behavior markup. You can find a complete schema in Custom Schema Example.

  <rsb:script xmlns:rsb="http://www.rssbus.com/ns/rsbscript/2">  
  
    <rsb:info title="StaticInsured" description="Custom Schema for the Elasticsearch insured data set.">  
      <!-- Column definitions -->
      <attr name="_id"                         xs:type="string"  other:xPath="_id"                                                         other:sourceField="_id"                 other:analyzed="true"  />
      <attr name="_score"                      xs:type="double"  other:xPath="_score"                                                      other:sourceField="_score"              other:analyzed="true"  />
      <attr name="name"                        xs:type="string"  other:xPath="_source/name"                                                other:sourceField="name"                other:analyzed="true"  />
      <attr name="address.street"              xs:type="string"  other:xPath="_source/address/street"                                      other:sourceField="address.street"      other:analyzed="true"  />
      <attr name="address.city"                xs:type="string"  other:xPath="_source/address/city"                                        other:sourceField="address.city"        other:analyzed="true"  />
      <attr name="address.state"               xs:type="string"  other:xPath="_source/address/state"                                       other:sourceField="address.state"       other:analyzed="true"  />
      <attr name="insured_ages"                xs:type="string"  other:xPath="_source/insured_ages"          other:valueFormat="aggregate" other:sourceField="insured_ages"        other:analyzed="false" />
      <attr name="insured_ages.0"              xs:type="integer" other:xPath="_source/insured_ages[0]"                                     other:sourceField="insured_ages"        other:analyzed="false" />
      <attr name="vehicles"                    xs:type="string"  other:xPath="_source/vehicles"              other:valueFormat="aggregate" other:sourceField="vehicles"            other:analyzed="true"  />
      <attr name="vehicles.0.year"             xs:type="integer" other:xPath="_source/vehicles[0]/year"                                    other:sourceField="vehicles.year"       other:analyzed="true"  />
      <attr name="vehicles.0.make"             xs:type="string"  other:xPath="_source/vehicles[0]/make"                                    other:sourceField="vehicles.make"       other:analyzed="true"  />
      <attr name="vehicles.0.model"            xs:type="string"  other:xPath="_source/vehicles[0]/model"                                   other:sourceField="vehicles.model"      other:analyzed="true"  />
      <attr name="vehicles.0.body_style"       xs:type="string"  other:xPath="_source/vehicles[0]/body_style"                              other:sourceField="vehicles.body_style" other:analyzed="true"  />
  
      <input name="rows@next" desc="Internal attribute used for paging through data."  />
    </rsb:info>  
  
  
    <rsb:set attr="es_index" value="auto"/>
    <rsb:set attr="es_type"  value="insured"/>
  
  </rsb:script>
  

In this section is a complete schema. The info section enables a relational view of an Elasticsearch object. For more details, see Custom Schema Definitions. The table below only supports SELECT commands. INSERT, UPDATE, and DELETE commands are not currently supported.

Use the es_index and es_type attributes to specify the name of the Elasticsearch type and index you want to retrieve and parse. You can use the es_index and es_type attributes to define multiple schemas for the same Elasticsearch type.

If es_type is not specified, the filename determines the Elasticsearch type that is parsed.

Copy the rows@next input as-is into your schema. The operations, such as elasticsearchadoSelect, are internal implementations and can also be copied as is.

<rsb:script xmlns:rsb="http://www.rssbus.com/ns/rsbscript/2">  

  <rsb:info title="StaticInsured" description="Custom Schema for the Elasticsearch insured data set.">  
    <!-- Column definitions -->
    <attr name="_id"                         xs:type="string"  other:xPath="_id"                                                         other:sourceField="_id"                 other:analyzed="true"  />
    <attr name="_score"                      xs:type="double"  other:xPath="_score"                                                      other:sourceField="_score"              other:analyzed="true"  />
    <attr name="name"                        xs:type="string"  other:xPath="_source/name"                                                other:sourceField="name"                other:analyzed="true"  />
    <attr name="address.street"              xs:type="string"  other:xPath="_source/address/street"                                      other:sourceField="address.street"      other:analyzed="true"  />
    <attr name="address.city"                xs:type="string"  other:xPath="_source/address/city"                                        other:sourceField="address.city"        other:analyzed="true"  />
    <attr name="address.state"               xs:type="string"  other:xPath="_source/address/state"                                       other:sourceField="address.state"       other:analyzed="true"  />
    <attr name="insured_ages"                xs:type="string"  other:xPath="_source/insured_ages"          other:valueFormat="aggregate" other:sourceField="insured_ages"        other:analyzed="false" />
    <attr name="insured_ages.0"              xs:type="integer" other:xPath="_source/insured_ages[0]"                                     other:sourceField="insured_ages"        other:analyzed="false" />
    <attr name="vehicles"                    xs:type="string"  other:xPath="_source/vehicles"              other:valueFormat="aggregate" other:sourceField="vehicles"            other:analyzed="true"  />
    <attr name="vehicles.0.year"             xs:type="integer" other:xPath="_source/vehicles[0]/year"                                    other:sourceField="vehicles.year"       other:analyzed="true"  />
    <attr name="vehicles.0.make"             xs:type="string"  other:xPath="_source/vehicles[0]/make"                                    other:sourceField="vehicles.make"       other:analyzed="true"  />
    <attr name="vehicles.0.model"            xs:type="string"  other:xPath="_source/vehicles[0]/model"                                   other:sourceField="vehicles.model"      other:analyzed="true"  />
    <attr name="vehicles.0.body_style"       xs:type="string"  other:xPath="_source/vehicles[0]/body_style"                              other:sourceField="vehicles.body_style" other:analyzed="true"  />

    <input name="rows@next" desc="Internal attribute used for paging through data."  />
  </rsb:info>  

  <rsb:set attr="es_index" value="auto"/>
  <rsb:set attr="es_type"  value="insured"/>

  <rsb:script method="GET">
    <rsb:call op="elasticsearchadoSelect">
      <rsb:push/>
    </rsb:call>
  </rsb:script>

  <rsb:script method="POST">
    <rsb:call op="elasticsearchadoModify">
      <rsb:push/>
    </rsb:call>
  </rsb:script>

  <rsb:script method="MERGE">
    <rsb:call op="elasticsearchadoModify">
      <rsb:push/>
    </rsb:call>
  </rsb:script>

  <rsb:script method="DELETE">
    <rsb:call op="elasticsearchadoModify">
      <rsb:push/>
    </rsb:call>
  </rsb:script>

</rsb:script>

This section provides a walk-through for writing Elasticsearch data access code in Python script.

For more information on the available data source entities and how to query them with SQL, see Data Model. For the SQL syntax, see SQL Compliance.

Connecting from Code

For information on how to connect with the elasticsearch.connector module and its related classes, see Connecting.

Executing SQL

Executing Stored Procedures

Batch Processing

Connecting with the cdata.elasticsearch Module:

import cdata.elasticsearch as mod
conn = mod.connect("Server=127.0.0.1;Port=9200;")

Once the connection is created, you can use it to execute subsequent SQL queries.

After connecting as described in Connecting, you can use the open connection to execute SQL statements.

Executing Queries

For example:

cur = conn.execute("SELECT Id, Name FROM [CData].[Elasticsearch].Employee")
rs = cur.fetchall()
for row in rs:
	print(row)

Parameterized Queries

Various Python collections, such as arrays and tuples, can act as additional arguments for the execute() method. This enables you to parameterize the queries executed and help to prevent SQL Injection.

For example:

cmd = "SELECT Id, Name FROM [CData].[Elasticsearch].Employee WHERE Industry = ?"
params = ["Floppy Disks"]
cur = conn.execute(cmd, params)
rs = cur.fetchall()
for row in rs:
	print(row)

The connection is also used to issue INSERT, UPDATE, and DELETE commands to the data source. Parameters can be used with these statements if desired.

Note that the connector does not support transactions. As with normal write operations, all SQL statements executed by this connector affect the data source immediately. Call the connection's commit() method following the execution.

Insert

cmd = "INSERT INTO [CData].[Elasticsearch].Employee (Id, Name) VALUES (?, ?)"
params = ["Jon Doe", "Floppy Disks"]
cur = conn.execute(cmd, params)
print("Records affected: ", cur.rowcount)

Update

cmd = "UPDATE [CData].[Elasticsearch].Employee SET Name = ? WHERE Id = ?"
params = ["Floppy Disks", "1"]
cur = conn.execute(cmd, params)
print("Records affected: ", cur.rowcount)

Delete

The following example removes an existing record from the table:

cmd = "DELETE FROM [CData].[Elasticsearch].Employee WHERE Id = ?"
params = ["1"]
cur = conn.execute(cmd, params)
print("Records affected: ", cur.rowcount)

You can execute stored procedures using either the execute() or callproc() method of the connection.

Calling Stored Procedures Using Execute()

cmd = "EXECUTE CreateTable TableName = ?"
params = ["Orders"]
conn.execute(cmd, params)

Calling Stored Procedures Using Callproc()

cur = conn.cursor()
params = ["Orders"]
cur.callproc("CreateTable", params)

This Python connector also supports writing to the data source via batch processing, using the cursor object's executemany() method. This requires both a SQL statement string and a data frame of values that act as a series of parameters for executing the SQL statement.

Note that the connector does not support transactions. As with normal write operations, all SQL statements executed by this connector affect the data source immediately. Call the connection's commit() method following the execution.

Insert

cur = conn.cursor()
cmd = "INSERT INTO [CData].[Elasticsearch].Employee (Id, Name) VALUES (?, ?)"
params = [["Jon Doe", "Floppy Disks"], ["Jon Doe", "Floppy Disks"]]
cur.executemany(cmd, params)
print("Records affected: ", cur.rowcount)

Update

cur = conn.cursor()
cmd = "UPDATE [CData].[Elasticsearch].Employee SET Name = ? WHERE Id = ?"
params = [["Floppy Disks", "1"], ["Floppy Disks", "1"]]
cur.executemany(cmd, params)
print("Records affected: ", cur.rowcount)

Delete

cur = conn.cursor()
cmd = "DELETE FROM [CData].[Elasticsearch].Employee WHERE Id = ?"
params = [["1"], ["1"]]
cur.executemany(cmd, params)
print("Records affected: ", cur.rowcount)

The connector is integrated with other tools and packages within Python.

Python Integration Guides

The following sections show how to create and use connections with the connector in common packages in Python:

• From SQLAlchemy
• From Pandas
• From Matplotlib
• From Petl

Complete List of Elasticsearch Integration Quickstarts

The CData Python Connector for Elasticsearch includes a Dialect class that enables integration with SQLAlchemy. Bear in mind that several aspects of connector functionality are not currently supported in SQLAlchemy 2.0 or above. If necessary, downgrade SQLAlchemy to version 1.4 or 1.3 before using this connector.

The following sections detail various aspects of this integration:

Connecting From SQLAlchemy

Reflecting Metadata With SQLAlchemy

Querying Data From SQLAlchemy

Modifying Data From SQLAlchemy

Connecting With a Dialect URL

from sqlalchemy import create_engine
engine = create_engine("elasticsearch:///?Server=127.0.0.1;Port=9200;")

For SQLAlchemy 2.0, the dialect name is elasticsearch_2. To establish a connection, use the following URL format:

from sqlalchemy import create_engine
engine = create_engine("elasticsearch_2:///?Server=127.0.0.1;Port=9200;")

SQLAlchemy can act as an Object-relational Map (ORM). This enables you to treat records of a database table as instantiable records. To leverage this functionality, you must reflect the underlying metadata in one of the following ways.

Note: The following examples employ SQLAlchemy 1.4.

Modeling Data Using a Mapping Class

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class [CData].[Elasticsearch].Employee(Base):
	__tablename__ = "[CData].[Elasticsearch].Employee"
	Id = Column(String, primary_key=True)
	Id = Column(String)
	Name = Column(String)

Automatically Reflecting Metadata

from sqlalchemy import MetaData
from sqlalchemy.ext.automap import automap_base
meta = MetaData()
abase = automap_base(metadata=meta)
abase.prepare(autoload_with=engine)
[CData].[Elasticsearch].Employee = abase.classes.[CData].[Elasticsearch].Employee

You can also reflect a single table with an inspector. When reflecting this way, providing a list of specific columns to map is optional:

from sqlalchemy import MetaData, Table
from sqlalchemy import inspect
meta = MetaData()
insp = inspect(engine)
[CData].[Elasticsearch].Employee_table = Table("[CData].[Elasticsearch].Employee", meta)
insp.reflect_table([CData].[Elasticsearch].Employee_table, ["Id","Name"])

After you use the steps in Connecting to connect, and use one of the methods in Reflecting Metadata to reflect some of the metadata, you can use a session object to query data.

Querying Data Using the Query Method

For example:

engine = create_engine("elasticsearch:///?Server=127.0.0.1;Port=9200;")
factory = sessionmaker(bind=engine)
session = factory()
for instance in session.query([CData].[Elasticsearch].Employee).filter_by(Industry="Floppy Disks"):
	print("Id: ", instance.Id)
	print("Id: ", instance.Id)
	print("Name: ", instance.Name)
	print("---------")

Querying Data Using the Execute Method

[CData].[Elasticsearch].Employee_table = [CData].[Elasticsearch].Employee.metadata.tables["[CData].[Elasticsearch].Employee"]
for instance in session.execute([CData].[Elasticsearch].Employee_table.select().where([CData].[Elasticsearch].Employee_table.c.Industry == "Floppy Disks")):
	print("Id: ", instance.Id)
	print("FullName: ", instance.Name)
	print("City: ", instance.BillingCity)
	print("---------")

Implicit Joining

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, String, Integer, DateTime, ForeignKey
from sqlalchemy.orm import sessionmaker, relationship

Base = declarative_base()
class Contact(Base):
	__tablename__ = "Contact"
	Id = Column(Integer, primary_key=True)
	Name = Column(String)
	Email = Column(String)
	BirthDate = Column(DateTime)
	AccountId = Column(String, ForeignKey("Account.Id"))
	Account_Link = relationship("Account", back_populates="Contact_Link")

class Account(Base):
	__tablename__ = "Account"
	Id = Column(String, primary_key=True)
	Name = Column(String)
	BillingCity = Column(String)
	NumberOfEmployees = Column(Integer)
	Contact_Link = relationship("Contact", order_by=Contact.Id, back_populates="Account_Link")

Once the relationship is established, the tables are queried simultaneously using the session's query() method. For example:

rs = session.query(Account, Contact).filter(Account.Id == Contact.AccountId)
for Ac, Ct in rs:
  print("AccountId: ", Ac.Id)
  print("AccountName: ", Ac.Name)
  print("ContactId: ", Ct.Id)
  print("ContactName: ", Ct.Name)

Other Join Forms

• Explicit condition (necessary if there are no foreign keys in your mapped classes):
  

  rs = session.query(Account, Contact).join(Contact, Account.Id == Contact.AccountId)
  for Ac, Ct in rs:

• Left-to-right relationship:
  

  rs = session.query(Account, Contact).join(Account.Contact_Link)
  for Ac, Ct in rs:

• Left-to-right relationship with explicit target:
  

  rs = session.query(Account, Contact).join(Contact, Account.Contact_Link)
  for Ac, Ct in rs:

• String form of a left-to-right relationship:
  

  rs = session.query(Account, Contact).join("Contact_Link")
  for Ac, Ct in rs:

SQLAlchemy ORM also exposes support for other clauses in SQL, such as ORDER BY, GROUP BY, LIMIT, and OFFSET. All of these are supported by this connector:

ORDER BY

rs = session.query([CData].[Elasticsearch].Employee).order_by([CData].[Elasticsearch].Employee.AnnualRevenue)
for instance in rs:
	print("Id: ", instance.Id)
	print("Id: ", instance.Id)
	print("Name: ", instance.Name)
	print("---------")

You can also use the session object's execute() method perform an ORDER BY. For example:

rs = session.execute([CData].[Elasticsearch].Employee_table.select().order_by([CData].[Elasticsearch].Employee_table.c.AnnualRevenue))
for instance in rs:

GROUP BY

rs = session.query(func.count([CData].[Elasticsearch].Employee.Id).label("CustomCount"), [CData].[Elasticsearch].Employee.Id).group_by([CData].[Elasticsearch].Employee.Id)
for instance in rs:
	print("Count: ", instance.CustomCount)
	print("Id: ", instance.Id)
	print("---------")

You can also use the session object's execute() method to perform a GROUP BY:

rs = session.execute([CData].[Elasticsearch].Employee_table.select().with_only_columns([func.count([CData].[Elasticsearch].Employee_table.c.Id).label("CustomCount"), [CData].[Elasticsearch].Employee_table.c.Id]).group_by([CData].[Elasticsearch].Employee_table.c.Id))
for instance in rs:

LIMIT and OFFSET

rs = session.query([CData].[Elasticsearch].Employee).limit(25).offset(100)
for instance in rs:
	print("Id: ", instance.Id)
	print("Id: ", instance.Id)
	print("Name: ", instance.Name)
	print("---------")

You can also use the session object's execute() method to set a LIMIT or OFFSET:

rs = session.execute([CData].[Elasticsearch].Employee_table.select().limit(25).offset(100))
for instance in rs:

Certain aggregate functions can also be used within SQLAlchemy by using the func module.

To import this module, execute:

from sqlalchemy.sql import func

Once func is imported, the following aggregate functions are available:

COUNT

rs = session.query(func.count([CData].[Elasticsearch].Employee.Id).label("CustomCount"), [CData].[Elasticsearch].Employee.Id).group_by([CData].[Elasticsearch].Employee.Id)
for instance in rs:
	print("Count: ", instance.CustomCount)
	print("Id: ", instance.Id)
	print("---------")

You can also execute COUNT using the session object's execute() method:

rs = session.execute([CData].[Elasticsearch].Employee_table.select().with_only_columns([func.count([CData].[Elasticsearch].Employee_table.c.Id).label("CustomCount"), [CData].[Elasticsearch].Employee_table.c.Id])group_by([CData].[Elasticsearch].Employee_table.c.Id))
for instance in rs:

SUM

This example calculates the cumulative amount of a numeric column in a set of groups.

rs = session.query(func.sum([CData].[Elasticsearch].Employee.AnnualRevenue).label("CustomSum"), [CData].[Elasticsearch].Employee.Id).group_by([CData].[Elasticsearch].Employee.Id)
for instance in rs:
	print("Sum: ", instance.CustomSum)
	print("Id: ", instance.Id)
	print("---------")

You can also invoke SUM using the session object's execute() method.

rs = session.execute([CData].[Elasticsearch].Employee_table.select().with_only_columns([func.sum([CData].[Elasticsearch].Employee_table.c.AnnualRevenue).label("CustomSum"), [CData].[Elasticsearch].Employee_table.c.Id]).group_by([CData].[Elasticsearch].Employee_table.c.Id))
for instance in rs:

AVG

rs = session.query(func.avg([CData].[Elasticsearch].Employee.AnnualRevenue).label("CustomAvg"), [CData].[Elasticsearch].Employee.Id).group_by([CData].[Elasticsearch].Employee.Id)
for instance in rs:
	print("Avg: ", instance.CustomAvg)
	print("Id: ", instance.Id)
	print("---------")

You can also use the session object's execute() method to invoke AVG:

rs = session.execute([CData].[Elasticsearch].Employee_table.select().with_only_columns([func.avg([CData].[Elasticsearch].Employee_table.c.AnnualRevenue).label("CustomAvg"), [CData].[Elasticsearch].Employee_table.c.Id]).group_by([CData].[Elasticsearch].Employee_table.c.Id))
for instance in rs:

MAX and MIN

rs = session.query(func.max([CData].[Elasticsearch].Employee.AnnualRevenue).label("CustomMax"), func.min([CData].[Elasticsearch].Employee.AnnualRevenue).label("CustomMin"), [CData].[Elasticsearch].Employee.Id).group_by([CData].[Elasticsearch].Employee.Id)
for instance in rs:
	print("Max: ", instance.CustomMax)
	print("Min: ", instance.CustomMin)
	print("Id: ", instance.Id)
	print("---------")

You can also use the session object's execute() method to invoke MAX and MIN:

rs = session.execute([CData].[Elasticsearch].Employee_table.select().with_only_columns([func.max([CData].[Elasticsearch].Employee_table.c.AnnualRevenue).label("CustomMax"), func.min([CData].[Elasticsearch].Employee_table.c.AnnualRevenue).label("CustomMin"), [CData].[Elasticsearch].Employee_table.c.Id]).group_by([CData].[Elasticsearch].Employee_table.c.Id))
for instance in rs:

Commands can be executed individually by the session with a call to "execute()".

Obtaining the Table Object

The query supplied to this method is constructed using the associated Table object of a mapped class. This Table object is obtained from the mapped class's metadata field, as below:

[CData].[Elasticsearch].Employee_table = [CData].[Elasticsearch].Employee.metadata.tables["[CData].[Elasticsearch].Employee"]

Once the table object is obtained, the write operations are executed in the following ways. The queries are executed immediately without the need for a call to "commit()":

Insert

The following example adds a new record to the table:

session.execute([CData].[Elasticsearch].Employee_table.insert(), {"Id": "Jon Doe", "Name": "Floppy Disks"})

Update

The following example modifies an existing record in the table:

session.execute([CData].[Elasticsearch].Employee_table.update().where([CData].[Elasticsearch].Employee_table.c.Id == "1").values(Id="Jon Doe", Name="Floppy Disks"))

Delete

The following example removes an existing record from the table:

session.execute([CData].[Elasticsearch].Employee_table.delete().where([CData].[Elasticsearch].Employee_table.c.Id == "1"))

When combined with the connector, Pandas can be used to generate data frames that contain your Elasticsearch data. Once created, a data frame can be passed to various other Python packages.

Connecting

import pandas as pd
from sqlalchemy import create_engine
engine = create_engine("elasticsearch:///?Server=127.0.0.1;Port=9200;")

Querying Data

df = pd.read_sql("""
	SELECT
	   Id,
	   Name,
     $exNumericCol;
	FROM [CData].[Elasticsearch].Employee;""", engine)
print(df)

Modifying Data

df = pd.DataFrame({"Id": ["Jon Doe"], "Name": ["Floppy Disks"]})
df.to_sql("[CData].[Elasticsearch].Employee", con=engine, if_exists="append", index=False)

Matplotlib contains a number of tools that can graphically model Elasticsearch data after being fed a data frame From Pandas.

Using PyPlot

from matplotlib import pyplot as plt

Once a Pandas data frame is obtained, it can be used to create a plot visualizing Elasticsearch data. For example, the following plot generates and displays a bar graph relating Id and AnnualRevenue values:

df.plot(kind="bar", x="Id", y=["AnnualRevenue"])
plt.show()

The connector can be used to create ETL applications and pipelines for CSV data in Python using Petl.

Install Required Modules

pip install petl

Connecting

import petl as etl
import cdata.elasticsearch as mod
cnxn = mod.connect("Server=127.0.0.1;Port=9200;")

Extract, Transform, and Load the Elasticsearch Data

sql = "SELECT	Id, Name FROM [CData].[Elasticsearch].Employee "
table1 = etl.fromdb(cnxn,sql)

Loading Data

etl.tocsv(table1,'output.csv')

Modifying Data

table1 = [['Id','Name'],['Jon Doe','Floppy Disks']]
etl.appenddb(table1,cnxn,'[CData].[Elasticsearch].Employee')

The extension supports schema discovery by using SQL queries to available System Tables.

Using SQL

• Tables and Views
• Columns
• Procedures

The connector possesses system tables that are used to discover the tables and views available in the data model. Of these system tables, "sys_tables" and "sys_views" are used to fetch information about the available tables and views respectively:

Tables

import cdata.elasticsearch as mod
conn = mod.connect("Server=127.0.0.1;Port=9200;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_tables"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

Views

import cdata.elasticsearch as mod
conn = mod.connect("Server=127.0.0.1;Port=9200;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_views"
cur.execute(cmd, params)
rs = cur.fetchall()
for row in rs:
	print(row)

The available columns for any given table are fetched from a system table called "sys_tablecolumns". A specific table name is provided in the WHERE criteria to restrict the table from which the column information is fetched:

import cdata.elasticsearch as mod
conn = mod.connect("Server=127.0.0.1;Port=9200;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_tablecolumns WHERE TableName = '[CData].[Elasticsearch].Employee'"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

Procedures

import cdata.elasticsearch as mod
conn = mod.connect("Server=127.0.0.1;Port=9200;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_procedures"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

Parameters

import cdata.elasticsearch as mod
conn = mod.connect("Server=127.0.0.1;Port=9200;")
cur = conn.cursor()
cmd = "SELECT * FROM sys_procedureparameters WHERE ProcedureName = 'CreateTable'"
cur.execute(cmd)
rs = cur.fetchall()
for row in rs:
	print(row)

This section details a selection of advanced features of the Elasticsearch connector.

User Defined Views

The connector supports the use of user defined views, virtual tables whose contents are decided by a pre-configured user defined query. These views are useful when you cannot directly control queries being issued to the drivers. For an overview of creating and configuring custom views, see User Defined Views .

SSL Configuration

Use SSL Configuration to adjust how connector handles TLS/SSL certificate negotiations. You can choose from various certificate formats;. For further information, see the SSLServerCert property under "Connection String Options" .

Firewall and Proxy

Configure the connector for compliance with Firewall and Proxy, including Windows proxies and HTTP proxies. You can also set up tunnel connections.

Caching Data

Caching Data enables faster access to data and reduces the number of API calls, improving performance. The connector supports a simple caching model where multiple connections can also share the cache over time. When configuring the cache connection, you can specify automatic or explicit data caching.

Query Processing

For further information, see Query Processing.

Logging

Exception Handling

The CData Python Connector for Elasticsearch supports the use of user defined views: user-defined virtual tables whose contents are decided by a preconfigured query. User defined views are useful in situations where you cannot directly control the query being issued to the driver; for example, when using the driver from a tool.

Use a user defined view to define predicates that are always applied. If you specify additional predicates in the query to the view, they are combined with the query already defined as part of the view.

There are two ways to create user defined views:

• Create a JSON-formatted configuration file defining the views you want.
• DDL statements.

Defining Views Using a Configuration File

You can also have multiple view definitions and control them using the UserDefinedViews connection property. When you use this property, only the specified views are seen by the connector.

This user defined view configuration file is formatted so that each root element defines the name of a view, and includes a child element, called query, which contains the custom SQL query for the view.

For example:

{
	"MyView": {
		"query": "SELECT * FROM [CData].[Elasticsearch].Employee WHERE MyColumn = 'value'"
	},
	"MyView2": {
		"query": "SELECT * FROM MyTable WHERE Id IN (1,2,3)"
	}
}

"UserDefinedViews", "C:\\Users\\yourusername\\Desktop\\tmp\\UserDefinedViews.json"

Defining Views Using DDL Statements

The connector is also capable of creating and altering the schema via DDL Statements such as CREATE LOCAL VIEW, ALTER LOCAL VIEW, and DROP LOCAL VIEW.

Create a View

To create a new view using DDL statements, provide the view name and query as follows:

CREATE LOCAL VIEW [MyViewName] AS SELECT * FROM Customers LIMIT 20;

If no JSON file exists, the above code creates one. The view is then created in the JSON configuration file and is now discoverable. The JSON file location is specified by the UserDefinedViews connection property.

Alter a View

To alter an existing view, provide the name of an existing view alongside the new query you would like to use instead:

ALTER LOCAL VIEW [MyViewName] AS SELECT * FROM Customers WHERE TimeModified > '3/1/2020';

The view is then updated in the JSON configuration file.

Drop a View

To drop an existing view, provide the name of an existing schema alongside the new query you would like to use instead.

DROP LOCAL VIEW [MyViewName]

This removes the view from the JSON configuration file. It can no longer be queried.

Schema for User Defined Views

Working with User Defined Views

SELECT * FROM Customers WHERE City = 'Raleigh';

SELECT * FROM UserViews.RCustomers WHERE Status = 'Active';

SELECT * FROM Customers WHERE City = 'Raleigh' AND Status = 'Active';

Customizing the SSL Configuration

By default, the connector attempts to negotiate TLS with the server. The server certificate is validated against the default system trusted certificate store. You can override how the certificate gets validated using the SSLServerCert connection property.

To specify another certificate, see the SSLServerCert connection property.

Client SSL Certificates

The Elasticsearch connector also supports setting client certificates. Set the following to connect using a client certificate.

• SSLClientCert: The name of the certificate store for the client certificate.
• SSLClientCertType: The type of key store containing the TLS/SSL client certificate.
• SSLClientCertPassword: The password for the TLS/SSL client certificate.
• SSLClientCertSubject: The subject of the TLS/SSL client certificate.

Connecting Through a Firewall or Proxy

HTTP Proxies

To authenticate to an HTTP proxy, set the following:

• ProxyServer: the hostname or IP address of the proxy server that you want to route HTTP traffic through.
• ProxyPort: the TCP port that the proxy server is running on.
• ProxyAuthScheme: the authentication method the connector uses when authenticating to the proxy server.
• ProxyUser: the username of a user account registered with the proxy server.
• ProxyPassword: the password associated with the ProxyUser.

Other Proxies

Set the following properties:

• To use a proxy-based firewall, set FirewallType, FirewallServer, and FirewallPort.
• To tunnel the connection, set FirewallType to TUNNEL.
• To authenticate, specify FirewallUser and FirewallPassword.
• To authenticate to a SOCKS proxy, additionally set FirewallType to SOCKS5.

Caching Data

Caching data provides several benefits, including faster access to data and reducing the number of API calls, which improve performance. The connector supports a simple caching model where multiple connections can also share the cache over time. You can enable and configure caching features by setting the necessary connection properties.

Contents

The sections in this chapter detail the connector's caching functionality and link to the corresponding connection properties, as well as SQL statements.

Configuring the Cache Connection

Configuring the Cache Connection describes the properties that you can set when configuring the cache database.

Caching Metadata

Caching Metadata describes the CacheMetadata property. This property determines whether or not to cache the table metadata to a file store.

Automatically Caching Data

Automatically Caching Data describes how the connector automatically refreshes the cache when the AutoCache property is set.

Explicitly Caching Data

Explicitly Caching Data describes how you can decide what data is stored in the cache and when it is updated.

Data Type Mapping

Data Type Mapping shows the mappings between the data types configured in the schema and the data types in the database.

Configuring the Caching Database

This section describes the properties for caching data to the persistent store of your choice.

CacheLocation

The CacheLocation property species the path to a file-system-based database. When caching is enabled, a file-system-based database is used by default. If CacheLocation is not specified, this database is stored at the path in Location. If neither of these connection properties are specified, the connector uses a platform-dependent default location.

CacheConnection

The CacheConnection property specifies a database driver and the connection string to the caching database.

CacheDriver and CacheProvider

Both the CacheDriver and CacheProvider properties are supported. Each specifies a database driver and the connection string to the caching database. CacheDriver is designed for Linux and MacOS; CacheProvider is Windows-based.

This section describes how to enable caching metadata and how to update the metadata cache.

Before being able to query data, the connector requires relevant metadata to be retrieved. By default, metadata is cached in memory and shared across connections. But if you want to persist across processes, or if metadata requests are expensive, the solution is to cache the metadata to disk.

Enable Caching Metadata

To enable caching of metadata, set CacheMetadata = true and see Configuring the Cache Connection for instructions on how to configure your connection string. The connector caches the metadata the first time it is needed and uses the metadata cache for subsequent requests.

Update the Metadata Cache

Because metadata is cached, changes to metadata on the live source, for example, adding or removing a column or attribute, are not automatically reflected in the metadata cache. To get updates to the live metadata, you need to delete or drop the cached data.

Automatically caching data is useful when you do not want to rebuild the cache for each query. When you query data for the first time, the connector automatically initializes and builds a cache in the background. When AutoCache = true, the connector uses the cache for subsequent query executions, resulting in faster response times.

Configuring Automatic Caching

Caching the Employee Table

The following example caches the Employee table in the file specified by the CacheLocation property of the connection string.

SELECT Id, Name FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'

Common Use Case

A common use for automatically caching data is to improve driver performance when making repeated requests to a live data source, such as building a report or creating a visualization. With auto caching enabled, repeated requests to the same data may be executed in a short period of time, but within an allowable tolerance (CacheTolerance) of what is considered "live" data.

With explicit caching (AutoCache = false), you decide exactly what data is cached and when to query the cache instead of the live data. Explicit caching gives you full control over the cache contents by using CACHE Statements. This section describes some strategies to use the caching features offered by the connector.

Creating the Cache

To load data in the cache, issue the following statement.

CACHE SELECT * FROM tableName WHERE ...

Once the statement is issued, any matching data in tableName is loaded into the corresponding table.

Updating the Cache

This section describes two ways to update the cache.

Updating with the SELECT Statement

The following example shows a statement that can update modified rows and add missing rows in the cached table. However, this statement does not delete extra rows that are already in the cache. This statement only merges the new rows or updates the existing rows.

CACHE SELECT * FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'

Updating with the TRUNCATE Statement

The following example shows a statement that can update modified rows and add missing rows in the cached table. This statement can also delete rows in the cache table that are not present in the live data source.

  CACHE WITH TRUNCATE SELECT * FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'
  

Query the Data in Online or Offline Mode

This section describes how to query the data in online or offline mode.

Online: Select Cached Tables

You can use the tableName#CACHE syntax to explicitly execute queries to the cache while still online, as shown in the following example.

SELECT * FROM [CData].[Elasticsearch].Employee#CACHE

Offline: Select Cached Tables

With Offline = true, SELECT statements always execute against the local cache database, regardless of whether you explicitly specify the cached table or not. Modification of the cache is disabled in Offline mode to prevent accidentally updating only the cached data. Executing a DELETE/UPDATE/INSERT statement while in Offline mode results in an exception.

The following example selects from the local cache but not the live data source because Offline = true.

SELECT * FROM [CData].[Elasticsearch].Employee WHERE Industry='Floppy Disks' ORDER BY Name ASC

Delete Data from the Cache

You can delete data from the cache by building a direct connection to the database. Note that the connector does not support manually deleting data from the cache.

Common Use Case

A common use for caching is to have an application always query the cached data and only update the cache at set intervals, such as once every day or every two hours. There are two ways in which this can be implemented:

• AutoCache = false and Offline = false. All queries issued by the application explicitly reference the tableName#CACHE table. When the cache needs to be updated, the application executes a tableName#CACHE ... statement to bring the cached data up to date.
• Offline = true. Caching is transparent to the application. All queries are executed against the table as normal, so most application code does not need to be aware that caching is done. To update the cached data, simply create a separate connection with Offline = false and execute a tableName#CACHE ... statement.

The connector maps types from the data source to the corresponding data type available in the chosen cache database. The following table shows the mappings between the data types configured in the schema and the data types in the database. Some schema types have synonyms which are all listed in the Schema column.

Data Type Mapping

Note: String columns can map to different data types depending on their length.

Query Processing

For sources that do not support SQL-92, the connector offloads as much of SQL statement processing as possible to Elasticsearch and then processes the rest of the query in memory (client-side). This results in optimal performance.

For data sources with limited query capabilities, the connector handles transformations of the SQL query to make it simpler for the connector. The goal is to make smart decisions based on the query capabilities of the data source to push down as much of the computation as possible. The Elasticsearch Query Evaluation component examines SQL queries and returns information indicating what parts of the query the connector is not capable of executing natively.

The Elasticsearch Query Slicer component is used in more specific cases to separate a single query into multiple independent queries. The client-side Query Engine makes decisions about simplifying queries, breaking queries into multiple queries, and pushing down or computing aggregations on the client-side while minimizing the size of the result set.

There's a significant trade-off in evaluating queries, even partially, client-side. There are always queries that are impossible to execute efficiently in this model, and some can be particularly expensive to compute in this manner. CData always pushes down as much of the query as is feasible for the data source to generate the most efficient query possible and provide the most flexible query capabilities.

More Information

Logging

Basic Logging

To begin capturing connector logging, set these properties:

• Logfile: A filepath that designates the name and location of the log file.
• Verbosity: A numerical value (1-5) that determines the amount of detail in the log. See the page in the Connection Properties section for an explanation of the five levels.
• MaxLogFileSize: When the limit is hit, a new log is created in the same folder with the date and time appended to the end. The default limit is 100 MB. Values lower than 100 kB will use 100 kB as the value instead.
• MaxLogFileCount: A string specifying the maximum file count of log files. When the limit is hit, a new log is created in the same folder with the date and time appended to the end and the oldest log file will be deleted. Minimum supported value is 2. A value of 0 or a negative value indicates no limit on the count.

Once these properties are set, the connector populates the log file as it carries out various tasks, such as when authentication is performed or queries are executed. If the specified file doesn't already exist, it is created.

Log Verbosity

The following list describes each level:

For normal operations, Verbosity should not be set to greater than 1. At higher verbosities you can log substantial amounts of data, which can delay execution times.

To refine the logged content further by showing/hiding specific categories of information, see LogModules.

Sensitive Data

• Verbosity 3: The full body of the request and the response, which includes all the data returned by the connector
• Verbosity 4: SSL certificates
• Verbosity 5: Any extra transfer data not included at Verbosity 3, such as non human-readable binary transfer data

Note: Although we mask sensitive values, such as passwords, in the connection string and any request in the log, it is always best practice to review the logs for any sensitive information before sharing outside your organization.

Advanced Logging

You may want to refine the exact information that is recorded to the log file. This can be accomplished using the LogModules property. This property allows you to filter the logging using a semicolon-separated list of logging modules.

Example property value:

LogModules=INFO;EXEC;SSL;SQL;META;

Note that the logfile filtering triggered by the Verbosity connection property takes precedence over the filtering imposed by this connection property. This means that operations of a higher verbosity level than the level specified in the Verbosity connection property are not printed in the logfile, even if they belong to one of the modules specified in this connection property.

The available modules and submodules are:

Exception Handling

Exceptions can be surfaced from either the API or the CData Python Connector for Elasticsearch. Each exception will have an error code, an error message, and a SQL state.

Error Codes

The error code classifies the type of error.

SQL State

The SQL state is used when throwing generic provider errors to the wrapper and indicates the success or failure of a call.

Some of the common SQL states are listed below:

Error Message

The error message provides more detailed reasoning about why the error occurred. It provides an explanation of the issue, and may include steps on how to resolve it.

The CData Python Connector for Elasticsearch supports several operations on data, including querying, deleting, modifying, and inserting.

SELECT Statements

See SELECT Statements for a syntax reference and examples.

See Data Model for information on the capabilities of the Elasticsearch API.

INSERT Statements

See INSERT Statements for a syntax reference and examples, as well as retrieving the new records' Ids.

UPDATE Statements

The primary key Id is required to update a record. See UPDATE Statements for a syntax reference and examples.

UPSERT Statements

An UPSERT updates a record if it exists and inserts the record if it does not. See UPSERT Statements for a syntax reference and examples.

DELETE Statements

The primary key Id is required to delete a record. See DELETE Statements for a syntax reference and examples.

CACHE Statements

CACHE statements allow granular control over the connector's caching functionality. For a syntax reference and examples, see CACHE Statements.

For more information on the caching feature, see Caching Data.

EXECUTE Statements

Use EXECUTE or EXEC statements to execute stored procedures. See EXECUTE Statements for a syntax reference and examples.

Names and Quoting

• Table and column names are considered identifier names; as such, they are restricted to the following characters: [A-Z, a-z, 0-9, _:@].
• To use a table or column name with characters not listed above, the name must be quoted using square brackets ([name]) in any SQL statement.
• Parameter names can optionally start with the @ symbol (e.g., @p1 or @CustomerName) and cannot be quoted.
• Strings must be quoted using single quotes (e.g., 'John Doe').

The connector provides functions that are similar to those that are available with most standard databases. These functions are implemented in the CData provider engine and thus are available across all data sources with the same consistent API. Three categories of functions are available: string, date, and math.

The connector interprets all SQL function inputs as either strings or column identifiers, so you need to escape all literals as strings, with single quotes. For example, contrast the SQL Server syntax and connector syntax for the DATENAME function:

• SQL Server:
  

  SELECT DATENAME(yy,GETDATE())

• connector:
  

  SELECT DATENAME('yy',GETDATE())

String Functions

These functions perform string manipulations and return a string value. See STRING Functions for more details.

Date Functions

These functions perform date and date time manipulations. See DATE Functions for more details.

Math Functions

These functions provide mathematical operations. See MATH Functions for more details.

Predicate Functions

These functions can be used to specify criteria in the WHERE clause of your SQL query. See Predicate Functions for more details.

ASCII(character_expression)

Returns the ASCII code value of the left-most character of the character expression.

• character_expression: The character expression.

                      SELECT ASCII('0');
                      --  Result: 48
                    

BASE64_ENCODE(input_binary)

Returns the Base64-encoded string form of a binary input.

• input_binary: The binary value to encode.

                        SELECT BASE64_ENCODE(BinaryData);
                    -- Result: 'QmFzZTY0RW5jb2RlZA=='
                    

BASE64_DECODE(input_string)

Returns the binary result of decoding a Base64-encoded string.

• input_string: The Base64-encoded string.

                        SELECT BASE64_DECODE('QmFzZTY0RW5jb2RlZA==');
                    -- Result: (binary output)
                    

CHAR(integer_expression)

Converts the integer ASCII code to the corresponding character.

• integer_expression: The integer from 0 through 255.

                      SELECT CHAR(48);
                      -- Result: '0'
                    

CHARINDEX(expressionToFind ,expressionToSearch [,start_location ])

Returns the starting position of the specified expression in the character string.

• expressionToFind: The character expression to find.
• expressionToSearch: The character expression, typically a column, to search.
• start_location: An optional character position to start searching for expressionToFind in expressionToSearch.

                      SELECT CHARINDEX('456', '0123456');
                      -- Result: 4

                      SELECT CHARINDEX('456', '0123456', 5);
                      -- Result: -1
                    

CHAR_LENGTH(character_expression),

Returns the number of UTF-8 characters present in the expression.

• character_expression: The set of characters to be evaluated for length.

				 SELECT CHAR_LENGTH('sample text') FROM Account LIMIT 1
				 -- Result: 11			
				

CONCAT(string_value1, string_value2, ..., string_valueN)

Returns the string that is the concatenation of two or more string values.

• string_value1: The first string to be concatenated.
• string_value2: The second string to be concatenated.
• string_valueN: (optional) Any additional strings to be concatenated.

                      SELECT CONCAT('Hello, ', 'world!');
                      -- Result: 'Hello, world!'
                    

CONTAINS(expressionToSearch, expressionToFind)

Returns 1 if expressionToFind is found within expressionToSearch; otherwise, 0.

• expressionToSearch: The character expression, typically a column, to search.
• expressionToFind: The character expression to find.

                      SELECT CONTAINS('0123456', '456');
                      -- Result: 1

                      SELECT CONTAINS('0123456', 'Not a number');
                      -- Result: 0
                    

ENDSWITH(character_expression, character_suffix)

Returns 1 if character_expression ends with character_suffix; otherwise, 0.

• character_expression: The character expression.
• character_suffix: The character suffix to search for.

                      SELECT ENDSWITH('0123456', '456');
                      -- Result: 1

                      SELECT ENDSWITH('0123456', '012');
                      -- Result: 0
                    

FILESIZE(uri)

Returns the number of bytes present in the file at the specified file path.

• uri: The path of the file from which to read the size.

				SELECT FILESIZE('C:/Users/User1/Desktop/myfile.txt');
				-- Result: 23684
				

FORMAT(value [, parseFormat], format )

Returns the value formatted with the specified format.

• value: The string to format.
• format: The string specifying the output syntax of the date or numeric format.
• parseFormat: The string specifying the input syntax of the date value. Not applicable to numeric types.

                      SELECT FORMAT(12.34, '#');
                      -- Result: 12

                      SELECT FORMAT(12.34, '#.###');
                      -- Result: 12.34

                      SELECT FORMAT(1234, '0.000E0');
                      -- Result: 1.234E3
                      
                      SELECT FORMAT('2019/01/01', 'yyyy-MM-dd');
                      -- Result: 2019-01-01
                      
                      SELECT FORMAT('20190101', 'yyyyMMdd', 'yyyy-MM-dd');
                      -- Result: '2019-01-01'
                    

HASHBYTES(algorithm, value)

Returns the hash of the input value as a byte array using the given algorithm. The supported algorithms are MD5, SHA1, SHA2_256, SHA2_512, SHA3_224, SHA3_256, SHA3_384, and SHA3_512.

• algorithm: The algorithm to use for hashing. Must be one of MD5, SHA1, SHA2_256, SHA2_512, SHA3_224, SHA3_256, SHA3_384, or SHA3_512.
• value: The value to hash. Must be either a string or byte array.

                      SELECT HASHBYTES('MD5', 'Test');
                      -- Result (byte array): 0x0CBC6611F5540BD0809A388DC95A615B
                    

INDEXOF(expressionToSearch, expressionToFind [,start_location ])

Returns the starting position of the specified expression in the character string.

• expressionToSearch: The character expression, typically a column, to search.
• expressionToFind: The character expression to find.
• start_location: An optional character position to start searching for expressionToFind in expressionToSearch.

                      SELECT INDEXOF('0123456', '456');
                      -- Result: 4

                      SELECT INDEXOF('0123456', '456', 5);
                      -- Result: -1
                    

ISALPHABETIC(character_expression)

Returns 1 if the character expression consists only of alphabetic characters; otherwise, 0.

• character_expression: The string expression to evaluate.

                      SELECT ISALPHABETIC('Hello');
                      -- Result: 1

                      SELECT ISALPHABETIC('Hello123');
                      -- Result: 0

                      SELECT ISALPHABETIC('Hello!');
                      -- Result: 0
                    

ISALPHANUMERIC(character_expression)

Returns 1 if the character expression consists only of alphabetic and numeric characters; otherwise, 0.

• character_expression: The string expression to evaluate.

                      SELECT ISALPHANUMERIC('Hello123');
                      -- Result: 1

                      SELECT ISALPHANUMERIC('123');
                      -- Result: 1

                      SELECT ISALPHANUMERIC('Hello.123');
                      -- Result: 0
                    

ISNUMERIC(character_expression)

Returns 1 if the character expression consists only of numeric digits and up to one decimal point; otherwise, 0.

• character_expression: The string expression to evaluate.

                      SELECT ISNUMERIC('123');
                      -- Result: 1

                      SELECT ISNUMERIC('123.45');
                      -- Result: 1

                      SELECT ISNUMERIC('123.45.67');
                      -- Result: 0

                      SELECT ISNUMERIC('12a3');
                      -- Result: 0
                    

JSON_EXTRACT(json, jsonpath)

Selects any value in a JSON array or object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to extract.
• jsonpath: The XPath used to select the nodes. The JSONPath must be a string constant. The values of the nodes selected will be returned in a token-separated list.

                      SELECT JSON_EXTRACT('{"test": {"data": 1}}', '$.test');
                      -- Result: '{"data":1}'

                      SELECT JSON_EXTRACT('{"test": {"data": 1}}', '$.test.data');
                      -- Result: 1

                      SELECT JSON_EXTRACT('{"test": {"data": [1, 2, 3]}}', '$.test.data[1]');
                      -- Result: 2
                    

LEFT ( character_expression , integer_expression )

Returns the specified number of characters counting from the left of the specified string.

• character_expression: The character expression.
• integer_expression: The positive integer that specifies how many characters will be returned counting from the left of character_expression.

                      SELECT LEFT('1234567890', 3);
                      -- Result: '123'
                    

LEN(string_expression)

Returns the number of characters of the specified string expression.

• string_expression: The string expression.

                      SELECT LEN('12345');
                      -- Result: 5
                    

LOCATE(substring,string)

Returns an integer representing how many characters into the string the substring appears.

• substring: The substring to find inside larger string.
• string: The larger string that is searched for the substring.
• start locations: An optional integer that sets the character position (offset) from which to start searching.

				SELECT LOCATE('sample','XXXXXsampleXXXXX');
				-- Result: 6

                SELECT LOCATE('sample', 'XXXXXsampleXXXXX', 7)
                -- Result: 0
				

LOWER ( character_expression )

Returns the character expression with the uppercase character data converted to lowercase.

• character_expression: The character expression.

                      SELECT LOWER('MIXED case');
                      -- Result: 'mixed case'
                    

LTRIM(character_expression)

Returns the character expression with leading blanks removed.

• character_expression: The character expression.

                      SELECT LTRIM('     trimmed');
                      -- Result: 'trimmed'
                    

MASK(string_expression, mask_character [, start_index [, end_index ]])

Replaces the characters between start_index and end_index with the mask_character within the string.

• string_expression: The string expression to be searched.
• mask_character: The character to mask with.
• start_index: The optional number of characters to leave unmasked at beginning of string. Defaults to 0.
• end_index: The optional number of characters to leave unmasked at end of string. Defaults to 0.

                        SELECT MASK('1234567890','*',);
                        -- Result: '**********'
                        SELECT MASK('1234567890','*', 4);
                        -- Result: '1234******'
                        SELECT MASK('1234567890','*', 4, 2);
                        -- Result: '1234****90'  
                    

NCHAR(integer_expression)

Returns the Unicode character with the specified integer code as defined by the Unicode standard.

• integer_expression: The integer from 0 through 65535 (0 through xFFFF).

OCTET_LENGTH(character_expression),

Returns the number of bytes present in the expression.

• character_expression: The set of characters to be be evaluated.

				 SELECT OCTET_LENGTH('text') FROM Account LIMIT 1
				 -- Result: 4
				

PATINDEX(pattern, expression)

Returns the starting position of the first occurrence of the pattern in the expression. Returns 0 if the pattern is not found.

• pattern: The character expression that contains the sequence to be found. The wild-card character % can be used only at the start or end of the expression.
• expression: The expression, typically a column, to search for the pattern.

                      SELECT PATINDEX('123%', '1234567890');
                      -- Result: 1

                      SELECT PATINDEX('%890', '1234567890');
                      -- Result: 8

                      SELECT PATINDEX('%456%', '1234567890');
                      -- Result: 4
                    

POSITION(expressionToFind IN expressionToSearch)

Returns the starting position of the specified expression in the character string.

• expressionToFind: The character expression to find.
• expressionToSearch: The character expression, typically a column, to search.

                      SELECT POSITION('456' IN '123456');
                      -- Result: 4

                      SELECT POSITION('x' IN '123456');
                      -- Result: 0
                    

QUOTENAME(character_string [, quote_character])

Returns a valid SQL Server-delimited identifier by adding the necessary delimiters to the specified Unicode string.

• character_string: The string of Unicode character data. The string is limited to 128 characters. Inputs greater than 128 characters return null.
• quote_character: An optional single character to be used as the delimiter. These include:
  • a single quotation mark (')
  • a left or right bracket ([])
  • a double quotation mark (")
  • a left or right parenthesis ( () )
  • a greater or less than sign (><)
  • a left or right brace ({})
  • a backtick (`)

  If quote_character is not specified brackets are used. If an unacceptable character is supplied, it returns NULL.

                      SELECT QUOTENAME('table_name');
                      -- Result: '[table_name]'

                      SELECT QUOTENAME('table_name', '"');
                      -- Result: '"table_name"'

                      SELECT QUOTENAME('table_name', '[');
                      -- Result: '[table_name]'
                    

REGEXP_REPLACE(expr, pattern [, replacement [, position [, occurrence [, match_type]]]])

Replaces occurrences of a regular expression pattern in the input string with a specified value and returns the resulting string.

• expr: The string expression to be searched.
• pattern: The regular expression pattern to match.
• replacement: (optional) The string to replace each matched occurrence of pattern with. Supports backreferences \1 through \9 and escape sequences \n, \r, \t, and \\. By default, this argument is an empty string, meaning matched portions are removed from the output string.
• position: (optional) The 1-based starting position used when searching for regular expression matches in expr. The default is 1. All characters prior to the starting position are included in the output string unaltered. Skipped characters are ignored when calculating regular expression matches, even if they match pattern.
• occurrence: (optional) Specifies whether all occurrences of pattern,, or only a specific occurrence of pattern are replaced. The default is 0, which means all occurrences of pattern are replaced with replacement. Set to 1 to only replace the first instance of the pattern; 2 to replace the second; etc.
• match_type: (optional) Modifiers used to customize matching behavior. Supported values are: 'c' (case-sensitive, default), 'i' (case-insensitive), 'm' (multiline), 'n' (dot matches newline), 'x' (extended mode). These can be freely combined by including the letters back to back. For example, 'im' applies the functionality of both 'i' and 'm'. The regular expression syntax used is that of the Extended mode ('x') ignores whitespace and allows inline comments. If the pattern needs to match a literal space, it must be explicitly escaped.

                      SELECT REGEXP_REPLACE('abc123def456', '\d+', 'NUM');
                      -- Result: 'abcNUMdefNUM'

                      SELECT REGEXP_REPLACE('Hello\nHELLO\nhello', '^hello', 'X', 1, 0, 'im');
                      -- Result: 'X\nX\nX'
                    

REPLACE(string_expression, string_pattern, string_replacement)

Replaces all occurrences of a string with another string.

• string_expression: The string expression to be searched. This can be a character or binary data type.
• string_pattern: The substring to be found. Cannot be an empty string.
• string_replacement: The replacement string.

                      SELECT REPLACE('1234567890', '456', '|');
                      -- Result: '123|7890'

                      SELECT REPLACE('123123123', '123', '.');
                      -- Result: '...'

                      SELECT REPLACE('1234567890', 'a', 'b');
                      -- Result: '1234567890'
                    

REPLICATE ( string_expression ,integer_expression )

Repeats the string value the specified number of times.

• string_expression: The string to replicate.
• integer_expression: The repeat count.

                      SELECT REPLACE('x', 5);
                      -- Result: 'xxxxx'
                    

REVERSE ( string_expression )

Returns the reverse order of the string expression.

• string_expression: The string.

                      SELECT REVERSE('1234567890');
                      -- Result: '0987654321'
                    

RIGHT ( character_expression , integer_expression )

Returns the right part of the string with the specified number of characters.

• character_expression: The character expression.
• integer_expression: The positive integer that specifies how many characters of the character expression will be returned.

                      SELECT RIGHT('1234567890', 3);
                      -- Result: '890'
                    

RTRIM(character_expression)

Returns the character expression after it removes trailing blanks.

• character_expression: The character expression.

                      SELECT RTRIM('trimmed     ');
                      -- Result: 'trimmed'
                    

SOUNDEX(character_expression)

Returns the four-character Soundex code, based on how the string sounds when spoken.

• character_expression: The alphanumeric expression of character data.

                      SELECT SOUNDEX('smith');
                      -- Result: 'S530'
                    

SPACE(repeatcount)

Returns the string that consists of repeated spaces.

• repeatcount: The number of spaces.

                      SELECT SPACE(5);
                      -- Result: '     '
                    

SPLIT(string, delimiter, offset)

Returns a section of the string between to delimiters.

• string: The string to split.
• delimiter: The character to split the string with.
• offset: The number of the split to return. Positive numbers are treated as offsets from the left, and negative numbers are treated as offsets from the right.

                      SELECT SPLIT('a/b/c/d', '/', 1);
                      -- Result: 'a'
                      SELECT SPLIT('a/b/c/d', '/', -2);
                      -- Result: 'c'
                    

STARTSWITH(character_expression, character_prefix)

Returns 1 if character_expression starts with character_prefix; otherwise, 0.

• character_expression: The character expression.
• character_prefix: The character prefix to search for.

                      SELECT STARTSWITH('0123456', '012');
                      -- Result: 1

                      SELECT STARTSWITH('0123456', '456');
                      -- Result: 0
                    

STR ( float_expression [ , integer_length [ , integer_decimal ] ] )

Returns the character data converted from the numeric data. For example, STR(123.45, 6, 1) returns 123.5.

• float_expression: The float expression.
• length: The optional total length to return. This includes decimal point, sign, digits, and spaces. The default is 10.
• decimal: The optional number of places to the right of the decimal point. The decimal must be less than or equal to 16.

                      SELECT STR('123.456');
                      -- Result: '123'

                      SELECT STR('123.456', 2);
                      -- Result: '**'

                      SELECT STR('123.456', 10, 2);
                      -- Result: '123.46'
                    

STUFF(character_expression , integer_start , integer_length , replaceWith_expression)

Inserts a string into another string. It deletes the specified length of characters in the first string at the start position and then inserts the second string into the first string at the start position.

• character_expression: The string expression.
• start: The integer value that specifies the location to start deletion and insertion. If start or length is negative, null is returned. If start is longer than the string to be modified, character_expression, null is returned.
• length: The integer that specifies the number of characters to delete. If length is longer than character_expression, deletion occurs up to the last character in replaceWith_expression.
• replaceWith_expression: The expression of character data that will replace length characters of character_expression beginning at the start value.

                      SELECT STUFF('1234567890', 3, 2, 'xx');
                      -- Result: '12xx567890'
                    

SUBSTRING(string_value FROM start FOR length)

Returns the part of the string with the specified length; starts at the specified index.

• string_value: The character string.
• start: The positive integer that specifies the start index of characters to return.
• length: Optional. The positive integer that specifies how many characters will be returned.

                      SELECT SUBSTRING('1234567890' FROM 3 FOR 2);
                      -- Result: '34'

                      SELECT SUBSTRING('1234567890' FROM 3);
                      -- Result: '34567890'
                    

                    SELECT SUBSTRING('1234567890', 3, 2)
                    --Result: '34'
                    SELECT SUBSTRING('1234567890', 3)
                    --Result: '34567890'
                    

TEXT_ENCODE(input_string, charset)

Returns binary output by encoding a string using the specified character set.

• input_string: The plain text string.
• charset: The character set to use, such as 'UTF-8', 'ISO-8859-1'.

                    SELECT TEXT_ENCODE('Café', 'UTF-8');
                    -- Result: (binary output)
                    

TEXT_DECODE(input_binary, charset)

Returns a string decoded from binary data using the specified character set.

• input_binary: The binary value to decode.
• charset: The character set used for decoding.

                    SELECT TEXT_DECODE(BinaryData, 'UTF-8');
                    -- Result: 'Café'
                    

TOSTRING(string_value1)

Converts the value of this instance to its equivalent string representation.

• string_value1: The string to be converted.

                      SELECT TOSTRING(123);
                      -- Result: '123'

                      SELECT TOSTRING(123.456);
                      -- Result: '123.456'

                      SELECT TOSTRING(null);
                      -- Result: ''
                    

TRIM(trimspec trimchar FROM string_value)

Returns the character expression with leading and/or trailing blanks removed.

• trimspec: Optional. If included must be one of the keywords BOTH, LEADING or TRAILING.
• trimchar: Optional. If included should be a one-character string value.
• string_value: The string value to trim.

                      SELECT TRIM('     trimmed     ');
                      -- Result: 'trimmed'

                      SELECT TRIM(LEADING FROM '     trimmed     ');
                      -- Result: 'trimmed     '

                      SELECT TRIM('-' FROM '-----trimmed-----');
                      -- Result: 'trimmed'

                      SELECT TRIM(BOTH '-' FROM '-----trimmed-----');
                      -- Result: 'trimmed'

                      SELECT TRIM(TRAILING '-' FROM '-----trimmed-----');
                      -- Result: '-----trimmed'
                    

UNICODE(ncharacter_expression)

Returns the integer value defined by the Unicode standard of the first character of the input expression.

• ncharacter_expression: The Unicode character expression.

UPPER ( character_expression )

Returns the character expression with lowercase character data converted to uppercase.

• character_expression: The character expression.

                      SELECT UPPER('MIXED case');
                      -- Result: 'MIXED CASE'
                    

XML_EXTRACT(xml, xpath [, separator])

Extracts an XML document using the specified XPath to flatten the XML. A comma is used to separate the outputs by default, but this can be changed by specifying the third parameter.

• xml: The XML document to extract.
• xpath: The XPath used to select the nodes. The nodes selected will be returned in a token-separated list.
• separator: The optional token used to separate the items in the flattened response. If this is not specified, the separator will be a comma.

                      SELECT XML_EXTRACT('<vowels><ch>a</ch><ch>e</ch><ch>i</ch><ch>o</ch><ch>u</ch></vowels>', '/vowels/ch');
                      -- Result: 'a,e,i,o,u'

                      SELECT XML_EXTRACT('<vowels><ch>a</ch><ch>e</ch><ch>i</ch><ch>o</ch><ch>u</ch></vowels>', '/vowels/ch', ';');
                      -- Result: 'a;e;i;o;u'
                    

ABS ( numeric_expression )

Returns the absolute (positive) value of the specified numeric expression.

• numeric_expression: The expression of an indeterminate numeric data type except for the bit data type.

                      SELECT ABS(15);
                      -- Result: 15

                      SELECT ABS(-15);
                      -- Result: 15
                    

ACOS ( float_expression )

Returns the arc cosine, the angle in radians whose cosine is the specified float expression.

• float_expression: The float expression that specifies the cosine of the angle to be returned. Values outside the range from -1 to 1 return null.

                      SELECT ACOS(0.5);
                      -- Result: 1.0471975511966
                    

ASIN ( float_expression )

Returns the arc sine, the angle in radians whose sine is the specified float expression.

• float_expression: The float expression that specifies the sine of the angle to be returned. Values outside the range from -1 to 1 return null.

                      SELECT ASIN(0.5);
                      -- Result: 0.523598775598299
                    

ATAN ( float_expression )

Returns the arc tangent, the angle in radians whose tangent is the specified float expression.

• float_expression: The float expression that specifies the tangent of the angle to be returned.

                      SELECT ATAN(10);
                      -- Result: 1.47112767430373
                    

ATN2 ( float_expression1 , float_expression2 )

Returns the angle in radians between the positive x-axis and the ray from the origin to the point (y, x) where x and y are the values of the two specified float expressions.

• float_expression1: The float expression that is the y-coordinate.
• float_expression2: The float expression that is the x-coordinate.

                      SELECT ATN2(1, 1);
                      -- Result: 0.785398163397448
                    

CEILING ( numeric_expression ) or CEIL( numeric_expression )

Returns the smallest integer greater than or equal to the specified numeric expression.

• numeric_expression: The expression of an indeterminate numeric data type except for the bit data type.

                      SELECT CEILING(1.3);
                      -- Result: 2

                      SELECT CEILING(1.5);
                      -- Result: 2

                      SELECT CEILING(1.7);
                      -- Result: 2
                    

COS ( float_expression )

Returns the trigonometric cosine of the specified angle in radians in the specified expression.

• float_expression: The float expression of the specified angle in radians.

                      SELECT COS(1);
                      -- Result: 0.54030230586814
                    

COT ( float_expression )

Returns the trigonometric cotangent of the angle in radians specified by float_expression.

• float_expression: The float expression of the angle in radians.

                      SELECT COT(1);
                      -- Result: 0.642092615934331
                    

DEGREES ( numeric_expression )

Returns the angle in degrees for the angle specified in radians.

• numeric_expression: The angle in radians, an expression of an indeterminate numeric data type except for the bit data type.

                      SELECT DEGREES(3.1415926);
                      -- Result: 179.999996929531
                    

EXP ( float_expression )

Returns the exponential value of the specified float expression. For example, EXP(LOG(20)) is 20.

• float_expression: The float expression.

                      SELECT EXP(2);
                      -- Result: 7.38905609893065
                    

EXPR ( expression )

Evaluates the expression.

• expression: The expression. Operators allowed are +, -, *, /, ==, !=, >, <, >=, and <=.

                      SELECT EXPR('1 + 2 * 3');
                      -- Result: 7

                      SELECT EXPR('1 + 2 * 3 == 7');
                      -- Result: true
                    

FLOOR ( numeric_expression )

Returns the largest integer less than or equal to the numeric expression.

• numeric_expression: The expression of an indeterminate numeric data type except for the bit data type.

                      SELECT FLOOR(1.3);
                      -- Result: 1

                      SELECT FLOOR(1.5);
                      -- Result: 1

                      SELECT FLOOR(1.7);
                      -- Result: 1
                    

GREATEST(int1,int2,....)

Returns the greatest of the supplied integers.

				SELECT GREATEST(3,5,8,10,1)
				-- Result: 10			
				

HEX(value)

Returns a the equivalent hex for the input value.

• value: A string or numerical value to be converted into hex.

				SELECT HEX(866849198);
				-- Result: 33AB11AE
				
				SELECT HEX('Sample Text');
				-- Result: 53616D706C652054657874
				

JSON_AVG(json, jsonpath)

Computes the average value of a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_AVG('[1,2,3,4,5]', '$[x]');
                      -- Result: 3

                      SELECT JSON_AVG('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 3

                      SELECT JSON_AVG('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 4.5
                    

JSON_COUNT(json, jsonpath)

Returns the number of elements in a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_COUNT('[1,2,3,4,5]', '$[x]');
                      -- Result: 5

                      SELECT JSON_COUNT('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 5

                      SELECT JSON_COUNT('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 2
                    

JSON_MAX(json, jsonpath)

Gets the maximum value in a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_MAX('[1,2,3,4,5]', '$[x]');
                      -- Result: 5

                      SELECT JSON_MAX('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 5

                      SELECT JSON_MAX('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[..3]');
                      -- Result: 4
                    

JSON_MIN(json, jsonpath)

Gets the minimum value in a JSON array within a JSON object. The path to the array is specified in the jsonpath argument. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_MIN('[1,2,3,4,5]', '$[x]');
                      -- Result: 1

                      SELECT JSON_MIN('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 1

                      SELECT JSON_MIN('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 4
                    

JSON_SUM(json, jsonpath)

Computes the summary value in JSON according to the JSONPath expression. Return value is numeric or null.

• json: The JSON document to compute.
• jsonpath: The JSONPath used to select the nodes. [x], [2..], [..8], or [1..12] are accepted. [x] selects all nodes.

                      SELECT JSON_SUM('[1,2,3,4,5]', '$[x]');
                      -- Result: 15

                      SELECT JSON_SUM('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[x]');
                      -- Result: 15

                      SELECT JSON_SUM('{"test": {"data": [1,2,3,4,5]}}', '$.test.data[3..]');
                      -- Result: 9
                    

LEAST(int1,int2,....)

Returns the least of the supplied integers.

				SELECT LEAST(3,5,8,10,1)
				-- Result: 1			
				

LOG ( float_expression [, base ] )

Returns the natural logarithm of the specified float expression.

• float_expression: The float expression.
• base: The optional integer argument that sets the base for the logarithm.

                      SELECT LOG(7.3890560);
                      -- Result: 1.99999998661119
                    

LOG10 ( float_expression )

Returns the base-10 logarithm of the specified float expression.

• float_expression: The expression of type float.

                      SELECT LOG10(10000);
                      -- Result: 4
                    

MOD(dividend,divisor)

Returns the integer value associated with the remainder when dividing the dividend by the divisor.

• dividend: The number to take the modulus of.
• divisor: The number to divide the dividend by when determining the modulus.

				SELECT MOD(10,3);
				-- Result: 1
				

NEGATE(real_number)

Returns the opposite to the real number input.

• real_number: The real number to find the opposite of.

				SELECT NEGATE(10);
				-- Result: -10
				
				SELECT NEGATE(-12.4)
				--Result: 12.4
				

PI ( )

Returns the constant value of pi.

                  SELECT PI()
                  -- Result: 3.14159265358979 
                

POWER ( float_expression , y )

Returns the value of the specified expression raised to the specified power.

• float_expression: The float expression.
• y: The power to raise float_expression to.

                      SELECT POWER(2, 10);
                      -- Result: 1024

                      SELECT POWER(2, -2);
                      -- Result: 0.25
                    

RADIANS ( float_expression )

Returns the angle in radians of the angle in degrees.

• float_expression: The degrees of the angle as a float expression.

                      SELECT RADIANS(180);
                      -- Result: 3.14159265358979
                    

RAND ( [ integer_seed ] )

Returns a pseudorandom float value from 0 through 1, exclusive.

• seed: The optional integer expression that specifies the seed value. If seed is not specified, a seed value at random will be assigned.

                      SELECT RAND();
                      -- This result may be different, since the seed is randomized
                      -- Result: 0.873159630165044

                      SELECT RAND(1);
                      -- This result will always be the same, since the seed is constant
                      -- Result: 0.248668584157093
                    

ROUND ( numeric_expression [ ,integer_length] [ ,function ] )

Returns the numeric value rounded to the specified length or precision.

• numeric_expression: The expression of a numeric data type.
• length: The optional precision to round the numeric expression to. When this is omitted, the default behavior will be to round to the nearest whole number.
• function: The optional type of operation to perform. When the function parameter is omitted or has a value of 0 (default), numeric_expression is rounded. When a value other than 0 is specified, numeric_expression is truncated.

                      SELECT ROUND(1.3, 0);
                      -- Result: 1

                      SELECT ROUND(1.55, 1);
                      -- Result: 1.6

                      SELECT ROUND(1.7, 0, 0);
                      -- Result: 2

                      SELECT ROUND(1.7, 0, 1);
                      -- Result: 1
                      
                      SELECT ROUND (1.24);
                      -- Result: 1.0
                    

SIGN ( numeric_expression )

Returns the positive sign (1), 0, or negative sign (-1) of the specified expression.

• numeric_expression: The expression of an indeterminate data type except for the bit data type.

                      SELECT SIGN(0);
                      -- Result: 0

                      SELECT SIGN(10);
                      -- Result: 1

                      SELECT SIGN(-10);
                      -- Result: -1
                    

SIN ( float_expression )

Returns the trigonometric sine of the angle in radians.

• float_expression: The float expression specifying the angle in radians.

                     SELECT SIN(1);
                     -- Result: 0.841470984807897
                    

SQRT ( float_expression )

Returns the square root of the specified float value.

• float_expression: The expression of type float.

                      SELECT SQRT(100);
                      -- Result: 10
                    

SQUARE ( float_expression )

Returns the square of the specified float value.

• float_expression: The expression of type float.

                      SELECT SQUARE(10);
                      -- Result: 100

                      SELECT SQUARE(-10);
                      -- Result: 100
                    

TAN ( float_expression )

Returns the tangent of the input expression.

• float_expression: The expression of type float.

                      SELECT TAN(1);
                      -- Result: 1.5574077246549
                    

TRUNC(decimal_number,precision)

Returns the supplied decimal number truncated to have the supplied decimal precision.

• decimal_number: The decimal value to truncate.
• precision: The number of decimal places to truncate the decimal number to.

				SELECT TRUNC(10.3423,2);
				-- Result: 10.34
				

_ROW_NUMBER_()

Returns a row index as an additional column.

				SELECT ColumnName, _ROW_NUMBER_() FROM TableName
				-- Result: ColumnData, 0
				ColumnData2, 1
				ColumnData3, 2
				

CURRENT_DATE()

Returns the current date value.

                  SELECT CURRENT_DATE();
                  -- Result: 2018-02-01
                

CURRENT_TIMESTAMP()

Returns the current time stamp of the database system as a datetime value. This value is equal to GETDATE and SYSDATETIME, and is always in the local timezone.

                  SELECT CURRENT_TIMESTAMP();
                  -- Result: 2018-02-01 03:04:05
                

DATEADD (datepart , integer_number , date [, dateformat])

Returns the datetime value that results from adding the specified number (a signed integer) to the specified date part of the date.

• datepart: The part of the date to add the specified number to. The valid values and abbreviations are
  • year (yy, yyyy)
  • quarter (qq, q)
  • month (mm, m)
  • week (wk, ww)
  • weekday (dw)
  • dayofyear (dy, y)
  • day (dd, d)
  • hour (hh)
  • minute (mi, n)
  • second (ss, s)
  • millisecond (ms)
• number: The number to be added.
• date: The expression of the datetime data type.
• dateformat: The optional output date format.

                  SELECT DATEADD('d', 5, '2018-02-01');
                  -- Result: 2018-02-06

                  SELECT DATEADD('hh', 5, '2018-02-01 00:00:00');
                  -- Result: 2018-02-01 05:00:00
                

DATEDIFF ( datepart , startdate , enddate )

Returns the difference (a signed integer) of the specified time interval between the specified start date and end date.

• datepart: The part of the date that is the time interval of the difference between the start date and end date. The valid values and abbreviations are:
  • Year (year, yyyy, yy)
  • Quarter (quarter, qq, q)
  • Month (month, mm, m)
  • Week (week, wk, ww)
  • Weekday (weekday, dw)
  • Dayofyear (dayofyear, dy, y)
  • Day (day, dd, d)
  • Hour (hour, hh)
  • Minute (minute, mi, n)
  • Second (second, ss, s)
  • Millisecond (millisecond, ms)
• startdate: The datetime expression of the start date.
• enddate: The datetime expression of the end date.

                  SELECT DATEDIFF('d', '2018-02-01', '2018-02-10');
                  -- Result: 9

                  SELECT DATEDIFF('hh', '2018-02-01 00:00:00', '2018-02-01 12:00:00');
                  -- Result: 12
                

DATE_FORMAT(date,format)

Returns the date or timestamp in the format specified. This function mirrors the MySQL DATE_FORMAT function.

• date: A date or timestamp string.
• format: The specifier string of the desired output format. The list of supported format specifiers comes from the MySQL DATE_FORMAT function (see link to MySQL documentation above).

					SELECT DATE_FORMAT('9/4/2021 3:11:53 AM','%h')
					-- Result: 03
				  

DATEFROMPARTS(integer_year, integer_month, integer_day)

Returns the datetime value for the specified year, month, and day.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.

                    SELECT DATEFROMPARTS(2018, 2, 1);
                    -- Result: 2018-02-01
                  

DATENAME(datepart , date)

Returns the character string that represents the specified date part of the specified date.

• datepart: The part of the date to return. The valid values and abbreviations are year (yy, yyyy), quarter (qq, q), month (mm, m), dayofyear (dy, y), day (dd, d), week (wk, ww), weekday (dw), hour (hh), minute (mi, n), second (ss, s), millisecond (ms), microsecond (mcs), and nanosecond (ns).
• date: The datetime expression.

                     SELECT DATENAME('yy', '2018-02-01');
                     -- Result: '2018'

                     SELECT DATENAME('dw', '2018-02-01');
                     -- Result: 'Thursday'
                   

DATEPART(datepart, date [,integer_datefirst])

Returns a character string that represents the specified date part of the specified date.

• datepart: The part of the date to return. The valid values and abbreviations are year (yy, yyyy), quarter (qq, q), month (mm, m), dayofyear (dy, y), day (dd, d), week (wk, ww), weekday (dw), hour (hh), minute (mi, n), second (ss, s), millisecond (ms), microsecond (mcs), nanosecond (ns), ISODOW, ISO_WEEK (isoweek, isowk,isoww), and ISOYEAR.
• date: The datetime string.
• datefirst: The optional integer representing the first day of the week. The default is 7, Sunday.

                    SELECT DATEPART('yy', '2018-02-01');
                    -- Result: 2018

                    SELECT DATEPART('dw', '2018-02-01');
                    -- Result: 5
                  

DATETIMEFROMPARTS(integer_year, integer_month, integer_day, integer_hour, integer_minute, integer_seconds, integer_milliseconds)

Returns the datetime value for the specified date parts.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.
• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.
• seconds: The integer expression specifying the seconds.
• milliseconds: The integer expression specifying the milliseconds.

                    SELECT DATETIMEFROMPARTS(2018, 2, 1, 1, 2, 3, 456);
                    -- Result: 2018-02-01 01:02:03.456
                  

DATETIME2FROMPARTS(integer_year, integer_month, integer_day, integer_hour, integer_minute, integer_seconds, integer_fractions, integer_precision)

Returns the datetime value for the specified date parts.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.
• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.
• seconds: The integer expression specifying the seconds.
• fractions: The integer expression specifying the fractions of the second.
• precision: The integer expression specifying the precision of the fraction.

				    SELECT DATETIME2FROMPARTS(2018, 2, 1, 1, 2, 3, 456, 3);
                    -- Result: 2018-02-01 01:02:03.456
                  

DATE_TRUNC(date, datepart)

Truncates the date to the precision of the given date part. Modeled after the Oracle TRUNC function.

• date: The datetime string that specifies the date.
• datepart: Refer to the Oracle documentation for valid datepart syntax.

				    SELECT DATE_TRUNC('05-04-2005', 'YY');
                    -- Result: '1/1/2005'
					
                    SELECT DATE_TRUNC('05-04-2005', 'MM');
                    -- Result: '5/1/2005'                    
                  

DATE_TRUNC2(datepart, date, [weekday])

Truncates the date to the precision of the given date part. Modeled after the PostgreSQL date_trunc function.

• datepart: One of 'millennium', 'century', 'decade', 'year', 'quarter', 'month', 'week', 'day', 'hour', 'minute' or 'second'.
• date: The datetime string that specifies the date.
• weekday: The optional day of the week to use as the first day for 'week'. One of 'sunday', 'monday', etc.

                    SELECT DATE_TRUNC2('year', '2020-02-04');
                    -- Result: '2020-01-01'

                    SELECT DATE_TRUNC2('week', '2020-02-04', 'monday');
                    -- Result: '2020-02-02', which is the previous Monday
                  

DAY(date)

Returns the integer that specifies the day component of the specified date.

• date: The datetime string that specifies the date.

                    SELECT DAY('2018-02-01');
                    -- Result: 1
                  

DAYNAME(date)

Returns the name of the day of the week of the specified date.

• date: The datetime string that specifies the date.

                    SELECT DAYNAME('8/18/2021');
                    -- Result: Wednesday
                  

DAYOFMONTH(date)

• date: The datetime string that specifies the date.

				  SELECT DAYOFMONTH('04/15/2000');
				  -- Result: 15
				  

DAYOFWEEK(date)

• date: The datetime string that specifies the date.

				  SELECT DAYOFWEEK('04/15/2000');
				  -- Result: 7
				  

DAYOFYEAR(date)

• date: The datetime string that specifies the date.

				  SELECT DAYOFYEAR('04/15/2000');
				  -- Result: 106
				  

EOMONTH(date [, integer_month_to_add ]) or LAST_DAY(date)

Returns the last day of the month that contains the specified date with an optional offset.

• date: The datetime expression specifying the date for which to return the last day of the month.
• integer_month_to_add: The optional integer expression specifying the number of months to add to the date before calculating the end of the month.

                  SELECT EOMONTH('2018-02-01');
                  -- Result: 2018-02-28
                  
                  SELECT LAST_DAY('2018-02-01');
                  -- Result: 2018-02-28

                  SELECT EOMONTH('2018-02-01', 2);
                  -- Result: 2018-04-30
                

EXTRACT(date_part FROM date_column_name)

Returns the last day of the month that contains the specified date with an optional offset.

• date_part: One of the following date components: YEAR, MONTH, DAY, HOUR, MINUTE, SECOND.
• date_column_name: The name of a date column in a table.

                  SELECT EXTRACT(YEAR FROM DateColumn)
                  -- Result: 2021
                

FDWEEK(date)

• date: The datetime string that specifies the date.
• weeks to add: An optional integer expression specifying the number of months to add to the date before calculating the first day of the week.

				  SELECT FDWEEK('02-08-2018');
				  -- Result: 2/4/2018

          SELECT FDWEEK('02-08-2018', 1)
          --Result: 02/11/2018
				  

FDMONTH(date)

• date: The datetime string that specifies the date.
• month to add: An optional integer expression specifying the number of months to add to the date before calculating the first day of the month.

				  SELECT FDMONTH('02-08-2018');
				  -- Result: 2/1/2018

          SELECT FDMONTH('02-08-2018', 1) 
          --Result: 03/01/2018
				  

FDQUARTER(date)

• date: The datetime string that specifies the date.
• quarters to add: An optional integer expression specifying the number of months to add to the date before calculating the first day of the quarter.

				  SELECT FDQUARTER('05-08-2018');
				  -- Result: 4/1/2018

          SELECT FDQUARTER('05-08-2018',1)
          --Result: 07/01/2018
				  

FILEMODIFIEDTIME(uri)

Returns the time stamp associated with the Date Modified of the relevant file.

• uri: An absolute path pointing to a file on the local file system.

				 SELECT FILEMODIFIEDTIME('C:/Documents/myfile.txt');
				 -- Result: 6/25/2019 10:06:58 AM
				 

FROM_DAYS(datevalue)

Returns a date derived from the number of days after 1582-10-15 (based upon the Gregorian calendar). This will be equivalent to the MYSQL FROM_DAYS function.

• datevalue: A integer value representing the number of days since 1582-10-15.

				SELECT FROM_DAYS(736000);
				-- Result: 2/6/2015
				

FROM_UNIXTIME(time, issecond)

Returns a representation of the unix_timestamp argument as a value in YYYY-MM-DD HH:MM:SS expressed in the current time zone.

• time: The time stamp value from epoch time. Milliseconds are accepted.
• issecond: Indicates the time stamp value is milliseconds to epoch time.

                      SELECT FROM_UNIXTIME(1540495231, 1);
                      -- Result: 2018-10-25 19:20:31

                      SELECT FROM_UNIXTIME(1540495357385, 0);
                      -- Result: 2018-10-25 19:22:37
                    

GETDATE()

Returns the current time stamp of the database system as a datetime value. This value is equal to CURRENT_TIMESTAMP and SYSDATETIME, and is always in the local timezone.

                  SELECT GETDATE();
                  -- Result: 2018-02-01 03:04:05
                

GETUTCDATE()

Returns the current time stamp of the database system formatted as a UTC datetime value. This value is equal to SYSUTCDATETIME.

In addition, GETUTCDATE can take an optional second parameter, a date and time that are converted to UTC.

                  SELECT GETUTCDATE();
                  -- For example, if the local timezone is Eastern European Time (GMT+2)
                  -- Result: 2018-02-01 05:04:05

                  SELECT GETUTCDATE('2020/08/31 13:56:00')
                  --Result: '2020-08-31 17:56:00'
                

HOUR(date)

Returns the hour component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT HOUR('02-02-2020 11:30:00');
				-- Result: 11
				

ISDATE(date, [date_format])

Returns 1 if the value is a valid date, time, or datetime value; otherwise, 0.

• date: The datetime string.
• date_format: The optional datetime format.

                      SELECT ISDATE('2018-02-01', 'yyyy-MM-dd');
                      -- Result: 1

                      SELECT ISDATE('Not a date');
                      -- Result: 0
                    

LAST_WEEK()

Returns a time stamp equivalent to exactly one week before the current date.

				SELECT LAST_WEEK();	//Assume the date is 3/17/2020	
			 -- Result: 3/10/2020 00:00:00
				

LAST_MONTH()

Returns a time stamp equivalent to exactly one month before the current date.

	
				SELECT LAST_MONTH(); //Assume the date is 3/17/2020
				-- Result: 2/17/2020 00:00:00
				

LAST_YEAR()

Returns a time stamp equivalent to exactly one year before the current date.

				SELECT LAST_YEAR();	//Assume the date is 3/17/2020	
				-- Result: 3/10/2019 00:00:00
				

LDWEEK(date)

Returns the last day of the provided week.

• date: The datetime string.
• weeks to add: An optional integer expression specifying the number of months to add to the date before calculating the last day of the week.

				SELECT LDWEEK('02-02-2020');
				-- Result: 2/8/2020
				

LDMONTH(date)

Returns the last day of the provided month.

• date: The datetime string.
• months to add: An optional integer expression specifying the number of months to add to the date before calculating the last day of the month.

				SELECT LDMONTH('02-02-2020');
				-- Result: 2/29/2020

        SELECT LDMONTH('02-08-2020', 1)
        --Result: 03/31/2020
				

LDQUARTER(date)

Returns the last day of the provided quarter.

• date: The datetime string.
• quarters to add: An optional integer expression specifying the number of months to add to the date before calculating the last day of the quarter.

				SELECT LDQUARTER('02-02-2020');
				-- Result: 3/31/2020

        SELECT LDQUARTER('02-02-2020',1)
        --Result: 06/30/2020
				

MAKEDATE(year, days)

Returns a date value from a year and a number of days.

• year: The year
• days: The number of days into the year. Value must be greater than 0.

          SELECT MAKEDATE(2020, 1);
          -- Result: 2020-01-01
        

MINUTE(date)

Returns the minute component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT MINUTE('02-02-2020 11:15:00');
				-- Result: 15
				

MONTH(date)

Returns the month component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT MONTH('02-02-2020');
				-- Result: 2
				

QUARTER(date)

Returns the quarter associated with the provided datetime.

• date: The datetime string that specifies the date.

				SELECT QUARTER('02-02-2020');
				-- Result: 1
				

SECOND(date)

Returns the second component from the provided datetime.

• date: The datetime string that specifies the date.

				SELECT SECOND('02-02-2020 11:15:23');
				-- Result: 23
				

SMALLDATETIMEFROMPARTS(integer_year, integer_month, integer_day, integer_hour, integer_minute)

Returns the datetime value for the specified date and time.

• year: The integer expression specifying the year.
• month: The integer expression specifying the month.
• day: The integer expression specifying the day.
• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.

                      SELECT SMALLDATETIMEFROMPARTS(2018, 2, 1, 1, 2);
                      -- Result: 2018-02-01 01:02:00
                    

STRTODATE(string,format)

Parses the provided string value and returns the corresponding datetime.

• string: The string value to be converted to datetime format.
• format: A format string which describes how to interpret the first string input. A few special formats are available as well, including UNIX, UNIXMILIS, TICKS, and FILETICKS.

				SELECT STRTODATE('03*04*2020','dd*MM*yyyy');
				-- Result: 4/3/2020
				

SYSDATETIME()

Returns the current time stamp as a datetime value of the database system. It is equal to GETDATE and CURRENT_TIMESTAMP, and is always in the local timezone.

                  SELECT SYSDATETIME();
                  -- Result: 2018-02-01 03:04:05
                

SYSUTCDATETIME()

Returns the current system date and time as a UTC datetime value. It is equal to GETUTCDATE.

                  SELECT SYSUTCDATETIME();
                  -- For example, if the local timezone is Eastern European Time (GMT+2)
                  -- Result: 2018-02-01 05:04:05
                

TIMEFROMPARTS(integer_hour, integer_minute, integer_seconds, integer_fractions, integer_precision)

Returns the time value for the specified time and with the specified precision.

• hour: The integer expression specifying the hour.
• minute: The integer expression specifying the minute.
• seconds: The integer expression specifying the seconds.
• fractions: The integer expression specifying the fractions of the second.
• precision : The integer expression specifying the precision of the fraction.

                      SELECT TIMEFROMPARTS(1, 2, 3, 456, 3);
                      -- Result: 01:02:03.456
                    

TO_DAYS(date)

Returns the number of days since 0000-00-01. This will only return a value for dates on or after 1582-10-15 (based upon the Gregorian calendar). This will be equivalent to the MYSQL TO_DAYS function.

• date: The datetime string that specifies the date.

				SELECT TO_DAYS('02-06-2015');
				-- Result: 736000
				

WEEK(date)

Returns the week (of the year) associated with the provided datetime.

• date: The datetime string that specifies the date.

				SELECT WEEK('02-17-2020 11:15:23');
				-- Result: 8
				

YEAR(date)

Returns the integer that specifies the year of the specified date.

• date: The datetime string.

                      SELECT YEAR('2018-02-01');
                      -- Result: 2018
                    

The following date literal functions can be used to filter date fields using relative intervals. Note that while the <, >, and = operators are supported for these functions, <= and >= are not.

L_TODAY()

The current day.

  SELECT * FROM MyTable WHERE MyDateField = L_TODAY()

L_YESTERDAY()

The previous day.

  SELECT * FROM MyTable WHERE MyDateField = L_YESTERDAY()

L_TOMORROW()

The following day.

  SELECT * FROM MyTable WHERE MyDateField = L_TOMORROW()

L_LAST_WEEK()

Every day in the preceding week.

  SELECT * FROM MyTable WHERE MyDateField = L_LAST_WEEK()

L_THIS_WEEK()

Every day in the current week.

  SELECT * FROM MyTable WHERE MyDateField = L_THIS_WEEK()

L_NEXT_WEEK()

Every day in the following week.

  SELECT * FROM MyTable WHERE MyDateField = L_NEXT_WEEK()

• L_LAST/L_THIS/L_NEXT MONTH
• L_LAST/L_THIS/L_NEXT QUARTER
• L_LAST/L_THIS/L_NEXT YEAR

L_LAST_N_DAYS(n)

The previous n days, excluding the current day.

  SELECT * FROM MyTable WHERE MyDateField = L_LAST_N_DAYS(3)

L_NEXT_N_DAYS(n)

The following n days, including the current day.

  SELECT * FROM MyTable WHERE MyDateField = L_NEXT_N_DAYS(3)

• L_LAST/L_NEXT_90_DAYS

L_LAST_N_WEEKS(n)

Every day in every week, starting n weeks before current week, and ending in the previous week.

  SELECT * FROM MyTable WHERE MyDateField = L_LAST_N_WEEKS(3)

L_NEXT_N_WEEKS(n)

Every day in every week, starting the following week, and ending n weeks in the future.

  SELECT * FROM MyTable WHERE MyDateField = L_NEXT_N_WEEKS(3)

• L_LAST/L_NEXT_N_MONTHS(n)
• L_LAST/L_NEXT_N_QUARTERS(n)
• L_LAST/L_NEXT_N_YEARS(n)

COMMON(expression, cutoff_frequency)

Used to explicitly specify the query type to send and thus will send 'expression' in a common terms query.

Example SQL Query:

SELECT * FROM employee WHERE COMMON(about) = 'like to build' 

{"common":{"about":{"query":"like to build"}}}

• expression: The expression to search for.
• cutoff_frequency: The cutoff frequency value used to allocate terms to the high or low frequency group. Can be an absolute frequency (>=1) or a relative frequency (0.0 .. 1.0).

FILTER(expression)

Used to explicitly specify the filter context and thus will send 'expression' in a filter context, rather than a query context. A filter context does not affect the calculated scores. This is useful when performing queries where you want part of the filter to be used to calculate scores but filter the results returned (without affecting the score) using additional criteria.

Example SQL Query:

SELECT * FROM employee WHERE FILTER(TERM(first_name)) = 'john' 

{"filter":{"bool":{"must":{"term":{"first_name":"john"}}}}}

• expression: Either a column or another function.

GEO_BOUNDING_BOX(column, top_left, bottom_right)

Used to specify a query to filter hits based on a point location using a bounding box.

Example SQL Query:

SELECT * FROM cities WHERE GEO_BOUNDING_BOX(location, '[-74.1,40.73]', '[-71.12,40.01]') 

{"bool":{"filter":{"geo_bounding_box":{"location":{"top_left":[-74.1,40.73],"bottom_right":[-71.12,40.01]}}},"must":[{"match_all":{}}]}}

• column: A Geo-point column to perform the GEO_BOUNDING_BOX filter on.
• top_left: The top-left coordinates of the bounding box. This value can be an array [shown in example], object of lat and lon values, comma-separated list, or a geohash of a latitude and longitude value.
• bottom_right: The bottom-right coordinates of the bounding box. This value can be an array [shown in example], object of lat and lon values, comma-separated list, or a geohash of a latitude and longitude value.

GEO_BOUNDING_BOX(column, top, left, bottom, right)

Used to specify a query to filter hits based on a point location using a bounding box.

Example SQL Query:

SELECT * FROM cities WHERE GEO_BOUNDING_BOX(location, -74.1, 40.73, -71.12, 40.01) 

{"bool":{"filter":{"geo_bounding_box":{"location":{"top":-74.1,"left":40.73,"bottom":-71.12,"right":40.01}}},"must":[{"match_all":{}}]}}

• column: A Geo-point column to perform the GEO_BOUNDING_BOX filter on.
• top: The top coordinate of the bounding box.
• left: The left coordinate of the bounding box.
• bottom: The bottom coordinate of the bounding box.
• right: The right coordinate of the bounding box.

GEO_DISTANCE(column, point_lat_lon, distance)

Used to specify a query to filter documents that include only the hits that exist within a specific distance from a geo point.

Example SQL Query:

SELECT * FROM cities WHERE GEO_DISTANCE(location, '40,-70', '12mi') 

{"bool":{"filter":{"geo_distance":{"location":"40,-70","distance":"12mi"}},"must":[{"match_all":{}}]}}

• column: A Geo-point column to perform the GEO_DISTANCE filter on.
• point_lat_lon: The coordinates of a geo point that will be used to measure the distance from. This value can be an array, object of lat and lon values, comma-separated list [shown in example], or a geohash of a latitude and longitude value.
• distance: The distance to search within from the specified geo point. This value takes an numeric value along with a distance unit. Common distance units are: mi (miles), yd (yards), ft (feet), in (inch), km (kilometers), m (meters). Please see Elastic documentation for complete list of distance units.

GEO_DISTANCE_RANGE(column, point_lat_lon, from_distance, to_distance)

Used to specify a query to filter documents that include only the hits that exist within a range from a specific geo point.

Example SQL Query:

SELECT * FROM cities WHERE GEO_DISTANCE_RANGE(location, 'drn5x1g8cu2y', '10mi', '20mi') 

{"bool":{"filter":{"geo_distance_range":{"location":"drn5x1g8cu2y","from":"10mi","to":"20mi"}},"must":[{"match_all":{}}]}}

• column: A Geo-point column to perform the GEO_DISTANCE_RANGE filter on.
• point_lat_lon: The coordinates of a geo point that will be used to measure the range from. This value can be an array, object of lat and lon values, comma-separated list, or a geohash [shown in example] of a latitude and longitude value.
• from_distance: The starting distance to calculate the range from the specified geo point. This value takes an numeric value along with a distance unit. Common distance units are: mi (miles), yd (yards), ft (feet), in (inch), km (kilometers), m (meters). Please see Elastic documentation for complete list of distance units.
• to_distance: The end distance to calculate the range from the specified geo point. This value takes an numeric value along with a distance unit. Common distance units are: mi (miles), yd (yards), ft (feet), in (inch), km (kilometers), m (meters). Please see Elastic documentation for complete list of distance units.

GEO_POLYGON(column, points)

Used to specify a query to filter hits that only fall within a polygon of points.

Example SQL Query:

SELECT * FROM cities WHERE GEO_POLYGON(location, '[{"lat":40,"lon":-70},{"lat":30,"lon":-80},{"lat":20,"lon":-90}]') 

{"bool":{"filter":{"geo_polygon":{"location":{"points":[{"lat":40,"lon":-70},{"lat":30,"lon":-80},{"lat":20,"lon":-90}]}}},"must":[{"match_all":{}}]}}

• column: A Geo-point column to perform the GEO_POLYGON filter on.
• points: A JSON array of points that make up a polygon. This value can be an array of arrays, object of lat and lon values [shown in example], comma-separated lists, or geohashes of a latitude and longitude value.

GEO_SHAPE(column, type, points [, relation])

Used to specify an inline shape query to filter documents using the geo_shape type to find documents that have a shape that intersects with the query shape.

Example SQL Query:

SELECT * FROM shapes WHERE GEO_SHAPE(my_shape, 'envelope', '[[13.0, 53.0], [14.0, 52.0]] 

{"bool":{"filter":{"geo_shape":{"my_shape":{"shape":{"type":"envelope","coordinates":[[13.0, 53.0], [14.0, 52.0]]}}}},"must":[{"match_all":{}}]}}

• column: A Geo-shape column to perform the GEO_SHAPE filter on.
• type: The type of shape to search for. Valid values: point, linestring, polygon, multipoint, multilinestring, multipolygon, geometrycollection, envelope, and circle. Please see Elastic documentation for further information regarding these shapes.
• points: The coordinates for the shape type specified. These coordinates and their structure will vary depending upon the shape type desired. Please see Elastic search documentation for further details.
• relation: The name of the spatial relation operator to use at search time. Valid values: intersects (default), disjoint, within, and contains. Please see Elastic documentation for further information regarding spatial relations.

INARRAY(column)

Used to search for values contained within a primitive array. Supports comparison operators based on the data type contained within the array, including the LIKE operator.

Example SQL Query:

SELECT * FROM employee WHERE INARRAY(skills) = 'coding' 

• column: A primitive array column to filter on.

MATCH(column)

Used to explicitly specify the query type to send and thus will send 'column' in a match query.

Example SQL Query:

SELECT * FROM employee WHERE MATCH(last_name) = 'SMITH' 

{"match":{"last_name":"SMITH"}}

• column: A column to perform the match query on.

MATCH_PHRASE(column)

Used to explicitly specify the query type to send and thus will send 'column' in a match phrase query.

Example SQL Query:

SELECT * FROM employee WHERE MATCH_PHRASE(about) = 'rides motorbikes' 

{"match_phrase":{"about":"rides motorbikes"}}

• column: A column to perform the match phrase query on.

MATCH_PHRASE_PREFIX(column)

Used to explicitly specify the query type to send and thus will send 'column' in a match phrase prefix query. The match phrase prefix query is the same as a match query except that it allows for prefix matches on the last term in the text.

Example SQL Query:

SELECT * FROM employee WHERE MATCH_PHRASE_PREFIX(about) = 'quick brown f' 

{"match_phrase_prefix":{"about":"quick brown f"}}

• expression: A column to perform the match phrase prefix query on.

TERM(column)

Used to explicitly specify the query type to send and thus will send 'column' in a term query.

Example SQL Query:

SELECT * FROM employee WHERE TERM(last_name) = 'jacobs' 

{"term":{"last_name":"jacobs"}}

• column: A column to perform the term query on.

DSLQuery([table,] dsl_json)

Used to explicitly specify the Elasticsearch DSL query to send in the request. Can be used along with other filters and the AND and OR operators.

DSL query JSON can contain a full 'bool' query object, a 'must', 'should', 'must_not', or 'filter' occurrence type, or just a clause object (which will append to a 'must' (default) or 'should' occurrence type depending on whether an AND or OR operator is used).

Example SQL Query (These examples generate the same query using a 'bool' object, 'must' occurrence type, and query object):

SELECT * FROM employee WHERE DSLQuery('{"bool":{"must":[{"query_string":{"default_field":"last_name","query":"\\"Smith\\""}}]}}')
SELECT * FROM employee WHERE DSLQuery('{"must":[{"query_string":{"default_field":"last_name","query":"\\"Smith\\""}}]}')
SELECT * FROM employee WHERE DSLQuery('{"query_string":{"default_field":"last_name","query":"\\"Smith\\""}}') 

{"bool":{"must":[{"query_string":{"default_field":"last_name","query":"\\"Smith\\""}}]}}

Example SQL Query (with OR operator):

SELECT * FROM employee WHERE Age < 10 OR DSLQuery('{"should":[{"query_string":{"default_field":"last_name","query":"\"Smith\""}}]}') 

{"bool":{"should":[{"range":{"age":{"lt":10}}},{"query_string":{"default_field":"last_name","query":"\"Smith\""}}]}}

Additionally you can specify the table that you want the DSLQuery to be issued on, this is useful when executing queries against multiple tables such as JOIN queries.

Example SQL Query:

SELECT * FROM employee JOIN job ON employee.jobid = job.id WHERE DSLQuery(employee, '{"bool":{"must":[{"query_string":{"default_field":"last_name","query":"\\"Smith\\""}}]}}') 

• column: A column to perform the term query on.

A SELECT statement can consist of the following basic clauses.

• SELECT
• INTO
• FROM
• JOIN
• WHERE
• GROUP BY
• HAVING
• UNION
• ORDER BY
• LIMIT

SELECT Syntax

The following syntax diagram outlines the syntax supported by the SQL engine of the connector:

SELECT {
  [ TOP <numeric_literal> | DISTINCT ]
  { 
    * 
    | { 
        <expression> [ [ AS ] <column_reference> ] 
        | { <table_name> | <correlation_name> } .* 
      } [ , ... ] 
  }
  { 
    FROM <table_reference> [ [ AS ] <identifier> ] 
  } [ , ... ]
  [ [  
      INNER | { { LEFT | RIGHT | FULL } [ OUTER ] } 
    ] JOIN <table_reference> [ ON <search_condition> ] [ [ AS ] <identifier> ] 
  ] [ ... ] 
  [ WHERE <search_condition> ]
  [ GROUP BY <column_reference> [ , ... ]
  [ HAVING <search_condition> ]
  [ UNION [ ALL ] <select_statement> ]
  [ 
    ORDER BY 
    <column_reference> [ ASC | DESC ] [ NULLS FIRST | NULLS LAST ]
  ]
  [ 
    LIMIT <expression>
    [ 
      { OFFSET | , }
      <expression> 
    ]
  ] 
} | SCOPE_IDENTITY() 

<expression> ::=
  | <column_reference>
  | @ <parameter> 
  | ?
  | COUNT( * | { [ DISTINCT ] <expression> } )
  | { AVG | MAX | MIN | SUM | COUNT } ( <expression> ) 
  | NULLIF ( <expression> , <expression> ) 
  | COALESCE ( <expression> , ... ) 
  | CASE <expression>
      WHEN { <expression> | <search_condition> } THEN { <expression> | NULL } [ ... ]
    [ ELSE { <expression> | NULL } ]
    END 
  | {RANK() | DENSE_RANK()} OVER ([PARTITION BY <column_reference>] {ORDER BY <column_reference>})
  | <literal>
  | <sql_function> 

<search_condition> ::= 
  {
    <expression> { = | > | < | >= | <= | <> | != | LIKE | NOT LIKE | IN | NOT IN | IS NULL | IS NOT NULL | AND | OR | CONTAINS | BETWEEN | IS DISTINCT FROM | IS NOT DISTINCT FROM } [ <expression> ]
  } [ { AND | OR } ... ] 

Examples

1. Return all columns:
   

   SELECT * FROM [CData].[Elasticsearch].Employee

2. Rename a column:
   

   SELECT [Name] AS MY_Name FROM [CData].[Elasticsearch].Employee

3. Cast a column's data as a different data type:
   

   SELECT CAST(AnnualRevenue AS VARCHAR) AS Str_AnnualRevenue FROM [CData].[Elasticsearch].Employee

4. Search data:
   

   SELECT * FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'

5. Return the number of items matching the query criteria:
   

   SELECT COUNT(*) AS MyCount FROM [CData].[Elasticsearch].Employee 

6. Return the number of unique items matching the query criteria:
   

   SELECT COUNT(DISTINCT Name) FROM [CData].[Elasticsearch].Employee 

7. Return the unique items matching the query criteria:
   

   SELECT DISTINCT Name FROM [CData].[Elasticsearch].Employee 

8. Sort a result set in ascending order:
   

   SELECT Id, Name FROM [CData].[Elasticsearch].Employee  ORDER BY Name ASC

9. Restrict a result set to the specified number of rows:
   

   SELECT Id, Name FROM [CData].[Elasticsearch].Employee LIMIT 10 

10. Parameterize a query to pass in inputs at execution time. This enables you to create prepared statements and mitigate SQL injection attacks.
    

    SELECT * FROM [CData].[Elasticsearch].Employee WHERE Industry = @param

Pseudo Columns

Some input-only fields are available in SELECT statements. These fields, called pseudo columns, do not appear as regular columns in the results, yet may be specified as part of the WHERE clause. You can use pseudo columns to access additional features from Elasticsearch.

    SELECT * FROM [CData].[Elasticsearch].Employee WHERE Pseudo = '@Pseudo'
    

Aggregate Functions

For SELECT examples using aggregate functions, see Aggregate Functions.

JOIN Queries

See JOIN Queries for SELECT query examples using JOINs.

Date Literal Functions

Date Literal Functions contains SELECT examples with date literal functions.

Predicate Functions

For SELECT examples using predicate functions, see Predicate Functions.

COUNT

Returns the number of rows matching the query criteria.

SELECT COUNT(*) FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'

COUNT(DISTINCT)

Returns the number of distinct, non-null field values matching the query criteria.

SELECT COUNT(DISTINCT Id) AS DistinctValues FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'

AVG

Returns the average of the column values.

SELECT Name, AVG(AnnualRevenue) FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'  GROUP BY Name

MIN

Returns the minimum column value.

SELECT MIN(AnnualRevenue), Name FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks' GROUP BY Name

MAX

Returns the maximum column value.

SELECT Name, MAX(AnnualRevenue) FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks' GROUP BY Name

SUM

Returns the total sum of the column values.

SELECT SUM(AnnualRevenue) FROM [CData].[Elasticsearch].Employee WHERE Industry = 'Floppy Disks'

The CData Python Connector for Elasticsearch supports standard SQL joins like the following examples.

Inner Join

An inner join selects only rows from both tables that match the join condition:

SELECT Customer.FirstName, Customer.LastName, Purchases.ItemName FROM Purchases INNER JOIN Customer ON Purchases.CustomerId = Customer.Id

Left Join

A left join selects all rows in the FROM table and only matching rows in the JOIN table:

 SELECT Customer.FirstName, Customer.LastName, Purchases.ItemName FROM Purchases LEFT JOIN Customer ON Purchases.CustomerId = Customer.Id

MAPFIELD(column, data_type)

Used to explicitly specify a mapping (by sending the 'unmapped_type' sort option) for a column that does not have a mapping associated with it, which will enable sorting on the column. By default, if a column does not have a mapping, an exception will be thrown containing an error message similar to: "No mapping found for [column] in order to sort on".

Example SQL Query:

SELECT * FROM employee ORDER BY MAPFIELD(start_date, 'long') DESC 

{"start_date":{"order":"desc", "unmapped_type": "long"}}

• column: The column to perform the order by on.
• data_type: The Elasticsearch data type to map the column to.

To create new records, use INSERT statements.

INSERT Syntax

The INSERT statement specifies the columns to be inserted and the new column values. You can specify the column values in a comma-separated list in the VALUES clause, as shown in the following example:

INSERT INTO <table_name> 
( <column_reference> [ , ... ] )
VALUES 
( { <expression> | NULL } [ , ... ] ) 
  

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

INSERT INTO [CData].[Elasticsearch].Employee (Name) VALUES ('Floppy Disks')

To modify existing records, use UPDATE statements.

Update Syntax

The UPDATE statement takes as input a comma-separated list of columns and new column values as name-value pairs in the SET clause, as shown in the following example:

UPDATE <table_name> SET <select_statement> | {<column_reference> = <expression> [ , ... ]} WHERE { Id = <expression>  } [ { AND | OR } ... ] 

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

The following is an example query:

UPDATE [CData].[Elasticsearch].Employee SET Name='Floppy Disks' WHERE Id = @myId

An UPSERT statement updates an existing record or creates a new record if an existing record is not identified.

UPSERT Syntax

The UPSERT syntax is the same as for INSERT. Elasticsearch uses the input provided in the VALUES clause to determine whether the record already exists. If the record does not exist, all columns required to insert the record must be specified. See Data Model for any table-specific information.

UPSERT INTO <table_name> 
( <column_reference> [ , ... ] )
VALUES 
( { <expression> | NULL } [ , ... ] ) 
  

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

The following is an example query:

UPSERT INTO [CData].[Elasticsearch].Employee (Name) VALUES ('Floppy Disks')

To delete information from a table, use DELETE statements.

DELETE Syntax

The DELETE statement requires the table name in the FROM clause and the row's primary key in the WHERE clause, as shown in the following example:

<delete_statement> ::= DELETE FROM <table_name> WHERE { Id = <expression> } [ { AND | OR } ... ]

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

The following is an example query:

DELETE FROM [CData].[Elasticsearch].Employee WHERE Id = @myId

When caching is enabled, CACHE statements provide complete control over the data that is cached and the table to which it is cached. The CACHE statement executes the SELECT statement specified and caches its results to a table with the same name in the cache database or to table specified in <cached_table_name>. The connector updates or inserts rows to the cache depending on whether or not they already exist in the cache, so the primary key, which is used to identify existing rows, must be included in the selected columns.

See Caching Data for more information on different caching strategies.

CACHE Statement Syntax

The cache statement may include the following options that alter its behavior:

CACHE [ <cached_table_name> ] [ WITH TRUNCATE | AUTOCOMMIT | SCHEMA ONLY | DROP EXISTING | ALTER SCHEMA ] <select_statement> 

WITH TRUNCATE

If this option is set, the connector removes existing rows in the cache table before adding the selected rows. Use this option if you want to refresh the entire cache table but keep its existing schema.

AUTOCOMMIT

If this option is set, the connector commits each row individually. Use this option if you want to ignore the rows that could not be cached due to some reason. By default, the entire result set is cached as a single transaction.

DROP EXISTING

If this option is set, the connector drops the existing cache table before caching the new results. Use this option if you want to refresh the entire cache table, including its schema.

SCHEMA ONLY

If this option is set, the connector creates the cache table based on the SELECT statement without executing the query.

ALTER SCHEMA

If this option is set, the connector alters the schema of the existing table in the cache if it does not match the schema of the SELECT statement. This option results in new columns or dropped columns, if the schema of the SELECT statement does not match the cached table.

Common Queries

Use the following cache statement to cache all rows of a table:

CACHE SELECT * FROM [CData].[Elasticsearch].Employee

Use the following cache statement to cache all rows of a table into the cache table Cached[CData].[Elasticsearch].Employee:

CACHE CachedEmployee SELECT * FROM [CData].[Elasticsearch].Employee

Use the following cache statement for incremental caching. The DateModified column may not exist in all tables. The cache statement shows how incremental caching would work if there were such a column. Also, notice that, in this case, the WITH TRUNCATE and DROP EXISTING options are specifically omitted, which would have deleted all existing rows.

CACHE CachedEmployee SELECT * FROM [CData].[Elasticsearch].Employee WHERE DateModified > '2013-04-04'

Use the following cache statements to create a table with all available columns that will then cache only a few of them. The sequence of statements cache only Id and Name even though the cache table Cached[CData].[Elasticsearch].Employee has all the columns in [CData].[Elasticsearch].Employee.

CACHE CachedEmployee SCHEMA ONLY SELECT * FROM [CData].[Elasticsearch].Employee
CACHE CachedEmployee SELECT Id, Name FROM [CData].[Elasticsearch].Employee

To execute stored procedures, you can use EXECUTE or EXEC statements.

EXEC and EXECUTE assign stored procedure inputs, referenced by name, to values or parameter names.

Stored Procedure Syntax

To execute a stored procedure as an SQL statement, use the following syntax:

 
{ EXECUTE | EXEC } <stored_proc_name> 
{
  [ @ ] <input_name> = <expression>
} [ , ... ]

<expression> ::=
  | @ <parameter> 
  | ?
  | <literal>

Example Statements

Reference stored procedure inputs by name:

EXECUTE my_proc @second = 2, @first = 1, @third = 3;

Execute a parameterized stored procedure statement:

EXECUTE my_proc second = @p1, first = @p2, third = @p3; 

PIVOT and UNPIVOT can be used to change a table-valued expression into another table.

PIVOT

PIVOT Synax

 
"SELECT 'AverageCost' AS Cost_Sorted_By_Production_Days, [0], [1], [2], [3], [4]
FROM
(
SELECT DaysToManufacture, StandardCost
FROM Production.Product
) AS SourceTable
PIVOT
(
AVG(StandardCost)
FOR DaysToManufacture IN ([0], [1], [2], [3], [4])
) AS PivotTable;"

UNPIVOT

UNPIVOT Sytax

 
"SELECT VendorID, Employee, Orders
FROM
(SELECT VendorID, Emp1, Emp2, Emp3, Emp4, Emp5
FROM pvt) p
UNPIVOT
(Orders FOR Employee IN
(Emp1, Emp2, Emp3, Emp4, Emp5)
)AS unpvt;"

For further information on PIVOT and UNPIVOT, see FROM clause plus JOIN, APPLY, PIVOT (Transact-SQL)

Use INSERT INTO SELECT queries to select a list of records from one table and insert those same records into another table as a group. Inserting batches of records in this way may result in improved query performance compared to using many individual INSERT INTO queries.

The table whose records are selected for insertion into another table can be either a real table or a user-defined temporary table.

Inserting Records from Real Tables

To insert a group of records from one real, non-temporary, source table into another destination table, you can use an INSERT INTO SELECT query. This type of query is formatted similarly to a standard INSERT INTO query, except the VALUES clause is substituted with a SELECT query targeting the source table. All records matched by the embedded SELECT query are inserted into the destination table.

If the source table and destination tables have different column names, you must map columns from the source table to the corresponding columns in the destination table you want to insert them into. Perform this mapping by specifying the destination table columns in the same order as the source table columns you want to match them with. For example:

INSERT INTO DestinationTable (A,B,C,D) SELECT Q,R,S,T FROM SourceTable

In this example, the first source column (Q) is inserted into the first destination column (A), the second source column (R) is inserted into the second destination column (B), and so on.

If the source table and destination table both have the same column list with the same names, you can use a streamlined query.

INSERT INTO DestinationTableWithSameColumns SELECT * FROM SourceTable

In this example, there is no need to specify a list of columns for either the source or destination table, because their metadata already matches.

Inserting Records from Temporary Tables

You can manually define and populate temporary tables to hold a list of records for later bulk insertion.

Populate the Temporary Table

To create a temporary table, you must give it a name ending in "#TEMP" and execute an INSERT INTO query using that name, as if that table already existed in the database. After executing the first INSERT INTO, the temporary table exists and can receive subsequent INSERTs. For example:

INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Name, MyCustomField__c) VALUES ('New Employee', '9000');
INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Name, MyCustomField__c) VALUES ('New Employee 2', '9001');
INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Name, MyCustomField__c) VALUES ('New Employee 3', '9002');

This creates a temporary table called [CData].[Elasticsearch].Employee#TEMP with two columns and three rows of data. Since type cannot be determined on the temporary table itself, all values are considered strings and later converted to the proper type when they are inserted together into the real (non-temporary) table of interest.

Insert Temporary Table Contents into Real Tables

Once your temporary table is populated, execute an INSERT INTO SELECT query targeting the real (non-temporary) table you want to insert the temporary table's records into. This is formatted similarly to a standard INSERT INTO query, except the VALUES clause is substituted with a SELECT query targeting the matching columns in the temporary table. For example:

INSERT INTO [CData].[Elasticsearch].Employee (Name, MyCustomField__c) SELECT Name, MyCustomField__c FROM [CData].[Elasticsearch].Employee#TEMP

Results

The LastResultInfo#TEMP temporary table contains details about the most recently executed query that uses the contents of a temporary table in an embedded SELECT clause, as is the case for INSERT INTO SELECT queries that use a temporary table as the source of records. This table is cleared and repopulated each time such a query is executed. LastResultInfo#TEMP includes information such as whether the query in question succeeded, and how many rows were affected by the query.

Temporary Table Lifespan

Temporary tables only last as long as the connection remains open. When the connection to Elasticsearch is closed, all temporary tables are cleared, including the LastResultInfo#TEMP table.

To perform multiple updates in a single request to Elasticsearch,first use the INSERT INTO syntax to insert a temporary table of data into Elasticsearch. This works by first populating a temporary table with the data you are going to submit to Elasticsearch. Once you have all of the data you want to update, use UPDATE SELECT FROM to pass the temporary table data into the table in Elasticsearch.

Populate the Temporary Table

The temporary table you are populating is dynamic and is created at run time the first time you insert to it. Temporary tables are denoted by a # appearing in their name. When using a temporary table to update, the temporary table must be named in the format [TableName]#TEMP, where TableName is the name of the table you are inserting to. For example:

INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Id, Name, MyCustomField__c) VALUES ('AX1000001', 'New Employee', '9000');
INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Id, Name, MyCustomField__c) VALUES ('AX1000002', 'New Employee 2', '9001');
INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Id, Name, MyCustomField__c) VALUES ('AX1000003', 'New Employee 3', '9002');

This creates a temporary table called [CData].[Elasticsearch].Employee#TEMP with three columns and three rows of data. Since type cannot be determined on the temporary table itself, all values are stored in memory as strings. The values are later converted to the proper type when they are submitted to the [CData].[Elasticsearch].Employee table.

Update the Actual Table

Once your temporary table is populated, it is now time to update the actual table in Elasticsearch. You can do this by performing an UPDATE to the actual table and selecting the input data from the temporary table. For example:

UPDATE [CData].[Elasticsearch].Employee (Id, Name, MyCustomField__c) SELECT Id, Name, MyCustomField__c FROM [CData].[Elasticsearch].Employee#TEMP

Results

The results of the query are stored in the LastResultInfo#TEMP temporary table. This table is cleared and repopulated the next time data is modified by passing in a temporary table. Please be aware that the LastResultInfo#TEMP table has no predefined schema. You need to check its metadata at run time before reading data.

Temporary Table Life Span

Temporary tables only last as long as the connection remains open. When the connection to Elasticsearch is closed, all temporary tables are cleared, including the LastResultInfo#TEMP table.

To perform multiple deletes in a single request to Elasticsearch, first use the INSERT INTO syntax to create an in-memory temporary table of data to be deleted. Once you have all of the data you want to delete added to temporary table, use DELETE FROM syntax to delete data from the live table in Elasticsearch. This functionality is also available via the standard Batch Processing API available in JDBC.

Populate the Temporary Table

The temporary table you are populating is dynamic and is created at run time the first time you insert to it. Temporary tables are denoted by a # appearing in their name. When using a temporary table to delete, the temporary table must be named in the format [TableName]#TEMP, where TableName is the name of the table you are inserting to. For example:

INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Id) VALUES ('AX1000001');
INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Id) VALUES ('AX1000002');
INSERT INTO [CData].[Elasticsearch].Employee#TEMP (Id) VALUES ('AX1000003');

This creates a temporary table called [CData].[Elasticsearch].Employee#TEMP with one column and three rows of data. Since type cannot be determined on the temporary table itself, all values are stored in memory as strings. They are later converted to the proper type when they are submitted to the [CData].[Elasticsearch].Employee table.

Delete from the Actual Table

Once your temporary table is populated, it is now time to insert to the actual table in Elasticsearch. You can do this by performing a DELETE from the actual table and selecting the input data from the temporary table. For example:

DELETE FROM [CData].[Elasticsearch].Employee WHERE EXISTS SELECT Id FROM [CData].[Elasticsearch].Employee#TEMP

In this example, the full contents of the [CData].[Elasticsearch].Employee#TEMP table are passed into the [CData].[Elasticsearch].Employee table. This results in fewer requests being submitted to Elasticsearch since multiple deletes may be submitted with each request, which is much better for performance if you have many records to delete.

Results

The results of the query are stored in the LastResultInfo#TEMP temporary table. This table is cleared and repopulated the next time data is modified by passing in a temporary table. Please be aware that the LastResultInfo#TEMP table has no predefined schema. You need to check its metadata at run time before reading data.

Temporary Table Life Span

Temporary tables only last as long as the connection remains opened. When the connection to Elasticsearch is closed, all temporary tables are cleared, including the LastResultInfo#TEMP table.

The CData Python Connector for Elasticsearch models Elasticsearch entities in relational Tables, Views, and Stored Procedures.

Tables

Searching with SQL describes in further detail how the tables are dynamically retrieved.

Views

Views are treated in a similar manner to Tables and thus exhibit similar behavior. There are some differences in the background though which are a direct result of how aliases work within Elasticsearch. (Note: In the following description, 'alias', 'index', 'type', and 'field' are referring to the Elasticsearch objects and not directly to anything within the connector).

Views (aliases) are tied to an index and thus span all the types within an index. Additionally aliases can span multiple indices. Therefore you may see an alias (view) listed multiple times under different schemas (index). When querying the view, regardless of the schema specified, data will be retrieved and returned for all indices and types associated with the corresponding alias. Thus the generated metadata will contain a column for each field within each type of each index associated with the alias.

Searching with SQL describes in further detail how the views are dynamically retrieved.

The ModifyIndexAliases stored procedure can be used to create index aliases within Elasticsearch.

In addition to the Elasticsearch aliases, an '_all' view is returned which enables querying the _all endpoint to retrieve data for all indices in a single query. Given how many indices and documents the _all view could cover, certain queries agains the '_all' view could be very expensive. Additionally, for scanning for table metadata, as governed by RowScanDepth, will be less accurate for '_all' views that cover very large or very heterogenous indices. See Automatic Schema Discovery for more information about this.

Stored Procedures

The connector models the data in Elasticsearch as a list of tables in a relational database that can be queried using standard SQL statements.

CData Python Connector for Elasticsearch Tables

General information about index templates

Columns

Views are similar to tables in the way that data is represented; however, views are read-only.

Queries can be executed against a view as if it were a normal table.

CData Python Connector for Elasticsearch Views

General information about index settings

Columns

General information about the installed X-Pack features

Columns

Stored procedures are function-like interfaces that extend the functionality of the connector beyond simple SELECT/INSERT/UPDATE/DELETE operations with Elasticsearch.

Stored procedures accept a list of parameters, perform their intended function, and then return any relevant response data from Elasticsearch, along with an indication of whether the procedure succeeded or failed.

CData Python Connector for Elasticsearch Stored Procedures

Submits a request to create an index with specified settings.

EXECUTE Example:

EXECUTE CreateIndex Index = 'firstindex', Alias = 'search', NumberOfShards = '3' 

Input

Result Set Columns

Creates a schema file for the collection.

CreateSchema

Creates a local schema file (.rsd) from an existing table or view in the data model.

The schema file is created in the directory set in the Location connection property when this procedure is executed. You can edit the file to include or exclude columns, rename columns, or adjust column datatypes.

The connector checks the Location to determine if the names of any .rsd files match a table or view in the data model. If there is a duplicate, the schema file will take precedence over the default instance of this table in the data model. If a schema file is present in Location that does not match an existing table or view, a new table or view entry is added to the data model of the connector.

Input

Result Set Columns

Submits an alias request to modify index aliases.

EXECUTE Example:

EXECUTE ModifyIndexAliases Action = 'add;add', Index = 'index_1;index_2', Alias = 'my_alias;my_alias' 

Note: The Index parameter supports the asterisk (*) character to perform a pattern match to add all matching indices to the alias.

Input

Result Set Columns

Procedure for updating index settings. Note that some settings may only be updated on a closed index.

See this documentation for information on index settings in Elasticsearch, and what they control and impact across indices and clusters. Make sure to reference documentation for the version of Elasticsearch being connected to.

EXECUTE Example:

EXECUTE UpdateIndexSettings IndexName='traffic', NumberOfReplicas='3'

Input