diff --git a/.assets/Changelog.md b/.assets/Changelog.md index 1949ae3..8552ad9 100644 --- a/.assets/Changelog.md +++ b/.assets/Changelog.md @@ -4,6 +4,7 @@ This page lists all pull requests that made significant changes to bc2adls. Pull request | Changes --------------- | --- +[101](https://github.com/microsoft/bc2adls/pull/101) | It is often desirable to query the data residing in the lake and use it inside Dynamics 365 Business Central (BC). Such data may either have been exported previously out of BC through the `bc2adls` tool, or general tabular data that has been sourced from external systems. This lights up many use cases where the lake becomes the datasource and can be looked up on demand from inside BC through AL language constructs. See [Querying data residing in the lake with bc2adls](/.assets/QueryLakeData.md) to know more. [79](https://github.com/microsoft/bc2adls/pull/79) | The step to clean up tracked deleted records from the export process has now been removed to make exports more efficient. This clean up step can instead be performed either by clicking on the action **Clear tracked deleted records** on the main setup page, or by invoking the new codeunit **ADLSE Clear Tracked Deletions** through a low- frequency custom job queue entry. [78](https://github.com/microsoft/bc2adls/pull/78) | Upgrading to new versions may lead the export configuration to enter an incorrect state, say, if a field that was being exported before gets obsoleted in the new version. This fix prevents such an occurence by raising an error during the upgrade process. If corrective actions, say, disabling such fields are not taken after multiple upgrade attempts, the bc2adls extension is uninstalled and upgrade is forced. A subsequent re-install of the extension will then disable such tables from being exported, so that the user can then react to the change in schema later on. [56](https://github.com/microsoft/bc2adls/pull/56) | The table ADLSE Run has now been added to the retention policy so that the logs for the executions can be cleared periodically, thus taking up less space in the database. diff --git a/.assets/QueryDataInTheLake.png b/.assets/QueryDataInTheLake.png new file mode 100644 index 0000000..c496f2e Binary files /dev/null and b/.assets/QueryDataInTheLake.png differ diff --git a/.assets/QueryLakeData.md b/.assets/QueryLakeData.md new file mode 100644 index 0000000..6430832 --- /dev/null +++ b/.assets/QueryLakeData.md @@ -0,0 +1,80 @@ +# Querying data residing in the lake with bc2adls + +It is often desirable to query the data residing in the lake and use it inside Dynamics 365 Business Central (BC). Such data may either have been exported previously out of BC through the `bc2adls` tool, or general tabular data that has been sourced from external systems. The following steps help you establish a mechanism to query such data directly inside BC through the AL constructs. + +Let's go through a few use cases that are enabled by this feature. +1. Data from BC that has been previously exported and archived into the lake may need to be looked up by the system or a user to see historical entities. +1. Data created on the lake by external systems (such as IoT devices or [Azure Synapse Link for Dataverse](https://learn.microsoft.com/en-us/power-apps/maker/data-platform/export-to-data-lake)) need to be looked up in BC to make relevant calculations. +1. Data lake can now be used as a cheaper single-storage solution for miscellaneous tabular data that can be queried by BC on-demand. + +## How it works +**Note the arrows that point from the lake database into BC in the diagram below.** Using the new façades [`ADLSE Query`](/businessCentral/src/Query/ADLSEQuery.Codeunit.al) and [`ADLSE Query Table`](/businessCentral/src/Query/ADLSEQueryTable.Codeunit.al), the AL developer issues a REST API call to the `AdlsProxy` Azure function app while passing information like the table and specific fields to be queried, filters to be applied, etc. The function app then formulates the request as an SQL query to the lake database, which in turn gets the relevant data from the `data` CDM folder in the storage account. The result is then returned as a Json response to BC so that records and corresponding fields in those records can be individually read via the AL language. Please see the documentation of the above façades for more details. + +![Architecture](/.assets/architecture.png "Flow of data") + +Currently the funcionality only supports, +- fetching a specific set (or all) fields in a filtered set of records that is sorted in a certain way, similar to the [Findset](https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/methods-auto/recordref/recordref-findset-method) call. +- counting the number of records in the lake, similar to the [Count](https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/methods-auto/recordref/recordref-count-method) call. +- checking if there are any records in the lake, similar to the [IsEmpty](https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/methods-auto/recordref/recordref-isempty-method) call. + +> **Note** +> 1. The approach suggested will **only work for tabular data** that have been structured into shared metadata tables as described in [Creating shared metadata tables](/.assets/SharedMetadataTables.md). For data that was not created through the `bc2adls` export, you may need to create such tables manually, as explained. +> 1. Since querying from BC requires a number of Azure components to work in tandem, please use this approach only for **non- business critical** processes that allow for network or process latency. +> 1. The architecture allows for a limited amount of data to be queried from the serverless SQL endpoint. You may get errors if the response is too large for BC to process. Therefore, it is highly recommended that you apply filtering to narrow the results and only fetch the fields that you require. + +## Setting it all up + +### Pre-requisites +- You have configured [shared metadata tables](/.assets/SharedMetadataTables.md) for your data on the lake. This may include tables that are unknown to BC. +- You have sufficient access to create Azure function apps on your subscription. +- You have [installed and configured](/.assets/Setup.md) `bc2adls`, and the tables and fields in BC to be queried from the lake have been added as per [these instructions](/.assets/Execution.md#exporting-data-from-bc). For tables that are meant to be imported only (and not exported to the lake), set the `Enabled for export` field to be `false`. This step is, of course, only relevant if you wish to read BC data from the lake via the [`ADLSE Query Table`](/businessCentral/src/Query/ADLSEQueryTable.Codeunit.al) façade. + +### Create and deploy function app to Azure +Start Visual Studio Code and open the folder [`adlsProxy`](/adlsProxy/). Follow the instructions given in [the documentation](https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-vs-code-csharp?tabs=in-process). I used the runtime stack as .NET 7 Isolated. Let's say you chose to name the Function App as `AdlsProxyX`. + +### Take note of the function app URL +Open the newly created function app `AdlsProxyX` in the Azure portal, under **Overview**, take a note of the value in the **URL** field. This should be the format `https://adlsproxyx.azurewebsites.net`. + +### Add a system managed identity for the Azure function +In the Azure function app, and follow [the instructions](https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=portal%2Chttp#add-a-system-assigned-identity) to add a system managed identity. This would create an identity named (usually) the same as the Function App. + +### Protect your function app using new AAD credentials +In the Azure function app, follow the instructions at [Create a new app registration automatically](https://learn.microsoft.com/en-us/azure/app-service/configure-authentication-provider-aad#--option-1-create-a-new-app-registration-automatically). This should create a brand new App registration that can be used to make requests on the function app. Take a note of the following values as they will be required later on, +- the `App (Client) ID` field, as well as, +- the newly created client secret stored as the [application setting](https://learn.microsoft.com/en-us/azure/azure-functions/functions-how-to-use-azure-function-app-settings?tabs=portal#settings) named `MICROSOFT_PROVIDER_AUTHENTICATION_SECRET`. Of course, you may just as well create a new secret on the new app registration and use it instead! + +### Take a note of the function keys +In the Azure function app, under **Functions**, you will notice a few functions that have been created. Go inside each of the functions and under `Function Keys`, make a note of the full text of the respective function key. +> It is recommended to go through the documentation at [Securing Azure functions](https://learn.microsoft.com/en-us/azure/azure-functions/security-concepts) in order to fully understand the different ways to authenticate and authorize functions. This may be handy if, say, you want only some credentials to access entity A, while some others can access entity B, but everyone can access entity C, etc. + +### Authorize the created system managed identity to query the data on the serverless SQL endpoint +Open the SQL query editor from the lake database in the Synapse studio opened from your Synapse workspace and execute the following query, + + CREATE LOGIN [AdlsProxyX] FROM EXTERNAL PROVIDER; + CREATE USER AdlsProxyX FROM LOGIN [AdlsProxyX]; + ALTER ROLE db_datareader ADD member AdlsProxyX; + +This will ensure that the function app has the necessary privileges to run SQL queries in the database. Please make sure that the above query has run in the context of the right database, and that you have replaced the word `AdlsProxyX` with the correct name of the system managed identity of the function app. + +### Authorize the created system managed identity to read the data on the lake +As queries from the Azure function will be executed in the context of the system managed identity of the function app, it needs to be assigned the **Storage Blob Data Reader** role on the storage account with the data files. + +### Enable BC to send queries to the function app +On the main setup page of the `bc2adls` extension, you will note a new fast tab called **Query data in the lake**. Fill out the fields in the following way, +- **Synapse Serverless SQL endpoint** Locate the Synapse workspace resource on the Azure portal and fill this with the value of the field **Serverless SQL endpoint** under **Overview**. +- **SQL Database Name** The name of the lake database that got created at the [Creating shared metadata tables](/.assets/SharedMetadataTables.md). +- **Client ID** The value of the app (client) id from the step [Protect your function app using new AAD credentials](#protect-your-function-app-using-new-aad-credentials) above. +- **Client secret** The value of the client secret from the step [Protect your function app using new AAD credentials](#protect-your-function-app-using-new-aad-credentials) above. +- **Function app url** The value of the url from the step [Take note of the function app URL](#take-note-of-the-function-app-url) above. +- **Function key FindSet** The value of the function key for the Findset function gathered at the step [Take a note of the function keys](#take-a-note-of-the-function-keys) above. +- **Function key IsEmpty** The value of the function key for the IsEmpty function gathered at the step [Take a note of the function keys](#take-a-note-of-the-function-keys) above. +- **Function key Count** The value of the function key for the Count function gathered at the step [Take a note of the function keys](#take-a-note-of-the-function-keys) above. + +![Screenshot](/.assets/QueryDataInTheLake.png "bc2adls setup page") + +## Making queries in AL +Phew, that was a lengthy configuration but it is finally time to query the lake! Open Visual Studio Code and go the place in your AL code where you want to query the lake and follow the examples given in the documentation for the two façades, +1. [`ADLSE Query`](/businessCentral/src/Query/ADLSEQuery.Codeunit.al) used for any tabular data, and +1. [`ADLSE Query Table`](/businessCentral/src/Query/ADLSEQueryTable.Codeunit.al) used for BC tables. + +Any errors that happen during the course of the Rest Api call to the function app are thrown up on the AL side. To troubleshoot further on the function app, it is recommended that you follow instructions at [Monitor executions in Azure functions](https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring). \ No newline at end of file diff --git a/.assets/Setup.md b/.assets/Setup.md index c9f4f00..3ecac0a 100644 --- a/.assets/Setup.md +++ b/.assets/Setup.md @@ -31,6 +31,8 @@ Let us take a look at the settings show in the sample screenshot below, - **Emit telemetry** The flag to enable or disable operational telemetry from this extension. It is set to True by default. - **Multi- company export** The flag to allow exporting data from multiple companies at the same time. You should enable this only after the export schema is finalized- in other words, ensure that at least one export for a company has been successful with all the desired tables and the desired fields in those tables. We recommend that the json files are manually checked in the outbound container before enabling this flag. Changes to the export schema (adding or removing tables as well as changing the field set to be exported) are not allowed as long as this flag is checked. +The fast tab **Query data in the lake** handles configuration in case you want to read (not export) data from the lake. Please refer to [Querying data residing in the lake with bc2adls](/.assets/QueryLakeData.md) for more details. + ![The Export to Azure Data Lake Storage page](/.assets/bcAdlsePage.png) > **Note** diff --git a/.assets/architecture.png b/.assets/architecture.png index 6e6fb45..7e05eb8 100644 Binary files a/.assets/architecture.png and b/.assets/architecture.png differ diff --git a/.assets/bc2adls_data_architecture.vsdx b/.assets/bc2adls_data_architecture.vsdx index 7807b71..d1af9b5 100644 Binary files a/.assets/bc2adls_data_architecture.vsdx and b/.assets/bc2adls_data_architecture.vsdx differ diff --git a/.gitignore b/.gitignore index c0a88fb..c06454a 100644 --- a/.gitignore +++ b/.gitignore @@ -349,6 +349,13 @@ MigrationBackup/ # Ionide (cross platform F# VS Code tools) working folder .ionide/ +# VSCode related +**/.vscode/* + # AL related **/*.app -**/.vscode/* \ No newline at end of file + +# AdlsProxy related +adlsProxy/bin/ +adlsProxy/obj/ +adlsProxy/local.settings.json \ No newline at end of file diff --git a/README.md b/README.md index 36d85b3..ecb7bc2 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ # Project -> **This tool is an experiment on Dynamics 365 Business Central with the sole purpose of discovering the possibilities of having data exported to an Azure Data Lake. To see the details of how this tool is supported, please visit [the Support page](./SUPPORT.md). In case you wish to use this tool for your next project and engage with us, you are welcome to write to bc2adls@microsoft.com. As we are a small team, please expect delays in getting back to you.** +> **This tool is an experiment on Dynamics 365 Business Central with the sole purpose of discovering the possibilities of having data synced to and from an Azure Data Lake. To see the details of how this tool is supported, please visit [the Support page](./SUPPORT.md). In case you wish to use this tool for your next project and engage with us, you are welcome to write to bc2adls@microsoft.com. As we are a small team, please expect delays in getting back to you.** ## Introduction -The **bc2adls** tool is used to export data from [Dynamics 365 Business Central](https://dynamics.microsoft.com/en-us/business-central/overview/) (BC) to [Azure Data Lake Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) and expose it in the [CDM folder](https://docs.microsoft.com/en-us/common-data-model/data-lake) format. The components involved are the following, -- the **[businessCentral](/tree/main/businessCentral/)** folder holds a [BC extension](https://docs.microsoft.com/en-gb/dynamics365/business-central/ui-extensions) called `Azure Data Lake Storage Export` (ADLSE) which enables export of incremental data updates to a container on the data lake. The increments are stored in the CDM folder format described by the `deltas.cdm.manifest.json manifest`. +The **bc2adls** tool is used to exchange data between [Dynamics 365 Business Central](https://dynamics.microsoft.com/en-us/business-central/overview/) (BC) and [Azure Data Lake Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) and expose it in the [CDM folder](https://docs.microsoft.com/en-us/common-data-model/data-lake) format in the lake. The components involved are the following, +- the **[businessCentral](/tree/main/businessCentral/)** folder holds a [BC extension](https://docs.microsoft.com/en-gb/dynamics365/business-central/ui-extensions) called `Azure Data Lake Storage Export` (ADLSE) which enables export of incremental data updates to a container on the data lake. The increments are stored in the CDM folder format described by the `deltas.cdm.manifest.json manifest`. It also provides a library to read the tabular data existing on the lake, including non- BC data. - the **[synapse](/tree/main/synapse/)** folder holds the templates needed to create an [Azure Synapse](https://azure.microsoft.com/en-gb/services/synapse-analytics/) pipeline that consolidates the increments into a final `data` CDM folder. The following diagram illustrates the flow of data through a usage scenario- the main points being, @@ -17,6 +17,7 @@ The following diagram illustrates the flow of data through a usage scenario- the - CDM: via the `data.cdm.manifest.json manifest` - CSV/Parquet: via the underlying files for each individual entity inside the `data` folder - Spark/SQL: via [shared metadata tables](/.assets/SharedMetadataTables.md) +- The reverse flow is also possible whereby data in the lake can be read into BC via AL constructs. ![Architecture](/.assets/architecture.png "Flow of data") @@ -24,6 +25,7 @@ More details: - [Installation and configuration](/.assets/Setup.md) - [Executing the export and pipeline](/.assets/Execution.md) - [Creating shared metadata tables](/.assets/SharedMetadataTables.md) +- [Querying data residing in the lake with bc2adls](/.assets/QueryLakeData.md) - [Frequently asked questions](/.assets/FAQs.md) - Webinars - [[Jan 2022] Webinar introducing bc2adls](https://www.microsoft.com/en-us/videoplayer/embed/RWSHHG) diff --git a/adlsProxy/AdlsProxy.csproj b/adlsProxy/AdlsProxy.csproj new file mode 100644 index 0000000..3c94b56 --- /dev/null +++ b/adlsProxy/AdlsProxy.csproj @@ -0,0 +1,28 @@ + + + net7.0 + v4 + Exe + enable + enable + + + + + + + + + + + PreserveNewest + + + PreserveNewest + Never + + + + + + \ No newline at end of file diff --git a/adlsProxy/CreateQuery.cs b/adlsProxy/CreateQuery.cs new file mode 100644 index 0000000..fe44eab --- /dev/null +++ b/adlsProxy/CreateQuery.cs @@ -0,0 +1,165 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +using Newtonsoft.Json.Linq; + +namespace AdlsProxy +{ + internal enum FilterType + { + Equals, + NotEquals, + GreaterThan, + GreaterThanOrEquals, + LessThan, + LessThanOrEquals + } + + /// + /// Creates an SQL query based on the JSON input passed. It is expected that the JSON is formatted in the following way, + /// { + /// "server": "Serverless SQL endpoint", + /// "database": "database name", + /// "entity": "custledgerentry_21", + /// "fields": [ "EntryNo-1", "CustomerNo-3", "PostingDate-4" ], // optional; if blank, return all fields. Only used by FindSet. + /// "filters": [ + /// { "op": "GreaterThanOrEquals", "field": "CustomerNo-3", "value": "40000" }, + /// { "op": "LessThan", "field": "EntryNo-1", "value": 1559 }, + /// { "op": "NotEquals", "field": "PostingDate-4", "value": "2021-03-23T00:00:00" } + /// ], // optional; if blank, return unfiltered set of all records + /// "orderBy": [ + /// { + /// "field": "PostingDate-4", + /// "ascending": false + /// }, + /// { + /// "field": "EntryNo-1" + /// } + /// ] // optional. Only used by FindSet. + /// } + /// + /// The SQL query formed as text. + + internal static class CreateQuery + { + public static string FindSet(JObject body, JToken database, JToken entity) + { + var selectFields = body["fields"] as JArray; + var filters = body["filters"] as JArray; + var orderBy = body["orderBy"] as JArray; + + var fieldListExpression = selectFields == null ? "*" : concatenateItems(selectFields, ",", t => $"[{t.ToString()}]"); + var filterExpression = filters == null ? "" : $" WHERE {concatenateItems(filters, " AND", filterTransformToken)}"; + var orderByExpression = orderBy == null ? "" : $" ORDER BY {concatenateItems(orderBy, ",", orderByTransformToken)}"; + return $"SELECT {fieldListExpression} FROM [{database}].[dbo].[{entity}]{filterExpression}{orderByExpression};"; + } + + public static string Count(JObject body, JToken database, JToken entity) + { + var filters = body["filters"] as JArray; + + var filterExpression = filters == null ? "" : $" WHERE {concatenateItems(filters, " AND", filterTransformToken)}"; + return $"SELECT COUNT(*) FROM [{database}].[dbo].[{entity}]{filterExpression};"; + } + + public static string IsEmpty(JObject body, JToken database, JToken entity) + { + var filters = body["filters"] as JArray; + + var filterExpression = filters == null ? "" : $" WHERE {concatenateItems(filters, " AND", filterTransformToken)}"; + return $"IF EXISTS (SELECT TOP 1 1 FROM [{database}].[dbo].[{entity}]{filterExpression}) SELECT 0 ELSE SELECT 1;"; + } + + private static string concatenateItems(IEnumerable list, string delimiter, Func transform) + { + string result = ""; + var counter = 0; + if (list != null) + { + foreach (var item in list) + { + result += $"{transform(item)}{delimiter} "; + counter++; + } + if (counter > 0) + { + // remove the last delimiter added + result = result.Remove(result.Length - $"{delimiter} ".Length); + } + } + return result; + } + + private static string filterTransformToken(JToken token) + { + var filter = token as JObject; + if (filter == null) + { + throw new ArgumentException($"Bad item {token} in the filters expression."); + } + var op = filter["op"]; + if (op == null || op.Type != JTokenType.String) + { + throw new ArgumentException($"Bad or missing operator in the filter {token}."); + } + if (!Enum.TryParse((filter["op"] ?? "").ToString(), true, out FilterType filterType)) + { + throw new ArgumentException($"Bad operator passed in the filter {token}."); + } + var field = filter["field"] as JToken; + if (field == null || field.Type != JTokenType.String) + { + throw new ArgumentException($"Bad or missing field in the expression {token}."); + } + var value = filter["value"]; + if (value == null) + { + throw new ArgumentException($"Missing value in the filter {token}."); + } + var valueTokenType = (filter["value"] ?? 0).Type; + var useQuotes = new[] { JTokenType.String, JTokenType.Date }.Contains(valueTokenType); + return $"[{filter["field"]}] {filterOperator(filterType)} {(useQuotes ? "'" : "")}{filter["value"]}{(useQuotes ? "'" : "")}"; + } + + private static string filterOperator(FilterType op) + { + switch (op) + { + case FilterType.Equals: + return "="; + case FilterType.NotEquals: + return "!="; + case FilterType.GreaterThan: + return ">"; + case FilterType.GreaterThanOrEquals: + return ">="; + case FilterType.LessThan: + return "<"; + case FilterType.LessThanOrEquals: + return "<="; + default: + throw new ArgumentException($"The filter operator {op} is not supported."); + } + } + + private static bool isQuotedValue(JToken value) + { + return (value.Type == JTokenType.String || value.Type == JTokenType.Date); + } + + private static string orderByTransformToken(JToken token) + { + var orderByItem = token as JObject; + if (orderByItem == null) + { + throw new ArgumentException($"Bad item {token} in the order by expression."); + } + var field = orderByItem["field"] as JToken; + if (field == null || field.Type != JTokenType.String) + { + throw new ArgumentException($"Bad or missing field in the expression {token} in the order by expression."); + } + bool orderByAscending = ((bool?)(orderByItem["ascending"] as JToken)) ?? true; + return $"[{field}]{(orderByAscending ? " ASC" : " DESC")}"; + } + } +} \ No newline at end of file diff --git a/adlsProxy/CreateResult.cs b/adlsProxy/CreateResult.cs new file mode 100644 index 0000000..2694739 --- /dev/null +++ b/adlsProxy/CreateResult.cs @@ -0,0 +1,62 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +using Newtonsoft.Json.Linq; +using Microsoft.Data.SqlClient; +using Microsoft.Extensions.Logging; + +namespace AdlsProxy +{ + internal static class CreateResult + { + public static JToken FindSet(ILogger logger, SqlDataReader reader) + { + IList columnNames = new List(); + for (int fldIndex = 0; fldIndex <= reader.FieldCount - 1; fldIndex++) + { + columnNames.Add(reader.GetName(fldIndex)); + } + + int recordCount = 0; + JArray queryResult = new JArray(); + while (reader.Read()) + { + IList fields = new List(); + for (int fldIndex = 0; fldIndex <= reader.FieldCount - 1; fldIndex++) + { + fields.Add(reader[fldIndex]); + } + queryResult.Add(tokenizeResultRecord(columnNames, fields)); + recordCount++; + } + logger.LogInformation($"[FindSet] Number of records found: {recordCount}."); + return queryResult; + } + + public static JToken Count(ILogger logger, SqlDataReader reader) + { + reader.Read(); + var result = (int)reader[0]; + logger.LogInformation($"[Count] Number of records found: {result}."); + return result; + } + + public static JToken IsEmpty(ILogger logger, SqlDataReader reader) + { + reader.Read(); + var result = ((int)reader[0]) == 0 ? false : true; + logger.LogInformation($"[IsEmpty] Records found: {!result}."); + return result; + } + + private static JObject tokenizeResultRecord(IList columnNames, IList values) + { + var result = new JObject(); + for (int fldIndex = 0; fldIndex <= columnNames.Count - 1; fldIndex++) + { + var field = values[fldIndex]; + result.Add(columnNames[fldIndex], field == null ? null : new JValue(field)); + } + return result; + } + } +} \ No newline at end of file diff --git a/adlsProxy/Functions.cs b/adlsProxy/Functions.cs new file mode 100644 index 0000000..ff3db2f --- /dev/null +++ b/adlsProxy/Functions.cs @@ -0,0 +1,46 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +using Microsoft.Azure.Functions.Worker; +using Microsoft.Azure.Functions.Worker.Http; +using Microsoft.Extensions.Logging; + +namespace AdlsProxy +{ + public class Functions + { + private readonly ILogger _logger; + + public Functions(ILoggerFactory loggerFactory) + { + _logger = loggerFactory.CreateLogger(); + } + + /// + /// Finds the records on a given entity based on optional filters and returns a JSON result. + /// + [Function("FindSet")] + public HttpResponseData FindSet([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req) + { + return ProcessQuery.Process(CreateQuery.FindSet, CreateResult.FindSet, req, this._logger); + } + + /// + /// Counts the records on a given entity based on optional filters and returns a JSON result. + /// + [Function("Count")] + public HttpResponseData Count([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req) + { + return ProcessQuery.Process(CreateQuery.Count, CreateResult.Count, req, this._logger); + } + + /// + /// Checks if a given entity is empty based on optional filters and returns a JSON result. + /// + [Function("IsEmpty")] + public HttpResponseData IsEmpty([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req) + { + return ProcessQuery.Process(CreateQuery.IsEmpty, CreateResult.IsEmpty, req, this._logger); + } + + } +} diff --git a/adlsProxy/ProcessQuery.cs b/adlsProxy/ProcessQuery.cs new file mode 100644 index 0000000..b8611bd --- /dev/null +++ b/adlsProxy/ProcessQuery.cs @@ -0,0 +1,100 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +using System.Net; +using Microsoft.Azure.Functions.Worker.Http; +using Microsoft.Data.SqlClient; +using Microsoft.Extensions.Logging; +using Newtonsoft.Json.Linq; + +namespace AdlsProxy +{ + internal static class ProcessQuery + { + public static HttpResponseData Process(Func queryCreate, Func resultCreate, HttpRequestData req, ILogger logger) + { + try + { + var bodyText = new StreamReader(req.Body).ReadToEnd(); + var bodyJson = JObject.Parse(bodyText); + if (bodyJson == null) + { + throw new ArgumentException("Body in the request must be in the correct JSON format."); + } + var dbServer = bodyJson["server"]; + if (dbServer == null || dbServer.Type != JTokenType.String) + { + throw new ArgumentException("Bad or missing SQL endpoint."); + } + var dbName = bodyJson["database"]; + if (dbName == null || dbServer.Type != JTokenType.String) + { + throw new ArgumentException("Bad or missing SQL database name."); + } + + var connParams = new SqlConnectionStringBuilder(); + connParams.DataSource = dbServer.ToString(); + connParams.InitialCatalog = dbName.ToString(); + connParams.Encrypt = true; + + // uncomment when testing locally- remember to add the attributes to the local.settings.json + // connParams.Authentication = SqlAuthenticationMethod.ActiveDirectoryServicePrincipal; + // connParams.UserID = Environment.GetEnvironmentVariable("SqlConnectionString_Auth_User"); // client ID + // connParams.Password = Environment.GetEnvironmentVariable("SqlConnectionString_Auth_Password"); // client secret + connParams.Authentication = SqlAuthenticationMethod.ActiveDirectoryManagedIdentity; + + logger.LogInformation($"Connection Parameters: {connParams.ConnectionString}"); + + JObject output = new JObject(); + using (SqlConnection connection = new SqlConnection(connParams.ConnectionString)) + { + connection.Open(); + + var entity = bodyJson["entity"] as JToken; + if (entity == null || entity.Type != JTokenType.String) + { + throw new ArgumentException("Bad or missing entity to be queried."); + } + + // form query + string sqlQuery = queryCreate(bodyJson, dbName, entity); + logger.LogInformation($"Query constructed: {sqlQuery}"); + SqlCommand command = new SqlCommand(sqlQuery, connection); + + // execute query + using (SqlDataReader reader = command.ExecuteReader()) + { + output.Add("result", resultCreate(logger, reader)); + } + } + + logger.LogInformation("Request processed."); + + var response = req.CreateResponse(HttpStatusCode.OK); + response.Headers.Add("Content-Type", "text/json; charset=utf-8"); + var outputAsText = output.ToString(); + response.WriteString(outputAsText); + logger.LogInformation($"Length of the response: {outputAsText.Length}."); + + return response; + } + catch (ArgumentException argEx) + { + logger.LogWarning($"Invalid input presented. {argEx.Message} \r\n {argEx.StackTrace}"); + + var response = req.CreateResponse(HttpStatusCode.BadRequest); + response.Headers.Add("Content-Type", "text/plain; charset=utf-8"); + response.WriteString(argEx.Message); + return response; + } + catch (Exception ex) + { + logger.LogError($"Exception! {ex.Message} \r\n {ex.StackTrace}"); + + var response = req.CreateResponse(HttpStatusCode.InternalServerError); + response.Headers.Add("Content-Type", "text/plain; charset=utf-8"); + response.WriteString("The server encountered an error processing your request. Please take a look at the server logs."); + return response; + } + } + } +} \ No newline at end of file diff --git a/adlsProxy/Program.cs b/adlsProxy/Program.cs new file mode 100644 index 0000000..61053a0 --- /dev/null +++ b/adlsProxy/Program.cs @@ -0,0 +1,9 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +using Microsoft.Extensions.Hosting; + +var host = new HostBuilder() + .ConfigureFunctionsWorkerDefaults() + .Build(); + +host.Run(); diff --git a/adlsProxy/Properties/launchSettings.json b/adlsProxy/Properties/launchSettings.json new file mode 100644 index 0000000..85288b8 --- /dev/null +++ b/adlsProxy/Properties/launchSettings.json @@ -0,0 +1,9 @@ +{ + "profiles": { + "AdlsProxy": { + "commandName": "Project", + "commandLineArgs": "--port 7024", + "launchBrowser": false + } + } +} \ No newline at end of file diff --git a/adlsProxy/host.json b/adlsProxy/host.json new file mode 100644 index 0000000..beb2e40 --- /dev/null +++ b/adlsProxy/host.json @@ -0,0 +1,11 @@ +{ + "version": "2.0", + "logging": { + "applicationInsights": { + "samplingSettings": { + "isEnabled": true, + "excludedTypes": "Request" + } + } + } +} \ No newline at end of file diff --git a/businessCentral/app.json b/businessCentral/app.json index df62eb3..3c06b63 100644 --- a/businessCentral/app.json +++ b/businessCentral/app.json @@ -3,8 +3,8 @@ "name": "Azure Data Lake Storage Export", "publisher": "The bc2adls team, Microsoft Denmark", "brief": "Sync data from Business Central to the Azure storage", - "description": "Exports data in chosen tables to the Azure Data Lake and keeps it in sync by incremental updates. Before you use this tool, please read the SUPPORT.md file at https://github.com/microsoft/bc2adls.", - "version": "1.3.12.5", + "description": "Exports and enables reading of data in chosen tables to the Azure Data Lake and keeps it in sync by incremental updates. Before you use this tool, please read the SUPPORT.md file at https://github.com/microsoft/bc2adls.", + "version": "1.3.14.0", "privacyStatement": "https://go.microsoft.com/fwlink/?LinkId=724009", "EULA": "https://go.microsoft.com/fwlink/?linkid=2009120", "help": "https://go.microsoft.com/fwlink/?LinkId=724011", @@ -24,7 +24,7 @@ "idRanges": [ { "from": 82560, - "to": 82575 + "to": 82580 } ], "target": "Cloud", diff --git a/businessCentral/src/ADLSE.Codeunit.al b/businessCentral/src/ADLSE.Codeunit.al index 3288114..21e5dc5 100644 --- a/businessCentral/src/ADLSE.Codeunit.al +++ b/businessCentral/src/ADLSE.Codeunit.al @@ -22,4 +22,6 @@ codeunit 82567 ADLSE internal procedure OnTableExported(TableID: Integer; LastTimeStampExported: BigInteger) begin end; + + } \ No newline at end of file diff --git a/businessCentral/src/ADLSECredentials.Codeunit.al b/businessCentral/src/ADLSECredentials.Codeunit.al index b259010..e059303 100644 --- a/businessCentral/src/ADLSECredentials.Codeunit.al +++ b/businessCentral/src/ADLSECredentials.Codeunit.al @@ -1,6 +1,6 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. See LICENSE in the project root for license information. -codeunit 82565 "ADLSE Credentials" +codeunit 82565 "ADLSE Credentials" implements "ADLSE ICredentials" { Access = Internal; // The max sizes of the fields are determined based on the recommendations listed at @@ -51,12 +51,21 @@ codeunit 82565 "ADLSE Credentials" CheckValueExists(ClientSecretKeyNameTok, ClientSecret); end; - [NonDebuggable] procedure GetTenantID(): Text begin exit(StorageTenantID); end; + procedure GetResource(): Text + begin + exit('https://storage.azure.com/'); + end; + + procedure GetScope(): Text + begin + exit('https://storage.azure.com/user_impersonation'); + end; + [NonDebuggable] procedure SetTenantID(NewTenantIdValue: Text): Text begin diff --git a/businessCentral/src/ADLSEField.Table.al b/businessCentral/src/ADLSEField.Table.al index 33ba1be..c4497d8 100644 --- a/businessCentral/src/ADLSEField.Table.al +++ b/businessCentral/src/ADLSEField.Table.al @@ -103,13 +103,14 @@ table 82562 "ADLSE Field" procedure CheckFieldToBeEnabled() var - Fld: Record Field; + Field: Record Field; ADLSESetup: Codeunit "ADLSE Setup"; ADLSEUtil: Codeunit "ADLSE Util"; begin - Fld.Get(Rec."Table ID", Rec."Field ID"); - ADLSEUtil.CheckFieldTypeForExport(Fld); - ADLSESetup.CheckFieldCanBeExported(Fld); + Field.Get(Rec."Table ID", Rec."Field ID"); + if not ADLSEUtil.CheckFieldTypeForExport(Field.Type) then + ADLSEUtil.RaiseFieldTypeNotSupportedError(Field.FieldName, Field.Type); + ADLSESetup.CheckFieldCanBeExported(Field); end; [TryFunction] diff --git a/businessCentral/src/ADLSEHttp.Codeunit.al b/businessCentral/src/ADLSEHttp.Codeunit.al index 319b1e1..c7d66dd 100644 --- a/businessCentral/src/ADLSEHttp.Codeunit.al +++ b/businessCentral/src/ADLSEHttp.Codeunit.al @@ -5,7 +5,7 @@ codeunit 82563 "ADLSE Http" Access = Internal; var - Credentials: Codeunit "ADLSE Credentials"; + Credentials: Interface "ADLSE ICredentials"; HttpMethod: Enum "ADLSE Http Method"; Url: Text; Body: Text; @@ -18,7 +18,7 @@ codeunit 82563 "ADLSE Http" UnsupportedMethodErr: Label 'Unsupported method: %1', Comment = '%1: http method name'; OAuthTok: Label 'https://login.microsoftonline.com/%1/oauth2/token', Comment = '%1: tenant id'; BearerTok: Label 'Bearer %1', Comment = '%1: access token'; - AcquireTokenBodyTok: Label 'resource=%1&scope=%2&client_id=%3&client_secret=%4&client_info=1&grant_type=client_credentials', Comment = '%1: encoded url, %2: encoded user impersonation, %3: client ID, %4: client secret'; + AcquireTokenBodyTok: Label 'resource=%1&scope=%2&client_id=%3&client_secret=%4&client_info=1&grant_type=client_credentials', Comment = '%1: resource, %2: scope, %3: client ID, %4: client secret'; procedure SetMethod(HttpMethodValue: Enum "ADLSE Http Method") begin @@ -39,7 +39,7 @@ codeunit 82563 "ADLSE Http" var ADLSEUtil: Codeunit "ADLSE Util"; begin - AdditionalRequestHeaders.Add(HeaderKey, ADLSEUtil.ConvertNumberToText(HeaderValue)); + AdditionalRequestHeaders.Add(HeaderKey, ADLSEUtil.ConvertVariantToText(HeaderValue)); end; procedure SetBody(BodyValue: Text) @@ -62,9 +62,9 @@ codeunit 82563 "ADLSE Http" exit(ContentTypePlainTextTok); end; - procedure SetAuthorizationCredentials(ADLSECredentials: Codeunit "ADLSE Credentials") + procedure SetAuthorizationCredentials(ADLSEICredentials: Interface "ADLSE ICredentials") begin - Credentials := ADLSECredentials; + Credentials := ADLSEICredentials; end; procedure GetResponseHeaderValue(HeaderKey: Text) Result: List of [Text] @@ -91,7 +91,7 @@ codeunit 82563 "ADLSE Http" var Client: HttpClient; Headers: HttpHeaders; - RequestMsg: HttpRequestMessage; + // RequestMsg: HttpRequestMessage; ResponseMsg: HttpResponseMessage; Content: HttpContent; HeaderKey: Text; @@ -108,19 +108,23 @@ codeunit 82563 "ADLSE Http" Headers.Add(HeaderKey, HeaderValue); end; end; - case HttpMethod of "ADLSE Http Method"::Get: Client.Get(Url, ResponseMsg); "ADLSE Http Method"::Put: begin - RequestMsg.Method('PUT'); - RequestMsg.SetRequestUri(Url); + // RequestMsg.Method('PUT'); + // RequestMsg.SetRequestUri(Url); AddContent(Content); Client.Put(Url, Content, ResponseMsg); end; - "ADLSE Http Method"::Delete: - Client.Delete(Url, ResponseMsg); + "ADLSE Http Method"::Post: + begin + // RequestMsg.Method('POST'); + // RequestMsg.SetRequestUri(Url); + AddContent(Content); + Client.Post(Url, Content, ResponseMsg); + end; else Error(UnsupportedMethodErr, HttpMethod); end; @@ -190,8 +194,8 @@ codeunit 82563 "ADLSE Http" RequestBody := StrSubstNo( AcquireTokenBodyTok, - 'https%3A%2F%2Fstorage.azure.com%2F', // url encoded form of https://storage.azure.com/ - 'https%3A%2F%2Fstorage.azure.com%2Fuser_impersonation', // url encoded form of https://storage.azure.com/user_impersonation + Credentials.GetResource(), + Credentials.GetScope(), Credentials.GetClientID(), Credentials.GetClientSecret()); Content.WriteFrom(RequestBody); diff --git a/businessCentral/src/ADLSEHttpMethod.Enum.al b/businessCentral/src/ADLSEHttpMethod.Enum.al index 4f0eb48..da79758 100644 --- a/businessCentral/src/ADLSEHttpMethod.Enum.al +++ b/businessCentral/src/ADLSEHttpMethod.Enum.al @@ -7,5 +7,5 @@ enum 82561 "ADLSE Http Method" value(0; Get) { } value(1; Put) { } - value(2; Delete) { } + value(3; Post) { } } \ No newline at end of file diff --git a/businessCentral/src/ADLSEICredentials.Interface.al b/businessCentral/src/ADLSEICredentials.Interface.al new file mode 100644 index 0000000..893f541 --- /dev/null +++ b/businessCentral/src/ADLSEICredentials.Interface.al @@ -0,0 +1,13 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +interface "ADLSE ICredentials" +{ + Access = Internal; + + procedure IsInitialized(): Boolean; + procedure GetClientID(): Text; + procedure GetClientSecret(): Text; + procedure GetTenantID(): Text; + procedure GetResource(): Text; + procedure GetScope(): Text; +} \ No newline at end of file diff --git a/businessCentral/src/ADLSESetup.Page.al b/businessCentral/src/ADLSESetup.Page.al index 5c1c41c..6f08f38 100644 --- a/businessCentral/src/ADLSESetup.Page.al +++ b/businessCentral/src/ADLSESetup.Page.al @@ -8,7 +8,7 @@ page 82560 "ADLSE Setup" SourceTable = "ADLSE Setup"; InsertAllowed = false; DeleteAllowed = false; - Caption = 'Export to Azure Data Lake Storage'; + Caption = 'Data sync to Azure Data Lake Storage'; layout { @@ -19,7 +19,7 @@ page 82560 "ADLSE Setup" Caption = 'Setup'; group(Account) { - Caption = 'Account'; + Caption = 'Storage account'; field(Container; Rec.Container) { ApplicationArea = All; @@ -106,6 +106,11 @@ page 82560 "ADLSE Setup" } } + part(Lookup; "ADLSE Setup Query") + { + ApplicationArea = All; + UpdatePropagation = Both; + } part(Tables; "ADLSE Setup Tables") { ApplicationArea = All; @@ -259,6 +264,7 @@ page 82560 "ADLSE Setup" ClientID: Text; [NonDebuggable] ClientSecret: Text; + OldLogsExist: Boolean; FailureNotificationID: Guid; ExportFailureNotificationMsg: Label 'Data from one or more tables failed to export on the last run. Please check the tables below to see the error(s).'; diff --git a/businessCentral/src/ADLSESetup.Table.al b/businessCentral/src/ADLSESetup.Table.al index 1d5f3f8..80302ed 100644 --- a/businessCentral/src/ADLSESetup.Table.al +++ b/businessCentral/src/ADLSESetup.Table.al @@ -75,6 +75,21 @@ table 82560 "ADLSE Setup" ADLSECurrentSession.CheckForNoActiveSessions(); end; } + + field(25; "Serverless SQL Endpoint"; Text[2048]) + { + Caption = 'Synapse Serverless SQL Endpoint'; + } + + field(26; "SQL Database"; Text[2048]) + { + Caption = 'SQL Database Name'; + } + + field(27; "Function App Url"; Text[2048]) + { + Caption = 'Function App Url'; + } } keys diff --git a/businessCentral/src/ADLSESetupFields.Page.al b/businessCentral/src/ADLSESetupFields.Page.al index 386e412..a147abf 100644 --- a/businessCentral/src/ADLSESetupFields.Page.al +++ b/businessCentral/src/ADLSESetupFields.Page.al @@ -20,21 +20,22 @@ page 82562 "ADLSE Setup Fields" field(FieldCaption; Rec.FieldCaption) { ApplicationArea = All; - Tooltip = 'Specifies the name of the field to be exported'; + Tooltip = 'Specifies the name of the field to be exported.'; } field("Field ID"; Rec."Field ID") { ApplicationArea = All; Caption = 'Number'; - Tooltip = 'Specifies the ID of the field to be exported'; + Tooltip = 'Specifies the ID of the field to be exported.'; Visible = false; } field(Enabled; Rec.Enabled) { ApplicationArea = All; - Tooltip = 'Specifies if the field will be exported'; + Caption = 'Enabled'; + Tooltip = 'Specifies if the field will be exported or imported.'; } field(ADLSFieldName; ADLSFieldName) @@ -59,7 +60,7 @@ page 82562 "ADLSE Setup Fields" { ApplicationArea = All; Caption = 'Type'; - Tooltip = 'Specifies the field type'; + Tooltip = 'Specifies the field type.'; Editable = false; Visible = false; } @@ -69,7 +70,7 @@ page 82562 "ADLSE Setup Fields" ApplicationArea = All; Caption = 'Obsolete State'; OptionCaption = 'No,Pending,Removed'; - Tooltip = 'Specifies the Obsolete State of the field'; + Tooltip = 'Specifies the Obsolete State of the field.'; Editable = false; Visible = false; } @@ -115,7 +116,7 @@ page 82562 "ADLSE Setup Fields" ADLSEUtil: Codeunit "ADLSE Util"; begin Fld.Get(Rec."Table ID", Rec."Field ID"); - ADLSFieldName := ADLSEUtil.GetDataLakeCompliantFieldName(Fld.FieldName, Fld."No."); + ADLSFieldName := ADLSEUtil.GetDataLakeCompliantFieldName(Fld.FieldName, Rec."Field ID"); FieldClassName := Fld.Class; FieldTypeName := Fld."Type Name"; FieldObsoleteState := Fld.ObsoleteState; diff --git a/businessCentral/src/ADLSESetupTables.Page.al b/businessCentral/src/ADLSESetupTables.Page.al index 2c50a2c..662e3e8 100644 --- a/businessCentral/src/ADLSESetupTables.Page.al +++ b/businessCentral/src/ADLSESetupTables.Page.al @@ -28,8 +28,7 @@ page 82561 "ADLSE Setup Tables" { ApplicationArea = All; Editable = true; - Caption = 'Enabled'; - Tooltip = 'Specifies the state of the table. Set this checkmark to export this table, otherwise not.'; + Tooltip = 'Specifies if the data in this table should be exported.'; } field(FieldsChosen; NumberFieldsChosenValue) { diff --git a/businessCentral/src/ADLSETable.Table.al b/businessCentral/src/ADLSETable.Table.al index 26dd5dc..093314c 100644 --- a/businessCentral/src/ADLSETable.Table.al +++ b/businessCentral/src/ADLSETable.Table.al @@ -23,7 +23,7 @@ table 82561 "ADLSE Table" field(3; Enabled; Boolean) { Editable = false; - Caption = 'Enabled'; + Caption = 'Enabled for export'; trigger OnValidate() begin diff --git a/businessCentral/src/ADLSEUtil.Codeunit.al b/businessCentral/src/ADLSEUtil.Codeunit.al index 3d65d1c..d6cd5bf 100644 --- a/businessCentral/src/ADLSEUtil.Codeunit.al +++ b/businessCentral/src/ADLSEUtil.Codeunit.al @@ -9,6 +9,7 @@ codeunit 82564 "ADLSE Util" AlphabetsUpperTxt: Label 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; NumeralsTxt: Label '1234567890'; FieldTypeNotSupportedErr: Label 'The field %1 of type %2 is not supported.', Comment = '%1 = field name, %2 = field type'; + TypeOfFieldNotSupportedErr: Label 'The type of field %1 is not supported.', Comment = '%1 = field name'; ConcatNameIdTok: Label '%1-%2', Comment = '%1: Name, %2: ID'; DateTimeExpandedFormatTok: Label '%1, %2 %3 %4 %5:%6:%7 GMT', Comment = '%1: weekday, %2: day, %3: month, %4: year, %5: hour, %6: minute, %7: second'; QuotedTextTok: Label '"%1"', Comment = '%1: text to be double- quoted'; @@ -16,6 +17,7 @@ codeunit 82564 "ADLSE Util" CommaSuffixedTok: Label '%1, ', Comment = '%1: text to be suffixed'; WholeSecondsTok: Label ':%1Z', Comment = '%1: seconds'; FractionSecondsTok: Label ':%1.%2Z', Comment = '%1: seconds, %2: milliseconds'; + UnmatchedEnumNameErr: Label 'No enum or option match in the field %1 found for the text %2.', Comment = '%1 is the field name, %2 is the text value for enum that failed parsing.'; procedure ToText(GuidValue: Guid): Text begin @@ -130,6 +132,14 @@ codeunit 82564 "ADLSE Util" exit(RecRef.Caption()); end; + procedure GetTableName(TableID: Integer) TableName: Text + var + RecRef: RecordRef; + begin + RecRef.Open(TableID); + TableName := RecRef.Name; + end; + procedure GetDataLakeCompliantTableName(TableID: Integer) TableName: Text var OrigTableName: Text; @@ -144,12 +154,12 @@ codeunit 82564 "ADLSE Util" exit(StrSubstNo(ConcatNameIdTok, GetDataLakeCompliantName(FieldName), FieldID)); end; - procedure GetTableName(TableID: Integer) TableName: Text + procedure GetDataLakeCompliantFieldName(TableID: Integer; FieldID: Integer): Text var - RecRef: RecordRef; + Field: Record Field; begin - RecRef.Open(TableID); - TableName := RecRef.Name; + Field.Get(TableID, FieldID); + exit(GetDataLakeCompliantFieldName(Field.FieldName, FieldID)); end; procedure GetDataLakeCompliantName(Name: Text) Result: Text @@ -172,75 +182,226 @@ codeunit 82564 "ADLSE Util" Result := ResultBuilder.ToText(); end; - procedure CheckFieldTypeForExport(Fld: Record Field) + procedure CheckFieldTypeForExport(FieldType: Option): Boolean + var + Field: Record Field; begin - case Fld.Type of - Fld.Type::BigInteger, - Fld.Type::Boolean, - Fld.Type::Code, - Fld.Type::Date, - Fld.Type::DateFormula, - Fld.Type::DateTime, - Fld.Type::Decimal, - Fld.Type::Duration, - Fld.Type::Guid, - Fld.Type::Integer, - Fld.Type::Option, - Fld.Type::Text, - Fld.Type::Time: - exit; + case FieldType of + Field.Type::BigInteger, + Field.Type::Boolean, + Field.Type::Code, + Field.Type::Date, + Field.Type::DateFormula, + Field.Type::DateTime, + Field.Type::Decimal, + Field.Type::Duration, + Field.Type::Guid, + Field.Type::Integer, + Field.Type::Option, + Field.Type::Text, + Field.Type::Time: + exit(true); end; - Error(FieldTypeNotSupportedErr, Fld."Field Caption", Fld.Type); + exit(false); end; - procedure ConvertFieldToText(Fld: FieldRef): Text + local procedure ConvertFieldToText(FieldRef: FieldRef): Text var DateTimeValue: DateTime; begin - case Fld.Type of - Fld.Type::BigInteger, - Fld.Type::Date, - Fld.Type::DateFormula, - Fld.Type::Decimal, - Fld.Type::Duration, - Fld.Type::Integer, - Fld.Type::Time: - exit(ConvertNumberToText(Fld.Value())); - Fld.Type::DateTime: + case FieldRef.Type of + FieldRef.Type::BigInteger, + FieldRef.Type::Date, + FieldRef.Type::DateFormula, + FieldRef.Type::Decimal, + FieldRef.Type::Duration, + FieldRef.Type::Integer, + FieldRef.Type::Time, + FieldRef.Type::Boolean: + exit(ConvertVariantToText(FieldRef.Value())); + FieldRef.Type::DateTime: begin - DateTimeValue := Fld.Value(); + DateTimeValue := FieldRef.Value(); if DateTimeValue = 0DT then exit(''); exit(ConvertDateTimeToText(DateTimeValue)); end; - Fld.Type::Option: - exit(Fld.GetEnumValueNameFromOrdinalValue(Fld.Value())); - Fld.Type::Boolean: - exit(Format(Fld.Value(), 0, 9)); - Fld.Type::Code, - Fld.Type::Guid, - Fld.Type::Text: - exit(ConvertStringToText(Fld.Value())); - else - Error(FieldTypeNotSupportedErr, Fld.Name(), Fld.Type); + FieldRef.Type::Option: + exit(FieldRef.GetEnumValueNameFromOrdinalValue(FieldRef.Value())); + FieldRef.Type::Code, + FieldRef.Type::Guid, + FieldRef.Type::Text: + exit(ConvertStringToText(FieldRef.Value())); end; + RaiseFieldTypeNotSupportedError(FieldRef.Name, FieldRef.Type); end; - local procedure ConvertStringToText(Val: Text): Text + local procedure RaiseFieldTypeNotSupportedError(FieldName: Text; FieldType: FieldType) begin - Val := Val.Replace('\', '\\'); // escape the escape character - Val := Val.Replace('"', '\"'); // escape the quote character - exit(StrSubstNo(QuotedTextTok, Val)); + Error(FieldTypeNotSupportedErr, FieldName, FieldType); + end; + + procedure RaiseFieldTypeNotSupportedError(FieldName: Text; FieldTypeOption: Option) + begin + Error(FieldTypeNotSupportedErr, FieldName, FieldTypeOption); end; - procedure ConvertNumberToText(Val: Integer): Text + procedure RaiseFieldTypeNotSupportedError(FieldName: Text) + begin + Error(TypeOfFieldNotSupportedErr, FieldName); + end; + + procedure ConvertVariantToJson(ValueVariant: Variant; var Result: JsonValue): Boolean + var + DateFormulaValue: DateFormula; + BigIntegerValue: BigInteger; + BooleanValue: Boolean; + DateValue: Date; + DateTimeValue: DateTime; + DecimalValue: Decimal; + DurationValue: Duration; + IntegerValue: Integer; + TextValue: Text; + TimeValue: Time; begin - exit(Format(Val, 0, 9)); + case true of + ValueVariant.IsBigInteger(): + begin + BigIntegerValue := ValueVariant; + Result.SetValue(BigIntegerValue); + exit(true); + end; + ValueVariant.IsDate(): + begin + DateValue := ValueVariant; + Result.SetValue(DateValue); + exit(true); + end; + ValueVariant.IsDateFormula(): + begin + DateFormulaValue := ValueVariant; + TextValue := ConvertVariantToText(DateFormulaValue); + Result.SetValue(TextValue); + exit(true); + end; + ValueVariant.IsDecimal(): + begin + DecimalValue := ValueVariant; + Result.SetValue(DecimalValue); + exit(true); + end; + ValueVariant.IsDuration(): + begin + DurationValue := ValueVariant; + Result.SetValue(DurationValue); + exit(true); + end; + ValueVariant.IsInteger(): + begin + IntegerValue := ValueVariant; + Result.SetValue(IntegerValue); + exit(true); + end; + ValueVariant.IsTime(): + begin + TimeValue := ValueVariant; + Result.SetValue(TimeValue); + exit(true); + end; + ValueVariant.IsDateTime(): + begin + DateTimeValue := ValueVariant; + Result.SetValue(DateTimeValue); + exit(true); + end; + ValueVariant.IsBoolean(): + begin + BooleanValue := ValueVariant; + Result.SetValue(BooleanValue); + exit(true); + end; + ValueVariant.IsByte(), + ValueVariant.IsChar(), + ValueVariant.IsCode(), + ValueVariant.IsGuid(), + ValueVariant.IsText(): + begin + TextValue := ValueVariant; + Result.SetValue(TextValue); + exit(true); + end; + ValueVariant.IsOption(): + ;// Option should be passed as Text + end; + exit(false); + end; + + procedure ConvertJsonToVariant(FieldRef: FieldRef; Value: JsonValue): Variant + var + DataFormulaVal: DateFormula; + DateTimeVal: DateTime; + TxtVal: Text; + GuidVal: Guid; + EnumIndex: Integer; + begin + case FieldRef.Type of + FieldType::BigInteger: + exit(Value.AsBigInteger()); + FieldType::Boolean: + exit(Value.AsBoolean()); + FieldType::Code: + exit(Value.AsCode()); + FieldType::Date: + begin + DateTimeVal := Value.AsDateTime(); + exit(DT2Date(DateTimeVal)); + end; + FieldType::DateFormula: + begin + TxtVal := Value.AsText(); + Evaluate(DataFormulaVal, TxtVal, 9); + exit(DataFormulaVal); + end; + FieldType::DateTime: + exit(Value.AsDateTime()); + FieldType::Decimal: + exit(Value.AsDecimal()); + FieldType::Duration: + exit(Value.AsDuration()); + FieldType::Guid: + begin + TxtVal := Value.AsText(); + Evaluate(GuidVal, TxtVal, 9); + exit(GuidVal); + end; + FieldType::Integer: + exit(Value.AsInteger()); + FieldType::Option: + begin + TxtVal := Value.AsText(); + for EnumIndex := 1 to FieldRef.EnumValueCount() do + if TxtVal = FieldRef.GetEnumValueName(EnumIndex) then + exit(FieldRef.GetEnumValueOrdinal(EnumIndex)); + Error(UnmatchedEnumNameErr, FieldRef.Name, TxtVal); + end; + FieldType::Text: + exit(Value.AsText()); + FieldType::Time: + exit(Value.AsTime()); + end; + RaiseFieldTypeNotSupportedError(FieldRef.Name, FieldRef.Type); + end; + + procedure ConvertStringToText(Val: Text): Text + begin + Val := Val.Replace('\', '\\'); // escape the escape character + Val := Val.Replace('"', '\"'); // escape the quote character + exit(StrSubstNo(QuotedTextTok, Val)); end; - local procedure ConvertNumberToText(Val: Variant): Text + procedure ConvertVariantToText(VariantVal: Variant): Text begin - exit(Format(Val, 0, 9)); + exit(Format(VariantVal, 0, 9)); end; local procedure ConvertDateTimeToText(Val: DateTime) Result: Text diff --git a/businessCentral/src/Query/ADLSEQuery.Codeunit.al b/businessCentral/src/Query/ADLSEQuery.Codeunit.al new file mode 100644 index 0000000..0b483ac --- /dev/null +++ b/businessCentral/src/Query/ADLSEQuery.Codeunit.al @@ -0,0 +1,242 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +codeunit 82577 "ADLSE Query" +{ + /// + /// This is the facade to generically query data in the data lake. + /// + /// Here is an example of how to use this facade. + /// + /// + /// procedure QuerySharedMetadataTableFromLake() + /// var + /// ADLSEQuery: Codeunit "ADLSE Query"; + /// RecordsFound: Integer; + /// EntryNo: Integer; + /// CustomerNo: Code[20]; + /// DocumentType: Text; + /// Timestamp: BigInteger; + /// PostingDate: Date; + /// ModifiedDateTime: DateTime; + /// begin + /// // initialize with the table name + /// ADLSEQuery.Init('custledgerentry_21'); + /// + /// // optionally set the connection string + /// ADLSEQuery.SetServer('serverless sql endpoint, XXXX.sql.azuresynapse.net'); + /// ADLSEQuery.SetDatabase('database name'); + /// + /// // set filtering + /// ADLSEQuery.SetRange('DocumentType-5', 'Invoice'); + /// ADLSEQuery.SetFilter('CustomerNo-3', '>=%1', '40000'); + /// ADLSEQuery.SetFilter('PostingDate-4', '>%1', 20211023D); + /// ADLSEQuery.SetRange('$Company', 'CRONUS USA, Inc.'); + /// + /// // set sequence of results + /// ADLSEQuery.SetOrderBy('PostingDate-4', false); + /// ADLSEQuery.SetOrderBy('EntryNo-1'); + /// + /// // set fields to be fetched + /// ADLSEQuery.AddLoadField('EntryNo-1'); + /// ADLSEQuery.AddLoadField('CustomerNo-3'); + /// ADLSEQuery.AddLoadField('DocumentType-5'); + /// ADLSEQuery.AddLoadField('PostingDate-4'); + /// ADLSEQuery.AddLoadField('SystemModifiedAt-2000000003'); + /// ADLSEQuery.AddLoadField('timestamp-0'); + /// + /// // make the find set query + /// if ADLSEQuery.FindSet() then + /// // records found + /// repeat + /// EntryNo := ADLSEQuery.Field('EntryNo-1').AsInteger(); // get an integer value + /// CustomerNo := ADLSEQuery.Field('CustomerNo-3').AsCode(); // get a code value + /// DocumentType := ADLSEQuery.Field('DocumentType-5').AsText(); // get a text value + /// PostingDate := DT2Date(ADLSEQuery.Field('PostingDate-4').AsDateTime()); // get a date value + /// ModifiedDateTime := ADLSEQuery.Field('SystemModifiedAt-2000000003').AsDateTime(); // get a datetime value + /// Timestamp := ADLSEQuery.Field('timestamp-0').AsBigInteger(); // get a big integer value + /// RecordsFound += 1; + /// until not ADLSEQuery.Next(); // Next will return false when there are no more records in the result set + /// Message('Records found: %1.', RecordsFound); + /// end; + /// + /// + /// + /// + /// The ADLSE Query facade provides a way to query data on + /// the lake, for any shared metadata table. + Access = Public; + + var + ADLSEQueryImpl: Codeunit "ADLSE Query Impl."; + + /// + /// States the table that needs to be queried from the lake. + /// + /// This procedure initializes the underlying query object and + /// must be the first call to be made when using this facade. + /// The name of the table to be queried. + procedure Init(TableName: Text) + begin + ADLSEQueryImpl.Init(TableName); + end; + + /// + /// Sets the SQL endpoint to be used in the connection string to make + /// queries on the lake. If the SQL endpoint is not set, the Synapse + /// Serverless SQL Endpoint configured on the setup page is used. + /// + /// The new SQL endpoint. + procedure SetServer(NewServer: Text) + begin + ADLSEQueryImpl.SetServer(NewServer); + end; + + /// + /// Sets the SQL database name to be used in the connection string to make + /// queries on the lake. If the SQL Database is not set, the SQL Database + /// Name configured on the setup page is used. + /// + /// The new database name. + procedure SetDatabase(NewDatabase: Text) + begin + ADLSEQueryImpl.SetDatabase(NewDatabase); + end; + + /// + /// Sets a simple filter, as a single value, on a field. + /// + /// The name of the field to be filtered on. + /// The value to which the field is to be + /// filtered. Convert option / enum values to their corresponding names + /// before passing them. + procedure SetRange(FieldName: Text; ValueVariant: Variant) + begin + ADLSEQueryImpl.SetRange(FieldName, ValueVariant); + end; + + /// + /// Sets a filter, based on a range of values, on a field. + /// + /// The name of the field to be filtered on. + /// The lower range value to which the field + /// is to be filtered. Convert option / enum values to their corresponding + /// names before passing them. + /// The upper range value to which the field + /// is to be filtered. Convert option / enum values to their corresponding + /// names before passing them. + procedure SetRange(FieldName: Text; FromValueVariant: Variant; ToValueVariant: Variant) + begin + ADLSEQueryImpl.SetRange(FieldName, FromValueVariant, ToValueVariant); + end; + + /// + /// Sets a generic filter on a field. + /// + /// The name of the field to be filtered on. + /// A text expression stating the type of + /// filter to be applied. The expressions supported are limited to those + /// stated in the enum "ADLSE Query Filter Operator". + /// The value to which the field is to be + /// filtered. Convert option / enum values to their corresponding names + /// before passing them. + procedure SetFilter(FieldName: Text; FilterExpression: Text; ValueVariant: Variant) + begin + ADLSEQueryImpl.SetFilter(FieldName, FilterExpression, ValueVariant); + end; + + /// + /// Orders the result against the field specified. The sorting is based on + /// the first field that is passed, followed by the next in a repeated call + /// of this procedure, and so on. The sorting will be in ascending order. + /// + /// The name of the field that the result should be + /// sorted against. + procedure SetOrderBy(FieldName: Text) + begin + ADLSEQueryImpl.SetOrderBy(FieldName); + end; + + /// + /// Orders the result against the field specified. The sorting is based on + /// the first field that is passed, followed by the next in a repeated call + /// of this procedure, and so on. + /// + /// The name of the field that the result should be + /// sorted against. + /// An optional boolean to specify the sorting + /// order- true implies sorted ascending and false, otherwise. + procedure SetOrderBy(FieldName: Text; Ascending: Boolean) + begin + ADLSEQueryImpl.SetOrderBy(FieldName, Ascending); + end; + + /// + /// States the field to be included in the result set when making a + /// FindSet call. If no fields are set, the result + /// contains all fields on the entity in the lake. + /// + /// The name of the field that should be included + /// in the result set. + procedure AddLoadField(FieldName: Text) + begin + ADLSEQueryImpl.AddLoadField(FieldName); + end; + + /// + /// Makes a call to fetch the records based on the filters and sorting + /// specified. The result points to the first record, if found. The fields + /// values can be requested by calling Field(). + /// Subsequent records can be pointed to by calling + /// Next(). + /// + /// True, if there are any records found; false otherwise. + procedure FindSet(): Boolean + begin + exit(ADLSEQueryImpl.FindSet()); + end; + + /// + /// Moves the cursor to the next record in the result set that was + /// populated originally by calling FindSet(). + /// + /// True, if the cursor could be moved to the next record; false + /// otherwise, say when the cursor is already on the last record. + procedure Next(): Boolean + begin + exit(ADLSEQueryImpl.Next()); + end; + + /// + /// The value of a field fetched from the record found as a result of the + /// FindSet() call. + /// + /// The name of the field that the result should be + /// sorted against. + /// The value of the field as a JsonValue variable. + /// + procedure Field(FieldName: Text): JsonValue + begin + exit(ADLSEQueryImpl.Field(FieldName)); + end; + + /// + /// Queries for records being present in the lake on a table with filters. + /// set previously. + /// + /// True if no records exist on the lake; false otherwise. + procedure IsEmpty(): Boolean + begin + exit(ADLSEQueryImpl.IsEmpty()); + end; + + /// + /// Queries for the number of records records being present in the lake on + /// a table with filters set previously. + /// + /// The number of records found. + procedure Count(): Integer + begin + exit(ADLSEQueryImpl.Count()); + end; + +} \ No newline at end of file diff --git a/businessCentral/src/Query/ADLSEQueryCredentials.Codeunit.al b/businessCentral/src/Query/ADLSEQueryCredentials.Codeunit.al new file mode 100644 index 0000000..5183bc9 --- /dev/null +++ b/businessCentral/src/Query/ADLSEQueryCredentials.Codeunit.al @@ -0,0 +1,180 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +codeunit 82575 "ADLSE Query Credentials" implements "ADLSE ICredentials" +{ + Access = Internal; + + var + ADLSECredentials: Codeunit "ADLSE Credentials"; + FunctionAppBaseUrl: Text; + [NonDebuggable] + ClientID: Text; + + [NonDebuggable] + ClientSecret: Text; + + [NonDebuggable] + FunctionKeys: Dictionary of [Text, Text]; + + Initialized: Boolean; + ValueNotFoundErr: Label 'No value found for %1.', Comment = '%1 = name of the key'; + ClientIdKeyNameTok: Label 'adlse-lookup-client-id', Locked = true; + ClientSecretKeyNameTok: Label 'adlse-lookup-client-secret', Locked = true; + FunctionKeysTok: Label 'adlse-function-keys', Locked = true; + ApiScopeTok: Label 'api://%1/user_impersonation', Locked = true; + + [NonDebuggable] + procedure Init(NewADLSECredentials: Codeunit "ADLSE Credentials"; NewFunctionAppUrl: Text) + begin + if IsInitialized() then + exit; + + ADLSECredentials := NewADLSECredentials; + + FunctionAppBaseUrl := NewFunctionAppUrl; + ClientID := GetSecret(ClientIdKeyNameTok); + ClientSecret := GetSecret(ClientSecretKeyNameTok); + InitFunctionKeys(); + + Initialized := true; + end; + + [NonDebuggable] + local procedure InitFunctionKeys() + var + FunctionKeysText: Text; + FunctionKeysJson: JsonObject; + Ky: Text; + ValJson: JsonToken; + ValText: Text; + begin + FunctionKeysText := GetSecret(FunctionKeysTok); + FunctionKeysJson.ReadFrom(FunctionKeysText); + foreach Ky in FunctionKeysJson.Keys() do begin + FunctionKeysJson.Get(Ky, ValJson); + ValText := ValJson.AsValue().AsText(); + FunctionKeys.Set(Ky, ValText); + end; + end; + + procedure IsInitialized(): Boolean + begin + exit(Initialized); + end; + + procedure Check() + var + ADLSESetup: Record "ADLSE Setup"; + NewADLSECredentials: Codeunit "ADLSE Credentials"; + begin + NewADLSECredentials.Check(); // pre-requisite + ADLSESetup.GetSingleton(); + ADLSESetup.TestField("Function App Url"); // must not be empty + + Init(NewADLSECredentials, ADLSESetup."Function App Url"); + CheckValueExists(ClientIdKeyNameTok, ClientID); + CheckValueExists(ClientSecretKeyNameTok, ClientSecret); + end; + + procedure GetFuntionAppBaseUrl(): Text + begin + exit(FunctionAppBaseUrl); + end; + + [NonDebuggable] + procedure GetClientID(): Text + begin + exit(ClientID); + end; + + [NonDebuggable] + procedure SetClientID(NewClientIDValue: Text): Text + begin + ClientID := NewClientIDValue; + SetSecret(ClientIdKeyNameTok, NewClientIDValue); + end; + + [NonDebuggable] + procedure GetClientSecret(): Text + begin + exit(ClientSecret); + end; + + [NonDebuggable] + procedure SetClientSecret(NewClientSecretValue: Text): Text + begin + ClientSecret := NewClientSecretValue; + SetSecret(ClientSecretKeyNameTok, NewClientSecretValue); + end; + + procedure GetTenantID(): Text + begin + exit(ADLSECredentials.GetTenantID()); + end; + + [NonDebuggable] + procedure GetResource(): Text + begin + exit(GetClientID()); + end; + + [NonDebuggable] + procedure GetScope(): Text + begin + exit(StrSubstNo(ApiScopeTok, GetClientID())); + end; + + [NonDebuggable] + procedure GetFunctionKey(FunctionName: Text): Text + begin + exit(FunctionKeys.Get(FunctionName)); + end; + + [NonDebuggable] + procedure SetFunctionKey(FunctionName: Text; KeyVal: Text): Text + var + JsonO: JsonObject; + AsText: Text; + Ky: Text; + begin + if not IsInitialized() then + InitFunctionKeys(); + + FunctionKeys.Set(FunctionName, KeyVal); + + foreach Ky in FunctionKeys.Keys do + JsonO.Add(Ky, FunctionKeys.Get(Ky)); + JsonO.WriteTo(AsText); + SetSecret(FunctionKeysTok, AsText); + end; + + [NonDebuggable] + local procedure GetSecret(KeyName: Text) Secret: Text + begin + if not IsolatedStorage.Contains(KeyName, IsolatedStorageDataScope()) then + exit(''); + IsolatedStorage.Get(KeyName, IsolatedStorageDataScope(), Secret); + end; + + [NonDebuggable] + local procedure SetSecret(KeyName: Text; Secret: Text) + begin + if EncryptionEnabled() then begin + IsolatedStorage.SetEncrypted(KeyName, Secret, IsolatedStorageDataScope()); + exit; + end; + IsolatedStorage.Set(KeyName, Secret, IsolatedStorageDataScope()); + end; + + [NonDebuggable] + local procedure CheckValueExists(KeyName: Text; Val: Text) + begin + if Val.Trim() = '' then + Error(ValueNotFoundErr, KeyName); + end; + + local procedure IsolatedStorageDataScope(): DataScope + begin + exit(DataScope::Module); // so that all companies share the same settings + end; +} \ No newline at end of file diff --git a/businessCentral/src/Query/ADLSEQueryFilterOperator.Enum.al b/businessCentral/src/Query/ADLSEQueryFilterOperator.Enum.al new file mode 100644 index 0000000..36f318b --- /dev/null +++ b/businessCentral/src/Query/ADLSEQueryFilterOperator.Enum.al @@ -0,0 +1,30 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +enum 82563 "ADLSE Query Filter Operator" +{ + Access = Internal; + + value(0; Equals) + { + } + + value(1; NotEquals) + { + } + + value(2; GreaterThan) + { + } + + value(3; GreaterThanOrEquals) + { + } + + value(4; LessThan) + { + } + + value(5; LessThanOrEquals) + { + } +} \ No newline at end of file diff --git a/businessCentral/src/Query/ADLSEQueryImpl.Codeunit.al b/businessCentral/src/Query/ADLSEQueryImpl.Codeunit.al new file mode 100644 index 0000000..86dd29d --- /dev/null +++ b/businessCentral/src/Query/ADLSEQueryImpl.Codeunit.al @@ -0,0 +1,282 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +codeunit 82574 "ADLSE Query Impl." +{ + Access = Internal; + + var + ADLSEQueryCredentials: Codeunit "ADLSE Query Credentials"; + Server: Text; + Database: Text; + TableName: Text; + SelectFields: JsonArray; + Filters: JsonArray; + OrderBys: JsonArray; + FindSetResult: JsonArray; + FindSetResultCurrentRowIndex: Integer; + IsInitialized: Boolean; + ServerCannotBeEmptyErr: Label 'Server cannot be empty. Have you called SetServer()?'; + DatabaseCannotBeEmptyErr: Label 'Database cannot be empty. Have you called SetDatabase()?'; + TableCannotBeEmptyErr: Label 'Table cannot be empty. Have you called Init()?'; + FieldMissingInResultSetErr: Label 'The field %1 is not present in the result. Make sure that the casing of the field name is correct and that this field has been loaded in the query.', Comment = '%1 is the field name'; + FilterExpressionNotSupportedErr: Label 'The filter expression %1 is not supported.', Comment = '%1 is the passed filter expression'; + FunctionApiUrlTok: Label '%1/api/%2', Comment = '%1 is the function app url, %2 is the function Api', Locked = true; + NotInitializedErr: Label 'The query api object needs to be initialized. Please call Init() first.'; + ApiCallFailedErr: Label 'The call to the URL %1 failed with status code %2: %3', Comment = '%1 is the url called for the api and %2 is the status code, %3 is the response body if any'; + ResultTokenMissingErr: Label 'Expected token ''result'' in response: %1', Comment = '%1 is the response from the function api call.'; + InvalidResponseErr: Label 'Invalid result in response: %1', Comment = '%1 is the response from the function api call.'; + NoRecordsFoundErr: Label 'No records found'; + UnexpectedJsonTokenForFieldErr: Label 'Expected a json value for the field %1. Got %2', Comment = '%1 is the field name queried, %2 is the text of the Json Token returned.'; + FieldAddedToOrderByErr: Label 'Field %1 has been added to the OrderBy clause already. You cannot add it twice.', Comment = '%1 is the field name'; + + procedure Init(NewTableName: Text) + var + ADLSESetup: Record "ADLSE Setup"; + begin + ADLSEQueryCredentials.Check(); + + ADLSESetup.GetSingleton(); + if Server = '' then + Server := ADLSESetup."Serverless SQL Endpoint"; + if Database = '' then + Database := ADLSESetup."SQL Database"; + + TableName := NewTableName; + Clear(SelectFields); + Clear(Filters); + Clear(OrderBys); + Clear(FindSetResult); + FindSetResultCurrentRowIndex := 0; + + IsInitialized := true; + end; + + procedure SetServer(NewServer: Text) + begin + Server := NewServer; + end; + + procedure SetDatabase(NewDatabase: Text) + begin + Database := NewDatabase; + end; + + procedure AddLoadField(NewFieldName: Text) + var + FieldToken: JsonToken; + FieldName: Text; + begin + CheckInitialized(); + foreach FieldToken in SelectFields do begin + FieldToken.WriteTo(FieldName); + if FieldName = NewFieldName then + exit; + end; + SelectFields.Add(NewFieldName); + end; + + procedure SetRange(FieldName: Text; ValueVariant: Variant) + begin + SetFilter(FieldName, "ADLSE Query Filter Operator"::Equals, ValueVariant); + end; + + procedure SetRange(FieldName: Text; FromValueVariant: Variant; ToValueVariant: Variant) + begin + SetFilter(FieldName, "ADLSE Query Filter Operator"::GreaterThanOrEquals, FromValueVariant); + SetFilter(FieldName, "ADLSE Query Filter Operator"::LessThanOrEquals, ToValueVariant); + end; + + procedure SetFilter(FieldName: Text; FilterExpression: Text; ValueVariant: Variant) + var + TrimmedExpression: Text; + begin + TrimmedExpression := FilterExpression.Replace(' ', ''); // remove all spaces + case TrimmedExpression of + '=%1': + SetFilter(FieldName, "ADLSE Query Filter Operator"::Equals, ValueVariant); + '<%1': + SetFilter(FieldName, "ADLSE Query Filter Operator"::LessThan, ValueVariant); + '<=%1': + SetFilter(FieldName, "ADLSE Query Filter Operator"::LessThanOrEquals, ValueVariant); + '>%1': + SetFilter(FieldName, "ADLSE Query Filter Operator"::GreaterThan, ValueVariant); + '>=%1': + SetFilter(FieldName, "ADLSE Query Filter Operator"::GreaterThanOrEquals, ValueVariant); + '<>%1': + SetFilter(FieldName, "ADLSE Query Filter Operator"::NotEquals, ValueVariant); + else + Error(FilterExpressionNotSupportedErr, FilterExpression); + end; + end; + + procedure SetFilter(FieldName: Text; FilterOp: enum "ADLSE Query Filter Operator"; ValueVariant: Variant) + var + ADLSEUtil: Codeunit "ADLSE Util"; + FilterObj: JsonObject; + ValueJson: JsonValue; + begin + CheckInitialized(); + FilterObj.Add('op', FilterOp.Names().Get(FilterOp.Ordinals().IndexOf(FilterOp.AsInteger()))); + FilterObj.Add('field', FieldName); + if not ADLSEUtil.ConvertVariantToJson(ValueVariant, ValueJson) then + ADLSEUtil.RaiseFieldTypeNotSupportedError(FieldName); + FilterObj.Add('value', ValueJson); + Filters.Add(FilterObj); + end; + + procedure SetOrderBy(FieldName: Text) + begin + SetOrderBy(FieldName, true); + end; + + procedure SetOrderBy(FieldName: Text; Ascending: Boolean) + var + OrderBy: JsonObject; + Token: JsonToken; + begin + CheckInitialized(); + if OrderBys.SelectToken('$[?(@.field==''' + FieldName + ''')]', Token) then + // field has been set to be ordered by already + Error(FieldAddedToOrderByErr, FieldName); + OrderBy.Add('field', FieldName); + OrderBy.Add('ascending', Ascending); + OrderBys.Add(OrderBy); + end; + + procedure FindSet(): Boolean + var + Payload: JsonObject; + Response: JsonToken; + begin + Payload := CreatePayload(); + + if SelectFields.Count() > 0 then + Payload.Add('fields', SelectFields); + if OrderBys.Count() > 0 then + Payload.Add('orderBy', OrderBys); + + Response := CallFunctionApi(GetFunctionFindSetToken(), Payload); + if not Response.IsArray() then + Error(InvalidResponseErr, Response); + + FindSetResultCurrentRowIndex := -1; + FindSetResult := Response.AsArray(); + if FindSetResult.Count() = 0 then + exit(false); + FindSetResultCurrentRowIndex := 0; // records found. Point at the first one. + exit(true); + end; + + procedure IsEmpty(): Boolean + var + Response: JsonToken; + begin + Response := CallFunctionApi(GetFunctionIsEmptyToken(), CreatePayload()); + if not Response.IsValue() then + Error(InvalidResponseErr, Response); + exit(Response.AsValue().AsBoolean()); + end; + + procedure Count(): Integer + var + Response: JsonToken; + begin + Response := CallFunctionApi(GetFunctionCountToken(), CreatePayload()); + if not Response.IsValue() then + Error(InvalidResponseErr, Response); + exit(Response.AsValue().AsInteger()); + end; + + local procedure CreatePayload() Payload: JsonObject + begin + if Server = '' then + Error(ServerCannotBeEmptyErr); + if Database = '' then + Error(DatabaseCannotBeEmptyErr); + if TableName = '' then + Error(TableCannotBeEmptyErr); + + Payload.Add('server', Server); + Payload.Add('database', Database); + Payload.Add('entity', TableName); + if Filters.Count() > 0 then + Payload.Add('filters', Filters); + end; + + [NonDebuggable] + local procedure CallFunctionApi(FunctionName: Text; Payload: JsonObject) ResultToken: JsonToken; + var + ADLSEHttp: Codeunit "ADLSE Http"; + Url: Text; + Request: Text; + Response: Text; + StatusCode: Integer; + Result: JsonObject; + begin + CheckInitialized(); + + ADLSEHttp.SetMethod("ADLSE Http Method"::Post); + Url := StrSubstNo(FunctionApiUrlTok, ADLSEQueryCredentials.GetFuntionAppBaseUrl(), FunctionName); + ADLSEHttp.SetUrl(Url); + ADLSEHttp.SetAuthorizationCredentials(ADLSEQueryCredentials); + ADLSEHttp.AddHeader('x-functions-key', ADLSEQueryCredentials.GetFunctionKey(FunctionName)); + ADLSEHttp.SetContentIsJson(); + + Payload.WriteTo(Request); + ADLSEHttp.SetBody(Request); + + if not ADLSEHttp.InvokeRestApi(Response, StatusCode) then + Error(ApiCallFailedErr, Url, StatusCode, Response); + + Result.ReadFrom(Response); + if not Result.Get('result', ResultToken) then + Error(ResultTokenMissingErr, Response); + end; + + procedure Next(): Boolean + begin + if FindSetResultCurrentRowIndex = FindSetResult.Count() - 1 then // at the last record + exit(false); + FindSetResultCurrentRowIndex += 1; + exit(true); + end; + + procedure Field(FieldName: Text): JsonValue + var + Result: JsonToken; + Value: JsonToken; + ValueAsText: Text; + begin + if FindSetResultCurrentRowIndex = -1 then + Error(NoRecordsFoundErr); + FindSetResult.Get(FindSetResultCurrentRowIndex, Result); + if not Result.AsObject().Get(FieldName, Value) then + Error(FieldMissingInResultSetErr, FieldName); + if not Value.IsValue() then begin + Value.WriteTo(ValueAsText); + Error(UnexpectedJsonTokenForFieldErr, FieldName, ValueAsText); + end; + exit(Value.AsValue()); + end; + + procedure GetFunctionFindSetToken(): Text + begin + exit('FindSet'); + end; + + procedure GetFunctionIsEmptyToken(): Text + begin + exit('IsEmpty'); + end; + + procedure GetFunctionCountToken(): Text + begin + exit('Count'); + end; + + local procedure CheckInitialized() + begin + if not IsInitialized then + Error(NotInitializedErr); + end; +} \ No newline at end of file diff --git a/businessCentral/src/Query/ADLSEQueryTable.Codeunit.al b/businessCentral/src/Query/ADLSEQueryTable.Codeunit.al new file mode 100644 index 0000000..88e2792 --- /dev/null +++ b/businessCentral/src/Query/ADLSEQueryTable.Codeunit.al @@ -0,0 +1,211 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +codeunit 82578 "ADLSE Query Table" +{ + /// + /// This is the facade to query data in the data lake for tables that have + /// already been configured in the main setup page. The data will be + /// filtered to the current company, if the table is per company. + /// + /// Here is an example of how to use this facade. + /// + /// + /// procedure QueryCustomerLedgerEntryFromLake() + /// var + /// CustLedgerEntry: Record "Cust. Ledger Entry"; + /// GLRegister: Record "G/L Register"; + /// ADLSEQueryTable: Codeunit "ADLSE Query Table"; + /// EntryNo: Integer; + /// CustomerNo: Code[20]; + /// DocumentType: Enum "Gen. Journal Document Type"; + /// Timestamp: BigInteger; + /// PostingDate: Date; + /// ModifiedDateTime: DateTime; + /// GLRegisterNo: Integer; + /// CreationTime: Time; + /// RecordsFound: Integer; + /// begin + /// // first set the table to be queried + /// ADLSEQueryTable.Open(Database::"Cust. Ledger Entry"); + /// + /// // set the filters to be applied + /// ADLSEQueryTable.SetRange(CustLedgerEntry.FieldNo("Document Type"), "Gen. Journal Document Type"::Invoice); + /// ADLSEQueryTable.SetFilter(CustLedgerEntry.FieldNo("Customer No."), '>=%1', '40000'); + /// ADLSEQueryTable.SetFilter(CustLedgerEntry.FieldNo("Posting Date"), '>%1', 20211023D); + /// + /// // set the result to be sorted first by descending order of Posting Date and then ascending order of Entry No. + /// ADLSEQueryTable.SetOrderBy(CustLedgerEntry.FieldNo("Posting Date"), false); + /// ADLSEQueryTable.SetOrderBy(CustLedgerEntry.FieldNo("Entry No.")); + /// + /// // make the find set query + /// if ADLSEQueryTable.FindSet() then + /// // records found + /// repeat + /// EntryNo := ADLSEQueryTable.Field(CustLedgerEntry.FieldNo("Entry No.")); // get an integer value + /// CustomerNo := ADLSEQueryTable.Field(CustLedgerEntry.FieldNo("Customer No.")); // get a code value + /// DocumentType := Enum::"Gen. Journal Document Type".FromInteger( + /// ADLSEQueryTable.Field(CustLedgerEntry.FieldNo("Document Type"))); // get an enum value + /// PostingDate := ADLSEQueryTable.Field(CustLedgerEntry.FieldNo("Posting Date")); // get a date value + /// ModifiedDateTime := ADLSEQueryTable.Field(CustLedgerEntry.FieldNo(SystemModifiedAt)); // get a datetime value + /// Timestamp := ADLSEQueryTable.Field(0); // get a big integer + /// RecordsFound += 1; + /// until not ADLSEQueryTable.Next(); // Next will return false when there are no more records in the result set + /// Message('Records found: %1.', RecordsFound); + /// end; + /// + /// + /// + /// + /// The ADLSE Query facade provides + /// a more general way to query data on the lake, especially when the + /// entity does not correspond to any table in Dynamics 365 Business + /// Central. All calls to the lake from this facade actually go through an + /// instance of the ADLSE Query facade. + /// + + Access = Public; + + var + ADLSEQueryTableImpl: Codeunit "ADLSE Query Table Impl."; + + /// + /// States the table that needs to be queried from the lake. This table + /// must be added to the list of tables in the setup page. + /// + /// This procedure initializes the underlying query object and + /// must be the first call to be made when using this facade. + /// The integer identifier for the table to be + /// queried. + procedure Open(TableID: Integer) + begin + ADLSEQueryTableImpl.Init(TableID); + end; + + /// + /// Sets a simple filter, as a single value, on a field. The field must be + /// present in the table and must also be enabled in the setup. + /// + /// The integer identifier for the field to be + /// filtered on. + /// The value to which the field is to be + /// filtered. + procedure SetRange(FieldID: Integer; ValueVariant: Variant) + begin + ADLSEQueryTableImpl.SetRange(FieldID, ValueVariant); + end; + + /// + /// Sets a filter, based on a range of values, on a field. The field must + /// be present in the table and must also be enabled in the setup. + /// + /// The integer identifier for the field to be + /// filtered on. + /// The lower range value to which the field + /// is to be filtered. + /// The upper range value to which the field + /// is to be filtered. + procedure SetRange(FieldID: Integer; FromValueVariant: Variant; ToValueVariant: Variant) + begin + ADLSEQueryTableImpl.SetRange(FieldID, FromValueVariant, ToValueVariant); + end; + + /// + /// Sets a generic filter on a field. + /// + /// The integer identifier for the field to be + /// filtered on. + /// A text expression stating the type of + /// filter to be applied. The expressions supported are limited to those + /// stated in the enum "ADLSE Query Filter Operator". + /// The value to which the field is to be + /// filtered. + procedure SetFilter(FieldID: Integer; FilterExpression: Text; ValueVariant: Variant) + begin + ADLSEQueryTableImpl.SetFilter(FieldID, FilterExpression, ValueVariant); + end; + + /// + /// Orders the result against the field specified. The sorting is based on + /// the first field that is passed, followed by the next in a repeated call + /// of this procedure, and so on. + /// + /// The integer identifier of the field that the + /// result should be sorted against. + procedure SetOrderBy(FieldID: Integer) + begin + ADLSEQueryTableImpl.SetOrderBy(FieldID); + end; + + /// + /// Orders the result against the field specified. The sorting is based on + /// the first field that is passed, followed by the next in a repeated call + /// to this procedure, and so on. + /// + /// The integer identifier of the field that the + /// result should be sorted against. + /// An optional boolean to specify the sorting + /// order- true implies sorted ascending and false, otherwise. + procedure SetOrderBy(FieldID: Integer; Ascending: Boolean) + begin + ADLSEQueryTableImpl.SetOrderBy(FieldID, Ascending); + end; + + /// + /// Makes a call to fetch the records based on the filters and sorting + /// specified. The result points to the first record, if found. The fields + /// values can be requested by calling Field(). + /// Subsequent records can be pointed to by calling + /// Next(). + /// + /// True, if there are any records found; false otherwise. + procedure FindSet(): Boolean + begin + exit(ADLSEQueryTableImpl.FindSet()); + end; + + /// + /// Moves the cursor to the next record in the result set that was + /// populated originally by calling FindSet(). + /// + /// True, if the cursor could be moved to the next record; false + /// otherwise, say when the cursor is already on the last record. + procedure Next(): Boolean + begin + exit(ADLSEQueryTableImpl.Next()); + end; + + /// + /// The value of a field fetched from the record found as a result of the + /// FindSet() call. + /// + /// The integer identifier of the field that the + /// result should be sorted against. + /// The value of the field. In case of Option/ Enum type fields, + /// this returns the ordinal integer value which may need to be converted + /// to the respective enum. + procedure Field(FieldID: Integer): Variant + begin + exit(ADLSEQueryTableImpl.Field(FieldID)); + end; + + /// + /// Queries for records being present in the lake on a table with filters. + /// set previously. + /// + /// True if no records exist on the lake; false otherwise. + procedure IsEmpty(): Boolean + begin + exit(ADLSEQueryTableImpl.IsEmpty()); + end; + + /// + /// Queries for the number of records records being present in the lake on + /// a table with filters set previously. + /// + /// The number of records found. + procedure Count(): Integer + begin + exit(ADLSEQueryTableImpl.Count()); + end; + +} \ No newline at end of file diff --git a/businessCentral/src/Query/ADLSEQueryTableImpl.Codeunit.al b/businessCentral/src/Query/ADLSEQueryTableImpl.Codeunit.al new file mode 100644 index 0000000..723c729 --- /dev/null +++ b/businessCentral/src/Query/ADLSEQueryTableImpl.Codeunit.al @@ -0,0 +1,143 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +codeunit 82576 "ADLSE Query Table Impl." +{ + Access = Internal; + + var + ADLSEQuery: Codeunit "ADLSE Query"; + ADLSEUtil: Codeunit "ADLSE Util"; + SystemFieldsList: List of [Integer]; + TableNumber: Integer; + TableNotConfiguredErr: Label 'The table %1 has not been configured on the setup page.', Comment = '%1 is the table caption'; + FieldNotConfiguredErr: Label 'The field with Id %1 on table %2 has not been configured on the setup page.', Comment = '%1 is the field ID and %2 is the table caption'; + FieldNotEnabledErr: Label 'The field %1 in table %2 is not enabled in the configuration', Comment = '%1 is the field caption, %2 is the table caption'; + + procedure Init(TableID: Integer) + var + ADLSETable: Record "ADLSE Table"; + ADLSECDMUtil: Codeunit "ADLSE CDM Util"; + begin + if not ADLSETable.Get(TableID) then + Error(TableNotConfiguredErr, ADLSEUtil.GetTableCaption(TableID)); + ADLSEQuery.Init(ADLSEUtil.GetDataLakeCompliantTableName(TableID).ToLower().Replace('-', '_')); + TableNumber := TableID; + + if ADLSEUtil.IsTablePerCompany(TableNumber) then + ADLSEQuery.SetRange(ADLSECDMUtil.GetCompanyFieldName(), CompanyName()); + + Clear(SystemFieldsList); + ADLSEUtil.AddSystemFields(SystemFieldsList); + end; + + procedure SetRange(FieldID: Integer; ValueVariant: Variant) + begin + CheckField(FieldID); + ADLSEQuery.SetRange(ADLSEUtil.GetDataLakeCompliantFieldName(TableNumber, FieldID), CurateVariant(FieldId, ValueVariant)); + end; + + procedure SetRange(FieldID: Integer; FromValueVariant: Variant; ToValueVariant: Variant) + begin + CheckField(FieldID); + ADLSEQuery.SetRange(ADLSEUtil.GetDataLakeCompliantFieldName(TableNumber, FieldID), CurateVariant(FieldId, FromValueVariant), CurateVariant(FieldId, ToValueVariant)); + end; + + procedure SetFilter(FieldID: Integer; FilterExpression: Text; ValueVariant: Variant) + begin + CheckField(FieldID); + ADLSEQuery.SetFilter(ADLSEUtil.GetDataLakeCompliantFieldName(TableNumber, FieldID), FilterExpression, CurateVariant(FieldId, ValueVariant)); + end; + + local procedure CurateVariant(FieldID: Integer; ValueVariant: Variant): Variant + var + FieldRef: FieldRef; + begin + FieldRef := GetFieldRef(FieldID); + if FieldRef.Type = FieldType::Option then + exit(FieldRef.GetEnumValueNameFromOrdinalValue(ValueVariant)); + exit(ValueVariant); + end; + + procedure SetOrderBy(FieldID: Integer) + begin + CheckField(FieldID); + ADLSEQuery.SetOrderBy(ADLSEUtil.GetDataLakeCompliantFieldName(TableNumber, FieldID)); + end; + + procedure SetOrderBy(FieldID: Integer; Ascending: Boolean) + begin + CheckField(FieldID); + ADLSEQuery.SetOrderBy(ADLSEUtil.GetDataLakeCompliantFieldName(TableNumber, FieldID), Ascending); + end; + + local procedure CheckField(FieldID: Integer) + var + ADLSEField: Record "ADLSE Field"; + begin + if SystemFieldsList.Contains(FieldID) then + exit; + if not ADLSEField.Get(TableNumber, FieldID) then + Error(FieldNotConfiguredErr, FieldID, ADLSEUtil.GetTableCaption(TableNumber)); + if not ADLSEField.Enabled then + Error(FieldNotEnabledErr, ADLSEField.FieldCaption, ADLSEUtil.GetTableCaption(TableNumber)); + end; + + procedure FindSet(): Boolean + var + ADLSEField: Record "ADLSE Field"; + SystemFieldID: Integer; + begin + ADLSEField.SetRange("Table ID", TableNumber); + ADLSEField.SetRange(Enabled, true); + if ADLSEField.FindSet() then + repeat + ADLSEQuery.AddLoadField(GetFieldNameOnTheLake(ADLSEField."Field ID")); + until ADLSEField.Next() = 0; + // also add System Audit fields + foreach SystemFieldID in SystemFieldsList do + ADLSEQuery.AddLoadField(GetFieldNameOnTheLake(SystemFieldID)); + + exit(ADLSEQuery.FindSet()); + end; + + procedure IsEmpty(): Boolean + begin + exit(ADLSEQuery.IsEmpty()); + end; + + procedure Count(): Integer + begin + exit(ADLSEQuery.Count()); + end; + + procedure Next(): Boolean + begin + exit(ADLSEQuery.Next()); + end; + + procedure Field(FieldID: Integer) VariantValue: Variant + var + Value: JsonValue; + begin + CheckField(FieldID); + Value := ADLSEQuery.Field(GetFieldNameOnTheLake(FieldID)); + + VariantValue := ADLSEUtil.ConvertJsonToVariant(GetFieldRef(FieldID), Value); + end; + + local procedure GetFieldRef(FieldID: Integer): FieldRef + var + RecordRef: RecordRef; + begin + RecordRef.Open(TableNumber); + exit(RecordRef.Field(FieldID)); + end; + + local procedure GetFieldNameOnTheLake(FieldID: Integer): Text + var + FieldRef: FieldRef; + begin + FieldRef := GetFieldRef(FieldID); + exit(ADLSEUtil.GetDataLakeCompliantFieldName(FieldRef.Name, FieldID)); + end; +} \ No newline at end of file diff --git a/businessCentral/src/Query/ADLSESetupQuery.Page.al b/businessCentral/src/Query/ADLSESetupQuery.Page.al new file mode 100644 index 0000000..dc7b41f --- /dev/null +++ b/businessCentral/src/Query/ADLSESetupQuery.Page.al @@ -0,0 +1,139 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. See LICENSE in the project root for license information. +page 82567 "ADLSE Setup Query" +{ + Caption = 'Query data in the lake'; + PageType = CardPart; + SourceTable = "ADLSE Setup"; + InsertAllowed = false; + DeleteAllowed = false; + + layout + { + area(Content) + { + group(Connection) + { + Caption = 'Default connection details'; + + field("Serverless SQL Endpoint"; Rec."Serverless SQL Endpoint") + { + Caption = 'Synapse Serverless SQL Endpoint'; + ApplicationArea = All; + ToolTip = 'Specifies the Synapse Serverless SQL Endpoint which hosts the SQL database holding the shared metadata tables.'; + } + + field("SQL Database"; Rec."SQL Database") + { + Caption = 'SQL Database Name'; + ApplicationArea = All; + ToolTip = 'Specifies the SQL database name holding the shared metadata tables.'; + } + } + + group(Authentication) + { + Caption = 'App registration'; + + field("Lookup Client ID"; LookupClientID) + { + Caption = 'Client ID'; + ApplicationArea = All; + ExtendedDatatype = Masked; + Tooltip = 'Specifies the application client ID for the Azure Function App that queries the Synapse serverless SQL endpoint.'; + + trigger OnValidate() + begin + ADLSEQueryCredentials.SetClientID(LookupClientID); + end; + } + + field("Lookup Client secret"; LookupClientSecret) + { + Caption = 'Client secret'; + ApplicationArea = All; + ExtendedDatatype = Masked; + Tooltip = 'Specifies the client secret for the Azure Function App that queries the Synapse serverless SQL endpoint.'; + + trigger OnValidate() + begin + ADLSEQueryCredentials.SetClientSecret(LookupClientSecret); + end; + } + } + + group(API) + { + Caption = 'Function Api'; + + field("Function App URL"; Rec."Function App Url") + { + Caption = 'Function app url'; + ApplicationArea = All; + ExtendedDatatype = URL; + ToolTip = 'Specifies the URL of the function app that queries the Synapse serverless SQL database.'; + } + + field("Function Key FindSet"; FunctionKeyFindSet) + { + Caption = 'Function key FindSet'; + ApplicationArea = All; + ExtendedDatatype = Masked; + ToolTip = 'Specifies a function key that authorizes the FindSet Api call on the function app.'; + + trigger OnValidate() + var + ADLSEQuery: Codeunit "ADLSE Query Impl."; + begin + ADLSEQueryCredentials.SetFunctionKey(ADLSEQuery.GetFunctionFindSetToken(), FunctionKeyFindSet); + end; + } + + field("Function Key IsEmpty"; FunctionKeyIsEmpty) + { + Caption = 'Function key IsEmpty'; + ApplicationArea = All; + ExtendedDatatype = Masked; + ToolTip = 'Specifies a function key that authorizes the IsEmpty Api call on the function app.'; + + trigger OnValidate() + var + ADLSEQuery: Codeunit "ADLSE Query Impl."; + begin + ADLSEQueryCredentials.SetFunctionKey(ADLSEQuery.GetFunctionIsEmptyToken(), FunctionKeyIsEmpty); + end; + } + + field("Function Key Count"; FunctionKeyCount) + { + Caption = 'Function key Count'; + ApplicationArea = All; + ExtendedDatatype = Masked; + ToolTip = 'Specifies a function key that authorizes the Count Api call on the function app.'; + + trigger OnValidate() + var + ADLSEQuery: Codeunit "ADLSE Query Impl."; + begin + ADLSEQueryCredentials.SetFunctionKey(ADLSEQuery.GetFunctionCountToken(), FunctionKeyCount); + end; + } + } + + } + } + + var + ADLSEQueryCredentials: Codeunit "ADLSE Query Credentials"; + [NonDebuggable] + [NonDebuggable] + LookupClientID: Text; + [NonDebuggable] + LookupClientSecret: Text; + [NonDebuggable] + FunctionKeyFindSet: Text; + [NonDebuggable] + FunctionKeyIsEmpty: Text; + [NonDebuggable] + FunctionKeyCount: Text; +} \ No newline at end of file