diff --git a/conf.py b/conf.py index d4c04ed5d..0f52caebb 100644 --- a/conf.py +++ b/conf.py @@ -26,7 +26,7 @@ # The full version, including alpha/beta/rc tags -release = '2022.1' +release = '2021.2.1.24' @@ -68,7 +68,7 @@ 'css/custom.css', # Relative to the _static path ] -html_logo = '_static/images/sqream_logo.png' +html_logo = '_static/images/SQream_logo_without background-15.png' # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. @@ -90,7 +90,7 @@ 'logo_only': True # Hide "SQream DB" title and only show logo , 'display_version': True # Display version at the top , 'style_external_links': True # Show little icon next to external links - , 'style_nav_header_background': '#0f9790' # SQream teal + , 'style_nav_header_background': '#133148' # SQream teal , 'navigation_depth': -1 , 'collapse_navigation': False , 'titles_only': True diff --git a/configuration_guides/admin_cluster_flags.rst b/configuration_guides/admin_cluster_flags.rst index 3c74819d6..115ca50ae 100644 --- a/configuration_guides/admin_cluster_flags.rst +++ b/configuration_guides/admin_cluster_flags.rst @@ -6,4 +6,4 @@ Cluster Administration Flags The **Cluster Administration Flags** page describes **Cluster** modification type flags, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: -* `Persisting Your Cache Directory `_ \ No newline at end of file +* `Setting Maximum CSV Row Length `_ \ No newline at end of file diff --git a/configuration_guides/admin_regular_flags.rst b/configuration_guides/admin_regular_flags.rst index 5300310b6..effc7ab40 100644 --- a/configuration_guides/admin_regular_flags.rst +++ b/configuration_guides/admin_regular_flags.rst @@ -5,27 +5,28 @@ Regular Administration Flags ************************* The **Regular Administration Flags** page describes **Regular** modification type flags, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: -* `Setting Bin Size `_ -* `Setting CUDA Memory `_ -* `Limiting Runtime to Utility Functions `_ -* `Enabling High Bin Control Granularity `_ -* `Reducing CPU Hashtable Sizes `_ -* `Setting Chunk Size for Copying from CPU to GPU `_ -* `Indicating GPU Synchronicity `_ -* `Enabling Modification of R&D Flags `_ -* `Checking for Post-Production CUDA Errors `_ -* `Enabling Modification of clientLogger_debug File `_ -* `Activating the NVidia Profiler Markers `_ -* `Appending String at End of Log Lines `_ -* `Monitoring and Printing Pinned Allocation Reports `_ -* `Increasing Chunk Size to Reduce Query Speed `_ -* `Adding Rechunker before Expensing Chunk Producer `_ -* `Setting the Buffer Size `_ -* `Setting Memory Used to Abort Server `_ -* `Splitting Large Reads for Concurrent Execution `_ -* `Setting Worker Amount to Handle Concurrent Reads `_ -* `Setting Implicit Casts in ORC Files `_ -* `Setting Timeout Limit for Locking Objects before Executing Statements `_ -* `Interpreting Decimal Literals as Double Instead of Numeric `_ -* `Interpreting VARCHAR as TEXT `_ -* `VARCHAR Identifiers `_ +* `Setting Bin Size `_ +* `Setting CUDA Memory `_ +* `Limiting Runtime to Utility Functions `_ +* `Enabling High Bin Control Granularity `_ +* `Reducing CPU Hashtable Sizes `_ +* `Setting Chunk Size for Copying from CPU to GPU `_ +* `Indicating GPU Synchronicity `_ +* `Enabling Modification of R&D Flags `_ +* `Checking for Post-Production CUDA Errors `_ +* `Enabling Modification of clientLogger_debug File `_ +* `Activating the NVidia Profiler Markers `_ +* `Appending String at End of Log Lines `_ +* `Monitoring and Printing Pinned Allocation Reports `_ +* `Increasing Chunk Size to Reduce Query Speed `_ +* `Adding Rechunker before Expensing Chunk Producer `_ +* `Setting the Buffer Size `_ +* `Maximum Pinned Percentage of Total RAM `_ +* `Setting Memory Used to Abort Server `_ +* `Splitting Large Reads for Concurrent Execution `_ +* `Setting Worker Amount to Handle Concurrent Reads `_ +* `Setting Implicit Casts in ORC Files `_ +* `Setting Timeout Limit for Locking Objects before Executing Statements `_ +* `Interpreting Decimal Literals as Double Instead of Numeric `_ +* `Interpreting VARCHAR as TEXT `_ +* `VARCHAR Identifiers `_ diff --git a/configuration_guides/admin_worker_flags.rst b/configuration_guides/admin_worker_flags.rst index 2be570695..908c628b4 100644 --- a/configuration_guides/admin_worker_flags.rst +++ b/configuration_guides/admin_worker_flags.rst @@ -5,7 +5,10 @@ Worker Administration Flags ************************* The **Worker Administration Flags** page describes **Worker** modification type flags, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: -* `Setting Total Device Memory Usage in SQream Instance `_ -* `Enabling Manually Setting Reported IP `_ -* `Setting Port Used for Metadata Server Connection `_ -* `Assigning Local Network IP `_ \ No newline at end of file +* `Setting Total Device Memory Usage in SQream Instance `_ +* `Enabling Manually Setting Reported IP `_ +* `Setting Port Used for Metadata Server Connection `_ +* `Assigning Local Network IP `_ +* `Enabling the Query Healer `_ +* `Configuring the Query Healer `_ +* `Adjusting Permitted Log-in Attempts `_ \ No newline at end of file diff --git a/configuration_guides/check_cuda_memory.rst b/configuration_guides/check_cuda_memory.rst index 84eec3f07..9ac78f767 100644 --- a/configuration_guides/check_cuda_memory.rst +++ b/configuration_guides/check_cuda_memory.rst @@ -8,4 +8,5 @@ The ``checkCudaMemory`` flag sets the pad device memory allocation with safety b The following describes the ``checkCudaMemory`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/compiler_gets_only_ufs.rst b/configuration_guides/compiler_gets_only_ufs.rst index 1190adc3e..00e8b44d8 100644 --- a/configuration_guides/compiler_gets_only_ufs.rst +++ b/configuration_guides/compiler_gets_only_ufs.rst @@ -8,4 +8,5 @@ The ``compilerGetsOnlyUFs`` flag sets the runtime to pass only utility functions The following describes the ``compilerGetsOnlyUFs`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/configuration_methods.rst b/configuration_guides/configuration_methods.rst deleted file mode 100644 index 00a70ab5c..000000000 --- a/configuration_guides/configuration_methods.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _configuration_methods: - -************************* -Configuration Methods -************************* -SQream provides two methods for configuration your instance of SQream. The current configuration method is based on cluster and session-based configuration, described in more detail below. Users can also use the previous configuration method done using a configuration file. - -The **Configuration Methods** page describes the following configurations methods: - -.. toctree:: - :maxdepth: 1 - :glob: - :titlesonly: - - current_configuration_method - previous_configuration_method - - - diff --git a/configuration_guides/configuring_sqream.rst b/configuration_guides/configuring_sqream.rst new file mode 100644 index 000000000..901af9e3d --- /dev/null +++ b/configuration_guides/configuring_sqream.rst @@ -0,0 +1,22 @@ +.. _configuring_sqream: + +************************* +Configuring SQream +************************* +The **Configuring SQream** page describes the following configuration topics: + +.. toctree:: + :maxdepth: 1 + :glob: + :titlesonly: + + current_method_configuration_levels + current_method_flag_types + current_method_configuration_roles + current_method_modification_methods + current_method_configuring_your_parameter_values + current_method_command_examples + current_method_showing_all_flags_in_the_catalog_table + current_method_all_configurations + + diff --git a/configuration_guides/cpu_reduce_hashtable_size.rst b/configuration_guides/cpu_reduce_hashtable_size.rst index c2b01604c..d32450cd3 100644 --- a/configuration_guides/cpu_reduce_hashtable_size.rst +++ b/configuration_guides/cpu_reduce_hashtable_size.rst @@ -1,11 +1,12 @@ .. _cpu_reduce_hashtable_size: ************************* -Enabling High Bin Control Granularity +Setting Hashtable Size for GROUP BY Queries ************************* -The ``copyToRestrictUtf8`` flag sets the custom bin size in the cache to enable high bin control granularity. +The ``cpuReduceHashtableSize`` flag sets the size of the hashtable in your CPU while running a ``GROUP BY`` query. -The following describes the ``checkCudaMemory`` flag: +The following describes the ``cpuReduceHashtableSize`` flag: -* **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Data type** - uint +* **Default value** - ``10000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/csv_limit_row_length.rst b/configuration_guides/csv_limit_row_length.rst index 03f31c697..41bc124df 100644 --- a/configuration_guides/csv_limit_row_length.rst +++ b/configuration_guides/csv_limit_row_length.rst @@ -8,4 +8,5 @@ The ``csvLimitRowLength`` flag sets the maximum supported CSV row length. The following describes the ``csvLimitRowLength`` flag: * **Data type** - uint -* **Default value** - ``100000`` \ No newline at end of file +* **Default value** - ``100000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/cuda_mem_cpy_max_size_bytes.rst b/configuration_guides/cuda_mem_cpy_max_size_bytes.rst index 371c9bda4..c28827077 100644 --- a/configuration_guides/cuda_mem_cpy_max_size_bytes.rst +++ b/configuration_guides/cuda_mem_cpy_max_size_bytes.rst @@ -8,4 +8,5 @@ The ``cudaMemcpyMaxSizeBytes`` flag sets the chunk size for copying from CPU to The following describes the ``cudaMemcpyMaxSizeBytes`` flag: * **Data type** - uint -* **Default value** - ``0`` \ No newline at end of file +* **Default value** - ``0`` +* **Allowed values** - 0-4000000000 \ No newline at end of file diff --git a/configuration_guides/cuda_mem_cpy_synchronous.rst b/configuration_guides/cuda_mem_cpy_synchronous.rst index 81e762071..2f8662947 100644 --- a/configuration_guides/cuda_mem_cpy_synchronous.rst +++ b/configuration_guides/cuda_mem_cpy_synchronous.rst @@ -8,4 +8,5 @@ The ``CudaMemcpySynchronous`` flag indicates if copying from/to GPU is synchrono The following describes the ``CudaMemcpySynchronous`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/cuda_mem_quota.rst b/configuration_guides/cuda_mem_quota.rst index 43f9d4943..d5f417978 100644 --- a/configuration_guides/cuda_mem_quota.rst +++ b/configuration_guides/cuda_mem_quota.rst @@ -9,3 +9,4 @@ The following describes the ``cudaMemQuota`` flag: * **Data type** - uint * **Default value** - ``90`` +* **Allowed values** - 0-100 diff --git a/configuration_guides/current_configuration_method.rst b/configuration_guides/current_configuration_method.rst deleted file mode 100644 index 5c1a64a85..000000000 --- a/configuration_guides/current_configuration_method.rst +++ /dev/null @@ -1,729 +0,0 @@ -.. _current_configuration_method: - -************************** -Configuring SQream -************************** -The **Configuring SQream** page describes SQream’s method for configuring your instance of SQream and includes the following topics: - -.. contents:: - :local: - :depth: 1 - -Overview ------ -Modifications that you make to your configurations are persistent based on whether they are made at the session or cluster level. Persistent configurations are modifications made to attributes that are retained after shutting down your system. - -Modifying Your Configuration ----- -The **Modifying Your Configuration** section describes the following: - -.. contents:: - :local: - :depth: 1 - -Modifying Your Configuration Using the Worker Configuration File -~~~~~~~~~~~ -You can modify your configuration using the **worker configuration file (config.json)**. Changes that you make to worker configuration files are persistent. Note that you can only set the attributes in your worker configuration file **before** initializing your SQream worker, and while your worker is active these attributes are read-only. - -The following is an example of a worker configuration file: - -.. code-block:: postgres - - { - “cluster”: “/home/test_user/sqream_testing_temp/sqreamdb”, - “gpu”: 0, - “licensePath”: “home/test_user/SQream/tests/license.enc”, - “machineIP”: “127.0.0.1”, - “metadataServerIp”: “127.0.0.1”, - “metadataServerPort”: “3105, - “port”: 5000, - “useConfigIP”” true, - “legacyConfigFilePath”: “home/SQream_develop/SqrmRT/utils/json/legacy_congif.json” - } - -You can access the legacy configuration file from the ``legacyConfigFilePath`` parameter shown above. If all (or most) of your workers require the same flag settings, you can set the ``legacyConfigFilePath`` attribute to the same legacy file. - -Modifying Your Configuration Using a Legacy Configuration File -~~~~~~~~~~~ -You can modify your configuration using a legacy configuration file. - -The Legacy configuration file provides access to the read/write flags used in SQream’s previous configuration method. A link to this file is provided in the **legacyConfigFilePath** parameter in the worker configuration file. - -The following is an example of the legacy configuration file: - -.. code-block:: postgres - - { - “developerMode”: true, - “reextentUse”: false, - “useClientLog”: true, - “useMetadataServer”” false - } - -Session vs Cluster Based Configuration -============================== -.. contents:: - :local: - :depth: 1 - -Cluster-Based Configuration --------------- -SQream uses cluster-based configuration, enabling you to centralize configurations for all workers on the cluster. Only flags set to the regular or cluster flag type have access to cluster-based configuration. Configurations made on the cluster level are persistent and stored at the metadata level. The parameter settings in this file are applied globally to all workers connected to it. - -For more information, see the following: - -* `Using SQream SQL `_ - modifying flag attributes from the CLI. -* `SQream Acceleration Studio `_ - modifying flag attributes from Studio. - -For more information on flag-based access to cluster-based configuration, see **Configuration Flag Types** below. - -Session-Based Configuration ----------------- -Session-based configurations are not persistent and are deleted when your session ends. This method enables you to modify all required configurations while avoiding conflicts between flag attributes modified on different devices at different points in time. - -The **SET flag_name** command is used to modify flag attributes. Any modifications you make with the **SET flag_name** command apply only to your open session, and are not saved when it ends - -For example, when the query below has completed executing, the values configured will be restored to its previous setting: - -.. code-block:: console - - set spoolMemoryGB=700; - select * from table a where date='2021-11-11' - -For more information, see the following: - -* `Using SQream SQL `_ - modifying flag attributes from the CLI. -* `SQream Acceleration Studio `_ - modifying flag attributes from Studio. - -Configuration Flag Types -========== -The flag type attribute can be set for each flag and determines its write access as follows: - -* **Administration:** session-based read/write flags that can be stored in the metadata file. -* **Cluster:** global cluster-based read/write flags that can be stored in the metadata file. -* **Worker:** single worker-based read-only flags that can be stored in the worker configuration file. - -The flag type determines which files can be accessed and which commands or commands sets users can run. - -The following table describes the file or command modification rights for each flag type: - -.. list-table:: - :widths: 20 20 20 20 - :header-rows: 1 - - * - **Flag Type** - - **Legacy Configuration File** - - **ALTER SYSTEM SET** - - **Worker Configuration File** - * - :ref:`Regular` - - Can modify - - Can modify - - Cannot modify - * - :ref:`Cluster` - - Cannot modify - - Can modify - - Cannot modify - * - :ref:`Worker` - - Cannot modify - - Cannot modify - - Can modify - -.. _regular_flag_types: - -Regular Flag Types ---------------------- -The following is an example of the correct syntax for running a **Regular** flag type command: - -.. code-block:: console - - SET spoolMemoryGB= 11; - executed - -The following table describes the Regular flag types: - -.. list-table:: - :widths: 2 5 10 - :header-rows: 1 - - * - **Command** - - **Description** - - **Example** - * - ``SET `` - - Used for modifying flag attributes. - - ``SET developerMode=true`` - * - ``SHOW / ALL`` - - Used to preset either a specific flag value or all flag values. - - ``SHOW `` - * - ``SHOW ALL LIKE`` - - Used as a wildcard character for flag names. - - ``SHOW `` - * - ``show_conf_UF`` - - Used to print all flags with the following attributes: - - * Flag name - * Default value - * Is developer mode (Boolean) - * Flag category - * Flag type - - ``rechunkThreshold,90,true,RND,regular`` - * - ``show_conf_extended UF`` - - Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. - - ``compilerGetsOnlyUFs,false,generic,regular,Makes runtime pass to compiler only`` - ``utility functions names,boolean,true,false`` - * - ``show_md_flag UF`` - - Used to show a specific flag/all flags stored in the metadata file. - - - * Example 1: ``* master=> ALTER SYSTEM SET heartbeatTimeout=111;`` - * Example 2: ``* master=> select show_md_flag(‘all’); heartbeatTimeout,111`` - * Example 3: ``* master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111`` - -.. _cluster_flag_types: - -Cluster Flag Types ---------------------- -The following is an example of the correct syntax for running a **Cluster** flag type command: - -.. code-block:: console - - ALTER SYSTEM RESET useMetadataServer; - executed - -The following table describes the Cluster flag types: - -.. list-table:: - :widths: 1 5 10 - :header-rows: 1 - - * - **Command** - - **Description** - - **Example** - * - ``ALTER SYSTEM SET `` - - Used to storing or modifying flag attributes in the metadata file. - - ``ALTER SYSTEM SET `` - * - ``ALTER SYSTEM RESET `` - - Used to remove a flag or all flag attributes from the metadata file. - - ``ALTER SYSTEM RESET `` - * - ``SHOW / ALL`` - - Used to print the value of a specified value or all flag values. - - ``SHOW `` - * - ``SHOW ALL LIKE`` - - Used as a wildcard character for flag names. - - ``SHOW `` - * - ``show_conf_UF`` - - Used to print all flags with the following attributes: - - * Flag name - * Default value - * Is developer mode (Boolean) - * Flag category - * Flag type - - ``rechunkThreshold,90,true,RND,regular`` - * - ``show_conf_extended UF`` - - Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. - - ``compilerGetsOnlyUFs,false,generic,regular,Makes runtime pass to compiler only`` - ``utility functions names,boolean,true,false`` - * - ``show_md_flag UF`` - - Used to show a specific flag/all flags stored in the metadata file. - - - * Example 1: ``* master=> ALTER SYSTEM SET heartbeatTimeout=111;`` - * Example 2: ``* master=> select show_md_flag(‘all’); heartbeatTimeout,111`` - * Example 3: ``* master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111`` - -.. _worker_flag_types: - -Worker Flag Types ---------------------- -The following is an example of the correct syntax for running a **Worker** flag type command: - -.. code-block:: console - - SHOW spoolMemoryGB; - -The following table describes the Worker flag types: - -.. list-table:: - :widths: 1 5 10 - :header-rows: 1 - - * - **Command** - - **Description** - - **Example** - * - ``ALTER SYSTEM SET `` - - Used to storing or modifying flag attributes in the metadata file. - - ``ALTER SYSTEM SET `` - * - ``ALTER SYSTEM RESET `` - - Used to remove a flag or all flag attributes from the metadata file. - - ``ALTER SYSTEM RESET `` - * - ``SHOW / ALL`` - - Used to print the value of a specified value or all flag values. - - ``SHOW `` - * - ``SHOW ALL LIKE`` - - Used as a wildcard character for flag names. - - ``SHOW `` - * - ``show_conf_UF`` - - Used to print all flags with the following attributes: - - * Flag name - * Default value - * Is developer mode (Boolean) - * Flag category - * Flag type - - ``rechunkThreshold,90,true,RND,regular`` - * - ``show_conf_extended UF`` - - Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. - - - ``compilerGetsOnlyUFs,false,generic,regular,Makes runtime pass to compiler only`` - ``utility functions names,boolean,true,false`` - * - ``show_md_flag UF`` - - Used to show a specific flag/all flags stored in the metadata file. - - - * Example 1: ``* master=> ALTER SYSTEM SET heartbeatTimeout=111;`` - * Example 2: ``* master=> select show_md_flag(‘all’); heartbeatTimeout,111`` - * Example 3: ``* master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111`` - -All Configurations ---------------------- -The following table describes the **Generic** and **Administration** configuration flags: - -.. list-table:: - :header-rows: 1 - :widths: 1 2 1 15 1 20 - :class: my-class - :name: my-name - - * - Flag Name - - Access Control - - Modification Type - - Description - - Data Type - - Default Value - - * - ``binSizes`` - - Administration - - Regular - - Sets the custom bin size in the cache to enable high granularity bin control. - - string - - - ``16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,`` - ``131072,262144,524288,1048576,2097152,4194304,8388608,16777216,`` - ``33554432,67108864,134217728,268435456,536870912,786432000,107374,`` - ``1824,1342177280,1610612736,1879048192,2147483648,2415919104,`` - ``2684354560,2952790016,3221225472`` - - * - ``checkCudaMemory`` - - Administration - - Regular - - Sets the pad device memory allocations with safety buffers to catch out-of-bounds writes. - - boolean - - ``FALSE`` - - * - ``compilerGetsOnlyUFs`` - - Administration - - Regular - - Sets the runtime to pass only utility functions names to the compiler. - - boolean - - ``FALSE`` - - * - ``copyToRestrictUtf8`` - - Administration - - Regular - - Sets the custom bin size in the cache to enable high granularity bin control. - - boolean - - ``FALSE`` - - * - ``cpuReduceHashtableSize`` - - Administration - - Regular - - Sets the hash table size of the CpuReduce. - - uint - - ``10000`` - - * - ``csvLimitRowLength`` - - Administration - - Cluster - - Sets the maximum supported CSV row length. - - uint - - ``100000`` - - * - ``cudaMemcpyMaxSizeBytes`` - - Administration - - Regular - - Sets the chunk size for copying from CPU to GPU. If set to 0, do not divide. - - uint - - ``0`` - - * - ``CudaMemcpySynchronous`` - - Administration - - Regular - - Indicates if copying from/to GPU is synchronous. - - boolean - - ``FALSE`` - - * - ``cudaMemQuota`` - - Administration - - Worker - - Sets the percentage of total device memory to be used by the instance. - - uint - - ``90`` - - * - ``developerMode`` - - Administration - - Regular - - Enables modifying R&D flags. - - boolean - - ``FALSE`` - - * - ``enableDeviceDebugMessages`` - - Administration - - Regular - - Activates the Nvidia profiler (nvprof) markers. - - boolean - - ``FALSE`` - - * - ``enableLogDebug`` - - Administration - - Regular - - Enables creating and logging in the clientLogger_debug file. - - boolean - - ``TRUE`` - - * - ``enableNvprofMarkers`` - - Administration - - Regular - - Activates the Nvidia profiler (nvprof) markers. - - boolean - - ``FALSE`` - - * - ``endLogMessage`` - - Administration - - Regular - - Appends a string at the end of every log line. - - string - - ``EOM`` - - - - * - ``varcharIdentifiers`` - - Administration - - Regular - - Activates using varchar as an identifier. - - boolean - - ``true`` - - - - * - ``extentStorageFileSizeMB`` - - Administration - - Cluster - - Sets the minimum size in mebibytes of extents for table bulk data. - - uint - - ``20`` - - * - ``gatherMemStat`` - - Administration - - Regular - - Monitors all pinned allocations and all **memcopies** to/from device, and prints a report of pinned allocations that were not memcopied to/from the device using the **dump_pinned_misses** utility function. - - boolean - - ``FALSE`` - - * - ``increaseChunkSizeBeforeReduce`` - - Administration - - Regular - - Increases the chunk size to reduce query speed. - - boolean - - ``FALSE`` - - * - ``increaseMemFactors`` - - Administration - - Regular - - Adds rechunker before expensive chunk producer. - - boolean - - ``TRUE`` - - * - ``leveldbWriteBufferSize`` - - Administration - - Regular - - Sets the buffer size. - - uint - - ``524288`` - - * - ``machineIP`` - - Administration - - Worker - - Manual setting of reported IP. - - string - - ``127.0.0.1`` - - - - - * - ``memoryResetTriggerMB`` - - Administration - - Regular - - Sets the size of memory used during a query to trigger aborting the server. - - uint - - ``0`` - - * - ``metadataServerPort`` - - Administration - - Worker - - Sets the port used to connect to the metadata server. SQream recommends using port ranges above 1024† because ports below 1024 are usually reserved, although there are no strict limitations. Any positive number (1 - 65535) can be used. - - uint - - ``3105`` - - * - ``mtRead`` - - Administration - - Regular - - Splits large reads to multiple smaller ones and executes them concurrently. - - boolean - - ``FALSE`` - - * - ``mtReadWorkers`` - - Administration - - Regular - - Sets the number of workers to handle smaller concurrent reads. - - uint - - ``30`` - - * - ``orcImplicitCasts`` - - Administration - - Regular - - Sets the implicit cast in orc files, such as **int** to **tinyint** and vice versa. - - boolean - - ``TRUE`` - - * - ``statementLockTimeout`` - - Administration - - Regular - - Sets the timeout (seconds) for acquiring object locks before executing statements. - - uint - - ``3`` - - * - ``useConfigIP`` - - Administration - - Worker - - Activates the machineIP (true). Setting to false ignores the machineIP and automatically assigns a local network IP. This cannot be activated in a cloud scenario (on-premises only). - - boolean - - ``FALSE`` - - * - ``useLegacyDecimalLiterals`` - - Administration - - Regular - - Interprets decimal literals as **Double** instead of **Numeric**. Used to preserve legacy behavior in existing customers. - - boolean - - ``FALSE`` - - * - ``useLegacyStringLiterals`` - - Administration - - Regular - - Interprets ASCII-only strings as **VARCHAR** instead of **TEXT**. Used to preserve legacy behavior in existing customers. - - boolean - - ``FALSE`` - - * - ``flipJoinOrder`` - - Generic - - Regular - - Reorders join to force equijoins and/or equijoins sorted by table size. - - boolean - - ``FALSE`` - - * - ``limitQueryMemoryGB`` - - Generic - - Worker - - Prevents a query from processing more memory than the flag’s value. - - uint - - ``100000`` - - * - ``cacheEvictionMilliseconds`` - - Generic - - Regular - - Sets how long the cache stores contents before being flushed. - - size_t - - ``2000`` - - - * - ``cacheDiskDir`` - - Generic - - Regular - - Sets the ondisk directory location for the spool to save files on. - - size_t - - Any legal string - - - * - ``cacheDiskGB`` - - Generic - - Regular - - Sets the amount of memory (GB) to be used by Spool on the disk. - - size_t - - ``128`` - - * - ``cachePartitions`` - - Generic - - Regular - - Sets the number of partitions that the cache is split into. - - size_t - - ``4`` - - - * - ``cachePersistentDir`` - - Generic - - Regular - - Sets the persistent directory location for the spool to save files on. - - string - - Any legal string - - - * - ``cachePersistentGB`` - - Generic - - Regular - - Sets the amount of data (GB) for the cache to store persistently. - - size_t - - ``128`` - - - * - ``cacheRamGB`` - - Generic - - Regular - - Sets the amount of memory (GB) to be used by Spool InMemory. - - size_t - - ``16`` - - - - - - - - * - ``logSysLevel`` - - Generic - - Regular - - - Determines the client log level: - 0 - L_SYSTEM, - 1 - L_FATAL, - 2 - L_ERROR, - 3 - L_WARN, - 4 - L_INFO, - 5 - L_DEBUG, - 6 - L_TRACE - - uint - - ``100000`` - - * - ``maxAvgBlobSizeToCompressOnGpu`` - - Generic - - Regular - - Sets the CPU to compress columns with size above (flag’s value) * (row count). - - uint - - ``120`` - - - * - ``sessionTag`` - - Generic - - Regular - - Sets the name of the session tag. - - string - - Any legal string - - - - * - ``spoolMemoryGB`` - - Generic - - Regular - - Sets the amount of memory (GB) to be used by the server for spooling. - - uint - - ``8`` - -Configuration Commands -========== -The configuration commands are associated with particular flag types based on permissions. - -The following table describes the commands or command sets that can be run based on their flag type. Note that the flag names described in the following table are described in the :ref:`Configuration Roles` section below. - -.. list-table:: - :header-rows: 1 - :widths: 1 2 10 17 - :class: my-class - :name: my-name - - * - Flag Type - - Command - - Description - - Example - * - Regular - - ``SET `` - - Used for modifying flag attributes. - - ``SET developerMode=true`` - * - Cluster - - ``ALTER SYSTEM SET `` - - Used to storing or modifying flag attributes in the metadata file. - - ``ALTER SYSTEM SET `` - * - Cluster - - ``ALTER SYSTEM RESET `` - - Used to remove a flag or all flag attributes from the metadata file. - - ``ALTER SYSTEM RESET `` - * - Regular, Cluster, Worker - - ``SHOW / ALL`` - - Used to print the value of a specified value or all flag values. - - ``SHOW `` - * - Regular, Cluster, Worker - - ``SHOW ALL LIKE`` - - Used as a wildcard character for flag names. - - ``SHOW `` - * - Regular, Cluster, Worker - - ``show_conf_UF`` - - Used to print all flags with the following attributes: - - * Flag name - * Default value - * Is developer mode (Boolean) - * Flag category - * Flag type - - - - - ``rechunkThreshold,90,true,RND,regular`` - * - Regular, Cluster, Worker - - ``show_conf_extended UF`` - - Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. - - ``spoolMemoryGB,15,false,generic,regular,Amount of memory (GB)`` - ``the server can use for spooling,”Statement that perform “”group by””,`` - ``“”order by”” or “”join”” operation(s) on large set of data will run`` - ``much faster if given enough spool memory, otherwise disk spooling will`` - ``be used resulting in performance hit.”,uint,,0-5000`` - * - Regular, Cluster, Worker - - ``show_md_flag UF`` - - Used to show a specific flag/all flags stored in the metadata file. - - - * Example 1: ``* master=> ALTER SYSTEM SET heartbeatTimeout=111;`` - * Example 2: ``* master=> select show_md_flag(‘all’); heartbeatTimeout,111`` - * Example 3: ``* master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111`` - -.. _configuration_roles: - -Configuration Roles -=========== -SQream divides flags into the following roles, each with their own set of permissions: - -* `Administration flags `_: can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command. -* `Generic flags `_: can be modified by standard users on a session basis. - -Showing All Flags in the Catalog Table -======= -SQream uses the **sqream_catalog.parameters** catalog table for showing all flags, providing the scope (default, cluster and session), description, default value and actual value. - -The following is the correct syntax for a catalog table query: - -.. code-block:: console - - SELECT * FROM sqream_catalog.settings - -The following is an example of a catalog table query: - -.. code-block:: console - - externalTableBlobEstimate, 100, 100, default, - varcharEncoding, ascii, ascii, default, Changes the expected encoding for Varchar columns - useCrcForTextJoinKeys, true, true, default, - hiveStyleImplicitStringCasts, false, false, default, - -This guide covers the configuration files and the ``SET`` statement. \ No newline at end of file diff --git a/configuration_guides/current_method_all_configurations.rst b/configuration_guides/current_method_all_configurations.rst new file mode 100644 index 000000000..00ad944c7 --- /dev/null +++ b/configuration_guides/current_method_all_configurations.rst @@ -0,0 +1,354 @@ +.. _current_method_all_configurations: + +************************** +All Configurations +************************** +The following table describes all **Generic** and **Administration** configuration flags: + +.. list-table:: + :header-rows: 1 + :widths: 1 2 1 15 1 20 + :class: my-class + :name: my-name + + * - Flag Name + - Access Control + - Modification Type + - Description + - Data Type + - Default Value + + * - ``binSizes`` + - Admin + - Regular + - Sets the custom bin size in the cache to enable high granularity bin control. + - string + - + ``16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,`` + ``131072,262144,524288,1048576,2097152,4194304,8388608,16777216,`` + ``33554432,67108864,134217728,268435456,536870912,786432000,107374,`` + ``1824,1342177280,1610612736,1879048192,2147483648,2415919104,`` + ``2684354560,2952790016,3221225472`` + + * - ``checkCudaMemory`` + - Admin + - Regular + - Sets the pad device memory allocations with safety buffers to catch out-of-bounds writes. + - boolean + - ``FALSE`` + + * - ``compilerGetsOnlyUFs`` + - Admin + - Regular + - Sets the runtime to pass only utility functions names to the compiler. + - boolean + - ``FALSE`` + + * - ``copyToRestrictUtf8`` + - Admin + - Regular + - Sets the custom bin size in the cache to enable high granularity bin control. + - boolean + - ``FALSE`` + + * - ``cpuReduceHashtableSize`` + - Admin + - Regular + - Sets the hash table size of the CpuReduce. + - uint + - ``10000`` + + * - ``csvLimitRowLength`` + - Admin + - Cluster + - Sets the maximum supported CSV row length. + - uint + - ``100000`` + + * - ``cudaMemcpyMaxSizeBytes`` + - Admin + - Regular + - Sets the chunk size for copying from CPU to GPU. If set to 0, do not divide. + - uint + - ``0`` + + * - ``CudaMemcpySynchronous`` + - Admin + - Regular + - Indicates if copying from/to GPU is synchronous. + - boolean + - ``FALSE`` + + * - ``cudaMemQuota`` + - Admin + - Worker + - Sets the percentage of total device memory to be used by the instance. + - uint + - ``90`` + + * - ``developerMode`` + - Admin + - Regular + - Enables modifying R&D flags. + - boolean + - ``FALSE`` + + * - ``enableDeviceDebugMessages`` + - Admin + - Regular + - Activates the Nvidia profiler (nvprof) markers. + - boolean + - ``FALSE`` + + * - ``enableLogDebug`` + - Admin + - Regular + - Enables creating and logging in the clientLogger_debug file. + - boolean + - ``TRUE`` + + * - ``enableNvprofMarkers`` + - Admin + - Regular + - Activates the Nvidia profiler (nvprof) markers. + - boolean + - ``FALSE`` + + * - ``endLogMessage`` + - Admin + - Regular + - Appends a string at the end of every log line. + - string + - ``EOM`` + + + + * - ``varcharIdentifiers`` + - Admin + - Regular + - Activates using varchar as an identifier. + - boolean + - ``true`` + + + + * - ``extentStorageFileSizeMB`` + - Admin + - Cluster + - Sets the minimum size in mebibytes of extents for table bulk data. + - uint + - ``20`` + + * - ``gatherMemStat`` + - Admin + - Regular + - Monitors all pinned allocations and all **memcopies** to/from device, and prints a report of pinned allocations that were not memcopied to/from the device using the **dump_pinned_misses** utility function. + - boolean + - ``FALSE`` + + * - ``increaseChunkSizeBeforeReduce`` + - Admin + - Regular + - Increases the chunk size to reduce query speed. + - boolean + - ``FALSE`` + + * - ``increaseMemFactors`` + - Admin + - Regular + - Adds rechunker before expensive chunk producer. + - boolean + - ``TRUE`` + + * - ``leveldbWriteBufferSize`` + - Admin + - Regular + - Sets the buffer size. + - uint + - ``524288`` + + * - ``machineIP`` + - Admin + - Worker + - Manual setting of reported IP. + - string + - ``127.0.0.1`` + + + + + * - ``memoryResetTriggerMB`` + - Admin + - Regular + - Sets the size of memory used during a query to trigger aborting the server. + - uint + - ``0`` + + * - ``metadataServerPort`` + - Admin + - Worker + - Sets the port used to connect to the metadata server. SQream recommends using port ranges above 1024† because ports below 1024 are usually reserved, although there are no strict limitations. Any positive number (1 - 65535) can be used. + - uint + - ``3105`` + + * - ``mtRead`` + - Admin + - Regular + - Splits large reads to multiple smaller ones and executes them concurrently. + - boolean + - ``FALSE`` + + * - ``mtReadWorkers`` + - Admin + - Regular + - Sets the number of workers to handle smaller concurrent reads. + - uint + - ``30`` + + * - ``orcImplicitCasts`` + - Admin + - Regular + - Sets the implicit cast in orc files, such as **int** to **tinyint** and vice versa. + - boolean + - ``TRUE`` + + * - ``statementLockTimeout`` + - Admin + - Regular + - Sets the timeout (seconds) for acquiring object locks before executing statements. + - uint + - ``3`` + + * - ``useConfigIP`` + - Admin + - Worker + - Activates the machineIP (true). Setting to false ignores the machineIP and automatically assigns a local network IP. This cannot be activated in a cloud scenario (on-premises only). + - boolean + - ``FALSE`` + + * - ``useLegacyDecimalLiterals`` + - Admin + - Regular + - Interprets decimal literals as **Double** instead of **Numeric**. Used to preserve legacy behavior in existing customers. + - boolean + - ``FALSE`` + + * - ``useLegacyStringLiterals`` + - Admin + - Regular + - Interprets ASCII-only strings as **VARCHAR** instead of **TEXT**. Used to preserve legacy behavior in existing customers. + - boolean + - ``FALSE`` + + * - ``flipJoinOrder`` + - Generic + - Regular + - Reorders join to force equijoins and/or equijoins sorted by table size. + - boolean + - ``FALSE`` + + * - ``limitQueryMemoryGB`` + - Generic + - Worker + - Prevents a query from processing more memory than the flag’s value. + - uint + - ``100000`` + + * - ``cacheEvictionMilliseconds`` + - Generic + - Regular + - Sets how long the cache stores contents before being flushed. + - size_t + - ``2000`` + + + * - ``cacheDiskDir`` + - Generic + - Regular + - Sets the ondisk directory location for the spool to save files on. + - size_t + - Any legal string + + + * - ``cacheDiskGB`` + - Generic + - Regular + - Sets the amount of memory (GB) to be used by Spool on the disk. + - size_t + - ``128`` + + * - ``cachePartitions`` + - Generic + - Regular + - Sets the number of partitions that the cache is split into. + - size_t + - ``4`` + + + * - ``cachePersistentDir`` + - Generic + - Regular + - Sets the persistent directory location for the spool to save files on. + - string + - Any legal string + + + * - ``cachePersistentGB`` + - Generic + - Regular + - Sets the amount of data (GB) for the cache to store persistently. + - size_t + - ``128`` + + + * - ``cacheRamGB`` + - Generic + - Regular + - Sets the amount of memory (GB) to be used by Spool InMemory. + - size_t + - ``16`` + + + + + + + + * - ``logSysLevel`` + - Generic + - Regular + - + Determines the client log level: + 0 - L_SYSTEM, + 1 - L_FATAL, + 2 - L_ERROR, + 3 - L_WARN, + 4 - L_INFO, + 5 - L_DEBUG, + 6 - L_TRACE + - uint + - ``100000`` + + * - ``maxAvgBlobSizeToCompressOnGpu`` + - Generic + - Regular + - Sets the CPU to compress columns with size above (flag’s value) * (row count). + - uint + - ``120`` + + + * - ``sessionTag`` + - Generic + - Regular + - Sets the name of the session tag. + - string + - Any legal string + + + + * - ``spoolMemoryGB`` + - Generic + - Regular + - Sets the amount of memory (GB) to be used by the server for spooling. + - uint + - ``8`` \ No newline at end of file diff --git a/configuration_guides/current_method_command_examples.rst b/configuration_guides/current_method_command_examples.rst new file mode 100644 index 000000000..2f75a4711 --- /dev/null +++ b/configuration_guides/current_method_command_examples.rst @@ -0,0 +1,36 @@ +.. _current_method_command_examples: + +************************** +Command Examples +************************** +This section includes the following command examples: + +.. contents:: + :local: + :depth: 1 + +Running a Regular Flag Type Command +--------------------- +The following is an example of running a **Regular** flag type command: + +.. code-block:: console + + SET spoolMemoryGB= 11; + executed + +Running a Worker Flag Type Command +--------------------- +The following is an example of running a **Worker** flag type command: + +.. code-block:: console + + SHOW spoolMemoryGB; + +Running a Cluster Flag Type Command +--------------------- +The following is an example of running a **Cluster** flag type command: + +.. code-block:: console + + ALTER SYSTEM RESET useMetadataServer; + executed \ No newline at end of file diff --git a/configuration_guides/current_method_configuration_levels.rst b/configuration_guides/current_method_configuration_levels.rst new file mode 100644 index 000000000..92c4b56d7 --- /dev/null +++ b/configuration_guides/current_method_configuration_levels.rst @@ -0,0 +1,33 @@ +.. _current_method_configuration_levels: + +************************** +Configuration Levels +************************** +SQream's configuration parameters are based on the following hierarchy: + +.. contents:: + :local: + :depth: 1 + +Cluster-Based Configuration +-------------- +Cluster-based configuration lets you centralize configurations for all workers on the cluster. Only Regular and Cluster flag types can be modified on the cluster level. These modifications are persistent and stored at the metadata level, which are applied globally to all workers in the cluster. + +.. note:: While cluster-based configuration was designed for configuring Workers, you can only configure Worker values set to the Regular or Cluster type. + +Worker-Based Configuration +-------------- +Worker-based configuration lets you modify the configuration belong to individual workers from the worker configuration file. + +For more information on making configurations from the worker configuration file, see `Modifying Your Configuration Using a Legacy Configuration File `_. + +Session-Based Configuration +-------------- +Session-based configurations are not persistent and are deleted when your session ends. This method enables you to modify all required configurations while avoiding conflicts between flag attributes modified on different devices at different points in time. The **SET flag_name** command is used to modify flag values on the session level. Any modifications you make with the **SET flag_name** command apply only to your open session, and are not saved when it ends. + +For example, when the query below has completed executing, the values configured will be restored to its previous setting: + +.. code-block:: console + + set spoolMemoryGB=700; + select * from table a where date='2021-11-11' diff --git a/configuration_guides/current_method_configuration_roles.rst b/configuration_guides/current_method_configuration_roles.rst new file mode 100644 index 000000000..11b7e4bfb --- /dev/null +++ b/configuration_guides/current_method_configuration_roles.rst @@ -0,0 +1,17 @@ +.. _current_method_configuration_roles: + +************************** +Configuration Roles +************************** +SQream divides flags into the following roles, each with their own set of permissions: + +* :ref:`admin_flags` - can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: **Comment** - *I don't think we need to mention the command here, as it's described below, and also not mentioned for Generic Flags.* + + * Regular + * Worker + * Cluster + +* :ref:`generic_flags` - can be modified by standard users on a session basis: + + * Regular + * Worker \ No newline at end of file diff --git a/configuration_guides/current_method_configuring_your_parameter_values.rst b/configuration_guides/current_method_configuring_your_parameter_values.rst new file mode 100644 index 000000000..d082db693 --- /dev/null +++ b/configuration_guides/current_method_configuring_your_parameter_values.rst @@ -0,0 +1,40 @@ +.. _current_method_configuring_your_parameter_values: + +************************** +Configuring Your Parameter Values +************************** +The method you must use to configure your parameter values depends on the configuration level. Each configuration level has its own command or set of commands used to configure values, as shown below: + ++-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| **Configuration Level** | ++=================================================================================================================================================================================================================================================================================================================+ +| **Regular, Worker, and Cluster** | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| **Command** | **Description** | **Example** | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``SET `` | Used for modifying flag attributes. | ``SET developerMode=true`` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``SHOW / ALL`` | Used to preset either a specific flag value or all flag values. | ``SHOW `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``SHOW ALL LIKE`` | Used as a wildcard character for flag names. | ``SHOW `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``show_conf_UF`` | Used to print all flags with the following attributes: | ``rechunkThreshold,90,true,RND,regular`` | +| | | | +| | * Flag name | | +| | * Default value | | +| | * Is Developer Mode (Boolean) | | +| | * Flag category | | +| | * Flag type | | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``show_conf_extended UF`` | Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. | ``rechunkThreshold,90,true,RND,regular`` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``show_md_flag UF`` | Used to show a specific flag/all flags stored in the metadata file. |* Example 1: ``* master=> ALTER SYSTEM SET heartbeatTimeout=111;`` | +| | |* Example 2: ``* master=> select show_md_flag(‘all’); heartbeatTimeout,111`` | +| | |* Example 3: ``* master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111`` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| **Worker and Cluster** | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``ALTER SYSTEM SET `` | Used for storing or modifying flag attributes in the metadata file. | ``ALTER SYSTEM SET `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``ALTER SYSTEM RESET `` | Used to remove a flag or all flag attributes from the metadata file. | ``ALTER SYSTEM RESET `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ \ No newline at end of file diff --git a/configuration_guides/current_method_flag_types.rst b/configuration_guides/current_method_flag_types.rst new file mode 100644 index 000000000..b1bedecba --- /dev/null +++ b/configuration_guides/current_method_flag_types.rst @@ -0,0 +1,20 @@ +.. _current_method_flag_types: + +************************** +Flag Types +************************** +SQream uses three flag types, **Cluster**, **Worker**, and **Regular**. Each of these flag types is associated with one of three hierarchical configuration levels described earlier, making it easier to configure your system. + +The highest level in the hierarchy is Cluster, which lets you set configurations across all workers in a given cluster. Modifying cluster values is **persistent**, meaning that any configurations you set are retained after shutting down your system. Configurations set at the Cluster level take the highest priority and override settings made on the Regular and Worker level **Comment** - *Confirm*. This is known as **cluster-based configuration**. Note that Cluster-based configuration lets you modify Cluster *and* Regular flag types. An example of a Cluster flag is **persisting your cache directory.** + +The second level is Worker, which lets you configure individual workers. Modifying Worker values are also **persistent**. This is known as **worker-based configuration**. Some examples of Worker flags includes **setting total device memory usage** and **setting metadata server connection port**. + +The lowest level is Regular, which means that modifying values of Regular flags affects only your current session and are not persistent. This means that they are automatically restored to their default value when the session ends. This is known as **session-based configuration**. Some examples of Regular flags includes **setting your bin size** and **setting CUDA memory**. + +To see each flag's default value, see one of the following: + +* The **Default Value** column in the :ref:`All Configurations` section. + + :: + +* The flag's individual description page, such as :ref:`Setting CUDA Memory`. \ No newline at end of file diff --git a/configuration_guides/current_method_modification_methods.rst b/configuration_guides/current_method_modification_methods.rst new file mode 100644 index 000000000..05825b07a --- /dev/null +++ b/configuration_guides/current_method_modification_methods.rst @@ -0,0 +1,50 @@ +.. _current_method_modification_methods: + +************************** +Modification Methods +************************** +SQream provides two different ways to modify your configurations. The current method is based on hierarchical configuration as described above. This method is based on making modifications on the **worker configuration file**, while you can still make modifications using the previous method using the **legacy configuration file**, both described below: + +.. contents:: + :local: + :depth: 1 + +Modifying Your Configuration Using the Worker Configuration File +------------------- +You can modify your configuration using the **worker configuration file (config.json)**. Changes that you make to worker configuration files are persistent. Note that you can only set the attributes in your worker configuration file **before** initializing your SQream worker, and while your worker is active these attributes are read-only. + +The following is an example of a worker configuration file: + +.. code-block:: postgres + + { + “cluster”: “/home/test_user/sqream_testing_temp/sqreamdb”, + “gpu”: 0, + “licensePath”: “home/test_user/SQream/tests/license.enc”, + “machineIP”: “127.0.0.1”, + “metadataServerIp”: “127.0.0.1”, + “metadataServerPort”: “3105, + “port”: 5000, + “useConfigIP”” true, + “legacyConfigFilePath”: “home/SQream_develop/SqrmRT/utils/json/legacy_congif.json” + } + +You can access the legacy configuration file from the ``legacyConfigFilePath`` parameter shown above. If all (or most) of your workers require the same flag settings, you can set the ``legacyConfigFilePath`` attribute to the same legacy file. + +Modifying Your Configuration Using a Legacy Configuration File +--------------------- +You can modify your configuration using a legacy configuration file. + +The Legacy configuration file provides access to the read/write flags used in SQream’s previous configuration method. A link to this file is provided in the **legacyConfigFilePath** parameter in the worker configuration file. + +The following is an example of the legacy configuration file: + +.. code-block:: postgres + + { + “developerMode”: true, + “reextentUse”: false, + “useClientLog”: true, + “useMetadataServer”” false + } +For more information on using the previous configuration method, see :ref:`previous_configuration_method`. \ No newline at end of file diff --git a/configuration_guides/current_method_showing_all_flags_in_the_catalog_table.rst b/configuration_guides/current_method_showing_all_flags_in_the_catalog_table.rst new file mode 100644 index 000000000..647a401b6 --- /dev/null +++ b/configuration_guides/current_method_showing_all_flags_in_the_catalog_table.rst @@ -0,0 +1,21 @@ +.. _current_method_showing_all_flags_in_the_catalog_table: + +************************** +Showing All Flags in the Catalog Table +************************** +SQream uses the **sqream_catalog.parameters** catalog table for showing all flags, providing the scope (default, cluster and session), description, default value and actual value. + +The following is the correct syntax for a catalog table query: + +.. code-block:: console + + SELECT * FROM sqream_catalog.settings + +The following is an example of a catalog table query: + +.. code-block:: console + + externalTableBlobEstimate, 100, 100, default, + varcharEncoding, ascii, ascii, default, Changes the expected encoding for Varchar columns + useCrcForTextJoinKeys, true, true, default, + hiveStyleImplicitStringCasts, false, false, default, \ No newline at end of file diff --git a/configuration_guides/developer_mode.rst b/configuration_guides/developer_mode.rst index fbb6c0cec..1fa0eb32c 100644 --- a/configuration_guides/developer_mode.rst +++ b/configuration_guides/developer_mode.rst @@ -8,4 +8,5 @@ The ``developerMode`` flag enables modifying R&D flags. The following describes the ``developerMode`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/enable_device_debug_messages.rst b/configuration_guides/enable_device_debug_messages.rst index c7d340b65..d48f0e94e 100644 --- a/configuration_guides/enable_device_debug_messages.rst +++ b/configuration_guides/enable_device_debug_messages.rst @@ -8,4 +8,5 @@ The ``enableDeviceDebugMessages`` flag checks for CUDA errors after producing ea The following describes the ``enableDeviceDebugMessages`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/enable_log_debug.rst b/configuration_guides/enable_log_debug.rst index 1566c4882..bfc0e5acb 100644 --- a/configuration_guides/enable_log_debug.rst +++ b/configuration_guides/enable_log_debug.rst @@ -8,4 +8,5 @@ The ``enableLogDebug`` flag enables creating and logging in the **clientLogger_d The following describes the ``enableLogDebug`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/enable_nv_prof_markers.rst b/configuration_guides/enable_nv_prof_markers.rst index 9edbf28e3..df18a10e7 100644 --- a/configuration_guides/enable_nv_prof_markers.rst +++ b/configuration_guides/enable_nv_prof_markers.rst @@ -8,4 +8,5 @@ The ``enableNvprofMarkers`` flag activates the NVidia Profiler (nvprof) markers. The following describes the ``enableNvprofMarkers`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/files_and_folders.txt b/configuration_guides/files_and_folders.txt new file mode 100644 index 000000000..0c8411b1b --- /dev/null +++ b/configuration_guides/files_and_folders.txt @@ -0,0 +1,67 @@ + Volume in drive C is Windows + Volume Serial Number is 565C-23BD + + Directory of C:\Users\Yaniv\Desktop\Local Reorganization Folder\configuration_guides + +03/22/2022 12:47 PM . +03/22/2022 12:47 PM .. +03/13/2022 11:57 AM 440 admin_cluster_flags.rst +03/09/2022 10:31 AM 403 admin_flags.rst +03/09/2022 11:19 AM 3,561 admin_regular_flags.rst +03/09/2022 10:31 AM 813 admin_worker_flags.rst +03/09/2022 11:22 AM 615 bin_sizes.rst +03/09/2022 12:10 PM 368 cache_disk_dir.rst +03/09/2022 12:14 PM 340 cache_disk_gb.rst +03/09/2022 11:59 AM 378 cache_eviction_milliseconds.rst +03/09/2022 11:42 AM 344 cache_partitions.rst +03/09/2022 12:02 PM 402 cache_persistent_dir.rst +03/09/2022 12:13 PM 377 cache_persistent_gb.rst +03/09/2022 11:53 AM 337 cache_ram_gb.rst +03/09/2022 11:23 AM 381 check_cuda_memory.rst +03/09/2022 11:24 AM 394 compiler_gets_only_ufs.rst +03/08/2022 05:38 PM 570 configuration_flags.rst +03/08/2022 05:38 PM 605 configuration_methods.rst +03/09/2022 11:24 AM 403 copy_to_restrict_utf8.rst +03/09/2022 11:38 AM 408 cpu_reduce_hashtable_size.rst +03/09/2022 11:25 AM 344 csv_limit_row_length.rst +03/09/2022 11:25 AM 424 cuda_mem_cpy_max_size_bytes.rst +03/09/2022 11:26 AM 369 cuda_mem_cpy_synchronous.rst +03/09/2022 11:26 AM 374 cuda_mem_quota.rst +03/20/2022 11:02 AM 24,433 current_configuration_method.rst +03/09/2022 11:27 AM 331 developer_mode.rst +03/09/2022 11:27 AM 397 enable_device_debug_messages.rst +03/09/2022 11:28 AM 383 enable_log_debug.rst +03/09/2022 11:28 AM 372 enable_nv_prof_markers.rst +03/09/2022 11:29 AM 302 end_log_message.rst +03/09/2022 10:31 AM 363 extent_storage_file_size_mb.rst +03/22/2022 12:47 PM 0 files_and_folders.txt +03/09/2022 11:29 AM 377 flip_join_order.rst +03/09/2022 11:30 AM 536 gather_mem_stat.rst +03/09/2022 10:31 AM 344 generic_flags.rst +03/09/2022 12:14 PM 1,837 generic_regular_flags.rst +03/09/2022 10:36 AM 381 generic_worker_flags.rst +03/09/2022 11:30 AM 410 increase_chunk_size_before_reduce.rst +03/09/2022 11:30 AM 379 increase_mem_factors.rst +03/09/2022 10:31 AM 308 index.rst +03/09/2022 11:31 AM 332 level_db_write_buffer_size.rst +03/09/2022 11:31 AM 385 limit_query_memory_gb.rst +03/09/2022 11:32 AM 441 log_sys_level.rst +03/09/2022 11:32 AM 328 machine_ip.rst +03/09/2022 11:33 AM 424 max_avg_blob_size_to_compress_on_gpu.rst +03/13/2022 04:54 PM 422 max_pinned_percentage_of_total_ram.rst +03/09/2022 11:33 AM 388 memory_reset_trigger_mb.rst +03/09/2022 11:33 AM 573 metadata_server_port.rst +03/09/2022 11:34 AM 371 mt_read.rst +03/09/2022 11:34 AM 367 mt_read_workers.rst +03/09/2022 11:35 AM 395 orc_implicit_casts.rst +03/08/2022 07:27 PM 9,074 previous_configuration_method.rst +03/09/2022 12:09 PM 313 session_tag.rst +03/09/2022 10:31 AM 2,822 spooling.rst +03/09/2022 11:35 AM 340 spool_memory_gb.rst +03/09/2022 11:35 AM 428 statement_lock_timeout.rst +03/09/2022 11:36 AM 487 use_config_ip.rst +03/09/2022 11:37 AM 494 use_legacy_decimal_literals.rst +03/13/2022 12:21 PM 461 use_legacy_string_literals.rst +03/09/2022 11:37 AM 354 varchar_identifiers.rst + 58 File(s) 63,002 bytes + 2 Dir(s) 898,522,271,744 bytes free diff --git a/configuration_guides/flip_join_order.rst b/configuration_guides/flip_join_order.rst index 341f12ada..806bde99d 100644 --- a/configuration_guides/flip_join_order.rst +++ b/configuration_guides/flip_join_order.rst @@ -8,4 +8,5 @@ The ``flipJoinOrder`` flag reorders join to force equijoins and/or equijoins sor The following describes the ``flipJoinOrder`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/gather_mem_stat.rst b/configuration_guides/gather_mem_stat.rst index 802e12b1f..3cc034194 100644 --- a/configuration_guides/gather_mem_stat.rst +++ b/configuration_guides/gather_mem_stat.rst @@ -8,4 +8,5 @@ The ``gatherMemStat`` flag monitors all pinned allocations and all **memcopies** The following describes the ``gatherMemStat`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/generic_flags.rst b/configuration_guides/generic_flags.rst index 2f17e8202..cb0241e31 100644 --- a/configuration_guides/generic_flags.rst +++ b/configuration_guides/generic_flags.rst @@ -11,7 +11,4 @@ The **Generic Flags** page describes the following flag types, which can be modi :glob: generic_regular_flags - generic_cluster_flags - generic_worker_flags - - + generic_worker_flags \ No newline at end of file diff --git a/configuration_guides/generic_regular_flags.rst b/configuration_guides/generic_regular_flags.rst index f8235ae07..2493fca76 100644 --- a/configuration_guides/generic_regular_flags.rst +++ b/configuration_guides/generic_regular_flags.rst @@ -6,16 +6,16 @@ Regular Generic Flags The **Regular Generic Flags** page describes **Regular** modification type flags, which can be modified by standard users on a session basis: -* `Flipping Join Order to Force Equijoins `_ -* `Determining Client Level `_ -* `Setting CPU to Compress Defined Columns `_ -* `Setting Query Memory Processing Limit `_ -* `Setting the Spool Memory `_ -* `Setting Cache Partitions `_ -* `Setting Cache Flushing `_ -* `Setting InMemory Spool Memory `_ -* `Setting Disk Spool Memory `_ -* `Setting Spool Saved File Directory Location `_ -* `Setting Data Stored Persistently on Cache `_ -* `Setting Persistent Spool Saved File Directory Location `_ -* `Setting Session Tag Name `_ \ No newline at end of file +* `Flipping Join Order to Force Equijoins `_ +* `Determining Client Level `_ +* `Setting CPU to Compress Defined Columns `_ +* `Setting Query Memory Processing Limit `_ +* `Setting the Spool Memory `_ +* `Setting Cache Partitions `_ +* `Setting Cache Flushing `_ +* `Setting InMemory Spool Memory `_ +* `Setting Disk Spool Memory `_ +* `Setting Spool Saved File Directory Location `_ +* `Setting Data Stored Persistently on Cache `_ +* `Setting Persistent Spool Saved File Directory Location `_ +* `Setting Session Tag Name `_ \ No newline at end of file diff --git a/configuration_guides/generic_worker_flags.rst b/configuration_guides/generic_worker_flags.rst index 97cee4ecf..cc9bdd6fa 100644 --- a/configuration_guides/generic_worker_flags.rst +++ b/configuration_guides/generic_worker_flags.rst @@ -3,6 +3,6 @@ ************************* Worker Generic Flags ************************* -The Worker Generic Flags** page describes **Worker** modification type flags, which can be modified by standard users on a session basis: +The **Worker Generic Flags** page describes **Worker** modification type flags, which can be modified by standard users on a session basis: - * `Persisting Your Cache Directory `_ \ No newline at end of file + * `Limits Available Query Processing Memory `_ \ No newline at end of file diff --git a/configuration_guides/healer_max_inactivity_hours.rst b/configuration_guides/healer_max_inactivity_hours.rst new file mode 100644 index 000000000..56ddb78fc --- /dev/null +++ b/configuration_guides/healer_max_inactivity_hours.rst @@ -0,0 +1,14 @@ +.. _healer_max_inactivity_hours: + +************************* +Configuring the Query Healer +************************* +The ``healerMaxInactivityHours`` flag is used for defining the threshold for creating a log recording a slow statement. The log includes information about the log memory, CPU and GPU. + +The following describes the ``healerMaxInactivityHours`` flag: + +* **Data type** - size_t +* **Default value** - ``5`` +* **Allowed values** - 1-4000000000 + +For related flags, see :ref:`is_healer_on`. \ No newline at end of file diff --git a/configuration_guides/increase_chunk_size_before_reduce.rst b/configuration_guides/increase_chunk_size_before_reduce.rst index 982d3db35..99d44a226 100644 --- a/configuration_guides/increase_chunk_size_before_reduce.rst +++ b/configuration_guides/increase_chunk_size_before_reduce.rst @@ -8,4 +8,5 @@ The ``increaseChunkSizeBeforeReduce`` flag increases the chunk size to reduce qu The following describes the ``increaseChunkSizeBeforeReduce`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/increase_mem_factors.rst b/configuration_guides/increase_mem_factors.rst index 166a57e14..e1f7898f5 100644 --- a/configuration_guides/increase_mem_factors.rst +++ b/configuration_guides/increase_mem_factors.rst @@ -8,4 +8,5 @@ The ``increaseMemFactors`` flag adds a rechunker before expensive chunk producer The following describes the ``increaseMemFactors`` flag: * **Data type** - boolean -* **Default value** - ``TRUE`` \ No newline at end of file +* **Default value** - ``true`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/index.rst b/configuration_guides/index.rst index 6a65853f0..6805bc188 100644 --- a/configuration_guides/index.rst +++ b/configuration_guides/index.rst @@ -11,8 +11,6 @@ The **Configuration Guides** page describes the following configuration informat :glob: spooling - configuration_methods + configuring_sqream configuration_flags - - - + previous_configuration_method \ No newline at end of file diff --git a/configuration_guides/is_healer_on.rst b/configuration_guides/is_healer_on.rst new file mode 100644 index 000000000..8b90654ec --- /dev/null +++ b/configuration_guides/is_healer_on.rst @@ -0,0 +1,14 @@ +.. _is_healer_on: + +************************* +Enabling the Query Healer +************************* +The ``is_healer_on`` flag enables the Query Healer, which periodically examines the progress of running statements and logs statements exceeding the ``healerMaxInactivityHours`` flag setting. + +The following describes the ``is_healer_on`` flag: + +* **Data type** - boolean +* **Default value** - ``true`` +* **Allowed values** - ``true``, ``false`` + +For related flags, see :ref:`healer_max_inactivity_hours`. \ No newline at end of file diff --git a/configuration_guides/level_db_write_buffer_size.rst b/configuration_guides/level_db_write_buffer_size.rst index c3cd60516..14eae1588 100644 --- a/configuration_guides/level_db_write_buffer_size.rst +++ b/configuration_guides/level_db_write_buffer_size.rst @@ -8,4 +8,5 @@ The ``leveldbWriteBufferSize`` flag sets the buffer size. The following describes the ``leveldbWriteBufferSize`` flag: * **Data type** - uint -* **Default value** - ``524288`` \ No newline at end of file +* **Default value** - ``524288`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/limit_query_memory_gb.rst b/configuration_guides/limit_query_memory_gb.rst index 7099674f2..38123f56d 100644 --- a/configuration_guides/limit_query_memory_gb.rst +++ b/configuration_guides/limit_query_memory_gb.rst @@ -1,11 +1,12 @@ .. _limit_query_memory_gb: ************************* -Setting Query Memory Processing Limit +Limiting Available Query Processing Memory ************************* The ``limitQueryMemoryGB`` flag prevents a query from processing more memory than the defined value. The following describes the ``limitQueryMemoryGB`` flag: * **Data type** - uint -* **Default value** - ``100000`` \ No newline at end of file +* **Default value** - ``100000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/log_sys_level.rst b/configuration_guides/log_sys_level.rst index 07e1e5800..031cee32f 100644 --- a/configuration_guides/log_sys_level.rst +++ b/configuration_guides/log_sys_level.rst @@ -16,4 +16,5 @@ The ``logSysLevel`` flag determines the client log level, as follows: The following describes the ``logSysLevel`` flag: * **Data type** - uint -* **Default value** - ``100000`` \ No newline at end of file +* **Default value** - ``1`` +* **Allowed values** - 0-6 \ No newline at end of file diff --git a/configuration_guides/login_max_retries.rst b/configuration_guides/login_max_retries.rst new file mode 100644 index 000000000..bf3ae6d40 --- /dev/null +++ b/configuration_guides/login_max_retries.rst @@ -0,0 +1,11 @@ +.. _login_max_retries: + +************************* +Adjusting Permitted Log-in Attempts +************************* +The ``loginMaxRetries`` flag sets the permitted log-in attempts. + +The following describes the ``loginMaxRetries`` flag: + +* **Data type** - size_t +* **Default value** - ``5`` \ No newline at end of file diff --git a/configuration_guides/machine_ip.rst b/configuration_guides/machine_ip.rst index 66a8a9a10..d184ed134 100644 --- a/configuration_guides/machine_ip.rst +++ b/configuration_guides/machine_ip.rst @@ -8,4 +8,5 @@ The ``machineIP`` flag enables you to manually set the reported IP. The following describes the ``machineIP`` flag: * **Data type** - string -* **Default value** - ``127.0.0.1`` \ No newline at end of file +* **Default value** - ``127.0.0.1`` +* **Allowed values** - char(16) \ No newline at end of file diff --git a/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst b/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst index 1081e0225..18d8f0408 100644 --- a/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst +++ b/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst @@ -8,4 +8,5 @@ The ``maxAvgBlobSizeToCompressOnGpu`` flag sets the CPU to compress columns with The following describes the ``maxAvgBlobSizeToCompressOnGpu`` flag: * **Data type** - uint -* **Default value** - ``120`` \ No newline at end of file +* **Default value** - ``120`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/max_pinned_percentage_of_total_ram.rst b/configuration_guides/max_pinned_percentage_of_total_ram.rst new file mode 100644 index 000000000..9ced5bde9 --- /dev/null +++ b/configuration_guides/max_pinned_percentage_of_total_ram.rst @@ -0,0 +1,12 @@ +.. _max_pinned_percentage_of_total_ram: + +************************* +Maximum Pinned Percentage of Total RAM +************************* +The ``maxPinnedPercentageOfTotalRAM`` flag sets the maximum percentage out of the available total CPU RAM that can be used by **pinned memory**. + +The following describes the ``machineIP`` flag: + +* **Data type** - uint +* **Default value** - ``70`` +* **Allowed values** - ``0-100`` \ No newline at end of file diff --git a/configuration_guides/memory_reset_trigger_mb.rst b/configuration_guides/memory_reset_trigger_mb.rst index bb8a383bb..c3430dda8 100644 --- a/configuration_guides/memory_reset_trigger_mb.rst +++ b/configuration_guides/memory_reset_trigger_mb.rst @@ -9,3 +9,4 @@ The following describes the ``memoryResetTriggerMB`` flag: * **Data type** - uint * **Default value** - ``0`` +* **Allowed values** - 0-4000000000 diff --git a/configuration_guides/metadata_server_port.rst b/configuration_guides/metadata_server_port.rst index 20f9a2db8..0dcd9bbe0 100644 --- a/configuration_guides/metadata_server_port.rst +++ b/configuration_guides/metadata_server_port.rst @@ -8,4 +8,5 @@ The ``metadataServerPort`` flag sets the port used to connect to the metadata se The following describes the ``metadataServerPort`` flag: * **Data type** - uint -* **Default value** - ``3105`` \ No newline at end of file +* **Default value** - ``3105`` +* **Allowed values** - 1-65535 \ No newline at end of file diff --git a/configuration_guides/mt_read.rst b/configuration_guides/mt_read.rst index 4aca17185..929d1c0e0 100644 --- a/configuration_guides/mt_read.rst +++ b/configuration_guides/mt_read.rst @@ -8,4 +8,5 @@ The ``mtRead`` flag splits large reads into multiple smaller ones and executes t The following describes the ``mtRead`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/mt_read_workers.rst b/configuration_guides/mt_read_workers.rst index 5f18fd4b3..90eb5cefd 100644 --- a/configuration_guides/mt_read_workers.rst +++ b/configuration_guides/mt_read_workers.rst @@ -9,3 +9,4 @@ The following describes the ``mtReadWorkers`` flag: * **Data type** - uint * **Default value** - ``30`` +* **Allowed values** - 1-4000000000 diff --git a/configuration_guides/orc_implicit_casts.rst b/configuration_guides/orc_implicit_casts.rst index 04cc903e9..e1b1287eb 100644 --- a/configuration_guides/orc_implicit_casts.rst +++ b/configuration_guides/orc_implicit_casts.rst @@ -8,4 +8,5 @@ The ``orcImplicitCasts`` flag sets the implicit cast in orc files, such as **int The following describes the ``orcImplicitCasts`` flag: * **Data type** - boolean -* **Default value** - ``TRUE`` +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/spool_memory_gb.rst b/configuration_guides/spool_memory_gb.rst index 9aa651a74..cb7135d03 100644 --- a/configuration_guides/spool_memory_gb.rst +++ b/configuration_guides/spool_memory_gb.rst @@ -8,4 +8,5 @@ The ``spoolMemoryGB`` flag sets the amount of memory (GB) available to the serve The following describes the ``spoolMemoryGB`` flag: * **Data type** - uint -* **Default value** - ``8`` \ No newline at end of file +* **Default value** - ``8`` +* **Allowed values** - 0-5000 \ No newline at end of file diff --git a/configuration_guides/spooling.rst b/configuration_guides/spooling.rst index 45c88d3ee..88d5a7980 100644 --- a/configuration_guides/spooling.rst +++ b/configuration_guides/spooling.rst @@ -65,5 +65,5 @@ The following is an example of setting ``spoolMemoryGB`` value in the previous c For more information about configuring the ``spoolMemoryGB`` flag, see the following: -* `Current configuration method `_ -* `Previous configuration method `_ \ No newline at end of file +* `Current configuration method `_ +* `Previous configuration method `_ \ No newline at end of file diff --git a/configuration_guides/statement_lock_timeout.rst b/configuration_guides/statement_lock_timeout.rst index 639f5d02d..ae552083a 100644 --- a/configuration_guides/statement_lock_timeout.rst +++ b/configuration_guides/statement_lock_timeout.rst @@ -9,3 +9,4 @@ The following describes the ``statementLockTimeout`` flag: * **Data type** - uint * **Default value** - ``3`` +* **Allowed values** - 0-4000000000 diff --git a/configuration_guides/use_config_ip.rst b/configuration_guides/use_config_ip.rst index 8779899b1..f195c9615 100644 --- a/configuration_guides/use_config_ip.rst +++ b/configuration_guides/use_config_ip.rst @@ -8,4 +8,5 @@ The ``useConfigIP`` flag activates the machineIP (``true``). Setting this flag t The following describes the ``useConfigIP`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` \ No newline at end of file +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/use_legacy_decimal_literals.rst b/configuration_guides/use_legacy_decimal_literals.rst index 63650a95b..9f26a5af1 100644 --- a/configuration_guides/use_legacy_decimal_literals.rst +++ b/configuration_guides/use_legacy_decimal_literals.rst @@ -8,4 +8,5 @@ The ``useLegacyDecimalLiterals`` flag interprets decimal literals as **Double** The following describes the ``useLegacyDecimalLiterals`` flag: * **Data type** - boolean -* **Default value** - ``FALSE`` +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/connecting_to_sqream/client_drivers/index.rst b/connecting_to_sqream/client_drivers/index.rst index e6f99b6fb..630888124 100644 --- a/connecting_to_sqream/client_drivers/index.rst +++ b/connecting_to_sqream/client_drivers/index.rst @@ -1,7 +1,7 @@ .. _client_drivers: ************************************ -Client Drivers for |latest_version| +Client Drivers for Version 2021.2.1.24 ************************************ The guides on this page describe how to use the Sqream DB client drivers and client applications with SQream. @@ -18,7 +18,7 @@ The following are applicable to all operating systems: * **JDBC** - recommended installation via ``mvn``: * `JDBC .jar file `_ - sqream-jdbc-4.5.3 (.jar) - * `JDBC driver `_ + * `JDBC driver `_ .. _python: @@ -26,7 +26,7 @@ The following are applicable to all operating systems: * **Python** - Recommended installation via ``pip``: * `Python .tar file `_ - pysqream v3.1.3 (.tar.gz) - * `Python driver `_ + * `Python driver `_ .. _nodejs: @@ -34,7 +34,7 @@ The following are applicable to all operating systems: * **Node.JS** - Recommended installation via ``npm``: * `Node.JS `_ - sqream-v4.2.4 (.tar.gz) - * `Node.JS driver `_ + * `Node.JS driver `_ .. _tableau_connector: @@ -42,7 +42,7 @@ The following are applicable to all operating systems: * **Tableau**: * `Tableau connector `_ - SQream (.taco) - * `Tableau manual installation `_ + * `Tableau manual installation `_ .. _powerbi_connector: @@ -50,7 +50,7 @@ The following are applicable to all operating systems: * **Power BI**: * `Power BI PowerQuery connector `_ - SQream (.mez) - * `Power BI manual installation `_ + * `Power BI manual installation `_ Windows @@ -68,12 +68,12 @@ Linux The following are applicable to Linux: * `SQream SQL (x86_64) `_ - sqream-sql-v2020.1.1_stable.x86_64.tar.gz -* `Sqream SQL CLI Reference `_ - Interactive command-line SQL client for Intel-based machines +* `Sqream SQL CLI Reference `_ - Interactive command-line SQL client for Intel-based machines :: * `SQream SQL*(IBM POWER9) `_ - sqream-sql-v2020.1.1_stable.ppc64le.tar.gz -* `Sqream SQL CLI Reference `_ - Interactive command-line SQL client for IBM POWER9-based machines +* `Sqream SQL CLI Reference `_ - Interactive command-line SQL client for IBM POWER9-based machines :: @@ -82,7 +82,7 @@ The following are applicable to Linux: :: * C++ connector - `libsqream-4.0 `_ -* `C++ shared object library `_ +* `C++ shared object library `_ .. toctree:: diff --git a/connecting_to_sqream/client_drivers/python/index.rst b/connecting_to_sqream/client_drivers/python/index.rst index c96ef5f2f..e03df343d 100644 --- a/connecting_to_sqream/client_drivers/python/index.rst +++ b/connecting_to_sqream/client_drivers/python/index.rst @@ -3,27 +3,34 @@ ************************* Python (pysqream) ************************* +The **Python** connector page describes the following: +.. contents:: + :local: + :depth: 1 + +Overview +============= The SQream Python connector is a set of packages that allows Python programs to connect to SQream DB. * ``pysqream`` is a pure Python connector. It can be installed with ``pip`` on any operating system, including Linux, Windows, and macOS. * ``pysqream-sqlalchemy`` is a SQLAlchemy dialect for ``pysqream`` -The connector supports Python 3.6.5 and newer. +The connector supports Python 3.6.5 and newer. The base ``pysqream`` package conforms to Python DB-API specifications `PEP-249 `_. -The base ``pysqream`` package conforms to Python DB-API specifications `PEP-249 `_. - -.. contents:: In this topic: - :local: - -Installing the Python connector +Installing the Python Connector ================================== Prerequisites ---------------- +Installing the Python connector includes the following prerequisites: + +.. contents:: + :local: + :depth: 1 -1. Python +Python ^^^^^^^^^^^^ The connector requires Python 3.6.5 or newer. To verify your version of Python: @@ -38,7 +45,7 @@ The connector requires Python 3.6.5 or newer. To verify your version of Python: .. warning:: If you're running on an older version, ``pip`` will fetch an older version of ``pysqream``, with version <3.0.0. This version is currently not supported. -2. PIP +PIP ^^^^^^^^^^^^ The Python connector is installed via ``pip``, the Python package manager and installer. @@ -60,35 +67,24 @@ We recommend upgrading to the latest version of ``pip`` before installing. To ve * On macOS, you may want to use virtualenv to install Python and the connector, to ensure compatibility with the built-in Python environment * If you encounter an error including ``SSLError`` or ``WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.`` - please be sure to reinstall Python with SSL enabled, or use virtualenv or Anaconda. -3. OpenSSL for Linux +OpenSSL for Linux ^^^^^^^^^^^^^^^^^^^^^^^^^^ - Some distributions of Python do not include OpenSSL. The Python connector relies on OpenSSL for secure connections to SQream DB. * To install OpenSSL on RHEL/CentOS - .. code-block:: console + .. code-block:: console - $ sudo yum install -y libffi-devel openssl-devel + $ sudo yum install -y libffi-devel openssl-devel * To install OpenSSL on Ubuntu - .. code-block:: console - - $ sudo apt-get install libssl-dev libffi-dev -y - -4. Cython (optional) -^^^^^^^^^^^^^^^^^^^^^^^^ - -Optional but highly recommended is Cython, which improves performance of Python applications. - - .. code-block:: console + .. code-block:: console - $ pip install cython + $ sudo apt-get install libssl-dev libffi-dev -y -Install via pip +Installing via PIP ----------------- - The Python connector is available via `PyPi `_. Install the connector with ``pip``: @@ -99,49 +95,59 @@ Install the connector with ``pip``: ``pip`` will automatically install all necessary libraries and modules. -Upgrading an existing installation +Upgrading an Existing Installation -------------------------------------- - -The Python drivers are updated periodically. -To upgrade an existing pysqream installation, use pip's ``-U`` flag. +The Python drivers are updated periodically. To upgrade an existing pysqream installation, use pip's ``-U`` flag: .. code-block:: console $ pip install pysqream pysqream-sqlalchemy -U - -Validate the installation +Validating Your Installation ----------------------------- +This section describes how to validate your installation. -Create a file called ``test.py``, containing the following: +**To validate your installation**: + +1. Create a file called ``test.py``, containing the following: .. literalinclude:: test.py :language: python :caption: pysqream Validation Script :linenos: -Make sure to replace the parameters in the connection with the respective parameters for your SQream DB installation. +2. Verify that the parameters in the connection have been replaced with your respective SQream installation parameters. -Run the test file to verify that you can connect to SQream DB: + :: -.. code-block:: console +3. Run the test file to verify that you can connect to SQream: + + .. code-block:: console - $ python test.py - Version: v2020.1 + $ python test.py + Version: v2020.1 + + If the validation was successful, you can build an application using the SQream Python connector. If you receive a connection error, verify the following: -If all went well, you are now ready to build an application using the SQream DB Python connector! + * You have access to a running SQream database. -If any connection error appears, verify that you have access to a running SQream DB and that the connection parameters are correct. + :: -SQLAlchemy examples + * The connection parameters are correct. + +SQLAlchemy Examples ======================== +SQLAlchemy is an **Object-Relational Mapper (ORM) for Python. When you install the SQream dialect (``pysqream-sqlalchemy``) you can use frameworks such as Pandas, TensorFlow, and Alembic to query SQream directly. -SQLAlchemy is an ORM for Python. +This section includes the following examples: -When you install the SQream DB dialect (``pysqream-sqlalchemy``) you can use frameworks like Pandas, TensorFlow, and Alembic to query SQream DB directly. +.. contents:: + :local: + :depth: 1 -A simple connection example +Standard Connection Example --------------------------------- +The following is a standard connection example: .. code-block:: python @@ -162,10 +168,9 @@ A simple connection example res = engine.execute('insert into test values (5), (6)') res = engine.execute('select * from test') -Pulling a table into Pandas +Pulling a Table into Pandas --------------------------------- - -In this example, we use the URL method to create the connection string. +The following example shows how to pull a table in Pandas. This examples uses the URL method to create the connection string: .. code-block:: python @@ -186,72 +191,76 @@ In this example, we use the URL method to create the connection string. table_df = pd.read_sql("select * from nba", con=engine) - API Examples =============== +This section includes the following examples: + +.. contents:: + :local: + :depth: 1 -Explaining the connection example +Describing Your Connection --------------------------------------- +This example shows how to describe the connection. -First, import the package and create a connection +**Describing your connection**: -.. code-block:: python +1. Import the package and create a connection: + + .. code-block:: python - # Import pysqream package + # Import pysqream package - import pysqream + import pysqream - """ - Connection parameters include: - * IP/Hostname - * Port - * database name - * username - * password - * Connect through load balancer, or direct to worker (Default: false - direct to worker) - * use SSL connection (default: false) - * Optional service queue (default: 'sqream') - """ - - # Create a connection object - - con = pysqream.connect(host='127.0.0.1', port=3108, database='raviga' - , username='rhendricks', password='Tr0ub4dor&3' - , clustered=True) + """ + Connection parameters include: + * IP/Hostname + * Port + * database name + * username + * password + * Connect through load balancer, or direct to worker (Default: false - direct to worker) + * use SSL connection (default: false) + * Optional service queue (default: 'sqream') + """ + + # Create a connection object -Then, run a query and fetch the results + con = pysqream.connect(host='127.0.0.1', port=3108, database='raviga' + , username='rhendricks', password='Tr0ub4dor&3' + , clustered=True) -.. code-block:: python +2. Run a query and fetch the results: + + .. code-block:: python - cur = con.cursor() # Create a new cursor - # Prepare and execute a query - cur.execute('select show_version()') + cur = con.cursor() # Create a new cursor + # Prepare and execute a query + cur.execute('select show_version()') - result = cur.fetchall() # `fetchall` gets the entire data set + result = cur.fetchall() # `fetchall` gets the entire data set - print (f"Version: {result[0][0]}") + print (f"Version: {result[0][0]}") -This should print the SQream DB version. For example ``v2020.1``. + The SQream version should be output, such as ``v2020.1``. -Finally, we will close the connection +3. Close the connection: -.. code-block:: python + .. code-block:: python - con.close() + con.close() -Using the cursor +Using the Cursor -------------------------------------------- - -The DB-API specification includes several methods for fetching results from the cursor. - -We will use the ``nba`` example. Here's a peek at the table contents: +The DB-API specification includes several methods for fetching results from the cursor. This sections shows an example using the ``nba`` table, which looks as follows: .. csv-table:: nba :file: nba-t10.csv :widths: auto :header-rows: 1 -Like before, we will import the library and create a :py:meth:`~Connection`, followed by :py:meth:`~Connection.execute` on a simple ``SELECT *`` query. +As before, you must import the library and create a :py:meth:`~Connection`, followed by :py:meth:`~Connection.execute` on a simple ``SELECT *`` query: .. code-block:: python @@ -265,9 +274,9 @@ Like before, we will import the library and create a :py:meth:`~Connection`, fol statement = 'SELECT * FROM nba' cur.execute(statement) -After executing the statement, we have a :py:meth:`Connection` cursor object waiting. A cursor is iterable, meaning that everytime we fetch, it advances the cursor to the next row. +When the statement has finished executing, you have a :py:meth:`Connection` cursor object waiting. A cursor is iterable, meaning that it advances the cursor to the next row when fetched. -Use :py:meth:`~Connection.fetchone` to get one record at a time: +You can use :py:meth:`~Connection.fetchone` to fetch one record at a time: .. code-block:: python @@ -275,14 +284,14 @@ Use :py:meth:`~Connection.fetchone` to get one record at a time: second_row = cur.fetchone() # Fetch one row at a time (second row) -To get several rows at a time, use :py:meth:`~Connection.fetchmany`: +To fetch several rows at a time, use :py:meth:`~Connection.fetchmany`: .. code-block:: python # executing `fetchone` twice is equivalent to this form: third_and_fourth_rows = cur.fetchmany(2) -To get all rows at once, use :py:meth:`~Connection.fetchall`: +To fetch all rows at once, use :py:meth:`~Connection.fetchall`: .. code-block:: python @@ -292,7 +301,7 @@ To get all rows at once, use :py:meth:`~Connection.fetchall`: # Close the connection when done con.close() -Here are the contents of the row variables we used: +The following is an example of the contents of the row variables used in our examples: .. code-block:: pycon @@ -308,12 +317,11 @@ Here are the contents of the row variables we used: .. note:: Calling a fetch command after all rows have been fetched will return an empty array (``[]``). -Reading result metadata +Reading Result Metadata ---------------------------- +When you execute a statement, the connection object also contains metadata about the result set, such as **column names**, **types**, etc). -When executing a statement, the connection object also contains metadata about the result set (e.g.column names, types, etc). - -The metadata is stored in the :py:attr:`Connection.description` object of the cursor. +The metadata is stored in the :py:attr:`Connection.description` object of the cursor: .. code-block:: pycon @@ -328,78 +336,112 @@ The metadata is stored in the :py:attr:`Connection.description` object of the cu >>> print(cur.description) [('Name', 'STRING', 24, 24, None, None, True), ('Team', 'STRING', 22, 22, None, None, True), ('Number', 'NUMBER', 1, 1, None, None, True), ('Position', 'STRING', 2, 2, None, None, True), ('Age (as of 2018)', 'NUMBER', 1, 1, None, None, True), ('Height', 'STRING', 4, 4, None, None, True), ('Weight', 'NUMBER', 2, 2, None, None, True), ('College', 'STRING', 21, 21, None, None, True), ('Salary', 'NUMBER', 4, 4, None, None, True)] -To get a list of column names, iterate over the ``description`` list: +You can fetch a list of column names by iterating over the ``description`` list: .. code-block:: pycon >>> [ i[0] for i in cur.description ] ['Name', 'Team', 'Number', 'Position', 'Age (as of 2018)', 'Height', 'Weight', 'College', 'Salary'] -Loading data into a table +Loading Data into a Table --------------------------- +This example shows how to load 10,000 rows of dummy data to an instance of SQream. -This example loads 10,000 rows of dummy data to a SQream DB instance +**To load data 10,000 rows of dummy data to an instance of SQream:** -.. code-block:: python +1. Run the following: + + .. code-block:: python - import pysqream - from datetime import date, datetime - from time import time + import pysqream + from datetime import date, datetime + from time import time - con = pysqream.connect(host='127.0.0.1', port=3108, database='master' - , username='rhendricks', password='Tr0ub4dor&3' - , clustered=True) + con = pysqream.connect(host='127.0.0.1', port=3108, database='master' + , username='rhendricks', password='Tr0ub4dor&3' + , clustered=True) - # Create a table for loading - create = 'create or replace table perf (b bool, t tinyint, sm smallint, i int, bi bigint, f real, d double, s varchar(12), ss text, dt date, dtt datetime)' - con.execute(create) +2. Create a table for loading: - # After creating the table, we can load data into it with the INSERT command + .. code-block:: python - # Create dummy data which matches the table we created - data = (False, 2, 12, 145, 84124234, 3.141, -4.3, "Marty McFly" , u"キウイは楽しい鳥です" , date(2019, 12, 17), datetime(1955, 11, 4, 1, 23, 0, 0)) - - - row_count = 10**4 + create = 'create or replace table perf (b bool, t tinyint, sm smallint, i int, bi bigint, f real, d double, s varchar(12), ss text, dt date, dtt datetime)' + con.execute(create) + +3. Load your data into table using the ``INSERT`` command. + + :: + +4. Create dummy data matching the table you created: + + .. code-block:: python - # Get a new cursor - cur = con.cursor() - insert = 'insert into perf values (?,?,?,?,?,?,?,?,?,?,?)' - start = time() - cur.executemany(insert, [data] * row_count) - print (f"Total insert time for {row_count} rows: {time() - start} seconds") + data = (False, 2, 12, 145, 84124234, 3.141, -4.3, "Marty McFly" , u"キウイは楽しい鳥です" , date(2019, 12, 17), datetime(1955, 11, 4, 1, 23, 0, 0)) + + row_count = 10**4 - # Close this cursor - cur.close() +5. Get a new cursor: + + .. code-block:: python + + cur = con.cursor() + insert = 'insert into perf values (?,?,?,?,?,?,?,?,?,?,?)' + start = time() + cur.executemany(insert, [data] * row_count) + print (f"Total insert time for {row_count} rows: {time() - start} seconds") + +6. Close this cursor: + + .. code-block:: python + + cur.close() - # Verify that the data was inserted correctly - # Get a new cursor - cur = con.cursor() - cur.execute('select count(*) from perf') - result = cur.fetchall() # `fetchall` collects the entire data set - print (f"Count of inserted rows: {result[0][0]}") +7. Verify that the data was inserted correctly. + + :: + +8. Get a new cursor: - # When done, close the cursor - cur.close() + .. code-block:: python + + cur = con.cursor() + cur.execute('select count(*) from perf') + result = cur.fetchall() # `fetchall` collects the entire data set + print (f"Count of inserted rows: {result[0][0]}") + +9. Close the cursor: + + .. code-block:: python + + cur.close() - # Close the connection - con.close() +10. Close the connection: + + .. code-block:: python + + con.close() -Reading data from a CSV file for load into a table +Reading Data from a CSV File for Loading into a Table ---------------------------------------------------------- +This example shows how to write a helper function to create an :ref:`insert` statement, by reading an existing table's metadata. -We will write a helper function to create an :ref:`insert` statement, by reading an existing table's metadata. +**To read data from a CSV file for loading into a table:** -.. code-block:: python +1. Run the following: + + .. code-block:: python - import pysqream - import datetime + import pysqream + import datetime - def insert_from_csv(cur, table_name, csv_filename, field_delimiter = ',', null_markers = []): - """ - We will first ask SQream DB for some table information. - This is important for understanding the number of columns, and will help - to create a matching INSERT statement + def insert_from_csv(cur, table_name, csv_filename, field_delimiter = ',', null_markers = []): + """ +2. Ask SQream for some table information. + + This is important for determining the number of columns, and helps create a matching ``INSERT`` statement: + + .. code-block:: python + """ column_info = cur.execute(f"SELECT * FROM {table_name} LIMIT 0").description @@ -414,85 +456,113 @@ We will write a helper function to create an :ref:`insert` statement, by reading except ValueError: return datetime.datetime.strptime(row[i], '%Y-%m-%d') - # Create enough placeholders (`?`) for the INSERT query string +3. Create enough placeholders (`?`) for the ``INSERT`` query string: + + + .. code-block:: python + qstring = ','.join(['?'] * len(column_info)) insert_statement = f"insert into {table_name} values ({qstring})" - # Open the CSV file +4. Open the CSV file: + + .. code-block:: python + with open(csv_filename, mode='r') as csv_file: csv_reader = csv.reader(csv_file, delimiter=field_delimiter) - # Execute the INSERT statement with the CSV data +5. Execute the ``INSERT`` statement with the CSV data: + + .. code-block:: python + cur.executemany(insert_statement, [row for row in csv_reader]) - con = pysqream.connect(host='127.0.0.1', port=3108, database='master' + con = pysqream.connect(host='127.0.0.1', port=3108, database='master' , username='rhendricks', password='Tr0ub4dor&3' , clustered=True) - cur = con.cursor() - insert_from_csv(cur, 'nba', 'nba.csv', field_delimiter = ',', null_markers = []) + cur = con.cursor() + insert_from_csv(cur, 'nba', 'nba.csv', field_delimiter = ',', null_markers = []) - con.close() + con.close() - -Using SQLAlchemy ORM to create tables and fill them with data +Using SQLAlchemy ORM to Create and Populate Tables ----------------------------------------------------------------------- +This section shows how to use the ORM to create and populate tables from Python objects. -You can also use the ORM to create tables and insert data to them from Python objects. +**To use SQLAlchemy ORM to create and populate tables:** -For example: +1. Run the following: -.. code-block:: python - - import sqlalchemy as sa - import pandas as pd - from sqlalchemy.engine.url import URL + .. code-block:: python + + import sqlalchemy as sa + import pandas as pd + from sqlalchemy.engine.url import URL - engine_url = URL('sqream' - , username='rhendricks' - , password='secret_passwor" - , host='localhost' - , port=5000 - , database='raviga' - , query={'use_ssl': False}) + engine_url = URL('sqream' + , username='rhendricks' + , password='secret_passwor" + , host='localhost' + , port=5000 + , database='raviga' + , query={'use_ssl': False}) - engine = sa.create_engine(engine_url) + engine = sa.create_engine(engine_url) + +2. Build a metadata object and bind it: + + .. code-block:: python - # Build a metadata object and bind it + metadata = sa.MetaData() + metadata.bind = engine - metadata = sa.MetaData() - metadata.bind = engine +3. Create a table in the local metadata: - # Create a table in the local metadata + .. code-block:: python - employees = sa.Table( - 'employees' - , metadata - , sa.Column('id', sa.Integer) - , sa.Column('name', sa.VARCHAR(32)) - , sa.Column('lastname', sa.VARCHAR(32)) - , sa.Column('salary', sa.Float) - ) + employees = sa.Table( + 'employees' + , metadata + , sa.Column('id', sa.Integer) + , sa.Column('name', sa.VARCHAR(32)) + , sa.Column('lastname', sa.VARCHAR(32)) + , sa.Column('salary', sa.Float) + ) - # The create_all() function uses the SQream DB engine object - # to create all the defined table objects. + The ``create_all()`` function uses the SQream engine object. - metadata.create_all(engine) +4. Create all the defined table objects: + + .. code-block:: python + + metadata.create_all(engine) - # Now that the table exists, we can insert data into it. +5. Populate your table. + + :: - # Build the data rows - insert_data = [ {'id': 1, 'name': 'Richard','lastname': 'Hendricks', 'salary': 12000.75} - ,{'id': 3, 'name': 'Bertram', 'lastname': 'Gilfoyle', 'salary': 8400.0} - ,{'id': 8, 'name': 'Donald', 'lastname': 'Dunn', 'salary': 6500.40} - ] +6. Build the data rows: - # Build the insert command - ins = employees.insert(insert_data) + .. code-block:: python + + insert_data = [ {'id': 1, 'name': 'Richard','lastname': 'Hendricks', 'salary': 12000.75} + ,{'id': 3, 'name': 'Bertram', 'lastname': 'Gilfoyle', 'salary': 8400.0} + ,{'id': 8, 'name': 'Donald', 'lastname': 'Dunn', 'salary': 6500.40} + ] + +7. Build the ``INSERT`` command: + + .. code-block:: python + + ins = employees.insert(insert_data) - # Execute the command - result = engine.execute(ins) +8. Execute the command: + + .. code-block:: python + + result = engine.execute(ins) For more information, see the :ref:`python_api_reference_guide`. \ No newline at end of file diff --git a/operational_guides/hdfs.rst b/external_storage_platforms/hdfs.rst similarity index 100% rename from operational_guides/hdfs.rst rename to external_storage_platforms/hdfs.rst diff --git a/operational_guides/external_data.rst b/external_storage_platforms/index.rst similarity index 52% rename from operational_guides/external_data.rst rename to external_storage_platforms/index.rst index 28fc3d237..92c35ee63 100644 --- a/operational_guides/external_data.rst +++ b/external_storage_platforms/index.rst @@ -1,9 +1,9 @@ -.. _external_data: +.. _external_storage_platforms: -********************************** -Working with External Data -********************************** -SQream supports the following external data sources: +*********************** +External Storage Platforms +*********************** +SQream supports the following external storage platforms: .. toctree:: :maxdepth: 1 diff --git a/operational_guides/s3.rst b/external_storage_platforms/s3.rst similarity index 100% rename from operational_guides/s3.rst rename to external_storage_platforms/s3.rst diff --git a/feature_guides/data_encryption.rst b/feature_guides/data_encryption.rst deleted file mode 100644 index dbc6152ed..000000000 --- a/feature_guides/data_encryption.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _data_encryption: - -*********************** -Data Encryption -*********************** -The **Data Encryption** page |icon-new_2022.1| describes the following: - -.. |icon-new_2022.1| image:: /_static/images/new_2022.1.png - :align: middle - :width: 110 - -.. toctree:: - :maxdepth: 1 - :titlesonly: - - data_encryption_overview - data_encryption_methods - data_encryption_types - data_encryption_syntax \ No newline at end of file diff --git a/feature_guides/data_encryption_methods.rst b/feature_guides/data_encryption_methods.rst deleted file mode 100644 index 84382f0e9..000000000 --- a/feature_guides/data_encryption_methods.rst +++ /dev/null @@ -1,20 +0,0 @@ -.. _data_encryption_methods: - -*********************** -Encryption Methods -*********************** -Data exists in one of following states and determines the encryption method: - -.. contents:: - :local: - :depth: 1 - -Encrypting Data in Transit ----------------- -**Data in transit** refers to data you use on a regular basis, usually stored on a database and accessed through applications or programs. This data is typically transferred between several physical or remote locations through email or uploading documents to the cloud. This type of data must therefore be protected while **in transit**. SQream encrypts data in transit using SSL when, for example, users insert data files from external repositories over a JDBC or ODBC connection. - -For more information, see `Use TLS/SSL When Possible `_. - -Encrypting Data at Rest ----------------- -**Data at rest** refers to data stored on your hard drive or on the cloud. Because this data can be potentially intercepted **physically**, it requires a form of encryption that protects your data wherever you store it. SQream faciliates encryption by letting you encrypt any columns located in your database that you want to keep private. \ No newline at end of file diff --git a/feature_guides/data_encryption_overview.rst b/feature_guides/data_encryption_overview.rst deleted file mode 100644 index a37325e52..000000000 --- a/feature_guides/data_encryption_overview.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. _data_encryption_overview: - -*********************** -Overview -*********************** -**Data Encryption** helps protect sensitive data at rest by concealing it from unauthorized users in the event of a breach. This is achieved by scrambling the content into an unreadable format based on encryption and decryption keys. Typically speaking, this data pertains to **PII (Personally Identifiable Information)**, which is sensitive information such as credit card numbers and other information related to an identifiable person. - -Users encrypt their data on a column basis by specifying ``column_name`` in the encryption syntax. - -The demand for confidentiality has steadily increased to protect the growing volumes of private data stored on computer systems and transmitted over the internet. To this end, regulatory bodies such as the **General Data Protection Regulation (GDPR)** have produced requirements to standardize and enforce compliance aimed at protecting customer data. - -Encryption can be used for the following: - -* Creating tables up to three encrypted columns. - - :: - -* Joining encrypted columns with other tables. - - :: - -* Selecting data from an encrypted column. - -For more information on the encryption syntax, see :ref:`data_encryption_syntax`. - -For more information on GDPR compliance requirements, see the `GDPR checklist `_. \ No newline at end of file diff --git a/feature_guides/data_encryption_syntax.rst b/feature_guides/data_encryption_syntax.rst deleted file mode 100644 index 3370698cf..000000000 --- a/feature_guides/data_encryption_syntax.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. _data_encryption_syntax: - -*********************** -Syntax -*********************** -The following is the syntax for encrypting a new table: - -.. code-block:: console - - CREATE TABLE ( - <(maximum string length)>, - - last_name <(maximum string length)>, - salary (<(maximum string length)>) ENCRYPT); - -The following is an example of encrypting a new table: - -.. code-block:: console - - CREATE TABLE client_name ( - first_name TEXT(128), - last_name TEXT(128), - salary INT(6) ENCRYPT); - -.. note:: Users without permissions cannot view the entire table as long as at least one column is encrypted. The (unique) encryption/decryption key is relevant only at the system level and is not held by users. - -Note that the master key is hard-coded in the system and cannot be changed. diff --git a/feature_guides/data_encryption_types.rst b/feature_guides/data_encryption_types.rst deleted file mode 100644 index ad6d96dc3..000000000 --- a/feature_guides/data_encryption_types.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _data_encryption_types: - -*********************** -Data Types -*********************** -Typically speaking, sensitive pertains to **PII (Personally Identifiable Information)**, which is sensitive information such as credit card numbers and other information related to an identifiable person. - -SQream's data encryption feature supports encrypting column-based data belonging to the following data types: - -* INT -* BIGINT -* TEXT - -For more information on the above data types, see :ref:`supported_data_types`. \ No newline at end of file diff --git a/feature_guides/query_healer.rst b/feature_guides/query_healer.rst new file mode 100644 index 000000000..8f0b56d08 --- /dev/null +++ b/feature_guides/query_healer.rst @@ -0,0 +1,34 @@ +.. _query_healer: + +*********************** +Query Healer +*********************** +The **Query Healer** page describes the following: + +.. contents:: + :local: + :depth: 1 + +Overview +---------- +The **Query Healer** periodically examines the progress of running statements, creating a log entry for all statements exceeding the ``healerMaxInactivityHours`` flag setting. The default setting of the ``healerMaxInactivityHours`` is five hours. + +The following is an example of a log record for a query stuck in the query detection phase for more than five hours: + +.. code-block:: console + + 2022/05/19::20:01:25|ERROR|Healer|(0x7f07147fc700)|Stuck query found. Statement ID: 72, Last chunk producer updated: 1 WriteTable, Started on: Thu May 19 14:01:25 2022, Last updated: Thu May 19 15:01:25 2022, Stuck time: 5 hours, Max allowed stuck query time: 5 hours + +The ``healerMaxInactivityHours`` log frequency is calculated as 5% of the flag setting. When set to to five hours (the default setting), the Query Healer triggers an examination every 15 minutes. + +Configuring the Healer +------------------ +The following **Administration Worker** flags are required to configure the Query Healer: + + * :ref:`is_healer_on` - Enables the Query Healer. + + :: + + * :ref:`healer_max_inactivity_hours` - Defines the threshold for creating a log recording a slow statement. The log includes information about the log memory, CPU and GPU. + +The ``healerMaxInactivityHours`` log frequency is calculated as 5% of the flag setting. For example, setting ``healerMaxInactivityHours`` to five hours (the default setting) triggers an examination every 15 minutes. \ No newline at end of file diff --git a/getting_started/preparing_your_machine_to_install_sqream.rst b/getting_started/preparing_your_machine_to_install_sqream.rst index 435f35de0..3747e5f5f 100644 --- a/getting_started/preparing_your_machine_to_install_sqream.rst +++ b/getting_started/preparing_your_machine_to_install_sqream.rst @@ -32,4 +32,4 @@ To prepare your machine to install SQream, do the following: For more information, see the following: * :ref:`recommended_pre-installation_configurations` -* `Hardware Guide `_ +* :ref:`hardware_guide` diff --git a/index.rst b/index.rst index d63e8aa7f..017193d2d 100644 --- a/index.rst +++ b/index.rst @@ -4,13 +4,13 @@ SQream DB Documentation ************************* -For SQream version 2022.1. +For SQream Version 2021.2.1.24. .. only:: html .. tip:: Want to read this offline? - `Download the documentation as a single PDF `_ . + `Download the documentation as a single PDF `_ . .. only:: pdf or latex @@ -35,7 +35,7 @@ SQream DB easily plugs in to third-party tools like :ref:`Tableau`_ + :ref:`first_steps` :ref:`sql_feature_support` @@ -47,17 +47,16 @@ SQream DB easily plugs in to third-party tools like :ref:`Tableau`_ + `Setting up SQream `_ :ref:`Best practices` + :ref:`connect_to_tableau` * - **Releases** - **Driver and Deployment** - **Help and Support** - * - - :ref:`2022.1<2022.1>` - + * - :ref:`2021.2<2021.2>` :ref:`2021.1<2021.1>` @@ -75,10 +74,12 @@ SQream DB easily plugs in to third-party tools like :ref:`Tableau` + :ref:`connect_to_tableau` - - :ref:`troubleshooting` guide + `Troubleshooting Page `_ - :ref:`information_for_support` + `Gathering Information for SQream Support `_ + @@ -89,7 +90,7 @@ If you couldn't find what you're looking for, we're always happy to help. Visit .. rubric:: Looking for older versions? -This version of the documentation is for SQream DB Version 2022.1. +This version of the documentation is for SQream DB |latest_version|. If you're looking for an older version of the documentation, versions 1.10 through 2019.2.1 are available at http://previous.sqream.com . @@ -104,6 +105,7 @@ If you're looking for an older version of the documentation, versions 1.10 throu installation_guides/index data_ingestion/index connecting_to_sqream/index + external_storage_platforms/index loading_and_unloading_data/index feature_guides/index operational_guides/index @@ -122,4 +124,4 @@ If you're looking for an older version of the documentation, versions 1.10 throu * :ref:`genindex` * :ref:`modindex` - * :ref:`search` + * :ref:`search` \ No newline at end of file diff --git a/loading_and_unloading_data/index.rst b/loading_and_unloading_data/index.rst index ae5c506bb..f14f79fc2 100644 --- a/loading_and_unloading_data/index.rst +++ b/loading_and_unloading_data/index.rst @@ -3,9 +3,10 @@ *********************** Loading and Unloading Data *********************** + The **Loading Data** section describes concepts and operations related to importing data into your SQream database: -* `Overview of loading data `_ - Describes best practices and considerations for loading data into SQream from a variety of sources and locations. +* :ref:`Overview of loading data ` - Describes best practices and considerations for loading data into SQream from a variety of sources and locations. * `Alternatives to loading data (foreign tables) `_ - Useful for running queries directly on external data without importing into your SQream database. @@ -25,12 +26,12 @@ The **Loading Data** section describes concepts and operations related to import * Troubleshooting - Describes troubleshooting solutions related to importing data from the following: - * `SAS Viya `_ + * :ref:`SAS Viya ` - * `Tableau `_ + * :ref:`Tableau ` The **Unloading Data** section describes concepts and operations related to exporting data from your SQream database: -* `Overview of unloading data `_ - Describes best practices and considerations for unloading data from SQream to a variety of sources and locations. +* `Overview of unloading data `_ - Describes best practices and considerations for unloading data from SQream to a variety of sources and locations. -* `The COPY TO statement `_ - Used for unloading data from a SQream database table or query to a file on a filesystem. \ No newline at end of file +* `The COPY TO statement `_ - Used for unloading data from a SQream database table or query to a file on a filesystem. diff --git a/operational_guides/access_control.rst b/operational_guides/access_control.rst index ffe687213..88a14d71b 100644 --- a/operational_guides/access_control.rst +++ b/operational_guides/access_control.rst @@ -4,581 +4,11 @@ Access Control ************** -.. contents:: - :local: - :depth: 1 - -Overview -========== -Access control provides authentication and authorization in SQream DB. SQream DB manages authentication and authorization using a role-based access control system (RBAC), like ANSI SQL and other SQL products. - -SQream DB has a default permissions system which is inspired by Postgres, but with more power. In most cases, this allows an administrator to set things up so that every object gets permissions set automatically. In SQream DB, users log in from any worker which verifies their roles and permissions from the metadata server. Each statement issues commands as the currently logged in role. - -Roles are defined at the cluster level, meaning they are valid for all databases in the cluster. To bootstrap SQream DB, a new install will always have one ``SUPERUSER`` role, typically named ``sqream``. To create more roles, you should first connect as this role. - -The following: - -* **Role** - a role can be a user, a group, or both. Roles can own database objects (e.g. tables), and can assign permissions on those objects to other roles. Roles can be members of other roles, meaning a user role can inherit permissions from its parent role. - - :: - -* **Authentication** - verifying the identity of the role. User roles have usernames (:term:`role names`) and passwords. - - :: - -* **Authorization** - checking the role has permissions to do a particular thing. The :ref:`grant` command is used for this. - -Managing Roles -===== -Roles are used for both users and groups. Roles are global across all databases in the SQream DB cluster. To use a ``ROLE`` as a user, it should have a password, the login permission, and connect permissions to the relevant databases. - -The Roles section describes the following role-related operations: - -.. contents:: - :local: - :depth: 1 - -Creating New Roles (Users) ------------------------------- -A user role can log in to the database, so it should have ``LOGIN`` permissions, as well as a password. - -For example: - -.. code-block:: postgres - - CREATE ROLE role_name ; - GRANT LOGIN to role_name ; - GRANT PASSWORD 'new_password' to role_name ; - GRANT CONNECT ON DATABASE database_name to role_name ; - -Examples: - -.. code-block:: postgres - - CREATE ROLE new_role_name ; - GRANT LOGIN TO new_role_name; - GRANT PASSWORD 'my_password' TO new_role_name; - GRANT CONNECT ON DATABASE master TO new_role_name; - -A database role may have a number of permissions that define what tasks it can perform. These are assigned using the :ref:`grant` command. - -Dropping Users ---------------- - -.. code-block:: postgres - - DROP ROLE role_name ; - -Examples: - -.. code-block:: postgres - - DROP ROLE admin_role ; - -Altering a User Name ------------------------- - -Renaming a user's role: - -.. code-block:: postgres - - ALTER ROLE role_name RENAME TO new_role_name ; - -Examples: - -.. code-block:: postgres - - ALTER ROLE admin_role RENAME TO copy_role ; - -.. _change_password: - -Changing User Passwords --------------------------- - -To change a user role's password, grant the user a new password. - -.. code-block:: postgres - - GRANT PASSWORD 'new_password' TO rhendricks; - -.. note:: Granting a new password overrides any previous password. Changing the password while the role has an active running statement does not affect that statement, but will affect subsequent statements. - -Altering Public Role Permissions ------------ - -There is a public role which always exists. Each role is granted to the ``PUBLIC`` role (i.e. is a member of the public group), and this cannot be revoked. You can alter the permissions granted to the public role. - -The ``PUBLIC`` role has ``USAGE`` and ``CREATE`` permissions on ``PUBLIC`` schema by default, therefore, new users can create, :ref:`insert`, :ref:`delete`, and :ref:`select` from objects in the ``PUBLIC`` schema. - - -Altering Role Membership (Groups) -------------------------- - -Many database administrators find it useful to group user roles together. By grouping users, permissions can be granted to, or revoked from a group with one command. In SQream DB, this is done by creating a group role, granting permissions to it, and then assigning users to that group role. - -To use a role purely as a group, omit granting it ``LOGIN`` and ``PASSWORD`` permissions. - -The ``CONNECT`` permission can be given directly to user roles, and/or to the groups they are part of. - -.. code-block:: postgres - - CREATE ROLE my_group; - -Once the group role exists, you can add user roles (members) using the ``GRANT`` command. For example: - -.. code-block:: postgres - - -- Add my_user to this group - GRANT my_group TO my_user; - - -To manage object permissions like databases and tables, you would then grant permissions to the group-level role (see :ref:`the permissions table` below. - -All member roles then inherit the permissions from the group. For example: - -.. code-block:: postgres - - -- Grant all group users connect permissions - GRANT CONNECT ON DATABASE a_database TO my_group; - - -- Grant all permissions on tables in public schema - GRANT ALL ON all tables IN schema public TO my_group; - -Removing users and permissions can be done with the ``REVOKE`` command: - -.. code-block:: postgres - - -- remove my_other_user from this group - REVOKE my_group FROM my_other_user; - -.. _permissions_table: - -Permissions -=========== -The following table displays the access control permissions: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Object/layer - - Permission - - Description - - * - all databases - - ``LOGIN`` - - use role to log into the system (the role also needs connect permission on the database it is connecting to) - - * - all databases - - ``PASSWORD`` - - the password used for logging into the system - - * - all databases - - ``SUPERUSER`` - - no permission restrictions on any activity - - * - database - - ``SUPERUSER`` - - no permission restrictions on any activity within that database (this does not include modifying roles or permissions) - - * - database - - ``CONNECT`` - - connect to the database - - * - database - - ``CREATE`` - - create schemas in the database - - * - database - - ``CREATE FUNCTION`` - - create and drop functions - - * - schema - - ``USAGE`` - - allows additional permissions within the schema - - * - schema - - ``CREATE`` - - create tables in the schema - - * - table - - ``SELECT`` - - :ref:`select` from the table - - * - table - - ``INSERT`` - - :ref:`insert` into the table - - * - table - - ``UPDATE`` - - :ref:`update` the value of certain columns in existing rows without creating a table - - * - table - - ``DELETE`` - - :ref:`delete` and :ref:`truncate` on the table - - * - table - - ``DDL`` - - drop and alter on the table - - * - table - - ``ALL`` - - all the table permissions - - * - function - - ``EXECUTE`` - - use the function - - * - function - - ``DDL`` - - drop and alter on the function - - * - function - - ``ALL`` - - all function permissions - -GRANT ------ - -:ref:`grant` gives permissions to a role, shown in the following syntax example: - -.. code-block:: postgres - - -- Grant permissions at the instance/ storage cluster level: - GRANT - - { SUPERUSER - | LOGIN - | PASSWORD '' - } - TO [, ...] - - -- Grant permissions at the database level: - GRANT {{CREATE | CONNECT| DDL | SUPERUSER | CREATE FUNCTION} [, ...] | ALL [PERMISSIONS]} - - ON DATABASE [, ...] - TO [, ...] - - -- Grant permissions at the schema level: - GRANT {{ CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [ - PERMISSIONS ]} - ON SCHEMA [, ...] - TO [, ...] - - -- Grant permissions at the object level: - GRANT {{SELECT | INSERT | DELETE | DDL } [, ...] | ALL [PERMISSIONS]} - ON { TABLE [, ...] | ALL TABLES IN SCHEMA [, ...]} - TO [, ...] - - -- Grant execute function permission: - GRANT {ALL | EXECUTE | DDL} ON FUNCTION function_name - TO role; - - -- Allows role2 to use permissions granted to role1 - GRANT [, ...] - TO - - -- Also allows the role2 to grant role1 to other roles: - GRANT [, ...] - TO - WITH ADMIN OPTION - -The following are some ``GRANT`` examples: - -.. code-block:: postgres - - GRANT LOGIN,superuser TO admin; - - GRANT CREATE FUNCTION ON database master TO admin; - - GRANT SELECT ON TABLE admin.table1 TO userA; - - GRANT EXECUTE ON FUNCTION my_function TO userA; - - GRANT ALL ON FUNCTION my_function TO userA; - - GRANT DDL ON admin.main_table TO userB; - - GRANT ALL ON all tables IN schema public TO userB; - - GRANT admin TO userC; - - GRANT superuser ON schema demo TO userA - - GRANT admin_role TO userB; - -REVOKE ------- - -:ref:`revoke` removes permissions from a role, shown in the following syntax example: - -.. code-block:: postgres - - -- Revoke permissions at the instance/ storage cluster level: - REVOKE - { SUPERUSER - | LOGIN - | PASSWORD - } - FROM [, ...] - - -- Revoke permissions at the database level: - REVOKE {{CREATE | CONNECT | DDL | SUPERUSER | CREATE FUNCTION}[, ...] |ALL [PERMISSIONS]} - ON DATABASE [, ...] - FROM [, ...] - - -- Revoke permissions at the schema level: - REVOKE { { CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [PERMISSIONS]} - ON SCHEMA [, ...] - FROM [, ...] - - -- Revoke permissions at the object level: - REVOKE { { SELECT | INSERT | DELETE | DDL } [, ...] | ALL } - ON { [ TABLE ] [, ...] | ALL TABLES IN SCHEMA - - [, ...] } - FROM [, ...] - - -- Removes access to permissions in role1 by role 2 - REVOKE [, ...] FROM [, ...] WITH ADMIN OPTION - - -- Removes permissions to grant role1 to additional roles from role2 - REVOKE [, ...] FROM [, ...] WITH ADMIN OPTION - - -Examples: - -.. code-block:: postgres - - REVOKE superuser on schema demo from userA; - - REVOKE delete on admin.table1 from userB; - - REVOKE login from role_test; - - REVOKE CREATE FUNCTION FROM admin; - -Default Permissions -------------------- - -The default permissions system (See :ref:`alter_default_permissions`) -can be used to automatically grant permissions to newly -created objects (See the departmental example below for one way it can be used). - -A default permissions rule looks for a schema being created, or a -table (possibly by schema), and is table to grant any permission to -that object to any role. This happens when the create table or create -schema statement is run. - - -.. code-block:: postgres - - - ALTER DEFAULT PERMISSIONS FOR target_role_name - [IN schema_name, ...] - FOR { TABLES | SCHEMAS } - { grant_clause | DROP grant_clause} - TO ROLE { role_name | public }; - - grant_clause ::= - GRANT - { CREATE FUNCTION - | SUPERUSER - | CONNECT - | CREATE - | USAGE - | SELECT - | INSERT - | UPDATE - | DELETE - | DDL - | EXECUTE - | ALL - } - - -Departmental Example -======================= - -You work in a company with several departments. - -The example below shows you how to manage permissions in a database shared by multiple departments, where each department has different roles for the tables by schema. It walks you through how to set the permissions up for existing objects and how to set up default permissions rules to cover newly created objects. - -The concept is that you set up roles for each new schema with the correct permissions, then the existing users can use these roles. - -A superuser must do new setup for each new schema which is a limitation, but superuser permissions are not needed at any other time, and neither are explicit grant statements or object ownership changes. - -In the example, the database is called ``my_database``, and the new or existing schema being set up to be managed in this way is called ``my_schema``. - -.. figure:: /_static/images/access_control_department_example.png - :scale: 60 % - - Our departmental example has four user group roles and seven users roles - -There will be a group for this schema for each of the following: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Group - - Activities - - * - database designers - - create, alter and drop tables - - * - updaters - - insert and delete data - - * - readers - - read data - - * - security officers - - add and remove users from these groups - -Setting up the department permissions ------------------------------------------- - -As a superuser, you connect to the system and run the following: - -.. code-block:: postgres - - -- create the groups - - CREATE ROLE my_schema_security_officers; - CREATE ROLE my_schema_database_designers; - CREATE ROLE my_schema_updaters; - CREATE ROLE my_schema_readers; - - -- grant permissions for each role - -- we grant permissions for existing objects here too, - -- so you don't have to start with an empty schema - - -- security officers - - GRANT connect ON DATABASE my_database TO my_schema_security_officers; - GRANT usage ON SCHEMA my_schema TO my_schema_security_officers; - - GRANT my_schema_database_designers TO my_schema_security_officers WITH ADMIN OPTION; - GRANT my_schema_updaters TO my_schema_security_officers WITH ADMIN OPTION; - GRANT my_schema_readers TO my_schema_security_officers WITH ADMIN OPTION; - - -- database designers - - GRANT connect ON DATABASE my_database TO my_schema_database_designers; - GRANT usage ON SCHEMA my_schema TO my_schema_database_designers; - - GRANT create,ddl ON SCHEMA my_schema TO my_schema_database_designers; - - -- updaters - - GRANT connect ON DATABASE my_database TO my_schema_updaters; - GRANT usage ON SCHEMA my_schema TO my_schema_updaters; - - GRANT SELECT,INSERT,DELETE ON ALL TABLES IN SCHEMA my_schema TO my_schema_updaters; - - -- readers - - GRANT connect ON DATABASE my_database TO my_schema_readers; - GRANT usage ON SCHEMA my_schema TO my_schema_readers; - - GRANT SELECT ON ALL TABLES IN SCHEMA my_schema TO my_schema_readers; - GRANT EXECUTE ON ALL FUNCTIONS TO my_schema_readers; - - - -- create the default permissions for new objects - - ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema - FOR TABLES GRANT SELECT,INSERT,DELETE TO my_schema_updaters; - - -- For every table created by my_schema_database_designers, give access to my_schema_readers: - - ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema - FOR TABLES GRANT SELECT TO my_schema_readers; - -.. note:: - * This process needs to be repeated by a user with ``SUPERUSER`` permissions each time a new schema is brought into this permissions management approach. - - * - By default, any new object created will not be accessible by our new ``my_schema_readers`` group. - Running a ``GRANT SELECT ...`` only affects objects that already exist in the schema or database. - - If you're getting a ``Missing the following permissions: SELECT on table 'database.public.tablename'`` error, make sure that - you've altered the default permissions with the ``ALTER DEFAULT PERMISSIONS`` statement. - -Creating new users in the departments ------------------------------------------ - -After the group roles have been created, you can now create user roles for each of your users. - -.. code-block:: postgres - - -- create the new database designer users - - CREATE ROLE ecodd; - GRANT LOGIN TO ecodd; - GRANT PASSWORD 'ecodds_secret_password' TO ecodd; - GRANT CONNECT ON DATABASE my_database TO ecodd; - GRANT my_schema_database_designers TO ecodd; - - CREATE ROLE ebachmann; - GRANT LOGIN TO ebachmann; - GRANT PASSWORD 'another_secret_password' TO ebachmann; - GRANT CONNECT ON DATABASE my_database TO ebachmann; - GRANT my_database_designers TO ebachmann; - - -- If a user already exists, we can assign that user directly to the group - - GRANT my_schema_updaters TO rhendricks; - - -- Create users in the readers group - - CREATE ROLE jbarker; - GRANT LOGIN TO jbarker; - GRANT PASSWORD 'action_jack' TO jbarker; - GRANT CONNECT ON DATABASE my_database TO jbarker; - GRANT my_schema_readers TO jbarker; - - CREATE ROLE lbream; - GRANT LOGIN TO lbream; - GRANT PASSWORD 'artichoke123' TO lbream; - GRANT CONNECT ON DATABASE my_database TO lbream; - GRANT my_schema_readers TO lbream; - - CREATE ROLE pgregory; - GRANT LOGIN TO pgregory; - GRANT PASSWORD 'c1ca6a' TO pgregory; - GRANT CONNECT ON DATABASE my_database TO pgregory; - GRANT my_schema_readers TO pgregory; - - -- Create users in the security officers group - - CREATE ROLE hoover; - GRANT LOGIN TO hoover; - GRANT PASSWORD 'mintchip' TO hoover; - GRANT CONNECT ON DATABASE my_database TO hoover; - GRANT my_schema_security_officers TO hoover; - - -.. todo: - create some example users - show that they have the right permission - try out the with admin option. we can't really do a security officer because - only superusers can create users and logins. see what can be done - need 1-2 users in each group, for at least 2 schemas/departments - this example will be very big just to show what this setup can do ... - example: a security officer for a department which will only have - read only access to a schema can only get that with admin option - access granted to them - -After this setup: - -* Database designers will be able to run any ddl on objects in the schema and create new objects, including ones created by other database designers -* Updaters will be able to insert and delete to existing and new tables -* Readers will be able to read from existing and new tables - -All this will happen without having to run any more ``GRANT`` statements. - -Any security officer will be able to add and remove users from these -groups. Creating and dropping login users themselves must be done by a -superuser. +.. toctree:: + :maxdepth: 1 + :titlesonly: + + access_control_overview + access_control_managing_roles + access_control_permissions + access_control_departmental_example \ No newline at end of file diff --git a/operational_guides/access_control_departmental_example.rst b/operational_guides/access_control_departmental_example.rst new file mode 100644 index 000000000..0a6b55e54 --- /dev/null +++ b/operational_guides/access_control_departmental_example.rst @@ -0,0 +1,185 @@ +.. _access_control_departmental_example: + +************** +Departmental Example +************** + +You work in a company with several departments. + +The example below shows you how to manage permissions in a database shared by multiple departments, where each department has different roles for the tables by schema. It walks you through how to set the permissions up for existing objects and how to set up default permissions rules to cover newly created objects. + +The concept is that you set up roles for each new schema with the correct permissions, then the existing users can use these roles. + +A superuser must do new setup for each new schema which is a limitation, but superuser permissions are not needed at any other time, and neither are explicit grant statements or object ownership changes. + +In the example, the database is called ``my_database``, and the new or existing schema being set up to be managed in this way is called ``my_schema``. + +Our departmental example has four user group roles and seven users roles + +There will be a group for this schema for each of the following: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Group + - Activities + + * - database designers + - create, alter and drop tables + + * - updaters + - insert and delete data + + * - readers + - read data + + * - security officers + - add and remove users from these groups + +Setting up the department permissions +------------------------------------------ + +As a superuser, you connect to the system and run the following: + +.. code-block:: postgres + + -- create the groups + + CREATE ROLE my_schema_security_officers; + CREATE ROLE my_schema_database_designers; + CREATE ROLE my_schema_updaters; + CREATE ROLE my_schema_readers; + + -- grant permissions for each role + -- we grant permissions for existing objects here too, + -- so you don't have to start with an empty schema + + -- security officers + + GRANT connect ON DATABASE my_database TO my_schema_security_officers; + GRANT usage ON SCHEMA my_schema TO my_schema_security_officers; + + GRANT my_schema_database_designers TO my_schema_security_officers WITH ADMIN OPTION; + GRANT my_schema_updaters TO my_schema_security_officers WITH ADMIN OPTION; + GRANT my_schema_readers TO my_schema_security_officers WITH ADMIN OPTION; + + -- database designers + + GRANT connect ON DATABASE my_database TO my_schema_database_designers; + GRANT usage ON SCHEMA my_schema TO my_schema_database_designers; + + GRANT create,ddl ON SCHEMA my_schema TO my_schema_database_designers; + + -- updaters + + GRANT connect ON DATABASE my_database TO my_schema_updaters; + GRANT usage ON SCHEMA my_schema TO my_schema_updaters; + + GRANT SELECT,INSERT,DELETE ON ALL TABLES IN SCHEMA my_schema TO my_schema_updaters; + + -- readers + + GRANT connect ON DATABASE my_database TO my_schema_readers; + GRANT usage ON SCHEMA my_schema TO my_schema_readers; + + GRANT SELECT ON ALL TABLES IN SCHEMA my_schema TO my_schema_readers; + GRANT EXECUTE ON ALL FUNCTIONS TO my_schema_readers; + + + -- create the default permissions for new objects + + ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema + FOR TABLES GRANT SELECT,INSERT,DELETE TO my_schema_updaters; + + -- For every table created by my_schema_database_designers, give access to my_schema_readers: + + ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema + FOR TABLES GRANT SELECT TO my_schema_readers; + +.. note:: + * This process needs to be repeated by a user with ``SUPERUSER`` permissions each time a new schema is brought into this permissions management approach. + + * + By default, any new object created will not be accessible by our new ``my_schema_readers`` group. + Running a ``GRANT SELECT ...`` only affects objects that already exist in the schema or database. + + If you're getting a ``Missing the following permissions: SELECT on table 'database.public.tablename'`` error, make sure that + you've altered the default permissions with the ``ALTER DEFAULT PERMISSIONS`` statement. + +Creating new users in the departments +----------------------------------------- + +After the group roles have been created, you can now create user roles for each of your users. + +.. code-block:: postgres + + -- create the new database designer users + + CREATE ROLE ecodd; + GRANT LOGIN TO ecodd; + GRANT PASSWORD 'ecodds_secret_password' TO ecodd; + GRANT CONNECT ON DATABASE my_database TO ecodd; + GRANT my_schema_database_designers TO ecodd; + + CREATE ROLE ebachmann; + GRANT LOGIN TO ebachmann; + GRANT PASSWORD 'another_secret_password' TO ebachmann; + GRANT CONNECT ON DATABASE my_database TO ebachmann; + GRANT my_database_designers TO ebachmann; + + -- If a user already exists, we can assign that user directly to the group + + GRANT my_schema_updaters TO rhendricks; + + -- Create users in the readers group + + CREATE ROLE jbarker; + GRANT LOGIN TO jbarker; + GRANT PASSWORD 'action_jack' TO jbarker; + GRANT CONNECT ON DATABASE my_database TO jbarker; + GRANT my_schema_readers TO jbarker; + + CREATE ROLE lbream; + GRANT LOGIN TO lbream; + GRANT PASSWORD 'artichoke123' TO lbream; + GRANT CONNECT ON DATABASE my_database TO lbream; + GRANT my_schema_readers TO lbream; + + CREATE ROLE pgregory; + GRANT LOGIN TO pgregory; + GRANT PASSWORD 'c1ca6a' TO pgregory; + GRANT CONNECT ON DATABASE my_database TO pgregory; + GRANT my_schema_readers TO pgregory; + + -- Create users in the security officers group + + CREATE ROLE hoover; + GRANT LOGIN TO hoover; + GRANT PASSWORD 'mintchip' TO hoover; + GRANT CONNECT ON DATABASE my_database TO hoover; + GRANT my_schema_security_officers TO hoover; + + +.. todo: + create some example users + show that they have the right permission + try out the with admin option. we can't really do a security officer because + only superusers can create users and logins. see what can be done + need 1-2 users in each group, for at least 2 schemas/departments + this example will be very big just to show what this setup can do ... + example: a security officer for a department which will only have + read only access to a schema can only get that with admin option + access granted to them + +After this setup: + +* Database designers will be able to run any ddl on objects in the schema and create new objects, including ones created by other database designers +* Updaters will be able to insert and delete to existing and new tables +* Readers will be able to read from existing and new tables + +All this will happen without having to run any more ``GRANT`` statements. + +Any security officer will be able to add and remove users from these +groups. Creating and dropping login users themselves must be done by a +superuser. \ No newline at end of file diff --git a/operational_guides/access_control_managing_roles.rst b/operational_guides/access_control_managing_roles.rst new file mode 100644 index 000000000..373b9a212 --- /dev/null +++ b/operational_guides/access_control_managing_roles.rst @@ -0,0 +1,124 @@ +.. _access_control_managing_roles: + +************** +Managing Roles +************** +Roles are used for both users and groups, and are global across all databases in the SQream cluster. For a ``ROLE`` to be used as a user, it requires a password and log-in and connect permissionss to the relevant databases. + +The Managing Roles section describes the following role-related operations: + +.. contents:: + :local: + :depth: 1 + +Creating New Roles (Users) +------------------------------ +A user role logging in to the database requires ``LOGIN`` permissions and as a password. + +The following is the syntax for creating a new role: + +.. code-block:: postgres + + CREATE ROLE ; + GRANT LOGIN to ; + GRANT PASSWORD <'new_password'> to ; + GRANT CONNECT ON DATABASE to ; + +The following is an example of creating a new role: + +.. code-block:: postgres + + CREATE ROLE new_role_name ; + GRANT LOGIN TO new_role_name; + GRANT PASSWORD 'my_password' to new_role_name; + GRANT CONNECT ON DATABASE master to new_role_name; + +A database role may have a number of permissions that define what tasks it can perform, which are assigned using the :ref:`grant` command. + +Dropping a User +------------------------------ +The following is the syntax for dropping a user: + +.. code-block:: postgres + + DROP ROLE ; + +The following is an example of dropping a user: + +.. code-block:: postgres + + DROP ROLE admin_role ; + +Altering a User Name +------------------------------ +The following is the syntax for altering a user name: + +.. code-block:: postgres + + ALTER ROLE RENAME TO ; + +The following is an example of altering a user name: + +.. code-block:: postgres + + ALTER ROLE admin_role RENAME TO copy_role ; + +Changing a User Password +------------------------------ +You can change a user role's password by granting the user a new password. + +The following is an example of changing a user password: + +.. code-block:: postgres + + GRANT PASSWORD <'new_password'> TO rhendricks; + +.. note:: Granting a new password overrides any previous password. Changing the password while the role has an active running statement does not affect that statement, but will affect subsequent statements. + +Altering Public Role Permissions +------------------------------ + +There is a public role which always exists. Each role is granted to the ``PUBLIC`` role (i.e. is a member of the public group), and this cannot be revoked. You can alter the permissions granted to the public role. + +The ``PUBLIC`` role has ``USAGE`` and ``CREATE`` permissions on ``PUBLIC`` schema by default, therefore, new users can create, :ref:`insert`, :ref:`delete`, and :ref:`select` from objects in the ``PUBLIC`` schema. + + +Altering Role Membership (Groups) +------------------------------ + +Many database administrators find it useful to group user roles together. By grouping users, permissions can be granted to, or revoked from a group with one command. In SQream DB, this is done by creating a group role, granting permissions to it, and then assigning users to that group role. + +To use a role purely as a group, omit granting it ``LOGIN`` and ``PASSWORD`` permissions. + +The ``CONNECT`` permission can be given directly to user roles, and/or to the groups they are part of. + +.. code-block:: postgres + + CREATE ROLE my_group; + +Once the group role exists, you can add user roles (members) using the ``GRANT`` command. For example: + +.. code-block:: postgres + + -- Add my_user to this group + GRANT my_group TO my_user; + + +To manage object permissions like databases and tables, you would then grant permissions to the group-level role (see :ref:`the permissions table` below. + +All member roles then inherit the permissions from the group. For example: + +.. code-block:: postgres + + -- Grant all group users connect permissions + GRANT CONNECT ON DATABASE a_database TO my_group; + + -- Grant all permissions on tables in public schema + GRANT ALL ON all tables IN schema public TO my_group; + +Removing users and permissions can be done with the ``REVOKE`` command: + +.. code-block:: postgres + + -- remove my_other_user from this group + REVOKE my_group FROM my_other_user; \ No newline at end of file diff --git a/operational_guides/access_control_overview.rst b/operational_guides/access_control_overview.rst new file mode 100644 index 000000000..080797fec --- /dev/null +++ b/operational_guides/access_control_overview.rst @@ -0,0 +1,20 @@ +.. _access_control_overview: + +************** +Overview +************** +Access control refers to SQream's authentication and authorization operations, managed using a **Role-Based Access Control (RBAC)** system, such as ANSI SQL or other SQL products. SQream's default permissions system is similar to Postgres, but is more powerful. SQream's method lets administrators prepare the system to automatically provide objects with their required permissions. + +SQream users can log in from any worker, which verify their roles and permissions from the metadata server. Each statement issues commands as the role that you're currently logged into. Roles are defined at the cluster level, and are valid for all databases in the cluster. To bootstrap SQream, new installations require one ``SUPERUSER`` role, typically named ``sqream``. You can only create new roles by connecting as this role. + +Access control refers to the following basic concepts: + + * **Role** - A role can be a user, a group, or both. Roles can own database objects (such as tables) and can assign permissions on those objects to other roles. Roles can be members of other roles, meaning a user role can inherit permissions from its parent role. + + :: + + * **Authentication** - Verifies the identity of the role. User roles have usernames (or **role names**) and passwords. + + :: + + * **Authorization** - Checks that a role has permissions to perform a particular operation, such as the :ref:`grant` command. \ No newline at end of file diff --git a/operational_guides/access_control_permissions.rst b/operational_guides/access_control_permissions.rst new file mode 100644 index 000000000..5afc23009 --- /dev/null +++ b/operational_guides/access_control_permissions.rst @@ -0,0 +1,218 @@ +.. _access_control_permissions: + +************** +Permissions +************** + +The following table displays the access control permissions: + ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Permission** | **Description** | ++====================+=========================================================================================================================+ +| **Object/Layer: All Databases** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``LOGIN`` | use role to log into the system (the role also needs connect permission on the database it is connecting to) | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``PASSWORD`` | the password used for logging into the system | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``SUPERUSER`` | no permission restrictions on any activity | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Database** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``SUPERUSER`` | no permission restrictions on any activity within that database (this does not include modifying roles or permissions) | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CONNECT`` | connect to the database | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CREATE`` | create schemas in the database | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CREATE FUNCTION``| create and drop functions | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Schema** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``USAGE`` | allows additional permissions within the schema | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CREATE`` | create tables in the schema | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Table** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``SELECT`` | :ref:`select` from the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``INSERT`` | :ref:`insert` into the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``UPDATE`` | UPDATE the value of certain columns in existing rows without creating a table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``DELETE`` | :ref:`delete` and :ref:`truncate` on the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``DDL`` | drop and alter on the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``ALL`` | all the table permissions | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Function** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``EXECUTE`` | use the function | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``DDL`` | drop and alter on the function | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``ALL`` | all function permissions | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ + + + + +GRANT +----- + +:ref:`grant` gives permissions to a role. + +.. code-block:: postgres + + -- Grant permissions at the instance/ storage cluster level: + GRANT + + { SUPERUSER + | LOGIN + | PASSWORD '' + } + TO [, ...] + + -- Grant permissions at the database level: + GRANT {{CREATE | CONNECT| DDL | SUPERUSER | CREATE FUNCTION} [, ...] | ALL [PERMISSIONS]} + + ON DATABASE [, ...] + TO [, ...] + + -- Grant permissions at the schema level: + GRANT {{ CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [ + PERMISSIONS ]} + ON SCHEMA [, ...] + TO [, ...] + + -- Grant permissions at the object level: + GRANT {{SELECT | INSERT | DELETE | DDL } [, ...] | ALL [PERMISSIONS]} + ON { TABLE [, ...] | ALL TABLES IN SCHEMA [, ...]} + TO [, ...] + + -- Grant execute function permission: + GRANT {ALL | EXECUTE | DDL} ON FUNCTION function_name + TO role; + + -- Allows role2 to use permissions granted to role1 + GRANT [, ...] + TO + + -- Also allows the role2 to grant role1 to other roles: + GRANT [, ...] + TO + WITH ADMIN OPTION + +``GRANT`` examples: + +.. code-block:: postgres + + GRANT LOGIN,superuser TO admin; + + GRANT CREATE FUNCTION ON database master TO admin; + + GRANT SELECT ON TABLE admin.table1 TO userA; + + GRANT EXECUTE ON FUNCTION my_function TO userA; + + GRANT ALL ON FUNCTION my_function TO userA; + + GRANT DDL ON admin.main_table TO userB; + + GRANT ALL ON all tables IN schema public TO userB; + + GRANT admin TO userC; + + GRANT superuser ON schema demo TO userA + + GRANT admin_role TO userB; + +REVOKE +------ + +:ref:`revoke` removes permissions from a role. + +.. code-block:: postgres + + -- Revoke permissions at the instance/ storage cluster level: + REVOKE + { SUPERUSER + | LOGIN + | PASSWORD + } + FROM [, ...] + + -- Revoke permissions at the database level: + REVOKE {{CREATE | CONNECT | DDL | SUPERUSER | CREATE FUNCTION}[, ...] |ALL [PERMISSIONS]} + ON DATABASE [, ...] + FROM [, ...] + + -- Revoke permissions at the schema level: + REVOKE { { CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [PERMISSIONS]} + ON SCHEMA [, ...] + FROM [, ...] + + -- Revoke permissions at the object level: + REVOKE { { SELECT | INSERT | DELETE | DDL } [, ...] | ALL } + ON { [ TABLE ] [, ...] | ALL TABLES IN SCHEMA + + [, ...] } + FROM [, ...] + + -- Removes access to permissions in role1 by role 2 + REVOKE [, ...] FROM [, ...] WITH ADMIN OPTION + + -- Removes permissions to grant role1 to additional roles from role2 + REVOKE [, ...] FROM [, ...] WITH ADMIN OPTION + + +Examples: + +.. code-block:: postgres + + REVOKE superuser on schema demo from userA; + + REVOKE delete on admin.table1 from userB; + + REVOKE login from role_test; + + REVOKE CREATE FUNCTION FROM admin; + +Default permissions +------------------- + +The default permissions system (See :ref:`alter_default_permissions`) +can be used to automatically grant permissions to newly +created objects (See the departmental example below for one way it can be used). + +A default permissions rule looks for a schema being created, or a +table (possibly by schema), and is table to grant any permission to +that object to any role. This happens when the create table or create +schema statement is run. + + +.. code-block:: postgres + + + ALTER DEFAULT PERMISSIONS FOR target_role_name + [IN schema_name, ...] + FOR { TABLES | SCHEMAS } + { grant_clause | DROP grant_clause} + TO ROLE { role_name | public }; + + grant_clause ::= + GRANT + { CREATE FUNCTION + | SUPERUSER + | CONNECT + | CREATE + | USAGE + | SELECT + | INSERT + | DELETE + | DDL + | EXECUTE + | ALL + } \ No newline at end of file diff --git a/operational_guides/index.rst b/operational_guides/index.rst index 57f15b0eb..16f9fbf6d 100644 --- a/operational_guides/index.rst +++ b/operational_guides/index.rst @@ -14,7 +14,6 @@ This section summarizes the following operational guides: access_control creating_or_cloning_a_storage_cluster - external_data foreign_tables delete_guide exporting_data diff --git a/readthedocs.yaml b/readthedocs.yaml new file mode 100644 index 000000000..77b6f73c2 --- /dev/null +++ b/readthedocs.yaml @@ -0,0 +1,28 @@ +# .readthedocs.yaml +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +# Required +version: 2 + +# Set the version of Python and other tools you might need +build: + os: ubuntu-22.04 + tools: + python: "3.11" + # You can also specify other tool versions: + # nodejs: "19" + # rust: "1.64" + # golang: "1.19" + +# Build documentation in the docs/ directory with Sphinx +sphinx: + configuration: conf.py + +# If using Sphinx, optionally build your docs in additional formats such as PDF +formats: [htmlzip,pdf] + +# Optionally declare the Python requirements required to build your docs +python: + install: + - requirements: requirements.txt \ No newline at end of file diff --git a/releases/2020.1.rst b/releases/2020.1.rst index e4928855e..bd2117282 100644 --- a/releases/2020.1.rst +++ b/releases/2020.1.rst @@ -24,7 +24,7 @@ Integrations * Load files directly from :ref:`HDFS`. SQream DB now comes with built-in, native HDFS support for directly loading data from Hadoop-based data lakes. Our focus on helping Hadoop customers do more with their data led us to develop this feature, which works out of the box. As a result, SQream DB can now not only read but also write data, and intermediate results back to HDFS for HIVE and other data consumers. SQream DB now fits seamlessly into a Hadoop data pipeline. -* Import :ref:`ORC files`, through :ref:`external_tables`. ORC files join Parquet as files that can be natively accessed and inserted into SQream DB tables. +* Import :ref:`ORC files`, through :ref:`foreign_tables`. ORC files join Parquet as files that can be natively accessed and inserted into SQream DB tables. * :ref:`Python driver (pysqream)` is now DB-API v2.0 compliant. Customers can write high-performance Python applications that make full use of SQream DB - connect, query, delete, and insert data. Data scientists can use pysqream with Pandas, Numpy, and AI/ML frameworks like TensorFlow for direct queries of huge datasets. diff --git a/releases/2020.3.rst b/releases/2020.3.rst index d072b15da..eb8ca8f62 100644 --- a/releases/2020.3.rst +++ b/releases/2020.3.rst @@ -26,9 +26,12 @@ The following list describes the new features: * ``TEXT`` is ramping up with new features (previously only available with VARCHARs): - * :ref:`substring`, :ref:`lower`, :ref:`ltrim`, :ref:`charindex`, :ref:`replace`, etc. + * `SUBSTRING `_ + * `LTRIM `_ + * `CHARINDEX `_ + * `REPLACE `_ - * Binary operators - :ref:`concat`, :ref:`like`, etc. + * Binary operators - `CONCAT `_ , `REPLACE `_ , etc. * Casts to and from ``TEXT`` diff --git a/releases/2021.1.2.rst b/releases/2021.1.2.rst index 43ce6db7d..6ea765973 100644 --- a/releases/2021.1.2.rst +++ b/releases/2021.1.2.rst @@ -40,17 +40,17 @@ String Literals Containing ASCII Characters Interepreted as TEXT ************ SQream now interprets all string literals, including those containing ASCII characters, as ``text``. -For more information, see `String Types `_. +For more information, see `String Types `_. Decimal Literals Interpreted as Numeric Columns ************ SQream now interprets literals containing decimal points as ``numeric`` instead of as ``double``. -For more information, see `Data Types `_. +For more information, see `Data Types `_. -Roles Area Added to Studio Version 5.3.3 +Roles Area Added to Studio Version 5.4.3 **************** -The **Roles** area has been added to `Studio version 5.3.3 `_. From the Roles area users can create and assign roles and manage user permissions. +The **Roles** area has been added to `Studio version 5.4.3 `_. From the Roles area users can create and assign roles and manage user permissions. Resolved Issues ------------- diff --git a/releases/2021.1.rst b/releases/2021.1.rst index b2b0dcfd8..246f53711 100644 --- a/releases/2021.1.rst +++ b/releases/2021.1.rst @@ -40,7 +40,7 @@ SQream now supports Numeric Data types for the following operations: * All aggregation types (not including Window functions). * Scalar functions (not including some trigonometric and logarithmic functions). -For more information, see `Numeric Data Types `_. +For more information, see `Numeric Data Types `_. Text Data Type ************ @@ -54,14 +54,14 @@ SQream now supports TEXT data types in all operations, which is default string d * Support text columns in queries with multiple distinct aggregates. * Text literal support for all functions. -For more information, see `String Types `_. +For more information, see `String Types `_. Supports Scalar Subqueries ************ SQream now supports running initial scalar subqueries. -For more information, see `Subqueries `_. +For more information, see `Subqueries `_. Literal Arguments ************ @@ -72,7 +72,7 @@ Simple Scalar SQL UDFs ************ SQream now supports simple scalar SQL UDF's. -For more information, see `Simple Scalar SQL UDF’s `_. +For more information, see `Simple Scalar SQL UDF’s `_. Logging Enhancements ************ @@ -91,7 +91,7 @@ Improved Presented License Information ************ SQream now displays information related to data size limitations, expiration date, type of license shown by the new UF. The **Utility Function (UF)** name is ``get_license_info()``. -For more information, see `GET_LICENSE_INFO `_. +For more information, see `GET_LICENSE_INFO `_. @@ -171,7 +171,7 @@ Operations and Configuration Changes Recommended SQream Configuration on Cloud ************ -For more information about AWS, see `Amazon S3 `_. +For more information about AWS, see `Amazon S3 `_. @@ -183,7 +183,6 @@ SQream now has a new ``runtimeGlobalFlags`` flag called ``WriteToFileThreads``. This flag configures the number of threads in the **WriteToFile** function. The default value is ``16``. -For more information about the ``runtimeGlobalFlags`` flag, see the **Runtime Global Flags** table in `Configuration `_. diff --git a/releases/2021.2.1.24.rst b/releases/2021.2.1.24.rst new file mode 100644 index 000000000..a89244934 --- /dev/null +++ b/releases/2021.2.1.24.rst @@ -0,0 +1,87 @@ +.. _2021.2.1.24: + +************************** +Release Notes 2021.2.1.24 +************************** +The 2021.2.1.24 release notes were released on 7/28/2022 and describe the following: + +.. contents:: + :local: + :depth: 1 + +Version Content +---------- +The 2021.2.1.24 Release Notes includes a query maintenance feature. + +New Features +---------- +The 2021.2.1.24 Release Notes include the following new features: + +.. contents:: + :local: + :depth: 1 + +Query Healer +************ +The new **Query Healer** feature periodically examines the progress of running statements, and is used for query maintenance. + +For more information, see `Query Healer `_. + +Resolved Issues +--------- +The following table lists the resolved issues for Version 2021.2.1.24: + ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| **SQ No.** | **Description** | ++=============+====================================================================================================================================+ +| SQ-10606 | Queries were getting stuck in the queue for a prolonged time. | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| SQ-10691 | The DB schema identifier was causing an error when running queries from joins suite. | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| SQ-10918 | The Workload Manager was only assigning jobs sequentially, delaying user SQLs assigned to workers running very large jobs. | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| SQ-10955 | Metadata filters were not being applied when users filtered by nullable dates using ``dateadd`` | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ + +Known Issues +--------- +The following table lists the known issues for Version 2021.2.1.24: + ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| **SQ No.** | **Description** | ++=============+====================================================================================================================================+ +| SQ-10071 | An error occurred on existing subqueries with ``TEXT`` and ``VARCHAR`` equality conditions. | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| SQ-10902 | Inserting a null value into non-null column was causing SQream to crash. | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ +| SQ-11088 | Specific workers caused low performance during compilation. | ++-------------+------------------------------------------------------------------------------------------------------------------------------------+ + +Operations and Configuration Changes +-------- +The following configuration flags were added: + + * :ref:`is_healer_on` + + :: + + * :ref:`healer_max_inactivity_hours` + +Naming Changes +------- +No relevant naming changes were made. + +Deprecated Features +------- +Version 2021.2.1.24 includes no deprecated features. + +End of Support +------- +The End of Support section is not relevant to Version 2021.2.1.24. + +.. toctree:: + :maxdepth: 2 + :glob: + :hidden: + + 2021.2.1.24 diff --git a/releases/2021.2.1.rst b/releases/2021.2.1.rst index f17bdd516..a84e5e0ad 100644 --- a/releases/2021.2.1.rst +++ b/releases/2021.2.1.rst @@ -21,30 +21,27 @@ CREATE TABLE ************ SQream now supports duplicating the column structure of an existing table using the ``LIKE`` clause. -For more information, see `Duplicating the Column Structure of an Existing Table `_. +For more information, see `Duplicating the Column Structure of an Existing Table `_. PERCENTILE FUNCTIONS ************ SQream now supports the following aggregation functions: -* :ref:`percentile_cont` -* :ref:`percentile_disc` -* :ref:`mode` +* `PERCENTILE_CONT `_ +* `PERCENTILE_DISC `_ +* `MODE `_ REGEX REPLACE ************ -SQream now supports the ``REGEXP_REPLACE`` function for finding and replacing text column substrings. - -For more information, see :ref:`regexp_replace`. +For more information, see `REGEXP_REPLACE `_ Delete Optimization ************ The ``DELETE`` statement can now delete values that contain multi-table conditions. -For more information, see `Deleting Values that Contain Multi-Table Conditions `_. - -For more information, see :ref:`regexp_replace`. +For more information, see `Deleting Values that Contain Multi-Table Conditions `_. +For more information, see `REGEXP_REPLACE `_ Performance Enhancements ------ @@ -61,8 +58,7 @@ The following table lists the issues that were resolved in Version 2021.2.1: * - SQ No. - Description * - SQ-8267 - - A method has been provided for including the ``GROUP BY`` and ``DISTINCT COUNT`` statements. - + - A method has been provided for including the ``GROUP BY`` and ``DISTINCT COUNT`` statements. Known Issues ------ diff --git a/releases/2021.2.rst b/releases/2021.2.rst index ec4773669..b134e9527 100644 --- a/releases/2021.2.rst +++ b/releases/2021.2.rst @@ -33,8 +33,8 @@ SQream now uses a new configuration system based on centralized configuration ac For more information, see the following: -* `Configuration `_ - describes how to configure your instance of SQream from a centralized location. -* `SQream Studio 5.4.2 `_ - configure your instance of SQream from Studio. +* `Configuration `_ - describes how to configure your instance of SQream from a centralized location. +* `SQream Studio 5.4.2 `_ - configure your instance of SQream from Studio. Qualifying Schemas Without Providing an Alias ************ @@ -131,8 +131,7 @@ NVARCHAR Data Type Renamed TEXT ************ The ``NVARCHAR`` data type has been renamed ``TEXT``. - -For more information on the ``TEXT`` data type, see `String (TEXT) `_ +For more information on the ``TEXT`` data type, see `String (TEXT) `_ End of Support ------ @@ -159,14 +158,14 @@ When upgrading from a SQream version earlier than 2021.2 you must upgrade your s $ cat /etc/sqream/sqream1_config.json |grep cluster $ ./upgrade_storage -For more information on upgrading your SQream version, see `Upgrading SQream Version `_. +For more information on upgrading your SQream version, see `Upgrading SQream Version `_. Upgrading Your Client Drivers ************ -For more information on the client drivers for version 2021.2, see `Client Drivers for 2021.2 `_. +For more information on the client drivers for version 2021.2, see `Client Drivers for 2021.2 `_. Configuring Your Instance of SQream ************ A new configuration method is used starting with Version 2021.2. -For more information about configuring your instance of SQream, see :ref:`configuration`. \ No newline at end of file +For more information about configuring your instance of SQream, see `Client Drivers for 2021.2 `_. \ No newline at end of file diff --git a/releases/2021.2_index.rst b/releases/2021.2_index.rst index 77a22b0ae..9bee5fd66 100644 --- a/releases/2021.2_index.rst +++ b/releases/2021.2_index.rst @@ -13,5 +13,6 @@ The 2021.2 Release Notes describe the following releases: :maxdepth: 1 :glob: + 2021.2.1.24 2021.2.1 2021.2 \ No newline at end of file diff --git a/releases/2022.1.rst b/releases/2022.1.rst deleted file mode 100644 index cdd6d98c5..000000000 --- a/releases/2022.1.rst +++ /dev/null @@ -1,137 +0,0 @@ -.. _2022.1: - -************************** -Release Notes 2022.1 -************************** -The 2022.1 release notes were released on 7/19/2022 and describe the following: - -.. contents:: - :local: - :depth: 1 - -Version Content ----------- -The 2022.1 Release Notes describes the following: - -* Enhanced security features. -* New data manipulation command. -* Additional data ingestion format. - -New Features ----------- -The 2022.1 Release Notes include the following new features: - -.. contents:: - :local: - :depth: 1 - -Data Encryption -************ -SQream now supports data encryption mechanisms in accordance with **General Data Protection Regulation (GDPR)** standards. - -Using the data encryption feature may lead to a maximum of a 10% increase in performance degradation. - -For more information, see `Data Encryption `_. - -Update Feature -************ -SQream now supports the DML **Update** feature, which is used for modifying the value of certain columns in existing rows. - -For more information, see `UPDATE `_. - -Avro Ingestion -************ -SQream now supports ingesting data from Avro files. - -For more information, see `Inserting Data from Avro `_. - -Known Issues ---------- -The following table lists the known issues for Version 2022.1: - -+-------------+-------------------------------------------------------------------------------------------+ -| **SQ No.** | **Description** | -+=============+===========================================================================================+ -| SQ-7732 | Reading numeric columns from an external Parquet file generated an error. | -+-------------+-------------------------------------------------------------------------------------------+ -| SQ-9889 | Running a query including Thai characters generated an internal runtime error. | -+-------------+-------------------------------------------------------------------------------------------+ -| SQ-10071 | Error on existing subqueries with TEXT and VARCHAR equality condition | -+-------------+-------------------------------------------------------------------------------------------+ -| SQ-10191 | The ``ALTER DEFAULT SCHEMA`` command was not functioning correctly. | -+-------------+-------------------------------------------------------------------------------------------+ -| SQ-10629 | Inserting data into a table significantly slowed down running queries. | -+-------------+-------------------------------------------------------------------------------------------+ -| SQ-10659 | Using a comment generated a compile error. | -+-------------+-------------------------------------------------------------------------------------------+ - -Resolved Issues ---------- -The following table lists the issues that were resolved in Version 2022.1: - -+-------------+-------------------------------------------------------------------------------------------+ -| **SQ No.** | **Description** | -+=============+===========================================================================================+ -| SQ-10111 | Reading numeric columns from an external Parquet file generated an error. | -+-------------+-------------------------------------------------------------------------------------------+ - -Operations and Configuration Changes --------- -No relevant operations and configuration changes were made. - -Naming Changes -------- -No relevant naming changes were made. - -Deprecated Features -------- -In SQream version 2022.1 the ``VARCHAR`` data type has been deprecated and replaced with ``TEXT``. SQream will maintain ``VARCHAR`` in all previous versions until completing the migration to ``TEXT``, at which point it will be deprecated in all earlier versions. SQream also provides an automated and secure tool to facilitate and simplify migration from ``VARCHAR`` to ``TEXT``. - -If you are using an earlier version of SQream, see the `Using Legacy String Literals `_ configuration flag. - -End of Support -------- -The End of Support section is not relevant to Version 2022.1. - -Upgrading to v2022.1 -------- -1. Generate a back-up of the metadata by running the following command: - - .. code-block:: console - - $ select backup_metadata('out_path'); - - .. tip:: SQream recommends storing the generated back-up locally in case needed. - - SQream runs the Garbage Collector and creates a clean backup tarball package. - -2. Shut down all SQream services. - - :: - -3. Extract the recently created back-up file. - - :: - -4. Replace your current metadata with the metadata you stored in the back-up file. - - :: - -5. Navigate to the new SQream package bin folder. - - :: - -6. Run the following command: - - .. code-block:: console - - $ ./upgrade_storage - - .. note:: Upgrading from a major version to another major version requires you to follow the **Upgrade Storage** step. This is described in Step 7 of the `Upgrading SQream Version `_ procedure. - -.. toctree:: - :maxdepth: 2 - :glob: - :hidden: - - 2022.1 diff --git a/releases/index.rst b/releases/index.rst index b1d065a9e..b3cfb3900 100644 --- a/releases/index.rst +++ b/releases/index.rst @@ -12,8 +12,6 @@ Release Notes * - Version - Release Date - * - :ref:`2022.1` - - July 19, 2022 * - :ref:`2021.2` - September 13, 2021 * - :ref:`2021.1` @@ -32,7 +30,6 @@ Release Notes :glob: :hidden: - 2022.1 2021.2_index 2021.1_index 2020.3_index diff --git a/requirements.txt b/requirements.txt index 806fe2730..575f023b1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,10 @@ # File: docs/requirements.txt # Defining the exact version will make sure things don't break -sphinx==3.5.3 -sphinx_rtd_theme==0.5.2 +sphinx==5.3.0 +sphinx_rtd_theme==1.2.0 +urllib3<=2.0.0 +openssl-python>=0.1.1 sphinx-notfound-page Pygments>=2.4.0 +pdftex diff --git a/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst b/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst index ec231c9a6..6db3c1e56 100644 --- a/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst +++ b/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst @@ -274,7 +274,7 @@ You can add and name new tabs for each statement that you need to execute, and S You can also rename the default tab name by double-clicking it and typing a new name and write multiple statements in tandem in the same tab by separating them with semicolons (``;``).If too many tabs to fit into the Statement Pane are open at the same time, the tab arrows are displayed. You can scroll through the tabs by clicking |icon-left| or |icon-right|, and close tabs by clicking |icon-close|. You can also close all tabs at once by clicking **Close all** located to the right of the tabs. -.. tip:: If this is your first time using SQream, see `Getting Started `_. + .. Keyboard shortcuts