diff --git a/_static/images/SAP_BO.png b/_static/images/SAP_BO.png new file mode 100644 index 000000000..413ce7d0e Binary files /dev/null and b/_static/images/SAP_BO.png differ diff --git a/_static/images/SAP_BO_2.png b/_static/images/SAP_BO_2.png new file mode 100644 index 000000000..91bc53d1a Binary files /dev/null and b/_static/images/SAP_BO_2.png differ diff --git a/_static/images/TPD_33.png b/_static/images/TPD_33.png new file mode 100644 index 000000000..f3235acc1 Binary files /dev/null and b/_static/images/TPD_33.png differ diff --git a/_static/images/chunking2.png b/_static/images/chunking2.png new file mode 100644 index 000000000..4922eb04a Binary files /dev/null and b/_static/images/chunking2.png differ diff --git a/_static/images/chunking_metadata2.png b/_static/images/chunking_metadata2.png new file mode 100644 index 000000000..4434cf655 Binary files /dev/null and b/_static/images/chunking_metadata2.png differ diff --git a/_static/images/color_table.png b/_static/images/color_table.png new file mode 100644 index 000000000..b815f9616 Binary files /dev/null and b/_static/images/color_table.png differ diff --git a/_static/images/dashboard.png b/_static/images/dashboard.png new file mode 100644 index 000000000..bd82bc7b4 Binary files /dev/null and b/_static/images/dashboard.png differ diff --git a/_static/images/execution_tree_1.png b/_static/images/execution_tree_1.png new file mode 100644 index 000000000..66ae30bbb Binary files /dev/null and b/_static/images/execution_tree_1.png differ diff --git a/_static/images/search_field.png b/_static/images/search_field.png new file mode 100644 index 000000000..9fe94bb4b Binary files /dev/null and b/_static/images/search_field.png differ diff --git a/_static/images/studio_icon_close.png b/_static/images/studio_icon_close.png index 72759ca70..c8bc59ba2 100644 Binary files a/_static/images/studio_icon_close.png and b/_static/images/studio_icon_close.png differ diff --git a/_static/images/studio_icon_copy.png b/_static/images/studio_icon_copy.png index 78576e0da..75fb1ace4 100644 Binary files a/_static/images/studio_icon_copy.png and b/_static/images/studio_icon_copy.png differ diff --git a/_static/images/studio_icon_ddl_edit.png b/_static/images/studio_icon_ddl_edit.png index a16dcb782..f6fdedd56 100644 Binary files a/_static/images/studio_icon_ddl_edit.png and b/_static/images/studio_icon_ddl_edit.png differ diff --git a/_static/images/studio_icon_dots.png b/_static/images/studio_icon_dots.png index 129691310..29776b3a2 100644 Binary files a/_static/images/studio_icon_dots.png and b/_static/images/studio_icon_dots.png differ diff --git a/_static/images/studio_icon_left.png b/_static/images/studio_icon_left.png index 2e779c72c..aef228943 100644 Binary files a/_static/images/studio_icon_left.png and b/_static/images/studio_icon_left.png differ diff --git a/_static/images/studio_icon_plain.png b/_static/images/studio_icon_plain.png new file mode 100644 index 000000000..7cc5c3b2c Binary files /dev/null and b/_static/images/studio_icon_plain.png differ diff --git a/_static/images/studio_icon_right.png b/_static/images/studio_icon_right.png index d4b84a247..e02c65a93 100644 Binary files a/_static/images/studio_icon_right.png and b/_static/images/studio_icon_right.png differ diff --git a/_static/images/studio_icon_search.png b/_static/images/studio_icon_search.png new file mode 100644 index 000000000..4e687f7e2 Binary files /dev/null and b/_static/images/studio_icon_search.png differ diff --git a/_static/images/third_party_connectors/pentaho/connection_tested_successfully_2.png b/_static/images/third_party_connectors/pentaho/connection_tested_successfully_2.png new file mode 100644 index 000000000..ea474c7cb Binary files /dev/null and b/_static/images/third_party_connectors/pentaho/connection_tested_successfully_2.png differ diff --git a/_static/images/third_party_connectors/pentaho/csv_file_input_options.png b/_static/images/third_party_connectors/pentaho/csv_file_input_options.png new file mode 100644 index 000000000..fbf64e0e5 Binary files /dev/null and b/_static/images/third_party_connectors/pentaho/csv_file_input_options.png differ diff --git a/_static/images/third_party_connectors/pentaho/csv_file_input_options_2.png b/_static/images/third_party_connectors/pentaho/csv_file_input_options_2.png new file mode 100644 index 000000000..db97cdda4 Binary files /dev/null and b/_static/images/third_party_connectors/pentaho/csv_file_input_options_2.png differ diff --git a/_static/images/third_party_connectors/pentaho/main_output_of_step.png b/_static/images/third_party_connectors/pentaho/main_output_of_step.png new file mode 100644 index 000000000..99190ef6c Binary files /dev/null and b/_static/images/third_party_connectors/pentaho/main_output_of_step.png differ diff --git a/guides/architecture/filesystem_and_filesystem_usage.rst b/architecture/filesystem_and_filesystem_usage.rst similarity index 100% rename from guides/architecture/filesystem_and_filesystem_usage.rst rename to architecture/filesystem_and_filesystem_usage.rst diff --git a/guides/architecture/index.rst b/architecture/index.rst similarity index 96% rename from guides/architecture/index.rst rename to architecture/index.rst index 7705abe91..5e4d6c867 100644 --- a/guides/architecture/index.rst +++ b/architecture/index.rst @@ -1,7 +1,7 @@ .. _architecture: *********************** -Architecture +System Architecture *********************** This topic includes guides that walk an end-user, database administrator, or system architect through the main ideas behind SQream DB. diff --git a/guides/architecture/internals_architecture.rst b/architecture/internals_architecture.rst similarity index 99% rename from guides/architecture/internals_architecture.rst rename to architecture/internals_architecture.rst index f25dfeb22..97019ec71 100644 --- a/guides/architecture/internals_architecture.rst +++ b/architecture/internals_architecture.rst @@ -1,7 +1,7 @@ .. _internals_architecture: *************************** -Internals and architecture +Internals and Architecture *************************** SQream DB internals diff --git a/guides/architecture/processes_and_network_architecture.rst b/architecture/processes_and_network_architecture.rst similarity index 96% rename from guides/architecture/processes_and_network_architecture.rst rename to architecture/processes_and_network_architecture.rst index 8a458f8d5..53a6afb6a 100644 --- a/guides/architecture/processes_and_network_architecture.rst +++ b/architecture/processes_and_network_architecture.rst @@ -1,7 +1,7 @@ .. _processes_and_network_architecture: ************************************* -Processes and network architecture +Processes and Network Architecture ************************************* A SQream DB installation contains several components: diff --git a/conf.py b/conf.py index 8c9b81b5a..551ecebdd 100644 --- a/conf.py +++ b/conf.py @@ -21,11 +21,14 @@ # -- Project information ----------------------------------------------------- project = 'SQream DB' -copyright = '2021 SQream' -author = 'Sean Tomarian' +copyright = '2022 SQream' +author = 'SQream Documentation' + # The full version, including alpha/beta/rc tags -release = '2020.3.1' + +release = '2020.3.2.1' + # -- General configuration --------------------------------------------------- @@ -108,4 +111,4 @@ base_version = release.split('-')[0] rst_epilog = """ .. |latest_version| replace:: v{} -""".format(base_version) \ No newline at end of file +""".format(base_version) diff --git a/configuration_guides/admin_cluster_flags.rst b/configuration_guides/admin_cluster_flags.rst new file mode 100644 index 000000000..6a1b63406 --- /dev/null +++ b/configuration_guides/admin_cluster_flags.rst @@ -0,0 +1,9 @@ +.. _admin_cluster_flags: + +************************* +Cluster Administration Flags +************************* + +The **Cluster Administration Flags** page describes **Cluster** modification type flags, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: + +* `Setting Maximum CSV Row Length `_ \ No newline at end of file diff --git a/configuration_guides/admin_flags.rst b/configuration_guides/admin_flags.rst new file mode 100644 index 000000000..e71b9f761 --- /dev/null +++ b/configuration_guides/admin_flags.rst @@ -0,0 +1,18 @@ +.. _admin_flags: + +************************* +Administration Flags +************************* + +The **Administration Flags** page describes the following flag types, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: + + +.. toctree:: + :maxdepth: 1 + :glob: + + admin_regular_flags + admin_cluster_flags + admin_worker_flags + + diff --git a/configuration_guides/admin_regular_flags.rst b/configuration_guides/admin_regular_flags.rst new file mode 100644 index 000000000..1391e6370 --- /dev/null +++ b/configuration_guides/admin_regular_flags.rst @@ -0,0 +1,32 @@ +.. _admin_regular_flags: + +************************* +Regular Administration Flags +************************* +The **Regular Administration Flags** page describes **Regular** modification type flags, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: + +* `Setting Bin Size `_ +* `Setting CUDA Memory `_ +* `Limiting Runtime to Utility Functions `_ +* `Enabling High Bin Control Granularity `_ +* `Reducing CPU Hashtable Sizes `_ +* `Setting Chunk Size for Copying from CPU to GPU `_ +* `Indicating GPU Synchronicity `_ +* `Enabling Modification of R&D Flags `_ +* `Checking for Post-Production CUDA Errors `_ +* `Enabling Modification of clientLogger_debug File `_ +* `Activating the NVidia Profiler Markers `_ +* `Appending String at End of Log Lines `_ +* `Monitoring and Printing Pinned Allocation Reports `_ +* `Increasing Chunk Size to Reduce Query Speed `_ +* `Adding Rechunker before Expensing Chunk Producer `_ +* `Setting the Buffer Size `_ +* `Maximum Pinned Percentage of Total RAM `_ +* `Setting Memory Used to Abort Server `_ +* `Splitting Large Reads for Concurrent Execution `_ +* `Setting Worker Amount to Handle Concurrent Reads `_ +* `Setting Implicit Casts in ORC Files `_ +* `Setting Timeout Limit for Locking Objects before Executing Statements `_ +* `Interpreting Decimal Literals as Double Instead of Numeric `_ +* `Interpreting VARCHAR as TEXT `_ +* `VARCHAR Identifiers `_ diff --git a/configuration_guides/admin_worker_flags.rst b/configuration_guides/admin_worker_flags.rst new file mode 100644 index 000000000..412955510 --- /dev/null +++ b/configuration_guides/admin_worker_flags.rst @@ -0,0 +1,11 @@ +.. _admin_worker_flags: + +************************* +Worker Administration Flags +************************* +The **Worker Administration Flags** page describes **Worker** modification type flags, which can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: + +* `Setting Total Device Memory Usage in SQream Instance `_ +* `Enabling Manually Setting Reported IP `_ +* `Setting Port Used for Metadata Server Connection `_ +* `Assigning Local Network IP `_ \ No newline at end of file diff --git a/configuration_guides/bin_sizes.rst b/configuration_guides/bin_sizes.rst new file mode 100644 index 000000000..9cdd8e0e8 --- /dev/null +++ b/configuration_guides/bin_sizes.rst @@ -0,0 +1,13 @@ +.. _bin_sizes: + +************************* +Setting Bin Size +************************* +The ``binSizes`` flag sets the custom bin size in the cache to enable high granularity over bin control. + +The following describes the ``binSizes`` flag: + +* **Data type** - string + +The **Default value** for the ``binSizes`` flag is ``16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,16777216,33554432,67108864,134217728,268435456,536870912,786432000,107374,1824,`` +``1342177280,1610612736,1879048192,2147483648,2415919104,2684354560,2952790016,3221225472`` \ No newline at end of file diff --git a/configuration_guides/cache_disk_dir.rst b/configuration_guides/cache_disk_dir.rst new file mode 100644 index 000000000..012955bc3 --- /dev/null +++ b/configuration_guides/cache_disk_dir.rst @@ -0,0 +1,12 @@ +.. _cache_disk_dir: + +************************* +Setting Spool Saved File Directory Location +************************* +The ``cacheDiskDir`` flag sets the ondisk directory location for the spool to save files on. + +The following describes the ``cacheDiskDir`` flag: + +* **Data type** - size_t +* **Default value** - ``128`` +* **Allowed values** - Any legal string \ No newline at end of file diff --git a/configuration_guides/cache_disk_gb.rst b/configuration_guides/cache_disk_gb.rst new file mode 100644 index 000000000..eb3d530cd --- /dev/null +++ b/configuration_guides/cache_disk_gb.rst @@ -0,0 +1,12 @@ +.. _cache_disk_gb: + +************************* +Setting Disk Spool Memory +************************* +The ``cacheDiskGB`` flag sets the amount of memory (GB) to be used by Spool on the disk. + +The following describes the ``cacheDiskGB`` flag: + +* **Data type** - size_t +* **Default value** - ``128`` +* **Allowed values** - 0-4000000000 \ No newline at end of file diff --git a/configuration_guides/cache_eviction_milliseconds.rst b/configuration_guides/cache_eviction_milliseconds.rst new file mode 100644 index 000000000..129d6281b --- /dev/null +++ b/configuration_guides/cache_eviction_milliseconds.rst @@ -0,0 +1,12 @@ +.. _cache_eviction_milliseconds: + +************************* +Setting Cache Flushing +************************* +The ``cacheEvictionMilliseconds`` flag sets how long the cache stores contents before being flushed. + +The following describes the ``cacheEvictionMilliseconds`` flag: + +* **Data type** - size_t +* **Default value** - ``2000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/cache_partitions.rst b/configuration_guides/cache_partitions.rst new file mode 100644 index 000000000..0637e347c --- /dev/null +++ b/configuration_guides/cache_partitions.rst @@ -0,0 +1,12 @@ +.. _cache_partitions: + +************************* +Setting Cache Partitions +************************* +The ``cachePartitions`` flag sets the number of partitions that the cache is split into. + +The following describes the ``cachePartitions`` flag: + +* **Data type** - size_t +* **Default value** - ``4`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/cache_persistent_dir.rst b/configuration_guides/cache_persistent_dir.rst new file mode 100644 index 000000000..c3e298189 --- /dev/null +++ b/configuration_guides/cache_persistent_dir.rst @@ -0,0 +1,12 @@ +.. _cache_persistent_dir: + +************************* +Setting Spool Persistent Saved File Directory Location +************************* +The ``cachePersistentDir`` flag sets the persistent directory location for the spool to save files on. + +The following describes the ``cachePersistentDir`` flag: + +* **Data type** - string +* **Default value** - ``/tmp`` +* **Allowed values** - Any legal string \ No newline at end of file diff --git a/configuration_guides/cache_persistent_gb.rst b/configuration_guides/cache_persistent_gb.rst new file mode 100644 index 000000000..418364e5c --- /dev/null +++ b/configuration_guides/cache_persistent_gb.rst @@ -0,0 +1,12 @@ +.. _cache_persistent_gb: + +************************* +Setting Data Stored Persistently on Cache +************************* +The ``cachePersistentGB`` flag sets the amount of data (GB) for the cache to store persistently . + +The following describes the ``cachePersistentGB`` flag: + +* **Data type** - size_t +* **Default value** - ``128`` +* **Allowed values** - 0-4000000000 \ No newline at end of file diff --git a/configuration_guides/cache_ram_gb.rst b/configuration_guides/cache_ram_gb.rst new file mode 100644 index 000000000..31d56613b --- /dev/null +++ b/configuration_guides/cache_ram_gb.rst @@ -0,0 +1,12 @@ +.. _cache_ram_gb: + +************************* +Setting InMemory Spool Memory +************************* +The ``cacheRamGB`` flag sets the amount of memory (GB) to be used by Spool InMemory. + +The following describes the ``cacheRamGB`` flag: + +* **Data type** - size_t +* **Default value** - ``16`` +* **Allowed values** - 0-4000000000 \ No newline at end of file diff --git a/configuration_guides/check_cuda_memory.rst b/configuration_guides/check_cuda_memory.rst new file mode 100644 index 000000000..9ac78f767 --- /dev/null +++ b/configuration_guides/check_cuda_memory.rst @@ -0,0 +1,12 @@ +.. _check_cuda_memory: + +************************* +Setting CUDA Memory +************************* +The ``checkCudaMemory`` flag sets the pad device memory allocation with safety buffers to catch out-of-bounds writes. + +The following describes the ``checkCudaMemory`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/compiler_gets_only_ufs.rst b/configuration_guides/compiler_gets_only_ufs.rst new file mode 100644 index 000000000..00e8b44d8 --- /dev/null +++ b/configuration_guides/compiler_gets_only_ufs.rst @@ -0,0 +1,12 @@ +.. _compiler_gets_only_ufs: + +************************* +Limiting Runtime to Utility Functions +************************* +The ``compilerGetsOnlyUFs`` flag sets the runtime to pass only utility functions names to the compiler. + +The following describes the ``compilerGetsOnlyUFs`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/configuration_flags.rst b/configuration_guides/configuration_flags.rst new file mode 100644 index 000000000..3a25dc9bd --- /dev/null +++ b/configuration_guides/configuration_flags.rst @@ -0,0 +1,20 @@ +.. _configuration_flags: + +************************* +Configuration Flags +************************* +SQream provides two methods for configuration your instance of SQream. The current configuration method is based on cluster and session-based configuration, described in more detail below. Users can also use the previous configuration method done using a configuration file. + +The **Configuration Methods** page describes the following configurations methods: + +.. toctree:: + :maxdepth: 1 + :glob: + :titlesonly: + + admin_flags + generic_flags + + + + diff --git a/configuration_guides/configuring_sqream.rst b/configuration_guides/configuring_sqream.rst new file mode 100644 index 000000000..901af9e3d --- /dev/null +++ b/configuration_guides/configuring_sqream.rst @@ -0,0 +1,22 @@ +.. _configuring_sqream: + +************************* +Configuring SQream +************************* +The **Configuring SQream** page describes the following configuration topics: + +.. toctree:: + :maxdepth: 1 + :glob: + :titlesonly: + + current_method_configuration_levels + current_method_flag_types + current_method_configuration_roles + current_method_modification_methods + current_method_configuring_your_parameter_values + current_method_command_examples + current_method_showing_all_flags_in_the_catalog_table + current_method_all_configurations + + diff --git a/configuration_guides/copy_to_restrict_utf8.rst b/configuration_guides/copy_to_restrict_utf8.rst new file mode 100644 index 000000000..5d5990243 --- /dev/null +++ b/configuration_guides/copy_to_restrict_utf8.rst @@ -0,0 +1,12 @@ +.. _copy_to_restrict_utf8: + +************************* +Enabling High Bin Control Granularity +************************* +The ``copyToRestrictUtf8`` flag sets sets the custom bin size in the cache to enable high bin control granularity. + +The following describes the ``copyToRestrictUtf8`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/cpu_reduce_hashtable_size.rst b/configuration_guides/cpu_reduce_hashtable_size.rst new file mode 100644 index 000000000..d32450cd3 --- /dev/null +++ b/configuration_guides/cpu_reduce_hashtable_size.rst @@ -0,0 +1,12 @@ +.. _cpu_reduce_hashtable_size: + +************************* +Setting Hashtable Size for GROUP BY Queries +************************* +The ``cpuReduceHashtableSize`` flag sets the size of the hashtable in your CPU while running a ``GROUP BY`` query. + +The following describes the ``cpuReduceHashtableSize`` flag: + +* **Data type** - uint +* **Default value** - ``10000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/csv_limit_row_length.rst b/configuration_guides/csv_limit_row_length.rst new file mode 100644 index 000000000..41bc124df --- /dev/null +++ b/configuration_guides/csv_limit_row_length.rst @@ -0,0 +1,12 @@ +.. _csv_limit_row_length: + +************************* +Setting Maximum CSV Row Length +************************* +The ``csvLimitRowLength`` flag sets the maximum supported CSV row length. + +The following describes the ``csvLimitRowLength`` flag: + +* **Data type** - uint +* **Default value** - ``100000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/cuda_mem_cpy_max_size_bytes.rst b/configuration_guides/cuda_mem_cpy_max_size_bytes.rst new file mode 100644 index 000000000..c28827077 --- /dev/null +++ b/configuration_guides/cuda_mem_cpy_max_size_bytes.rst @@ -0,0 +1,12 @@ +.. _cuda_mem_cpy_max_size_bytes: + +************************* +Setting Chunk Size for Copying from CPU to GPU +************************* +The ``cudaMemcpyMaxSizeBytes`` flag sets the chunk size for copying from CPU to GPU. If this value is set to ``0``, do not divide. + +The following describes the ``cudaMemcpyMaxSizeBytes`` flag: + +* **Data type** - uint +* **Default value** - ``0`` +* **Allowed values** - 0-4000000000 \ No newline at end of file diff --git a/configuration_guides/cuda_mem_cpy_synchronous.rst b/configuration_guides/cuda_mem_cpy_synchronous.rst new file mode 100644 index 000000000..2f8662947 --- /dev/null +++ b/configuration_guides/cuda_mem_cpy_synchronous.rst @@ -0,0 +1,12 @@ +.. _cuda_mem_cpy_synchronous: + +************************* +Indicating GPU Synchronicity +************************* +The ``CudaMemcpySynchronous`` flag indicates if copying from/to GPU is synchronous. + +The following describes the ``CudaMemcpySynchronous`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/cuda_mem_quota.rst b/configuration_guides/cuda_mem_quota.rst new file mode 100644 index 000000000..d5f417978 --- /dev/null +++ b/configuration_guides/cuda_mem_quota.rst @@ -0,0 +1,12 @@ +.. _cuda_mem_quota: + +************************* +Setting Total Device Memory Usage in SQream Instance +************************* +The ``cudaMemQuota`` flag sets the percentage of total device memory used by your instance of SQream. + +The following describes the ``cudaMemQuota`` flag: + +* **Data type** - uint +* **Default value** - ``90`` +* **Allowed values** - 0-100 diff --git a/configuration_guides/current_method_all_configurations.rst b/configuration_guides/current_method_all_configurations.rst new file mode 100644 index 000000000..00ad944c7 --- /dev/null +++ b/configuration_guides/current_method_all_configurations.rst @@ -0,0 +1,354 @@ +.. _current_method_all_configurations: + +************************** +All Configurations +************************** +The following table describes all **Generic** and **Administration** configuration flags: + +.. list-table:: + :header-rows: 1 + :widths: 1 2 1 15 1 20 + :class: my-class + :name: my-name + + * - Flag Name + - Access Control + - Modification Type + - Description + - Data Type + - Default Value + + * - ``binSizes`` + - Admin + - Regular + - Sets the custom bin size in the cache to enable high granularity bin control. + - string + - + ``16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,`` + ``131072,262144,524288,1048576,2097152,4194304,8388608,16777216,`` + ``33554432,67108864,134217728,268435456,536870912,786432000,107374,`` + ``1824,1342177280,1610612736,1879048192,2147483648,2415919104,`` + ``2684354560,2952790016,3221225472`` + + * - ``checkCudaMemory`` + - Admin + - Regular + - Sets the pad device memory allocations with safety buffers to catch out-of-bounds writes. + - boolean + - ``FALSE`` + + * - ``compilerGetsOnlyUFs`` + - Admin + - Regular + - Sets the runtime to pass only utility functions names to the compiler. + - boolean + - ``FALSE`` + + * - ``copyToRestrictUtf8`` + - Admin + - Regular + - Sets the custom bin size in the cache to enable high granularity bin control. + - boolean + - ``FALSE`` + + * - ``cpuReduceHashtableSize`` + - Admin + - Regular + - Sets the hash table size of the CpuReduce. + - uint + - ``10000`` + + * - ``csvLimitRowLength`` + - Admin + - Cluster + - Sets the maximum supported CSV row length. + - uint + - ``100000`` + + * - ``cudaMemcpyMaxSizeBytes`` + - Admin + - Regular + - Sets the chunk size for copying from CPU to GPU. If set to 0, do not divide. + - uint + - ``0`` + + * - ``CudaMemcpySynchronous`` + - Admin + - Regular + - Indicates if copying from/to GPU is synchronous. + - boolean + - ``FALSE`` + + * - ``cudaMemQuota`` + - Admin + - Worker + - Sets the percentage of total device memory to be used by the instance. + - uint + - ``90`` + + * - ``developerMode`` + - Admin + - Regular + - Enables modifying R&D flags. + - boolean + - ``FALSE`` + + * - ``enableDeviceDebugMessages`` + - Admin + - Regular + - Activates the Nvidia profiler (nvprof) markers. + - boolean + - ``FALSE`` + + * - ``enableLogDebug`` + - Admin + - Regular + - Enables creating and logging in the clientLogger_debug file. + - boolean + - ``TRUE`` + + * - ``enableNvprofMarkers`` + - Admin + - Regular + - Activates the Nvidia profiler (nvprof) markers. + - boolean + - ``FALSE`` + + * - ``endLogMessage`` + - Admin + - Regular + - Appends a string at the end of every log line. + - string + - ``EOM`` + + + + * - ``varcharIdentifiers`` + - Admin + - Regular + - Activates using varchar as an identifier. + - boolean + - ``true`` + + + + * - ``extentStorageFileSizeMB`` + - Admin + - Cluster + - Sets the minimum size in mebibytes of extents for table bulk data. + - uint + - ``20`` + + * - ``gatherMemStat`` + - Admin + - Regular + - Monitors all pinned allocations and all **memcopies** to/from device, and prints a report of pinned allocations that were not memcopied to/from the device using the **dump_pinned_misses** utility function. + - boolean + - ``FALSE`` + + * - ``increaseChunkSizeBeforeReduce`` + - Admin + - Regular + - Increases the chunk size to reduce query speed. + - boolean + - ``FALSE`` + + * - ``increaseMemFactors`` + - Admin + - Regular + - Adds rechunker before expensive chunk producer. + - boolean + - ``TRUE`` + + * - ``leveldbWriteBufferSize`` + - Admin + - Regular + - Sets the buffer size. + - uint + - ``524288`` + + * - ``machineIP`` + - Admin + - Worker + - Manual setting of reported IP. + - string + - ``127.0.0.1`` + + + + + * - ``memoryResetTriggerMB`` + - Admin + - Regular + - Sets the size of memory used during a query to trigger aborting the server. + - uint + - ``0`` + + * - ``metadataServerPort`` + - Admin + - Worker + - Sets the port used to connect to the metadata server. SQream recommends using port ranges above 1024† because ports below 1024 are usually reserved, although there are no strict limitations. Any positive number (1 - 65535) can be used. + - uint + - ``3105`` + + * - ``mtRead`` + - Admin + - Regular + - Splits large reads to multiple smaller ones and executes them concurrently. + - boolean + - ``FALSE`` + + * - ``mtReadWorkers`` + - Admin + - Regular + - Sets the number of workers to handle smaller concurrent reads. + - uint + - ``30`` + + * - ``orcImplicitCasts`` + - Admin + - Regular + - Sets the implicit cast in orc files, such as **int** to **tinyint** and vice versa. + - boolean + - ``TRUE`` + + * - ``statementLockTimeout`` + - Admin + - Regular + - Sets the timeout (seconds) for acquiring object locks before executing statements. + - uint + - ``3`` + + * - ``useConfigIP`` + - Admin + - Worker + - Activates the machineIP (true). Setting to false ignores the machineIP and automatically assigns a local network IP. This cannot be activated in a cloud scenario (on-premises only). + - boolean + - ``FALSE`` + + * - ``useLegacyDecimalLiterals`` + - Admin + - Regular + - Interprets decimal literals as **Double** instead of **Numeric**. Used to preserve legacy behavior in existing customers. + - boolean + - ``FALSE`` + + * - ``useLegacyStringLiterals`` + - Admin + - Regular + - Interprets ASCII-only strings as **VARCHAR** instead of **TEXT**. Used to preserve legacy behavior in existing customers. + - boolean + - ``FALSE`` + + * - ``flipJoinOrder`` + - Generic + - Regular + - Reorders join to force equijoins and/or equijoins sorted by table size. + - boolean + - ``FALSE`` + + * - ``limitQueryMemoryGB`` + - Generic + - Worker + - Prevents a query from processing more memory than the flag’s value. + - uint + - ``100000`` + + * - ``cacheEvictionMilliseconds`` + - Generic + - Regular + - Sets how long the cache stores contents before being flushed. + - size_t + - ``2000`` + + + * - ``cacheDiskDir`` + - Generic + - Regular + - Sets the ondisk directory location for the spool to save files on. + - size_t + - Any legal string + + + * - ``cacheDiskGB`` + - Generic + - Regular + - Sets the amount of memory (GB) to be used by Spool on the disk. + - size_t + - ``128`` + + * - ``cachePartitions`` + - Generic + - Regular + - Sets the number of partitions that the cache is split into. + - size_t + - ``4`` + + + * - ``cachePersistentDir`` + - Generic + - Regular + - Sets the persistent directory location for the spool to save files on. + - string + - Any legal string + + + * - ``cachePersistentGB`` + - Generic + - Regular + - Sets the amount of data (GB) for the cache to store persistently. + - size_t + - ``128`` + + + * - ``cacheRamGB`` + - Generic + - Regular + - Sets the amount of memory (GB) to be used by Spool InMemory. + - size_t + - ``16`` + + + + + + + + * - ``logSysLevel`` + - Generic + - Regular + - + Determines the client log level: + 0 - L_SYSTEM, + 1 - L_FATAL, + 2 - L_ERROR, + 3 - L_WARN, + 4 - L_INFO, + 5 - L_DEBUG, + 6 - L_TRACE + - uint + - ``100000`` + + * - ``maxAvgBlobSizeToCompressOnGpu`` + - Generic + - Regular + - Sets the CPU to compress columns with size above (flag’s value) * (row count). + - uint + - ``120`` + + + * - ``sessionTag`` + - Generic + - Regular + - Sets the name of the session tag. + - string + - Any legal string + + + + * - ``spoolMemoryGB`` + - Generic + - Regular + - Sets the amount of memory (GB) to be used by the server for spooling. + - uint + - ``8`` \ No newline at end of file diff --git a/configuration_guides/current_method_command_examples.rst b/configuration_guides/current_method_command_examples.rst new file mode 100644 index 000000000..2f75a4711 --- /dev/null +++ b/configuration_guides/current_method_command_examples.rst @@ -0,0 +1,36 @@ +.. _current_method_command_examples: + +************************** +Command Examples +************************** +This section includes the following command examples: + +.. contents:: + :local: + :depth: 1 + +Running a Regular Flag Type Command +--------------------- +The following is an example of running a **Regular** flag type command: + +.. code-block:: console + + SET spoolMemoryGB= 11; + executed + +Running a Worker Flag Type Command +--------------------- +The following is an example of running a **Worker** flag type command: + +.. code-block:: console + + SHOW spoolMemoryGB; + +Running a Cluster Flag Type Command +--------------------- +The following is an example of running a **Cluster** flag type command: + +.. code-block:: console + + ALTER SYSTEM RESET useMetadataServer; + executed \ No newline at end of file diff --git a/configuration_guides/current_method_configuration_levels.rst b/configuration_guides/current_method_configuration_levels.rst new file mode 100644 index 000000000..47ccf32b3 --- /dev/null +++ b/configuration_guides/current_method_configuration_levels.rst @@ -0,0 +1,33 @@ +.. _current_method_configuration_levels: + +************************** +Configuration Levels +************************** +SQream's configuration parameters are based on the following hierarchy: + +.. contents:: + :local: + :depth: 1 + +Cluster-Based Configuration +-------------- +Cluster-based configuration lets you centralize configurations for all workers on the cluster. Only Regular and Cluster flag types can be modified on the cluster level. These modifications are persistent and stored at the metadata level, which are applied globally to all workers in the cluster. + +.. note:: While cluster-based configuration was designed for configuring Workers, you can only configure Worker values set to the Regular or Cluster type. + +Worker-Based Configuration +-------------- +Worker-based configuration lets you modify the configuration belong to individual workers from the worker configuration file. + +For more information on making configurations from the worker configuration file, see `Modifying Your Configuration Using a Legacy Configuration File `_. + +Session-Based Configuration +-------------- +Session-based configurations are not persistent and are deleted when your session ends. This method enables you to modify all required configurations while avoiding conflicts between flag attributes modified on different devices at different points in time. The **SET flag_name** command is used to modify flag values on the session level. Any modifications you make with the **SET flag_name** command apply only to your open session, and are not saved when it ends. + +For example, when the query below has completed executing, the values configured will be restored to its previous setting: + +.. code-block:: console + + set spoolMemoryGB=700; + select * from table a where date='2021-11-11' \ No newline at end of file diff --git a/configuration_guides/current_method_configuration_roles.rst b/configuration_guides/current_method_configuration_roles.rst new file mode 100644 index 000000000..11b7e4bfb --- /dev/null +++ b/configuration_guides/current_method_configuration_roles.rst @@ -0,0 +1,17 @@ +.. _current_method_configuration_roles: + +************************** +Configuration Roles +************************** +SQream divides flags into the following roles, each with their own set of permissions: + +* :ref:`admin_flags` - can be modified by administrators on a session and cluster basis using the ``ALTER SYSTEM SET`` command: **Comment** - *I don't think we need to mention the command here, as it's described below, and also not mentioned for Generic Flags.* + + * Regular + * Worker + * Cluster + +* :ref:`generic_flags` - can be modified by standard users on a session basis: + + * Regular + * Worker \ No newline at end of file diff --git a/configuration_guides/current_method_configuring_your_parameter_values.rst b/configuration_guides/current_method_configuring_your_parameter_values.rst new file mode 100644 index 000000000..d082db693 --- /dev/null +++ b/configuration_guides/current_method_configuring_your_parameter_values.rst @@ -0,0 +1,40 @@ +.. _current_method_configuring_your_parameter_values: + +************************** +Configuring Your Parameter Values +************************** +The method you must use to configure your parameter values depends on the configuration level. Each configuration level has its own command or set of commands used to configure values, as shown below: + ++-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| **Configuration Level** | ++=================================================================================================================================================================================================================================================================================================================+ +| **Regular, Worker, and Cluster** | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| **Command** | **Description** | **Example** | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``SET `` | Used for modifying flag attributes. | ``SET developerMode=true`` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``SHOW / ALL`` | Used to preset either a specific flag value or all flag values. | ``SHOW `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``SHOW ALL LIKE`` | Used as a wildcard character for flag names. | ``SHOW `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``show_conf_UF`` | Used to print all flags with the following attributes: | ``rechunkThreshold,90,true,RND,regular`` | +| | | | +| | * Flag name | | +| | * Default value | | +| | * Is Developer Mode (Boolean) | | +| | * Flag category | | +| | * Flag type | | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``show_conf_extended UF`` | Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. | ``rechunkThreshold,90,true,RND,regular`` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``show_md_flag UF`` | Used to show a specific flag/all flags stored in the metadata file. |* Example 1: ``* master=> ALTER SYSTEM SET heartbeatTimeout=111;`` | +| | |* Example 2: ``* master=> select show_md_flag(‘all’); heartbeatTimeout,111`` | +| | |* Example 3: ``* master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111`` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| **Worker and Cluster** | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``ALTER SYSTEM SET `` | Used for storing or modifying flag attributes in the metadata file. | ``ALTER SYSTEM SET `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ +| ``ALTER SYSTEM RESET `` | Used to remove a flag or all flag attributes from the metadata file. | ``ALTER SYSTEM RESET `` | ++-----------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------+ \ No newline at end of file diff --git a/configuration_guides/current_method_flag_types.rst b/configuration_guides/current_method_flag_types.rst new file mode 100644 index 000000000..b1bedecba --- /dev/null +++ b/configuration_guides/current_method_flag_types.rst @@ -0,0 +1,20 @@ +.. _current_method_flag_types: + +************************** +Flag Types +************************** +SQream uses three flag types, **Cluster**, **Worker**, and **Regular**. Each of these flag types is associated with one of three hierarchical configuration levels described earlier, making it easier to configure your system. + +The highest level in the hierarchy is Cluster, which lets you set configurations across all workers in a given cluster. Modifying cluster values is **persistent**, meaning that any configurations you set are retained after shutting down your system. Configurations set at the Cluster level take the highest priority and override settings made on the Regular and Worker level **Comment** - *Confirm*. This is known as **cluster-based configuration**. Note that Cluster-based configuration lets you modify Cluster *and* Regular flag types. An example of a Cluster flag is **persisting your cache directory.** + +The second level is Worker, which lets you configure individual workers. Modifying Worker values are also **persistent**. This is known as **worker-based configuration**. Some examples of Worker flags includes **setting total device memory usage** and **setting metadata server connection port**. + +The lowest level is Regular, which means that modifying values of Regular flags affects only your current session and are not persistent. This means that they are automatically restored to their default value when the session ends. This is known as **session-based configuration**. Some examples of Regular flags includes **setting your bin size** and **setting CUDA memory**. + +To see each flag's default value, see one of the following: + +* The **Default Value** column in the :ref:`All Configurations` section. + + :: + +* The flag's individual description page, such as :ref:`Setting CUDA Memory`. \ No newline at end of file diff --git a/configuration_guides/current_method_modification_methods.rst b/configuration_guides/current_method_modification_methods.rst new file mode 100644 index 000000000..05825b07a --- /dev/null +++ b/configuration_guides/current_method_modification_methods.rst @@ -0,0 +1,50 @@ +.. _current_method_modification_methods: + +************************** +Modification Methods +************************** +SQream provides two different ways to modify your configurations. The current method is based on hierarchical configuration as described above. This method is based on making modifications on the **worker configuration file**, while you can still make modifications using the previous method using the **legacy configuration file**, both described below: + +.. contents:: + :local: + :depth: 1 + +Modifying Your Configuration Using the Worker Configuration File +------------------- +You can modify your configuration using the **worker configuration file (config.json)**. Changes that you make to worker configuration files are persistent. Note that you can only set the attributes in your worker configuration file **before** initializing your SQream worker, and while your worker is active these attributes are read-only. + +The following is an example of a worker configuration file: + +.. code-block:: postgres + + { + “cluster”: “/home/test_user/sqream_testing_temp/sqreamdb”, + “gpu”: 0, + “licensePath”: “home/test_user/SQream/tests/license.enc”, + “machineIP”: “127.0.0.1”, + “metadataServerIp”: “127.0.0.1”, + “metadataServerPort”: “3105, + “port”: 5000, + “useConfigIP”” true, + “legacyConfigFilePath”: “home/SQream_develop/SqrmRT/utils/json/legacy_congif.json” + } + +You can access the legacy configuration file from the ``legacyConfigFilePath`` parameter shown above. If all (or most) of your workers require the same flag settings, you can set the ``legacyConfigFilePath`` attribute to the same legacy file. + +Modifying Your Configuration Using a Legacy Configuration File +--------------------- +You can modify your configuration using a legacy configuration file. + +The Legacy configuration file provides access to the read/write flags used in SQream’s previous configuration method. A link to this file is provided in the **legacyConfigFilePath** parameter in the worker configuration file. + +The following is an example of the legacy configuration file: + +.. code-block:: postgres + + { + “developerMode”: true, + “reextentUse”: false, + “useClientLog”: true, + “useMetadataServer”” false + } +For more information on using the previous configuration method, see :ref:`previous_configuration_method`. \ No newline at end of file diff --git a/configuration_guides/current_method_showing_all_flags_in_the_catalog_table.rst b/configuration_guides/current_method_showing_all_flags_in_the_catalog_table.rst new file mode 100644 index 000000000..647a401b6 --- /dev/null +++ b/configuration_guides/current_method_showing_all_flags_in_the_catalog_table.rst @@ -0,0 +1,21 @@ +.. _current_method_showing_all_flags_in_the_catalog_table: + +************************** +Showing All Flags in the Catalog Table +************************** +SQream uses the **sqream_catalog.parameters** catalog table for showing all flags, providing the scope (default, cluster and session), description, default value and actual value. + +The following is the correct syntax for a catalog table query: + +.. code-block:: console + + SELECT * FROM sqream_catalog.settings + +The following is an example of a catalog table query: + +.. code-block:: console + + externalTableBlobEstimate, 100, 100, default, + varcharEncoding, ascii, ascii, default, Changes the expected encoding for Varchar columns + useCrcForTextJoinKeys, true, true, default, + hiveStyleImplicitStringCasts, false, false, default, \ No newline at end of file diff --git a/configuration_guides/developer_mode.rst b/configuration_guides/developer_mode.rst new file mode 100644 index 000000000..1fa0eb32c --- /dev/null +++ b/configuration_guides/developer_mode.rst @@ -0,0 +1,12 @@ +.. _developer_mode: + +************************* +Enabling Modification of R&D Flags +************************* +The ``developerMode`` flag enables modifying R&D flags. + +The following describes the ``developerMode`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/enable_device_debug_messages.rst b/configuration_guides/enable_device_debug_messages.rst new file mode 100644 index 000000000..d48f0e94e --- /dev/null +++ b/configuration_guides/enable_device_debug_messages.rst @@ -0,0 +1,12 @@ +.. _enable_device_debug_messages: + +************************* +Checking for Post-Production CUDA Errors +************************* +The ``enableDeviceDebugMessages`` flag checks for CUDA errors after producing each chunk. + +The following describes the ``enableDeviceDebugMessages`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/enable_log_debug.rst b/configuration_guides/enable_log_debug.rst new file mode 100644 index 000000000..bfc0e5acb --- /dev/null +++ b/configuration_guides/enable_log_debug.rst @@ -0,0 +1,12 @@ +.. _enable_log_debug: + +************************* +Enabling Modification of clientLogger_debug File +************************* +The ``enableLogDebug`` flag enables creating and logging in the **clientLogger_debug** file. + +The following describes the ``enableLogDebug`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/enable_nv_prof_markers.rst b/configuration_guides/enable_nv_prof_markers.rst new file mode 100644 index 000000000..df18a10e7 --- /dev/null +++ b/configuration_guides/enable_nv_prof_markers.rst @@ -0,0 +1,12 @@ +.. _enable_nv_prof_markers: + +************************* +Activating the NVidia Profiler Markers +************************* +The ``enableNvprofMarkers`` flag activates the NVidia Profiler (nvprof) markers. + +The following describes the ``enableNvprofMarkers`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/end_log_message.rst b/configuration_guides/end_log_message.rst new file mode 100644 index 000000000..46a1c71ae --- /dev/null +++ b/configuration_guides/end_log_message.rst @@ -0,0 +1,11 @@ +.. _end_log_message: + +************************* +Appending String at End of Log Lines +************************* +The ``endLogMessage`` flag appends a string at the end of each log line. + +The following describes the ``endLogMessage`` flag: + +* **Data type** - string +* **Default value** - ``EOM`` \ No newline at end of file diff --git a/configuration_guides/extent_storage_file_size_mb.rst b/configuration_guides/extent_storage_file_size_mb.rst new file mode 100644 index 000000000..dd4cddec7 --- /dev/null +++ b/configuration_guides/extent_storage_file_size_mb.rst @@ -0,0 +1,11 @@ +.. _extent_storage_file_size_mb: + +************************* +Setting Minimum Extent Size for Bulk Table Data +************************* +The ``extentStorageFileSizeMB`` flag sets the minimum size in mebibytes of extents for bulk table data. + +The following describes the ``extentStorageFileSizeMB`` flag: + +* **Data type** - uint +* **Default value** - ``20`` \ No newline at end of file diff --git a/configuration_guides/files_and_folders.txt b/configuration_guides/files_and_folders.txt new file mode 100644 index 000000000..0c8411b1b --- /dev/null +++ b/configuration_guides/files_and_folders.txt @@ -0,0 +1,67 @@ + Volume in drive C is Windows + Volume Serial Number is 565C-23BD + + Directory of C:\Users\Yaniv\Desktop\Local Reorganization Folder\configuration_guides + +03/22/2022 12:47 PM . +03/22/2022 12:47 PM .. +03/13/2022 11:57 AM 440 admin_cluster_flags.rst +03/09/2022 10:31 AM 403 admin_flags.rst +03/09/2022 11:19 AM 3,561 admin_regular_flags.rst +03/09/2022 10:31 AM 813 admin_worker_flags.rst +03/09/2022 11:22 AM 615 bin_sizes.rst +03/09/2022 12:10 PM 368 cache_disk_dir.rst +03/09/2022 12:14 PM 340 cache_disk_gb.rst +03/09/2022 11:59 AM 378 cache_eviction_milliseconds.rst +03/09/2022 11:42 AM 344 cache_partitions.rst +03/09/2022 12:02 PM 402 cache_persistent_dir.rst +03/09/2022 12:13 PM 377 cache_persistent_gb.rst +03/09/2022 11:53 AM 337 cache_ram_gb.rst +03/09/2022 11:23 AM 381 check_cuda_memory.rst +03/09/2022 11:24 AM 394 compiler_gets_only_ufs.rst +03/08/2022 05:38 PM 570 configuration_flags.rst +03/08/2022 05:38 PM 605 configuration_methods.rst +03/09/2022 11:24 AM 403 copy_to_restrict_utf8.rst +03/09/2022 11:38 AM 408 cpu_reduce_hashtable_size.rst +03/09/2022 11:25 AM 344 csv_limit_row_length.rst +03/09/2022 11:25 AM 424 cuda_mem_cpy_max_size_bytes.rst +03/09/2022 11:26 AM 369 cuda_mem_cpy_synchronous.rst +03/09/2022 11:26 AM 374 cuda_mem_quota.rst +03/20/2022 11:02 AM 24,433 current_configuration_method.rst +03/09/2022 11:27 AM 331 developer_mode.rst +03/09/2022 11:27 AM 397 enable_device_debug_messages.rst +03/09/2022 11:28 AM 383 enable_log_debug.rst +03/09/2022 11:28 AM 372 enable_nv_prof_markers.rst +03/09/2022 11:29 AM 302 end_log_message.rst +03/09/2022 10:31 AM 363 extent_storage_file_size_mb.rst +03/22/2022 12:47 PM 0 files_and_folders.txt +03/09/2022 11:29 AM 377 flip_join_order.rst +03/09/2022 11:30 AM 536 gather_mem_stat.rst +03/09/2022 10:31 AM 344 generic_flags.rst +03/09/2022 12:14 PM 1,837 generic_regular_flags.rst +03/09/2022 10:36 AM 381 generic_worker_flags.rst +03/09/2022 11:30 AM 410 increase_chunk_size_before_reduce.rst +03/09/2022 11:30 AM 379 increase_mem_factors.rst +03/09/2022 10:31 AM 308 index.rst +03/09/2022 11:31 AM 332 level_db_write_buffer_size.rst +03/09/2022 11:31 AM 385 limit_query_memory_gb.rst +03/09/2022 11:32 AM 441 log_sys_level.rst +03/09/2022 11:32 AM 328 machine_ip.rst +03/09/2022 11:33 AM 424 max_avg_blob_size_to_compress_on_gpu.rst +03/13/2022 04:54 PM 422 max_pinned_percentage_of_total_ram.rst +03/09/2022 11:33 AM 388 memory_reset_trigger_mb.rst +03/09/2022 11:33 AM 573 metadata_server_port.rst +03/09/2022 11:34 AM 371 mt_read.rst +03/09/2022 11:34 AM 367 mt_read_workers.rst +03/09/2022 11:35 AM 395 orc_implicit_casts.rst +03/08/2022 07:27 PM 9,074 previous_configuration_method.rst +03/09/2022 12:09 PM 313 session_tag.rst +03/09/2022 10:31 AM 2,822 spooling.rst +03/09/2022 11:35 AM 340 spool_memory_gb.rst +03/09/2022 11:35 AM 428 statement_lock_timeout.rst +03/09/2022 11:36 AM 487 use_config_ip.rst +03/09/2022 11:37 AM 494 use_legacy_decimal_literals.rst +03/13/2022 12:21 PM 461 use_legacy_string_literals.rst +03/09/2022 11:37 AM 354 varchar_identifiers.rst + 58 File(s) 63,002 bytes + 2 Dir(s) 898,522,271,744 bytes free diff --git a/configuration_guides/flip_join_order.rst b/configuration_guides/flip_join_order.rst new file mode 100644 index 000000000..806bde99d --- /dev/null +++ b/configuration_guides/flip_join_order.rst @@ -0,0 +1,12 @@ +.. _flip_join_order: + +************************* +Flipping Join Order to Force Equijoins +************************* +The ``flipJoinOrder`` flag reorders join to force equijoins and/or equijoins sorted by table size. + +The following describes the ``flipJoinOrder`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/gather_mem_stat.rst b/configuration_guides/gather_mem_stat.rst new file mode 100644 index 000000000..3cc034194 --- /dev/null +++ b/configuration_guides/gather_mem_stat.rst @@ -0,0 +1,12 @@ +.. _gather_mem_stat: + +************************* +Monitoring and Printing Pinned Allocation Reports +************************* +The ``gatherMemStat`` flag monitors all pinned allocations and all **memcopies** to and from a device, and prints a report of pinned allocations that were not **memcopied** to and from the device using the **dump_pinned_misses** utility function. + +The following describes the ``gatherMemStat`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/generic_flags.rst b/configuration_guides/generic_flags.rst new file mode 100644 index 000000000..cb0241e31 --- /dev/null +++ b/configuration_guides/generic_flags.rst @@ -0,0 +1,14 @@ +.. _generic_flags: + +************************* +Generic Flags +************************* + +The **Generic Flags** page describes the following flag types, which can be modified by standard users on a session basis: + +.. toctree:: + :maxdepth: 1 + :glob: + + generic_regular_flags + generic_worker_flags \ No newline at end of file diff --git a/configuration_guides/generic_regular_flags.rst b/configuration_guides/generic_regular_flags.rst new file mode 100644 index 000000000..b78ae3808 --- /dev/null +++ b/configuration_guides/generic_regular_flags.rst @@ -0,0 +1,21 @@ +.. _generic_regular_flags: + +************************* +Regular Generic Flags +************************* + +The **Regular Generic Flags** page describes **Regular** modification type flags, which can be modified by standard users on a session basis: + +* `Flipping Join Order to Force Equijoins `_ +* `Determining Client Level `_ +* `Setting CPU to Compress Defined Columns `_ +* `Setting Query Memory Processing Limit `_ +* `Setting the Spool Memory `_ +* `Setting Cache Partitions `_ +* `Setting Cache Flushing `_ +* `Setting InMemory Spool Memory `_ +* `Setting Disk Spool Memory `_ +* `Setting Spool Saved File Directory Location `_ +* `Setting Data Stored Persistently on Cache `_ +* `Setting Persistent Spool Saved File Directory Location `_ +* `Setting Session Tag Name `_ \ No newline at end of file diff --git a/configuration_guides/generic_worker_flags.rst b/configuration_guides/generic_worker_flags.rst new file mode 100644 index 000000000..74b65458d --- /dev/null +++ b/configuration_guides/generic_worker_flags.rst @@ -0,0 +1,8 @@ +.. _generic_worker_flags: + +************************* +Worker Generic Flags +************************* +The **Worker Generic Flags** page describes **Worker** modification type flags, which can be modified by standard users on a session basis: + + * `Limits Available Query Processing Memory `_ \ No newline at end of file diff --git a/configuration_guides/increase_chunk_size_before_reduce.rst b/configuration_guides/increase_chunk_size_before_reduce.rst new file mode 100644 index 000000000..99d44a226 --- /dev/null +++ b/configuration_guides/increase_chunk_size_before_reduce.rst @@ -0,0 +1,12 @@ +.. _increase_chunk_size_before_reduce: + +************************* +Increasing Chunk Size to Reduce Query Speed +************************* +The ``increaseChunkSizeBeforeReduce`` flag increases the chunk size to reduce query speed. + +The following describes the ``increaseChunkSizeBeforeReduce`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/increase_mem_factors.rst b/configuration_guides/increase_mem_factors.rst new file mode 100644 index 000000000..e1f7898f5 --- /dev/null +++ b/configuration_guides/increase_mem_factors.rst @@ -0,0 +1,12 @@ +.. _increase_mem_factors: + +************************* +Adding Rechunker before Expensing Chunk Producer +************************* +The ``increaseMemFactors`` flag adds a rechunker before expensive chunk producer. + +The following describes the ``increaseMemFactors`` flag: + +* **Data type** - boolean +* **Default value** - ``true`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/index.rst b/configuration_guides/index.rst new file mode 100644 index 000000000..6805bc188 --- /dev/null +++ b/configuration_guides/index.rst @@ -0,0 +1,16 @@ +.. _configuration_guides: + +************************* +Configuration Guides +************************* + +The **Configuration Guides** page describes the following configuration information: + +.. toctree:: + :maxdepth: 1 + :glob: + + spooling + configuring_sqream + configuration_flags + previous_configuration_method \ No newline at end of file diff --git a/configuration_guides/level_db_write_buffer_size.rst b/configuration_guides/level_db_write_buffer_size.rst new file mode 100644 index 000000000..14eae1588 --- /dev/null +++ b/configuration_guides/level_db_write_buffer_size.rst @@ -0,0 +1,12 @@ +.. _level_db_write_buffer_size: + +************************* +Setting the Buffer Size +************************* +The ``leveldbWriteBufferSize`` flag sets the buffer size. + +The following describes the ``leveldbWriteBufferSize`` flag: + +* **Data type** - uint +* **Default value** - ``524288`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/limit_query_memory_gb.rst b/configuration_guides/limit_query_memory_gb.rst new file mode 100644 index 000000000..38123f56d --- /dev/null +++ b/configuration_guides/limit_query_memory_gb.rst @@ -0,0 +1,12 @@ +.. _limit_query_memory_gb: + +************************* +Limiting Available Query Processing Memory +************************* +The ``limitQueryMemoryGB`` flag prevents a query from processing more memory than the defined value. + +The following describes the ``limitQueryMemoryGB`` flag: + +* **Data type** - uint +* **Default value** - ``100000`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/log_sys_level.rst b/configuration_guides/log_sys_level.rst new file mode 100644 index 000000000..031cee32f --- /dev/null +++ b/configuration_guides/log_sys_level.rst @@ -0,0 +1,20 @@ +.. _log_sys_level: + +************************* +Determining Client Level +************************* +The ``logSysLevel`` flag determines the client log level, as follows: + +* **0** - L_SYSTEM +* **1** - L_FATAL +* **2** - L_ERROR +* **3** - L_WARN +* **4** - L_INFO +* **5** - L_DEBUG +* **6** - L_TRACE + +The following describes the ``logSysLevel`` flag: + +* **Data type** - uint +* **Default value** - ``1`` +* **Allowed values** - 0-6 \ No newline at end of file diff --git a/configuration_guides/machine_ip.rst b/configuration_guides/machine_ip.rst new file mode 100644 index 000000000..d184ed134 --- /dev/null +++ b/configuration_guides/machine_ip.rst @@ -0,0 +1,12 @@ +.. _machine_ip: + +************************* +Enabling Manually Setting Reported IP +************************* +The ``machineIP`` flag enables you to manually set the reported IP. + +The following describes the ``machineIP`` flag: + +* **Data type** - string +* **Default value** - ``127.0.0.1`` +* **Allowed values** - char(16) \ No newline at end of file diff --git a/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst b/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst new file mode 100644 index 000000000..18d8f0408 --- /dev/null +++ b/configuration_guides/max_avg_blob_size_to_compress_on_gpu.rst @@ -0,0 +1,12 @@ +.. _max_avg_blob_size_to_compress_on_gpu: + +************************* +Setting CPU to Compress Defined Columns +************************* +The ``maxAvgBlobSizeToCompressOnGpu`` flag sets the CPU to compress columns with size above * . + +The following describes the ``maxAvgBlobSizeToCompressOnGpu`` flag: + +* **Data type** - uint +* **Default value** - ``120`` +* **Allowed values** - 1-4000000000 \ No newline at end of file diff --git a/configuration_guides/max_pinned_percentage_of_total_ram.rst b/configuration_guides/max_pinned_percentage_of_total_ram.rst new file mode 100644 index 000000000..9ced5bde9 --- /dev/null +++ b/configuration_guides/max_pinned_percentage_of_total_ram.rst @@ -0,0 +1,12 @@ +.. _max_pinned_percentage_of_total_ram: + +************************* +Maximum Pinned Percentage of Total RAM +************************* +The ``maxPinnedPercentageOfTotalRAM`` flag sets the maximum percentage out of the available total CPU RAM that can be used by **pinned memory**. + +The following describes the ``machineIP`` flag: + +* **Data type** - uint +* **Default value** - ``70`` +* **Allowed values** - ``0-100`` \ No newline at end of file diff --git a/configuration_guides/memory_reset_trigger_mb.rst b/configuration_guides/memory_reset_trigger_mb.rst new file mode 100644 index 000000000..c3430dda8 --- /dev/null +++ b/configuration_guides/memory_reset_trigger_mb.rst @@ -0,0 +1,12 @@ +.. _memory_reset_trigger_mb: + +************************* +Setting Memory Used to Abort Server +************************* +The ``memoryResetTriggerMB`` flag sets the size of memory used during a query to trigger aborting the server. + +The following describes the ``memoryResetTriggerMB`` flag: + +* **Data type** - uint +* **Default value** - ``0`` +* **Allowed values** - 0-4000000000 diff --git a/configuration_guides/metadata_server_port.rst b/configuration_guides/metadata_server_port.rst new file mode 100644 index 000000000..0dcd9bbe0 --- /dev/null +++ b/configuration_guides/metadata_server_port.rst @@ -0,0 +1,12 @@ +.. _metadata_server_port: + +************************* +Setting Port Used for Metadata Server Connection +************************* +The ``metadataServerPort`` flag sets the port used to connect to the metadata server. SQream recommends using port ranges above 1024 because ports below 1024 are usually reserved, although there are no strict limitations. You can use any positive number (1 - 65535) while setting this flag. + +The following describes the ``metadataServerPort`` flag: + +* **Data type** - uint +* **Default value** - ``3105`` +* **Allowed values** - 1-65535 \ No newline at end of file diff --git a/configuration_guides/mt_read.rst b/configuration_guides/mt_read.rst new file mode 100644 index 000000000..929d1c0e0 --- /dev/null +++ b/configuration_guides/mt_read.rst @@ -0,0 +1,12 @@ +.. _mt_read: + +************************* +Splitting Large Reads for Concurrent Execution +************************* +The ``mtRead`` flag splits large reads into multiple smaller ones and executes them concurrently. + +The following describes the ``mtRead`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` diff --git a/configuration_guides/mt_read_workers.rst b/configuration_guides/mt_read_workers.rst new file mode 100644 index 000000000..90eb5cefd --- /dev/null +++ b/configuration_guides/mt_read_workers.rst @@ -0,0 +1,12 @@ +.. _mt_read_workers: + +************************* +Setting Worker Amount to Handle Concurrent Reads +************************* +The ``mtReadWorkers`` flag sets the number of workers to handle smaller concurrent reads. + +The following describes the ``mtReadWorkers`` flag: + +* **Data type** - uint +* **Default value** - ``30`` +* **Allowed values** - 1-4000000000 diff --git a/configuration_guides/orc_implicit_casts.rst b/configuration_guides/orc_implicit_casts.rst new file mode 100644 index 000000000..e1b1287eb --- /dev/null +++ b/configuration_guides/orc_implicit_casts.rst @@ -0,0 +1,12 @@ +.. _orc_implicit_casts: + +************************* +Setting Implicit Casts in ORC Files +************************* +The ``orcImplicitCasts`` flag sets the implicit cast in orc files, such as **int** to **tinyint** and vice versa. + +The following describes the ``orcImplicitCasts`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/guides/operations/configuration.rst b/configuration_guides/previous_configuration_method.rst similarity index 91% rename from guides/operations/configuration.rst rename to configuration_guides/previous_configuration_method.rst index feceb5e56..ff48e7cc1 100644 --- a/guides/operations/configuration.rst +++ b/configuration_guides/previous_configuration_method.rst @@ -1,13 +1,13 @@ -.. _configuration: +.. _previous_configuration_method: -*********************** -Configuration -*********************** +************************** +Configuring SQream Using the Previous Configuration Method +************************** +The **Configuring SQream Using the Previous Configuration Method** page describes SQream’s previous method for configuring your instance of SQream, and includes the following topics: -This guide covers the configuration files and the ``SET`` statement. - -Configuration files -========================== +.. contents:: + :local: + :depth: 1 By default, configuration files are stored in ``/etc/sqream``. @@ -36,9 +36,8 @@ A very minimal configuration file looks like this: In the example above, the worker will start on port 5000, and will use GPU #0. -Frequently set parameters -============================ - +Frequently Set Parameters +------ .. todo list-table:: Compiler flags :widths: auto @@ -235,10 +234,10 @@ Frequently set parameters .. point to the best practices as well -.. warning:: JSON files can't contain any comments +.. warning:: JSON files can't contain any comments. -Recommended configuration file -===================================== +Recommended Configuration File +----- .. code-block:: json @@ -266,15 +265,4 @@ Recommended configuration file "cluster":"/home/sqream/sqream_storage", "licensePath":"/etc/sqream/license.enc" } - } - -Changing settings temporarily -=================================== - -The ``SET`` statement can modify one of the configuration settings for the session or connection. - -For example, to set the query plan's logging interval to "every 3 seconds" for subsequent statements: - -.. code-block:: psql - - t=> SET nodeInfoLoggingSec=3; SELECT * FROM nba; + } \ No newline at end of file diff --git a/configuration_guides/session_tag.rst b/configuration_guides/session_tag.rst new file mode 100644 index 000000000..12a98f01c --- /dev/null +++ b/configuration_guides/session_tag.rst @@ -0,0 +1,12 @@ +.. _session_tag: + +************************* +Setting Session Tag Name +************************* +The ``sessionTag`` flag sets the name of the session tag. + +The following describes the ``sessionTag`` flag: + +* **Data type** - string +* **Default value** - ``default`` +* **Allowed values** - Any legal string \ No newline at end of file diff --git a/configuration_guides/spool_memory_gb.rst b/configuration_guides/spool_memory_gb.rst new file mode 100644 index 000000000..cb7135d03 --- /dev/null +++ b/configuration_guides/spool_memory_gb.rst @@ -0,0 +1,12 @@ +.. _spool_memory_gb: + +************************* +Setting the Spool Memory +************************* +The ``spoolMemoryGB`` flag sets the amount of memory (GB) available to the server for spooling. + +The following describes the ``spoolMemoryGB`` flag: + +* **Data type** - uint +* **Default value** - ``8`` +* **Allowed values** - 0-5000 \ No newline at end of file diff --git a/configuration_guides/spooling.rst b/configuration_guides/spooling.rst new file mode 100644 index 000000000..88d5a7980 --- /dev/null +++ b/configuration_guides/spooling.rst @@ -0,0 +1,69 @@ +.. _spooling: + +************************** +Configuring the Spooling Feature +************************** +The **Configuring the Spooling Feature** page includes the following topics: + +.. contents:: + :local: + :depth: 1 + + +Overview +---------- +From the SQream Acceleration Studio you can allocate the amount of memory (GB) available to the server for spooling using the ``spoolMemoryGB`` flag. SQream recommends setting the ``spoolMemoryGB`` flag to 90% of the ``limitQueryMemoryGB`` flag. The ``limitQueryMemoryGB`` flag is the total memory you’ve allocated for processing queries. + +In addition, the ``limitQueryMemoryGB`` defines how much total system memory is used by each worker. SQream recommends setting ``limitQueryMemoryGB`` to 5% less than the total host memory divided by the amount of ``sqreamd`` workers on host. + +Note that ``spoolMemoryGB`` must bet set to less than the ``limitQueryMemoryGB``. + +Example Configurations +---------- +The **Example Configurations** section shows the following example configurations: + +.. contents:: + :local: + :depth: 1 + +Example 1 - Recommended Settings +~~~~~~~~~~~ +The following is an example of the recommended settings for a machine with 512GB of RAM and 4 workers: + +.. code-block:: console + + limitQueryMemoryGB - ⌊(512 * 0.95 / 4)⌋ → ~ 486 / 4 → 121 + spoolMemoryGB - ⌊( 0.9 * limitQueryMemoryGB )⌋ → ⌊( 0.9 * 121 )⌋ → 108 + +Example 2 - Setting Spool Memory +~~~~~~~~~~~ +The following is an example of setting ``spoolMemoryGB`` value in the current configuration method per-worker for 512GB of RAM and 4 workers: + +.. code-block:: console + + { + “cluster”: “/home/test_user/sqream_testing_temp/sqreamdb”, + “gpu”: 0, + “licensePath”: “home/test_user/SQream/tests/license.enc”, + “machineIP”: “127.0.0.1”, + “metadataServerIp”: “127.0.0.1”, + “metadataServerPort”: “3105, + “port”: 5000, + “useConfigIP”” true, + “limitQueryMemoryGB" : 121, + “spoolMemoryGB" : 108 + “legacyConfigFilePath”: “home/SQream_develop/SqrmRT/utils/json/legacy_congif.json” + } + +The following is an example of setting ``spoolMemoryGB`` value in the previous configuration method per-worker for 512GB of RAM and 4 workers: + +.. code-block:: console + + “runtimeFlags”: { + “limitQueryMemoryGB” : 121, + “spoolMemoryGB” : 108 + +For more information about configuring the ``spoolMemoryGB`` flag, see the following: + +* `Current configuration method `_ +* `Previous configuration method `_ \ No newline at end of file diff --git a/configuration_guides/statement_lock_timeout.rst b/configuration_guides/statement_lock_timeout.rst new file mode 100644 index 000000000..ae552083a --- /dev/null +++ b/configuration_guides/statement_lock_timeout.rst @@ -0,0 +1,12 @@ +.. _statement_lock_timeout: + +************************* +Setting Timeout Limit for Locking Objects before Executing Statements +************************* +The ``statementLockTimeout`` flag sets the timeout (seconds) for acquiring object locks before executing statements. + +The following describes the ``statementLockTimeout`` flag: + +* **Data type** - uint +* **Default value** - ``3`` +* **Allowed values** - 0-4000000000 diff --git a/configuration_guides/use_config_ip.rst b/configuration_guides/use_config_ip.rst new file mode 100644 index 000000000..f195c9615 --- /dev/null +++ b/configuration_guides/use_config_ip.rst @@ -0,0 +1,12 @@ +.. _use_config_ip: + +************************* +Assigning Local Network IP +************************* +The ``useConfigIP`` flag activates the machineIP (``true``). Setting this flag to ``false`` ignores the machineIP and automatically assigns a local network IP. This cannot be activated in a cloud scenario (on-premises only). + +The following describes the ``useConfigIP`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/use_legacy_decimal_literals.rst b/configuration_guides/use_legacy_decimal_literals.rst new file mode 100644 index 000000000..9f26a5af1 --- /dev/null +++ b/configuration_guides/use_legacy_decimal_literals.rst @@ -0,0 +1,12 @@ +.. _use_legacy_decimal_literals: + +************************* +Interpreting Decimal Literals as Double Instead of Numeric +************************* +The ``useLegacyDecimalLiterals`` flag interprets decimal literals as **Double** instead of **Numeric**. This flag is used to preserve legacy behavior in existing customers. + +The following describes the ``useLegacyDecimalLiterals`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/use_legacy_string_literals.rst b/configuration_guides/use_legacy_string_literals.rst new file mode 100644 index 000000000..164801084 --- /dev/null +++ b/configuration_guides/use_legacy_string_literals.rst @@ -0,0 +1,12 @@ +.. _use_legacy_string_literals: + +************************* +Using Legacy String Literals +************************* +The ``useLegacyStringLiterals`` flag interprets ASCII-only strings as **VARCHAR** instead of **TEXT**. This flag is used to preserve legacy behavior in existing customers. + +The following describes the ``useLegacyStringLiterals`` flag: + +* **Data type** - boolean +* **Default value** - ``false`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/configuration_guides/varchar_identifiers.rst b/configuration_guides/varchar_identifiers.rst new file mode 100644 index 000000000..889e5c16e --- /dev/null +++ b/configuration_guides/varchar_identifiers.rst @@ -0,0 +1,12 @@ +.. _varchar_identifiers: + +************************* +Interpreting VARCHAR as TEXT +************************* +The ``varcharIdentifiers`` flag activates using **varchar** as an identifier. + +The following describes the ``varcharIdentifiers`` flag: + +* **Data type** - boolean +* **Default value** - ``true`` +* **Allowed values** - ``true``, ``false`` \ No newline at end of file diff --git a/guides/client_drivers/cpp/connect_test.cpp b/connecting_to_sqream/client_drivers/cpp/connect_test.cpp similarity index 100% rename from guides/client_drivers/cpp/connect_test.cpp rename to connecting_to_sqream/client_drivers/cpp/connect_test.cpp diff --git a/guides/client_drivers/cpp/index.rst b/connecting_to_sqream/client_drivers/cpp/index.rst similarity index 82% rename from guides/client_drivers/cpp/index.rst rename to connecting_to_sqream/client_drivers/cpp/index.rst index 703643b0f..dcb76cfbc 100644 --- a/guides/client_drivers/cpp/index.rst +++ b/connecting_to_sqream/client_drivers/cpp/index.rst @@ -13,7 +13,7 @@ This tutorial shows how to write a C++ program that uses this driver. :local: -Installing the C++ driver +Installing the C++ Driver ================================== Prerequisites @@ -21,13 +21,13 @@ Prerequisites The SQream DB C++ driver was built on 64-bit Linux, and is designed to work with RHEL 7 and Ubuntu 16.04 and newer. -Getting the library +Getting the Library --------------------- -The C++ driver is provided as a tarball containing the compiled ``libsqream.so`` file and a header ``sqream.h``. Get the driver from the `SQream Drivers page `_. The library can be integrated into your C++-based applications or projects. +The C++ driver is provided as a tarball containing the compiled ``libsqream.so`` file and a header ``sqream.h``. Get the driver from the `SQream Drivers page `_. The library can be integrated into your C++-based applications or projects. -Extract the tarball archive +Extract the Tarball Archive ----------------------------- Extract the library files from the tarball @@ -41,7 +41,7 @@ Examples Assuming there is a SQream DB worker to connect to, we'll connect to it using the application and run some statements. -Testing the connection to SQream DB +Testing the Connection to SQream -------------------------------------------- Download this file by right clicking and saving to your computer :download:`connect_test.cpp `. @@ -52,7 +52,7 @@ Download this file by right clicking and saving to your computer :download:`conn :linenos: -Compiling and running the application +Compiling and Running the Application ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To build this code, place the library and header file in ./libsqream-3.0/ and run @@ -64,7 +64,7 @@ To build this code, place the library and header file in ./libsqream-3.0/ and ru Modify the ``-I`` and ``-L`` arguments to match the ``.so`` library and ``.h`` file if they are in another directory. -Creating a table and inserting values +Creating a Table and Inserting Values -------------------------------------------- Download this file by right clicking and saving to your computer :download:`insert_test.cpp `. @@ -75,7 +75,7 @@ Download this file by right clicking and saving to your computer :download:`inse :linenos: -Compiling and running the application +Compiling and Running the Application ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To build this code, use @@ -83,5 +83,4 @@ To build this code, use .. code-block:: console $ g++ -Wall -Ilibsqream-3.0 -Llibsqream-3.0 -lsqream insert_test.cpp -o insert_test - $ ./insert_test - + $ ./insert_test \ No newline at end of file diff --git a/guides/client_drivers/cpp/insert_test.cpp b/connecting_to_sqream/client_drivers/cpp/insert_test.cpp similarity index 100% rename from guides/client_drivers/cpp/insert_test.cpp rename to connecting_to_sqream/client_drivers/cpp/insert_test.cpp diff --git a/connecting_to_sqream/client_drivers/index.rst b/connecting_to_sqream/client_drivers/index.rst new file mode 100644 index 000000000..e0e88fbd2 --- /dev/null +++ b/connecting_to_sqream/client_drivers/index.rst @@ -0,0 +1,110 @@ +.. _client_drivers: + +************************************ +Client Drivers for Version 2020.3.2.1 +************************************ + +The guides on this page describe how to use the Sqream DB client drivers and client applications with SQream. + +Client Driver Downloads +============================= + +All Operating Systems +--------------------------- +The following are applicable to all operating systems: + +.. _jdbc: + +* **JDBC** - recommended installation via ``mvn``: + + * `JDBC .jar file `_ - sqream-jdbc-4.5.3 (.jar) + * `JDBC driver `_ + + +.. _python: + +* **Python** - Recommended installation via ``pip``: + + * `Python .tar file `_ - pysqream v3.1.3 (.tar.gz) + * `Python driver `_ + + +.. _nodejs: + +* **Node.JS** - Recommended installation via ``npm``: + + * `Node.JS `_ - sqream-v4.2.4 (.tar.gz) + * `Node.JS driver `_ + + +.. _tableau_connector: + +* **Tableau**: + + * `Tableau connector `_ - SQream (.taco) + * `Tableau manual installation `_ + + +.. _powerbi_connector: + +* **Power BI**: + + * `Power BI PowerQuery connector `_ - SQream (.mez) + * `Power BI manual installation `_ + + +Windows +-------------- +The following are applicable to Windows: + +* **ODBC installer** - SQream Drivers v2020.2.0, with Tableau customizations. Please contact your `SQream represenative `_ for this installer. + + For more information on installing and configuring ODBC on Windows, see :ref:`Install and configure ODBC on Windows `. + + +* **Net driver** - `SQream .Net driver v3.0.2 `_ + + + +Linux +-------------- +The following are applicable to Linux: + +* `SQream SQL (x86_64) `_ - sqream-sql-v2020.1.1_stable.x86_64.tar.gz +* `Sqream SQL CLI Reference `_ - Interactive command-line SQL client for Intel-based machines + + :: + +* `SQream SQL*(IBM POWER9) `_ - sqream-sql-v2020.1.1_stable.ppc64le.tar.gz +* `Sqream SQL CLI Reference `_ - Interactive command-line SQL client for IBM POWER9-based machines + + :: + +* ODBC Installer - Please contact your SQream representative for this installer. + + :: + +* C++ connector - `libsqream-4.0 `_ +* `C++ shared object library `_ + + +.. toctree:: + :maxdepth: 4 + :caption: Client Driver Documentation: + :titlesonly: + + jdbc/index + python/index + nodejs/index + odbc/index + cpp/index + + + +.. rubric:: Need help? + +If you couldn't find what you're looking for, we're always happy to help. Visit `SQream's support portal `_ for additional support. + +.. rubric:: Looking for older drivers? + +If you're looking for an older version of SQream DB drivers, versions 1.10 through 2019.2.1 are available at https://sqream.com/product/client-drivers/. \ No newline at end of file diff --git a/connecting_to_sqream/client_drivers/jdbc/index.rst b/connecting_to_sqream/client_drivers/jdbc/index.rst new file mode 100644 index 000000000..832987efa --- /dev/null +++ b/connecting_to_sqream/client_drivers/jdbc/index.rst @@ -0,0 +1,171 @@ +.. _java_jdbc: + +************************* +JDBC +************************* +The SQream JDBC driver lets you connect to SQream using many Java applications and tools. This page describes how to write a Java application using the JDBC interface. The JDBC driver requires Java 1.8 or newer. + +The JDBC page includes the following sections: + +.. contents:: + :local: + :depth: 1 + +Installing the JDBC Driver +================================== +The **Installing the JDBC Driver** section describes the following: + +.. contents:: + :local: + :depth: 1 + +Prerequisites +---------------- +The SQream JDBC driver requires Java 1.8 or newer, and SQream recommends using Oracle Java or OpenJDK.: + +* **Oracle Java** - Download and install `Java 8 `_ from Oracle for your platform. + + :: + +* **OpenJDK** - Install `OpenJDK `_ + + :: + +* **Windows** - SQream recommends installing `Zulu 8 `_ + +Getting the JAR file +--------------------- +SQream provides the JDBC driver as a zipped JAR file, available for download from the :ref:`client drivers download page`. This JAR file can be integrated into your Java-based applications or projects. + +Extracting the ZIP Archive +------------------------- +Run the following command to extract the JAR file from the ZIP archive: + +.. code-block:: console + + $ unzip sqream-jdbc-4.3.0.zip + +Setting Up the Class Path +---------------------------- +To use the driver, you must include the JAR named ``sqream-jdbc-.jar`` in the class path, either by inserting it in the ``CLASSPATH`` environment variable, or by using flags on the relevant Java command line. + +For example, if the JDBC driver has been unzipped to ``/home/sqream/sqream-jdbc-4.3.0.jar``, the following command is used to run application: + +.. code-block:: console + + $ export CLASSPATH=/home/sqream/sqream-jdbc-4.3.0.jar:$CLASSPATH + $ java my_java_app + +Alternatively, you can pass ``-classpath`` to the Java executable file: + +.. code-block:: console + + $ java -classpath .:/home/sqream/sqream-jdbc-4.3.0.jar my_java_app + +Connecting to SQream Using a JDBC Application +============================================== +You can connect to SQream using one of the following JDBC applications: + +.. contents:: + :local: + :depth: 1 + +Driver Class +-------------- +Use ``com.sqream.jdbc.SQDriver`` as the driver class in the JDBC application. + +Connection String +-------------------- +JDBC drivers rely on a connection string. + +The following is the syntax for SQream: + +.. code-block:: text + + jdbc:Sqream:///;user=;password=sqream;[; ...] + +Connection Parameters +^^^^^^^^^^^^^^^^^^^^^^^^ +The following table shows the connection string parameters: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Item + - State + - Default + - Description + * - ```` + - Mandatory + - None + - Hostname and port of the SQream DB worker. For example, ``127.0.0.1:5000``, ``sqream.mynetwork.co:3108`` + * - ```` + - Mandatory + - None + - Database name to connect to. For example, ``master`` + * - ``username=`` + - Mandatory + - None + - Username of a role to use for connection. For example, ``username=rhendricks`` + * - ``password=`` + - Mandatory + - None + - Specifies the password of the selected role. For example, ``password=Tr0ub4dor&3`` + * - ``service=`` + - Optional + - ``sqream`` + - Specifices service queue to use. For example, ``service=etl`` + * - ```` + - Optional + - ``false`` + - Specifies SSL for this connection. For example, ``ssl=true`` + * - ```` + - Optional + - ``true`` + - Connect via load balancer (use only if exists, and check port). + * - ```` + - Optional + - ``true`` + - Enables on-demand loading, and defines double buffer size for result. The ``fetchSize`` parameter is rounded according to chunk size. For example, ``fetchSize=1`` loads one row and is rounded to one chunk. If the fetchSize is 100,600, a chunk size of 100,000 loads, and is rounded to, two chunks. + * - ```` + - Optional + - ``true`` + - Defines the bytes size for inserting a buffer before flushing data to the server. Clients running a parameterized insert (network insert) can define the amount of data to collect before flushing the buffer. + * - ```` + - Optional + - ``true`` + - Defines the logger level as either ``debug`` or ``trace``. + * - ```` + - Optional + - ``true`` + - Enables the file appender and defines the file name. The file name can be set as either the file name or the file path. + +Connection String Examples +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The following is an example of a SQream cluster with load balancer and no service queues (with SSL): + +.. code-block:: text + + jdbc:Sqream://sqream.mynetwork.co:3108/master;user=rhendricks;password=Tr0ub4dor&3;ssl=true;cluster=true + +The following is a minimal example for a local standalone SQream database: + +.. code-block:: text + + jdbc:Sqream://127.0.0.1:5000/master;user=rhendricks;password=Tr0ub4dor&3 + +The following is an example of a SQream cluster with load balancer and a specific service queue named ``etl``, to the database named ``raviga`` + +.. code-block:: text + + jdbc:Sqream://sqream.mynetwork.co:3108/raviga;user=rhendricks;password=Tr0ub4dor&3;cluster=true;service=etl + +Sample Java Program +-------------------- +You can download the :download:`JDBC Application Sample File ` below by right-clicking and saving it to your computer. + +.. literalinclude:: sample.java + :language: java + :caption: JDBC Application Sample + :linenos: diff --git a/guides/client_drivers/jdbc/sample.java b/connecting_to_sqream/client_drivers/jdbc/sample.java similarity index 100% rename from guides/client_drivers/jdbc/sample.java rename to connecting_to_sqream/client_drivers/jdbc/sample.java diff --git a/guides/client_drivers/nodejs/index.rst b/connecting_to_sqream/client_drivers/nodejs/index.rst similarity index 98% rename from guides/client_drivers/nodejs/index.rst rename to connecting_to_sqream/client_drivers/nodejs/index.rst index 8ddd0d0bc..b634179ed 100644 --- a/guides/client_drivers/nodejs/index.rst +++ b/connecting_to_sqream/client_drivers/nodejs/index.rst @@ -33,7 +33,7 @@ If you need to install the driver in an offline system, see the offline method b Install from an offline package ------------------------------------- -The Node driver is provided as a tarball for download from the `SQream Drivers page `_ . +The Node driver is provided as a tarball for download from the `SQream Drivers page `_ . After downloading the tarball, use ``npm`` to install the offline package. @@ -360,6 +360,7 @@ BIGINT support The Node.JS connector supports fetching ``BIGINT`` values from SQream DB. However, some applications may encounter an error when trying to serialize those values. The error that appears is: + .. code-block:: none TypeError: Do not know how to serialize a BigInt @@ -378,5 +379,4 @@ For example: ? value.toString() : value // return everything else unchanged )); - console.log(json); // [{"test": "1"}] - + console.log(json); // [{"test": "1"}] \ No newline at end of file diff --git a/guides/client_drivers/nodejs/sample.js b/connecting_to_sqream/client_drivers/nodejs/sample.js similarity index 100% rename from guides/client_drivers/nodejs/sample.js rename to connecting_to_sqream/client_drivers/nodejs/sample.js diff --git a/guides/client_drivers/odbc/index.rst b/connecting_to_sqream/client_drivers/odbc/index.rst similarity index 100% rename from guides/client_drivers/odbc/index.rst rename to connecting_to_sqream/client_drivers/odbc/index.rst diff --git a/guides/client_drivers/odbc/install_configure_odbc_linux.rst b/connecting_to_sqream/client_drivers/odbc/install_configure_odbc_linux.rst similarity index 100% rename from guides/client_drivers/odbc/install_configure_odbc_linux.rst rename to connecting_to_sqream/client_drivers/odbc/install_configure_odbc_linux.rst diff --git a/guides/client_drivers/odbc/install_configure_odbc_windows.rst b/connecting_to_sqream/client_drivers/odbc/install_configure_odbc_windows.rst similarity index 91% rename from guides/client_drivers/odbc/install_configure_odbc_windows.rst rename to connecting_to_sqream/client_drivers/odbc/install_configure_odbc_windows.rst index ee51dc5c9..4972e3057 100644 --- a/guides/client_drivers/odbc/install_configure_odbc_windows.rst +++ b/connecting_to_sqream/client_drivers/odbc/install_configure_odbc_windows.rst @@ -1,10 +1,10 @@ .. _install_odbc_windows: **************************************** -Install and configure ODBC on Windows +Install and Configure ODBC on Windows **************************************** -The ODBC driver for Windows is provided as a self-contained installer. +The ODBC driver for Windows is provided as a self-contained installer. This tutorial shows you how to install and configure ODBC on Windows. @@ -12,7 +12,7 @@ This tutorial shows you how to install and configure ODBC on Windows. :local: :depth: 2 -Installing the ODBC driver +Installing the ODBC Driver ================================== Prerequisites @@ -23,12 +23,9 @@ Prerequisites Visual Studio 2015 Redistributables ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The ODBC driver requires Microsoft's Visual Studio 2015 Redistributable packages installed. +To install the ODBC driver you must first install Microsoft's **Visual C++ Redistributable for Visual Studio 2015**. To install Visual C++ Redistributable for Visual Studio 2015, see the `Install Instructions `_. -Follow the instructions for download and installation on Microsoft's website https://www.microsoft.com/en-us/download/details.aspx?id=48145 - - -Administrator privileges +Administrator Privileges ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The SQream DB ODBC driver requires administrator privileges on your computer to add the DSNs (data source names). @@ -43,7 +40,7 @@ Install the driver by following the on-screen instructions in the easy-to-follow .. note:: The installer will install the driver in ``C:\Program Files\SQream Technologies\ODBC Driver`` by default. This path is changable during the installation. -2. Selecting components +2. Selecting Components ^^^^^^^^^^^^^^^^^^^^^^^^^^ The installer includes additional components, like JDBC and Tableau customizations. @@ -87,7 +84,7 @@ ODBC driver configurations are done via DSNs. Each DSN represents one SQream DB -Connection parameters +Connection Parameters ----------------------- .. list-table:: @@ -134,4 +131,4 @@ After installing the ODBC driver, you may experience the following error: code 126: The specified module could not be found. (c:\Program Files\SQream Technologies\ODBC Driver\sqreamOdbc64.dll) -This is an issue with the Visual Studio Redistributable packages. Verify you've correctly installed them, as described in the :ref:`Visual Studio 2015 Redistributables ` section above. \ No newline at end of file +This is an issue with the Visual Studio Redistributable packages. Verify you've correctly installed them, as described in the :ref:`Visual Studio 2015 Redistributables ` section above. diff --git a/connecting_to_sqream/client_drivers/python/index.rst b/connecting_to_sqream/client_drivers/python/index.rst new file mode 100644 index 000000000..e03df343d --- /dev/null +++ b/connecting_to_sqream/client_drivers/python/index.rst @@ -0,0 +1,568 @@ +.. _pysqream: + +************************* +Python (pysqream) +************************* +The **Python** connector page describes the following: + +.. contents:: + :local: + :depth: 1 + +Overview +============= +The SQream Python connector is a set of packages that allows Python programs to connect to SQream DB. + +* ``pysqream`` is a pure Python connector. It can be installed with ``pip`` on any operating system, including Linux, Windows, and macOS. + +* ``pysqream-sqlalchemy`` is a SQLAlchemy dialect for ``pysqream`` + +The connector supports Python 3.6.5 and newer. The base ``pysqream`` package conforms to Python DB-API specifications `PEP-249 `_. + +Installing the Python Connector +================================== + +Prerequisites +---------------- +Installing the Python connector includes the following prerequisites: + +.. contents:: + :local: + :depth: 1 + +Python +^^^^^^^^^^^^ + +The connector requires Python 3.6.5 or newer. To verify your version of Python: + +.. code-block:: console + + $ python --version + Python 3.7.3 + + +.. note:: If both Python 2.x and 3.x are installed, you can run ``python3`` and ``pip3`` instead of ``python`` and ``pip`` respectively for the rest of this guide + +.. warning:: If you're running on an older version, ``pip`` will fetch an older version of ``pysqream``, with version <3.0.0. This version is currently not supported. + +PIP +^^^^^^^^^^^^ +The Python connector is installed via ``pip``, the Python package manager and installer. + +We recommend upgrading to the latest version of ``pip`` before installing. To verify that you are on the latest version, run the following command: + +.. code-block:: console + + $ python -m pip install --upgrade pip + Collecting pip + Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB) + |████████████████████████████████| 1.4MB 1.6MB/s + Installing collected packages: pip + Found existing installation: pip 19.1.1 + Uninstalling pip-19.1.1: + Successfully uninstalled pip-19.1.1 + Successfully installed pip-19.3.1 + +.. note:: + * On macOS, you may want to use virtualenv to install Python and the connector, to ensure compatibility with the built-in Python environment + * If you encounter an error including ``SSLError`` or ``WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.`` - please be sure to reinstall Python with SSL enabled, or use virtualenv or Anaconda. + +OpenSSL for Linux +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Some distributions of Python do not include OpenSSL. The Python connector relies on OpenSSL for secure connections to SQream DB. + +* To install OpenSSL on RHEL/CentOS + + .. code-block:: console + + $ sudo yum install -y libffi-devel openssl-devel + +* To install OpenSSL on Ubuntu + + .. code-block:: console + + $ sudo apt-get install libssl-dev libffi-dev -y + +Installing via PIP +----------------- +The Python connector is available via `PyPi `_. + +Install the connector with ``pip``: + +.. code-block:: console + + $ pip install pysqream pysqream-sqlalchemy + +``pip`` will automatically install all necessary libraries and modules. + +Upgrading an Existing Installation +-------------------------------------- +The Python drivers are updated periodically. To upgrade an existing pysqream installation, use pip's ``-U`` flag: + +.. code-block:: console + + $ pip install pysqream pysqream-sqlalchemy -U + +Validating Your Installation +----------------------------- +This section describes how to validate your installation. + +**To validate your installation**: + +1. Create a file called ``test.py``, containing the following: + +.. literalinclude:: test.py + :language: python + :caption: pysqream Validation Script + :linenos: + +2. Verify that the parameters in the connection have been replaced with your respective SQream installation parameters. + + :: + +3. Run the test file to verify that you can connect to SQream: + + .. code-block:: console + + $ python test.py + Version: v2020.1 + + If the validation was successful, you can build an application using the SQream Python connector. If you receive a connection error, verify the following: + + * You have access to a running SQream database. + + :: + + * The connection parameters are correct. + +SQLAlchemy Examples +======================== +SQLAlchemy is an **Object-Relational Mapper (ORM) for Python. When you install the SQream dialect (``pysqream-sqlalchemy``) you can use frameworks such as Pandas, TensorFlow, and Alembic to query SQream directly. + +This section includes the following examples: + +.. contents:: + :local: + :depth: 1 + +Standard Connection Example +--------------------------------- +The following is a standard connection example: + +.. code-block:: python + + import sqlalchemy as sa + from sqlalchemy.engine.url import URL + + engine_url = URL('sqream' + , username='rhendricks' + , password='secret_passwor" + , host='localhost' + , port=5000 + , database='raviga' + , query={'use_ssl': False}) + + engine = sa.create_engine(engine_url) + + res = engine.execute('create table test (ints int)') + res = engine.execute('insert into test values (5), (6)') + res = engine.execute('select * from test') + +Pulling a Table into Pandas +--------------------------------- +The following example shows how to pull a table in Pandas. This examples uses the URL method to create the connection string: + +.. code-block:: python + + import sqlalchemy as sa + import pandas as pd + from sqlalchemy.engine.url import URL + + + engine_url = URL('sqream' + , username='rhendricks' + , password='secret_passwor" + , host='localhost' + , port=5000 + , database='raviga' + , query={'use_ssl': False}) + + engine = sa.create_engine(engine_url) + + table_df = pd.read_sql("select * from nba", con=engine) + +API Examples +=============== +This section includes the following examples: + +.. contents:: + :local: + :depth: 1 + +Describing Your Connection +--------------------------------------- +This example shows how to describe the connection. + +**Describing your connection**: + +1. Import the package and create a connection: + + .. code-block:: python + + # Import pysqream package + + import pysqream + + """ + Connection parameters include: + * IP/Hostname + * Port + * database name + * username + * password + * Connect through load balancer, or direct to worker (Default: false - direct to worker) + * use SSL connection (default: false) + * Optional service queue (default: 'sqream') + """ + + # Create a connection object + + con = pysqream.connect(host='127.0.0.1', port=3108, database='raviga' + , username='rhendricks', password='Tr0ub4dor&3' + , clustered=True) + +2. Run a query and fetch the results: + + .. code-block:: python + + cur = con.cursor() # Create a new cursor + # Prepare and execute a query + cur.execute('select show_version()') + + result = cur.fetchall() # `fetchall` gets the entire data set + + print (f"Version: {result[0][0]}") + + The SQream version should be output, such as ``v2020.1``. + +3. Close the connection: + + .. code-block:: python + + con.close() + +Using the Cursor +-------------------------------------------- +The DB-API specification includes several methods for fetching results from the cursor. This sections shows an example using the ``nba`` table, which looks as follows: + +.. csv-table:: nba + :file: nba-t10.csv + :widths: auto + :header-rows: 1 + +As before, you must import the library and create a :py:meth:`~Connection`, followed by :py:meth:`~Connection.execute` on a simple ``SELECT *`` query: + +.. code-block:: python + + import pysqream + con = pysqream.connect(host='127.0.0.1', port=3108, database='master' + , username='rhendricks', password='Tr0ub4dor&3' + , clustered=True) + + cur = con.cursor() # Create a new cursor + # The select statement: + statement = 'SELECT * FROM nba' + cur.execute(statement) + +When the statement has finished executing, you have a :py:meth:`Connection` cursor object waiting. A cursor is iterable, meaning that it advances the cursor to the next row when fetched. + +You can use :py:meth:`~Connection.fetchone` to fetch one record at a time: + +.. code-block:: python + + first_row = cur.fetchone() # Fetch one row at a time (first row) + + second_row = cur.fetchone() # Fetch one row at a time (second row) + +To fetch several rows at a time, use :py:meth:`~Connection.fetchmany`: + +.. code-block:: python + + # executing `fetchone` twice is equivalent to this form: + third_and_fourth_rows = cur.fetchmany(2) + +To fetch all rows at once, use :py:meth:`~Connection.fetchall`: + +.. code-block:: python + + # To get all rows at once, use `fetchall` + remaining_rows = cur.fetchall() + + # Close the connection when done + con.close() + +The following is an example of the contents of the row variables used in our examples: + +.. code-block:: pycon + + >>> print(first_row) + ('Avery Bradley', 'Boston Celtics', 0, 'PG', 25, '6-2', 180, 'Texas', 7730337) + >>> print(second_row) + ('Jae Crowder', 'Boston Celtics', 99, 'SF', 25, '6-6', 235, 'Marquette', 6796117) + >>> print(third_and_fourth_rows) + [('John Holland', 'Boston Celtics', 30, 'SG', 27, '6-5', 205, 'Boston University', None), ('R.J. Hunter', 'Boston Celtics', 28, 'SG', 22, '6-5', 185, 'Georgia State', 1148640)] + >>> print(remaining_rows) + [('Jonas Jerebko', 'Boston Celtics', 8, 'PF', 29, '6-10', 231, None, 5000000), ('Amir Johnson', 'Boston Celtics', 90, 'PF', 29, '6-9', 240, None, 12000000), ('Jordan Mickey', 'Boston Celtics', 55, 'PF', 21, '6-8', 235, 'LSU', 1170960), ('Kelly Olynyk', 'Boston Celtics', 41, 'C', 25, '7-0', 238, 'Gonzaga', 2165160), + [...] + +.. note:: Calling a fetch command after all rows have been fetched will return an empty array (``[]``). + +Reading Result Metadata +---------------------------- +When you execute a statement, the connection object also contains metadata about the result set, such as **column names**, **types**, etc). + +The metadata is stored in the :py:attr:`Connection.description` object of the cursor: + +.. code-block:: pycon + + >>> import pysqream + >>> con = pysqream.connect(host='127.0.0.1', port=3108, database='master' + ... , username='rhendricks', password='Tr0ub4dor&3' + ... , clustered=True) + >>> cur = con.cursor() + >>> statement = 'SELECT * FROM nba' + >>> cur.execute(statement) + + >>> print(cur.description) + [('Name', 'STRING', 24, 24, None, None, True), ('Team', 'STRING', 22, 22, None, None, True), ('Number', 'NUMBER', 1, 1, None, None, True), ('Position', 'STRING', 2, 2, None, None, True), ('Age (as of 2018)', 'NUMBER', 1, 1, None, None, True), ('Height', 'STRING', 4, 4, None, None, True), ('Weight', 'NUMBER', 2, 2, None, None, True), ('College', 'STRING', 21, 21, None, None, True), ('Salary', 'NUMBER', 4, 4, None, None, True)] + +You can fetch a list of column names by iterating over the ``description`` list: + +.. code-block:: pycon + + >>> [ i[0] for i in cur.description ] + ['Name', 'Team', 'Number', 'Position', 'Age (as of 2018)', 'Height', 'Weight', 'College', 'Salary'] + +Loading Data into a Table +--------------------------- +This example shows how to load 10,000 rows of dummy data to an instance of SQream. + +**To load data 10,000 rows of dummy data to an instance of SQream:** + +1. Run the following: + + .. code-block:: python + + import pysqream + from datetime import date, datetime + from time import time + + con = pysqream.connect(host='127.0.0.1', port=3108, database='master' + , username='rhendricks', password='Tr0ub4dor&3' + , clustered=True) + +2. Create a table for loading: + + .. code-block:: python + + create = 'create or replace table perf (b bool, t tinyint, sm smallint, i int, bi bigint, f real, d double, s varchar(12), ss text, dt date, dtt datetime)' + con.execute(create) + +3. Load your data into table using the ``INSERT`` command. + + :: + +4. Create dummy data matching the table you created: + + .. code-block:: python + + data = (False, 2, 12, 145, 84124234, 3.141, -4.3, "Marty McFly" , u"キウイは楽しい鳥です" , date(2019, 12, 17), datetime(1955, 11, 4, 1, 23, 0, 0)) + + row_count = 10**4 + +5. Get a new cursor: + + .. code-block:: python + + cur = con.cursor() + insert = 'insert into perf values (?,?,?,?,?,?,?,?,?,?,?)' + start = time() + cur.executemany(insert, [data] * row_count) + print (f"Total insert time for {row_count} rows: {time() - start} seconds") + +6. Close this cursor: + + .. code-block:: python + + cur.close() + +7. Verify that the data was inserted correctly. + + :: + +8. Get a new cursor: + + .. code-block:: python + + cur = con.cursor() + cur.execute('select count(*) from perf') + result = cur.fetchall() # `fetchall` collects the entire data set + print (f"Count of inserted rows: {result[0][0]}") + +9. Close the cursor: + + .. code-block:: python + + cur.close() + +10. Close the connection: + + .. code-block:: python + + con.close() + +Reading Data from a CSV File for Loading into a Table +---------------------------------------------------------- +This example shows how to write a helper function to create an :ref:`insert` statement, by reading an existing table's metadata. + +**To read data from a CSV file for loading into a table:** + +1. Run the following: + + .. code-block:: python + + import pysqream + import datetime + + def insert_from_csv(cur, table_name, csv_filename, field_delimiter = ',', null_markers = []): + """ +2. Ask SQream for some table information. + + This is important for determining the number of columns, and helps create a matching ``INSERT`` statement: + + .. code-block:: python + + """ + + column_info = cur.execute(f"SELECT * FROM {table_name} LIMIT 0").description + + + def parse_datetime(v): + try: + return datetime.datetime.strptime(row[i], '%Y-%m-%d %H:%M:%S.%f') + except ValueError: + try: + return datetime.datetime.strptime(row[i], '%Y-%m-%d %H:%M:%S') + except ValueError: + return datetime.datetime.strptime(row[i], '%Y-%m-%d') + +3. Create enough placeholders (`?`) for the ``INSERT`` query string: + + + .. code-block:: python + + qstring = ','.join(['?'] * len(column_info)) + insert_statement = f"insert into {table_name} values ({qstring})" + +4. Open the CSV file: + + .. code-block:: python + + with open(csv_filename, mode='r') as csv_file: + csv_reader = csv.reader(csv_file, delimiter=field_delimiter) + +5. Execute the ``INSERT`` statement with the CSV data: + + .. code-block:: python + + cur.executemany(insert_statement, [row for row in csv_reader]) + + + con = pysqream.connect(host='127.0.0.1', port=3108, database='master' + , username='rhendricks', password='Tr0ub4dor&3' + , clustered=True) + + cur = con.cursor() + insert_from_csv(cur, 'nba', 'nba.csv', field_delimiter = ',', null_markers = []) + + con.close() + +Using SQLAlchemy ORM to Create and Populate Tables +----------------------------------------------------------------------- +This section shows how to use the ORM to create and populate tables from Python objects. + +**To use SQLAlchemy ORM to create and populate tables:** + +1. Run the following: + + .. code-block:: python + + import sqlalchemy as sa + import pandas as pd + from sqlalchemy.engine.url import URL + + + engine_url = URL('sqream' + , username='rhendricks' + , password='secret_passwor" + , host='localhost' + , port=5000 + , database='raviga' + , query={'use_ssl': False}) + + engine = sa.create_engine(engine_url) + +2. Build a metadata object and bind it: + + .. code-block:: python + + metadata = sa.MetaData() + metadata.bind = engine + +3. Create a table in the local metadata: + + .. code-block:: python + + employees = sa.Table( + 'employees' + , metadata + , sa.Column('id', sa.Integer) + , sa.Column('name', sa.VARCHAR(32)) + , sa.Column('lastname', sa.VARCHAR(32)) + , sa.Column('salary', sa.Float) + ) + + The ``create_all()`` function uses the SQream engine object. + +4. Create all the defined table objects: + + .. code-block:: python + + metadata.create_all(engine) + +5. Populate your table. + + :: + +6. Build the data rows: + + .. code-block:: python + + insert_data = [ {'id': 1, 'name': 'Richard','lastname': 'Hendricks', 'salary': 12000.75} + ,{'id': 3, 'name': 'Bertram', 'lastname': 'Gilfoyle', 'salary': 8400.0} + ,{'id': 8, 'name': 'Donald', 'lastname': 'Dunn', 'salary': 6500.40} + ] + +7. Build the ``INSERT`` command: + + .. code-block:: python + + ins = employees.insert(insert_data) + +8. Execute the command: + + .. code-block:: python + + result = engine.execute(ins) + +For more information, see the :ref:`python_api_reference_guide`. \ No newline at end of file diff --git a/connecting_to_sqream/client_drivers/python/nba-t10.csv b/connecting_to_sqream/client_drivers/python/nba-t10.csv new file mode 100644 index 000000000..c3d41e729 --- /dev/null +++ b/connecting_to_sqream/client_drivers/python/nba-t10.csv @@ -0,0 +1,9 @@ +Avery Bradley,Boston Celtics,0,PG,25,2-Jun,180,Texas,7730337 +Jae Crowder,Boston Celtics,99,SF,25,6-Jun,235,Marquette,6796117 +John Holland,Boston Celtics,30,SG,27,5-Jun,205,Boston University, +R.J. Hunter,Boston Celtics,28,SG,22,5-Jun,185,Georgia State,1148640 +Jonas Jerebko,Boston Celtics,8,PF,29,10-Jun,231,,5000000 +Amir Johnson,Boston Celtics,90,PF,29,9-Jun,240,,12000000 +Jordan Mickey,Boston Celtics,55,PF,21,8-Jun,235,LSU,1170960 +Kelly Olynyk,Boston Celtics,41,C,25,Jul-00,238,Gonzaga,2165160 +Terry Rozier,Boston Celtics,12,PG,22,2-Jun,190,Louisville,1824360 diff --git a/guides/client_drivers/python/test.py b/connecting_to_sqream/client_drivers/python/test.py similarity index 100% rename from guides/client_drivers/python/test.py rename to connecting_to_sqream/client_drivers/python/test.py diff --git a/guides/third_party_tools/connect.sas b/connecting_to_sqream/client_platforms/connect.sas similarity index 100% rename from guides/third_party_tools/connect.sas rename to connecting_to_sqream/client_platforms/connect.sas diff --git a/connecting_to_sqream/client_platforms/connect2.sas b/connecting_to_sqream/client_platforms/connect2.sas new file mode 100644 index 000000000..10fcdb0a2 --- /dev/null +++ b/connecting_to_sqream/client_platforms/connect2.sas @@ -0,0 +1,27 @@ +options sastrace='d,d,d,d' +sastraceloc=saslog +nostsuffix +msglevel=i +sql_ip_trace=(note,source) +DEBUG=DBMS_SELECT; + +options validvarname=any; + +libname sqlib jdbc driver="com.sqream.jdbc.SQDriver" + classpath="/opt/sqream/sqream-jdbc-4.0.0.jar" + URL="jdbc:Sqream://sqream-cluster.piedpiper.com:3108/raviga;cluster=true" + user="rhendricks" + password="Tr0ub4dor3" + schema="public" + PRESERVE_TAB_NAMES=YES + PRESERVE_COL_NAMES=YES; + +proc sql; + title 'Customers table'; + select * + from sqlib.customers; +quit; + +data sqlib.customers; + set sqlib.customers; +run; \ No newline at end of file diff --git a/connecting_to_sqream/client_platforms/connect3.sas b/connecting_to_sqream/client_platforms/connect3.sas new file mode 100644 index 000000000..c1bf11dcf --- /dev/null +++ b/connecting_to_sqream/client_platforms/connect3.sas @@ -0,0 +1,17 @@ +options sastrace='d,d,d,d' +sastraceloc=saslog +nostsuffix +msglevel=i +sql_ip_trace=(note,source) +DEBUG=DBMS_SELECT; + +options validvarname=any; + +libname sqlib jdbc driver="com.sqream.jdbc.SQDriver" + classpath="/opt/sqream/sqream-jdbc-4.0.0.jar" + URL="jdbc:Sqream://sqream-cluster.piedpiper.com:3108/raviga;cluster=true" + user="rhendricks" + password="Tr0ub4dor3" + schema="public" + PRESERVE_TAB_NAMES=YES + PRESERVE_COL_NAMES=YES; \ No newline at end of file diff --git a/guides/third_party_tools/index.rst b/connecting_to_sqream/client_platforms/index.rst similarity index 81% rename from guides/third_party_tools/index.rst rename to connecting_to_sqream/client_platforms/index.rst index a3335d463..15047ab54 100644 --- a/guides/third_party_tools/index.rst +++ b/connecting_to_sqream/client_platforms/index.rst @@ -1,10 +1,9 @@ -.. _third_party_tools: +.. _client_platforms: ************************************ -Third party tools +Client Platforms ************************************ - -These topics explain how to install and connect a variety of third party tools. +These topics explain how to install and connect a variety of third party tools. Browse the articles below, in the sidebar, or use the search to find the information you need. @@ -22,15 +21,18 @@ If you are looking for a tool that is not listed, SQream and our partners can he :caption: In this section: :titlesonly: - sql_workbench - tableau - r + informatica + microstrategy + pentaho php - xxspotfire + power_bi + r + sap_businessobjects sas_viya - xxtalend + sql_workbench + tableau + talend + tibco_spotfire xxdiagnosing_common_connectivity_issues -.. image:: /_static/images/connectivity_ecosystem.png - - +.. image:: /_static/images/connectivity_ecosystem.png \ No newline at end of file diff --git a/connecting_to_sqream/client_platforms/informatica.rst b/connecting_to_sqream/client_platforms/informatica.rst new file mode 100644 index 000000000..6bc50b22a --- /dev/null +++ b/connecting_to_sqream/client_platforms/informatica.rst @@ -0,0 +1,173 @@ +.. _informatica: + +************************* +Connect to SQream Using Informatica Cloud Services +************************* + +Overview +========= +The **Connecting to SQream Using Informatica Cloud Services** page is quick start guide for connecting to SQream using Informatica cloud services. + +It describes the following: + +.. contents:: + :local: + +Establishing a Connection between SQream and Informatica +----------------- +The **Establishing a Connection between SQream and Informatica** page describes how to establish a connection between SQream and the Informatica data integration Cloud. + +**To establish a connection between SQream and the Informatica data integration Cloud:** + +1. Go to the `Informatica Cloud homepage `_. + + :: + +2. Do one of the following: + + 1. Log in using your credentials. + + :: + + 2. Log in using your SAML Identity Provider. + +3. From the **Services** window, select **Administrator** or click **Show all services** to show all services. + + + The SQream dashboard is displayed. + + + :: + + +4. In the menu on the left, click **Runtime Environments**. + + + The **Runtime Environments** panel is displayed. + + :: + +5. Click **Download Secure Agent**. + + :: + +6. When the **Download the Secure Agent** panel is displayed, do the following: + + 1. Select a platform (Windows 64 or Linux 64). + + :: + + + 2. Click **Copy** and save the token on your local hard drive. + + The token is used in combination with your user name to authorize the agent to access your account. + + +7. Click **Download**. + + The installation begins. + + :: + +8. When the **Informatica Cloud Secure Agent Setup** panel is displayed, click **Next**. + + + :: + + +9. Provide your **User Name** and **Install Token** and click **Register**. + + :: + + + +10. From the Runtime Environments panel, click **New Runtime Environment**. + + + The **New Secure Agent Group** window is displayed. + + :: + +11. On the New Secure Agent Group window, click **OK** to connect your Runtime Environment with the running agent. + + .. note:: If you do not download Secure Agent, you will not be able to connect your Runtime Environment with the running agent and continue establishing a connection between SQream and the Informatica data integration Cloud. + +Establishing a Connection In Your Environment +----------------- + +The **Establishing a Connection In Your Environment** describes the following: + +.. contents:: + :local: + +Establishing an ODBC DSN Connection In Your Environment +~~~~~~~~~~~~~ +After establishing a connection between SQream and Informatica you can establish an ODBC DSN connection in your environment. + +**To establish an ODBC connection in your environment:** + +1. Click **Add**. + + :: + +2. Click **Configure**. + + .. note:: Verify that **Use Server Picker** is selected. + +3. Click **Test**. + + :: + +4. Verify that the connection has tested successfully. + + :: + +5. Click **Save**. + + :: + +6. Click **Actions** > **Publish**. + +Establishing a JDBC Connection In Your Environment +~~~~~~~~~~~~~ +After establishing a connection between SQream and Informatica you can establish a JDBC connection in your environment. + +**To establish a JDBC connection in your environment:** + +1. Create a new DB connection by clicking **Connections** > **New Connection**. + + The **New Connection** window is displayed. + + :: + + +2. In the **JDBC_IC Connection Properties** section, in the **JDBC Connection URL** field, establish a JDBC connection by providing the correct connection string. + + For connection string examples, see `Connection Strings `_. + + :: + +3. Click **Test**. + + :: + +4. Verify that the connection has tested successfully. + + :: + +5. Click **Save**. + + :: + +6. Click **Actions** > **Publish**. + +Supported SQream Driver Versions +--------------- + +SQream supports the following SQream driver versions: + +* **JDBC** - Version 4.3.4 and above. + + :: + +* **ODBC** - Version 4.0.0 and above. diff --git a/connecting_to_sqream/client_platforms/microstrategy.rst b/connecting_to_sqream/client_platforms/microstrategy.rst new file mode 100644 index 000000000..6cad19be2 --- /dev/null +++ b/connecting_to_sqream/client_platforms/microstrategy.rst @@ -0,0 +1,185 @@ +.. _microstrategy: + + +************************* +Connect to SQream Using MicroStrategy +************************* + +.. _ms_top: + +Overview +--------------- +This document is a Quick Start Guide that describes how to install MicroStrategy and connect a datasource to the MicroStrategy dasbhoard for analysis. + + + +The **Connecting to SQream Using MicroStrategy** page describes the following: + + +.. contents:: + :local: + + + + + + +What is MicroStrategy? +================ +MicroStrategy is a Business Intelligence software offering a wide variety of data analytics capabilities. SQream uses the MicroStrategy connector for reading and loading data into SQream. + +MicroStrategy provides the following: + +* Data discovery +* Advanced analytics +* Data visualization +* Embedded BI +* Banded reports and statements + + +For more information about Microstrategy, see `MicroStrategy `_. + + + +:ref:`Back to Overview ` + + + + + +Connecting a Data Source +======================= + +1. Activate the **MicroStrategy Desktop** app. The app displays the Dossiers panel to the right. + + :: + +2. Download the most current version of the `SQream JDBC driver `_. + + :: + +3. Click **Dossiers** and **New Dossier**. The **Untitled Dossier** panel is displayed. + + :: + +4. Click **New Data**. + + :: + +5. From the **Data Sources** panel, select **Databases** to access data from tables. The **Select Import Options** panel is displayed. + + :: + +6. Select one of the following: + + * Build a Query + * Type a Query + * Select Tables + + :: + +7. Click **Next**. + + :: + +8. In the Data Source panel, do the following: + + 1. From the **Database** dropdown menu, select **Generic**. The **Host Name**, **Port Number**, and **Database Name** fields are removed from the panel. + + :: + + 2. In the **Version** dropdown menu, verify that **Generic DBMS** is selected. + + :: + + 3. Click **Show Connection String**. + + :: + + 4. Select the **Edit connection string** checkbox. + + :: + + 5. From the **Driver** dropdown menu, select a driver for one of the following connectors: + + * **JDBC** - The SQream driver is not integrated with MicroStrategy and does not appear in the dropdown menu. However, to proceed, you must select an item, and in the next step you must specify the path to the SQream driver that you installed on your machine. + * **ODBC** - SQreamDB ODBC + + :: + + 6. In the **Connection String** text box, type the relevant connection string and path to the JDBC jar file using the following syntax: + + .. code-block:: console + + $ jdbc:Sqream:///;user=;password=sqream;[; ...] + + The following example shows the correct syntax for the JDBC connector: + + .. code-block:: console + + jdbc;MSTR_JDBC_JAR_FOLDER=C:\path\to\jdbc\folder;DRIVER=;URL={jdbc:Sqream:///;user=;password=;[; ...];} + + The following example shows the correct syntax for the ODBC connector: + + .. code-block:: console + + odbc:Driver={SqreamODBCDriver};DSN={SQreamDB ODBC};Server=;Port=;Database=;User=;Password=;Cluster=; + + For more information about the available **connection parameters** and other examples, see `Connection Parameters `_. + + 7. In the **User** and **Password** fields, fill out your user name and password. + + :: + + 8. In the **Data Source Name** field, type **SQreamDB**. + + :: + + 9. Click **Save**. The SQreamDB that you picked in the Data Source panel is displayed. + + +9. In the **Namespace** menu, select a namespace. The tables files are displayed. + + :: + +10. Drag and drop the tables into the panel on the right in your required order. + + :: + +11. **Recommended** - Click **Prepare Data** to customize your data for analysis. + + :: + +12. Click **Finish**. + + :: + +13. From the **Data Access Mode** dialog box, select one of the following: + + + * Connect Live + * Import as an In-memory Dataset + +Your populated dashboard is displayed and is ready for data discovery and analytics. + + + + + + +.. _supported_sqream_drivers: + +:ref:`Back to Overview ` + +Supported SQream Drivers +================ + +The following list shows the supported SQream drivers and versions: + +* **JDBC** - Version 4.3.3 and higher. +* **ODBC** - Version 4.0.0. + + +.. _supported_tools_and_operating_systems: + +:ref:`Back to Overview ` diff --git a/guides/third_party_tools/odbc-sqream.tdc b/connecting_to_sqream/client_platforms/odbc-sqream.tdc similarity index 100% rename from guides/third_party_tools/odbc-sqream.tdc rename to connecting_to_sqream/client_platforms/odbc-sqream.tdc diff --git a/connecting_to_sqream/client_platforms/pentaho.rst b/connecting_to_sqream/client_platforms/pentaho.rst new file mode 100644 index 000000000..fa8146c41 --- /dev/null +++ b/connecting_to_sqream/client_platforms/pentaho.rst @@ -0,0 +1,253 @@ +.. _pentaho_data_integration: + +************************* +Connecting to SQream Using Pentaho Data Integration +************************* +.. _pentaho_top: + +Overview +========= +This document is a Quick Start Guide that describes how to install Pentaho, create a transformation, and define your output. + +The Connecting to SQream Using Pentaho page describes the following: + +* :ref:`Installing Pentaho ` +* :ref:`Installing and setting up the JDBC driver ` +* :ref:`Creating a transformation ` +* :ref:`Defining your output ` +* :ref:`Importing your data ` + +.. _install_pentaho: + +Installing Pentaho +~~~~~~~~~~~~~~~~~ +To install PDI, see the `Pentaho Community Edition (CE) Installation Guide `_. + +The **Pentaho Community Edition (CE) Installation Guide** describes how to do the following: + +* Downloading the PDI software. +* Installing the **JRE (Java Runtime Environment)** and **JDK (Java Development Kit)**. +* Setting up the JRE and JDK environment variables for PDI. + +:ref:`Back to Overview ` + +.. _install_set_up_jdbc_driver: + +Installing and Setting Up the JDBC Driver +~~~~~~~~~~~~~~~~~ +After installing Pentaho you must install and set up the JDBC driver. This section explains how to set up the JDBC driver using Pentaho. These instructions use Spoon, the graphical transformation and job designer associated with the PDI suite. + +You can install the driver by copying and pasting the SQream JDBC .jar file into your **/design-tools/data-integration/lib** directory. + +**NOTE:** Contact your SQream license account manager for the JDBC .jar file. + +:ref:`Back to Overview ` + +.. _create_transformation: + +Creating a Transformation +~~~~~~~~~~~~~~~~~~ +After installing Pentaho you can create a transformation. + +**To create a transformation:** + +1. Use the CLI to open the PDI client for your operating system (Windows): + + .. code-block:: console + + $ spoon.bat + +2. Open the spoon.bat file from its folder location. + +:: + +3. In the **View** tab, right-click **Transformations** and click **New**. + + A new transformation tab is created. + +4. In the **Design** tab, click **Input** to show its file contents. + +:: + +5. Drag and drop the **CSV file input** item to the new transformation tab that you created. + +:: + +6. Double-click **CSV file input**. The **CSV file input** panel is displayed. + +:: + +7. In the **Step name** field, type a name. + +:: + +8. To the right of the **Filename** field, click **Browse**. + +:: + +9. Select the file that you want to read from and click **OK**. + +:: + +10. In the CSV file input window, click **Get Fields**. + +:: + +11. In the **Sample data** window, enter the number of lines you want to sample and click **OK**. The default setting is **100**. + + The tool reads the file and suggests the field name and type. + +12. In the CSV file input window, click **Preview**. + +:: + +13. In the **Preview size** window, enter the number of rows you want to preview and click **OK**. The default setting is **1000**. + +:: + +14. Verify that the preview data is correct and click **Close**. + +:: + +15. Click **OK** in the **CSV file input** window. + +:ref:`Back to Overview ` + +.. _define_output: + +Defining Your Output +----------------- +After creating your transformation you must define your output. + +**To define your output:** + +1. In the **Design** tab, click **Output**. + + The Output folder is opened. + +2. Drag and drop **Table output** item to the Transformation window. + +:: + +3. Double-click **Table output** to open the **Table output** dialog box. + +:: + +4. From the **Table output** dialog box, type a **Step name** and click **New** to create a new connection. Your **steps** are the building blocks of a transformation, such as file input or a table output. + + The **Database Connection** window is displayed with the **General** tab selected by default. + +5. Enter or select the following information in the Database Connection window and click **Test**. + + The following table shows and describes the information that you need to fill out in the Database Connection window: + + .. list-table:: + :widths: 6 31 73 + :header-rows: 1 + + * - No. + - Element Name + - Description + * - 1 + - Connection name + - Enter a name that uniquely describes your connection, such as **sampledata**. + * - 2 + - Connection type + - Select **Generic database**. + * - 3 + - Access + - Select **Native (JDBC)**. + * - 4 + - Custom connection URL + - Insert **jdbc:Sqream:///;user=;password=;[; ...];**. The IP is a node in your SQream cluster and is the name or schema of the database you want to connect to. Verify that you have not used any leading or trailing spaces. + * - 5 + - Custom driver class name + - Insert **com.sqream.jdbc.SQDriver**. Verify that you have not used any leading or trailing spaces. + * - 6 + - Username + - Your SQreamdb username. If you leave this blank, you will be prompted to provide it when you connect. + * - 7 + - Password + - Your password. If you leave this blank, you will be prompted to provide it when you connect. + + The following message is displayed: + +.. image:: /_static/images/third_party_connectors/pentaho/connection_tested_successfully_2.png + +6. Click **OK** in the window above, in the Database Connection window, and Table Output window. + +:ref:`Back to Overview ` + +.. _import_data: + +Importing Data +----------------- +After defining your output you can begin importing your data. + +For more information about backing up users, permissions, or schedules, see `Backup and Restore Pentaho Repositories `_ + +**To import data:** + +1. Double-click the **Table output** connection that you just created. + +:: + +2. To the right of the **Target schema** field, click **Browse** and select a schema name. + +:: + +3. Click **OK**. The selected schema name is displayed in the **Target schema** field. + +:: + +4. Create a new hop connection between the **CSV file input** and **Table output** steps: + + 1. On the CSV file input step item, click the **new hop connection** icon. + + .. image:: /_static/images/third_party_connectors/pentaho/csv_file_input_options.png + + 2. Drag an arrow from the **CSV file input** step item to the **Table output** step item. + + .. image:: /_static/images/third_party_connectors/pentaho/csv_file_input_options_2.png + + 3. Release the mouse button. The following options are displayed. + + :: + + 4. Select **Main output of step**. + + .. image:: /_static/images/third_party_connectors/pentaho/main_output_of_step.png + +:: + +5. Double-click **Table output** to open the **Table output** dialog box. + +:: + +6. In the **Target table** field, define a target table name. + +:: + +7. Click **SQL** to open the **Simple SQL editor.** + +:: + +8. In the **Simple SQL editor**, click **Execute**. + + The system processes and displays the results of the SQL statements. + +9. Close all open dialog boxes. + +:: + +10. Click the play button to execute the transformation. + + .. image:: /_static/images/third_party_connectors/pentaho/execute_transformation.png + + The **Run Options** dialog box is displayed. + +11. Click **Run**. + + The **Execution Results** are displayed. + +:ref:`Back to Overview ` \ No newline at end of file diff --git a/connecting_to_sqream/client_platforms/php.rst b/connecting_to_sqream/client_platforms/php.rst new file mode 100644 index 000000000..4410a8e0a --- /dev/null +++ b/connecting_to_sqream/client_platforms/php.rst @@ -0,0 +1,74 @@ +.. _php: + +***************************** +Connect to SQream Using PHP +***************************** + +Overview +========== +PHP is an open source scripting language that executes scripts on servers. The **Connect to PHP** page explains how to connect to a SQream cluster, and describes the following: + +.. contents:: + :local: + :depth: 1 + +Installing PHP +------------------- +**To install PHP:** + +1. Download the JDBC driver installer from the `SQream Drivers page `_. + + :: + +2. Create a DSN. + + :: + +3. Install the **uODBC** extension for your PHP installation. + + For more information, navigate to `PHP Documentation `_ and see the topic menu on the right side of the page. + +Configuring PHP +------------------- +You can configure PHP in one of the following ways: + +* When compiling, configure PHP to enable uODBC using ``./configure --with-pdo-odbc=unixODBC,/usr/local``. + + :: + +* Install ``php-odbc`` and ``php-pdo`` along with PHP using your distribution package manager. SQream recommends a minimum of version 7.1 for the best results. + +.. note:: PHP's string size limitations truncates fetched text, which you can override by doing one of the following: + + * Increasing the **php.ini** default setting, such as the *odbc.defaultlrl* to **10000**. + + :: + + * Setting the size limitation in your code before making your connection using **ini_set("odbc.defaultlrl", "10000");**. + + :: + + * Setting the size limitation in your code before fetchng your result using **odbc_longreadlen($result, "10000");**. + +Operating PHP +------------------- +After configuring PHP, you can test your connection. + +**To test your connection:** + +#. Create a test connection file using the correct parameters for your SQream installation, as shown below: + + .. literalinclude:: test.php + :language: php + :emphasize-lines: 4 + :linenos: + + The following is an example of a valid DSN line: + + .. code:: php + + $dsn = "odbc:Driver={SqreamODBCDriver};Server=192.168.0.5;Port=5000;Database=master;User=rhendricks;Password=super_secret;Service=sqream"; + +#. Run the PHP file either directly with PHP (``php test.php``) or through a browser. + + For more information about supported DSN parameters, see :ref:`dsn_params`. \ No newline at end of file diff --git a/connecting_to_sqream/client_platforms/power_bi.rst b/connecting_to_sqream/client_platforms/power_bi.rst new file mode 100644 index 000000000..da788cc6e --- /dev/null +++ b/connecting_to_sqream/client_platforms/power_bi.rst @@ -0,0 +1,143 @@ +.. _power_bi: + +************************* +Connect to SQream Using Power BI Desktop +************************* + +Overview +========= +**Power BI Desktop** lets you connect to SQream and use underlying data as with other data sources in Power BI Desktop. + +SQream integrates with Power BI Desktop to do the following: + +* Extract and transform your datasets into usable visual models in approximately one minute. + + :: + +* Use **DAX** functions **(Data Analysis Expressions)** to analyze your datasets. + + :: + +* Refresh datasets as needed or by using scheduled jobs. + +SQream uses Power BI for extracting data sets using the following methods: + +* **Direct query** - Direct queries lets you connect easily with no errors, and refreshes Power BI artifacts, such as graphs and reports, in a considerable amount of time in relation to the time taken for queries to run using the `SQream SQL CLI Reference guide `_. + + :: + +* **Import** - Lets you extract datasets from remote databases. + +The **Connect to SQream Using Power BI** page describes the following: + +.. contents:: + :local: + :depth: 1 + +Prerequisites +------------------- +To connect to SQream, the following must be installed: + +* **ODBC data source administrator** - 32 or 64, depending on your operating system. For Windows users, the ODBC data source administrator is embedded within the operating system. + +* **SQream driver** - The SQream application required for interacting with the ODBC according to the configuration specified in the ODBC administrator tool. + +Installing Power BI Desktop +------------------- +**To install Power BI Desktop:** + +1. Download `Power BI Desktop 64x `_. + + :: + +2. Download and configure your ODBC driver. + + For more information about configuring your ODBC driver, see `ODBC `_. + +3. Navigate to **Windows** > **Documents** and create a folder called **Power BI Desktop Custom Connectors**. + + :: + +4. In the **Power BI Desktop** folder, create a folder called **Custom Connectors**. + + +5. From the Client Drivers page, download the **PowerQuery.mez** file. + + :: + +5. Save the PowerQuery.mez file in the **Custom Connectors** folder you created in Step 3. + + :: + +6. Open the Power BI application. + + :: + +7. Navigate to **File** > **Options and Settings** > **Option** > **Security** > **Data Extensions**, and select **(Not Recommended) Allow any extension to load without validation or warning**. + + :: + +8. Restart the Power BI Desktop application. + + :: + +9. From the **Get Data** menu, select **SQream**. + + :: + +10. Click **Connect** and provide the information shown in the following table: + + .. list-table:: + :widths: 6 31 + :header-rows: 1 + + * - Element Name + - Description + * - Server + - Provide the network address to your database server. You can use a hostname or an IP address. + * - Port + - Provide the port that the database is responding to at the network address. + * - Database + - Provide the name of your database or the schema on your database server. + * - User + - Provide a SQreamdb username. + * - Passwords + - Provide a password for your user. + +11. Under **Data Connectivity mode**, select **DirectQuery mode**. + + :: + +12. Click **Connect**. + + :: + +13. Provide your user name and password and click **Connect**. + +Best Practices for Power BI +--------------- +SQream recommends using Power BI in the following ways for acquiring the best performance metrics: + +* Creating bar, pie, line, or plot charts when illustrating one or more columns. + + :: + +* Displaying trends and statuses using visual models. + + :: + +* Creating a unified view using **PowerQuery** to connect different data sources into a single dashboard. + +Supported SQream Driver Versions +--------------- +SQream supports the following SQream driver versions: + +* The **PowerQuery Connector** is an additional layer on top of the ODBC. + + :: + +* SQream Driver Installation (ODBC v4.1.1) - Contact your administrator for the link to download ODBC v4.1.1. + +Related Information +------------------- +For more information, see the `Glossary `_. \ No newline at end of file diff --git a/guides/third_party_tools/r.rst b/connecting_to_sqream/client_platforms/r.rst similarity index 98% rename from guides/third_party_tools/r.rst rename to connecting_to_sqream/client_platforms/r.rst index 7d9e7659f..c84bf901b 100644 --- a/guides/third_party_tools/r.rst +++ b/connecting_to_sqream/client_platforms/r.rst @@ -1,12 +1,12 @@ .. _r: ***************************** -Connecting with R +Connect to SQream Using R ***************************** You can use R to interact with a SQream DB cluster. -This tutorial is a guide that will show you how to connect R to SQream DB. +This tutorial is a guide that will show you how to connect R to SQream DB. .. contents:: In this topic: :local: @@ -148,4 +148,4 @@ A full example 3 3 91 22356 3 4 4 84 17232 4 5 5 13 14315 5 - > close(ch) \ No newline at end of file + > close(ch) diff --git a/connecting_to_sqream/client_platforms/sap_businessobjects.rst b/connecting_to_sqream/client_platforms/sap_businessobjects.rst new file mode 100644 index 000000000..285c8de5e --- /dev/null +++ b/connecting_to_sqream/client_platforms/sap_businessobjects.rst @@ -0,0 +1,60 @@ +.. _sap_businessobjects: + +************************* +Connecting to SQream Using SAP BusinessObjects +************************* +The **Connecting to SQream Using SAP BusinessObjects** guide includes the following sections: + +.. contents:: + :local: + :depth: 1 + +Overview +========== +The **Connecting to SQream Using SAP BusinessObjects** guide describes the best practices for configuring a connection between SQream and the SAP BusinessObjects BI platform. SAP BO's multi-tier architecture includes both client and server components, and this guide describes integrating SQream with SAP BO's object client tools using a generic JDBC connector. The instructions in this guide are relevant to both the **Universe Design Tool (UDT)** and the **Information Design Tool (IDT)**. This document only covers how to establish a connection using the generic out-of-the-box JDBC connectors, and does not cover related business object products, such as the **Business Objects Data Integrator**. + +The **Define a new connection** window below shows the generic JDBC driver, which you can use to establish a new connection to a database. + +.. image:: /_static/images/SAP_BO_2.png + +SAP BO also lets you customize the interface to include a SQream data source. + +Establising a New Connection Using a Generic JDCB Connector +========== +This section shows an example of using a generic JDBC connector to establish a new connection. + +**To establish a new connection using a generic JDBC connector:** + +1. In the fields, provide a user name, password, database URL, and JDBC class. + + The following is the correct format for the database URL: + + .. code-block:: console + +
jdbc:Sqream://:3108/
+	  
+   SQream recommends quickly testing your connection to SQream by selecting the Generic JDBC data source in the **Define a new connection** window. When you connect using a generic JDBC data source you do not need to modify your configuration files, but are limited to the out-of-the-box settings defined in the default **jdbc.prm** file.
+   
+   .. note:: Modifying the jdbc.prm file for the generic driver impacts all other databases using the same driver.
+
+   For more information, see `Connection String Examples `_.
+
+2. (Optonal)If you are using the generic JDBC driver specific to SQream, modify the jdbc.sbo file to include the SQream JDBC driver location by adding the following lines under the Database section of the file:
+
+   .. code-block:: console
+
+      Database Active="Yes" Name="SQream JDBC data source">
+      
+      
+      C:\Program Files\SQream Technologies\JDBC Driver\2021.2.0-4.5.3\sqream-jdbc-4.5.3.jar
+      
+      
+      
+      com.sqream.jdbc.SQDriver
+
+      
+      
+
+3. Restart the BusinessObjects server.
+
+   When the connection is established, **SQream** is listed as a driver selection.
\ No newline at end of file
diff --git a/guides/third_party_tools/sas_viya.rst b/connecting_to_sqream/client_platforms/sas_viya.rst
similarity index 98%
rename from guides/third_party_tools/sas_viya.rst
rename to connecting_to_sqream/client_platforms/sas_viya.rst
index 50f79c28b..744d4b771 100644
--- a/guides/third_party_tools/sas_viya.rst
+++ b/connecting_to_sqream/client_platforms/sas_viya.rst
@@ -1,10 +1,10 @@
 .. _connect_to_sas_viya:
 
 *************************
-Connecting to SAS Viya
+Connect to SQream Using SAS Viya
 *************************
 
-You can use SAS Viya to connect to a SQream DB cluster. This tutorial is a guide that will show you how to connect to SAS Viya.
+You can use SAS Viya to connect to a SQream DB cluster. This tutorial is a guide that will show you how to connect to SAS Viya. 
 
 .. contents:: In this topic:
    :local:
diff --git a/guides/third_party_tools/sql_workbench.rst b/connecting_to_sqream/client_platforms/sql_workbench.rst
similarity index 91%
rename from guides/third_party_tools/sql_workbench.rst
rename to connecting_to_sqream/client_platforms/sql_workbench.rst
index 5329527e8..6fe9713af 100644
--- a/guides/third_party_tools/sql_workbench.rst
+++ b/connecting_to_sqream/client_platforms/sql_workbench.rst
@@ -1,10 +1,10 @@
 .. _connect_to_sql_workbench:
 
 *****************************
-Connecting to SQL Workbench
+Connect to SQream Using SQL Workbench
 *****************************
 
-You can use SQL Workbench to interact with a SQream DB cluster. SQL Workbench/J is a free SQL query tool, and is designed to run on any JRE-enabled environment.
+You can use SQL Workbench to interact with a SQream DB cluster. SQL Workbench/J is a free SQL query tool, and is designed to run on any JRE-enabled environment. 
 
 This tutorial is a guide that will show you how to connect SQL Workbench to SQream DB.
 
@@ -16,7 +16,7 @@ Installing SQL Workbench with the SQream DB installer (Windows only)
 
 SQream DB's driver installer for Windows can install the Java prerequisites and SQL Workbench for you.
 
-#. Get the JDBC driver installer available for download from the `SQream Drivers page `_. The Windows installer takes care of the Java prerequisites and subsequent configuration.
+#. Get the JDBC driver installer available for download from the `SQream Drivers page `_. The Windows installer takes care of the Java prerequisites and subsequent configuration.
 
 #. Install the driver by following the on-screen instructions in the easy-to-follow installer.
    By default, the installer does not install SQL Workbench. Make sure to select the item!
@@ -54,7 +54,7 @@ For Windows, SQream recommends Zulu 8 https://www.azul.com/downloads/zulu-commun
 Get the SQream DB JDBC driver
 -------------------------------
 
-SQream DB's JDBC driver is provided as a zipped JAR file, available for download from the `SQream Drivers page `_. 
+SQream DB's JDBC driver is provided as a zipped JAR file, available for download from the `SQream Drivers page `_. 
 
 Download and extract the JAR file from the zip archive.
 
@@ -132,4 +132,4 @@ If you installed SQL Workbench manually, you can set a customization to help SQL
       
       workbench.db.sqreamdb.schema.retrieve.change.catalog=true
 
-#. Save the file and restart SQL Workbench
\ No newline at end of file
+#. Save the file and restart SQL Workbench
diff --git a/connecting_to_sqream/client_platforms/tableau.rst b/connecting_to_sqream/client_platforms/tableau.rst
new file mode 100644
index 000000000..1d2ca17b6
--- /dev/null
+++ b/connecting_to_sqream/client_platforms/tableau.rst
@@ -0,0 +1,215 @@
+.. _tableau:
+
+*************************
+Connecting to SQream Using Tableau
+*************************
+
+Overview
+=====================
+SQream's Tableau connector plugin, based on standard JDBC, enables storing and fast querying large volumes of data.
+
+The **Connecting to SQream Using Tableau** page is a Quick Start Guide that describes how install Tableau and the JDBC driver and connect to SQream for data analysis. It also describes using best practices and troubleshoot issues that may occur while installing Tableau. SQream supports both Tableau Desktop and Tableau Server on Windows, MacOS, and Linux distributions.
+
+For more information on SQream's integration with Tableau, see `Tableau's Extension Gallery `_.
+
+The Connecting to SQream Using Tableau page describes the following:
+
+.. contents::
+   :local:
+   :depth: 1
+
+Installing the JDBC Driver and Tableau Connector Plugin
+-------------------
+This section describes how to install the JDBC driver using the fully-integrated Tableau connector plugin (Tableau Connector, or **.taco** file). SQream has been tested with Tableau versions 9.2 and newer.
+
+You can connect to SQream using Tableau by doing one of the following:
+
+   * **For MacOS or Linux** - See :ref:`Installing the JDBC Driver `.
+
+.. _tableau_jdbc_installer:
+   
+Installing the JDBC Driver
+-------------------
+If you are using MacOS, Linux, or the Tableau server, after installing the Tableau Desktop application you can install the JDBC driver manually. When the driver is installed, you can connect to SQream.
+
+**To install the JDBC driver:**
+
+1. Download the JDBC installer and SQream Tableau connector (.taco) file from the :ref:`from the client drivers page`.
+
+    ::
+
+2. Based on your operating system, your Tableau driver directory is located in one of the following places:
+
+   * **Tableau Desktop on MacOS:** *~/Library/Tableau/Drivers*
+   
+      ::
+	  
+   * **Tableau Desktop on Windows:** *C:\\Program Files\\Tableau\\Drivers*
+      
+      ::
+   
+   * **Tableau on Linux**: */opt/tableau/tableau_driver/jdbc*
+	  
+   Note the following when installing the JDBC driver:
+
+   * You must have read permissions on the .jar file.
+   
+      ::
+	  
+   * Tableau requires a JDBC 4.0 or later driver.
+   
+      ::
+	  
+   * Tableau requires a Type 4 JDBC driver.
+   
+      ::
+	  
+   * The latest 64-bit version of Java 8 is installed.
+
+3. Install the **SQreamDB.taco** file by moving the SQreamDB.taco file into the Tableau connectors directory.
+   
+   Based on the installation method that you used, your Tableau driver directory is located in one of the following places:
+
+   * **Tableau Desktop on Windows:** *C:\\Users\\\\My Tableau Repository\\Connectors*
+   
+      ::
+	  
+   * **Tableau Desktop on MacOS:** *~/My Tableau Repository/Connectors*
+
+You can now restart Tableau Desktop or Server to begin using the SQream driver by connecting to SQream as described in the section below.
+
+Connecting to SQream
+---------------------
+After installing the JDBC driver you can connect to SQream.
+
+**To connect to SQream:**
+
+#. Start Tableau Desktop.
+
+    ::
+	
+#. In the **Connect** menu, in the **To a Server** sub-menu, click **More...**.
+
+   More connection options are displayed.
+
+    ::
+	
+#. Select **SQream DB by SQream Technologies**.
+
+   The **New Connection** dialog box is displayed.
+
+    ::
+	
+#. In the New Connection dialog box, fill in the fields and click **Sign In**.
+
+  The following table describes the fields:
+   
+  .. list-table:: 
+     :widths: 15 38 38
+     :header-rows: 1
+   
+     * - Item
+       - Description
+       - Example
+     * - Server
+       - Defines the server of the SQream worker.
+       - ``127.0.0.1`` or ``sqream.mynetwork.co``
+     * - Port
+       - Defines the TCP port of the SQream worker.
+       - ``3108`` when using a load balancer, or ``5100`` when connecting directly to a worker with SSL.
+     * - Database
+       - Defines the database to establish a connection with.
+       - ``master``
+     * - Cluster
+       - Enables (``true``) or disables (``false``) the load balancer. After enabling or disabling the load balance, verify the connection.
+       - 
+     * - Username
+       - Specifies the username of a role to use when connecting.
+       - ``rhendricks``	 
+     * - Password
+       - Specifies the password of the selected role.
+       - ``Tr0ub4dor&3``
+     * - Require SSL (recommended)
+       - Sets SSL as a requirement for establishing this connection.
+       - 
+
+The connection is established and the data source page is displayed.
+  
+.. _set_up_sqream_tables_as_data_sources:
+
+Setting Up SQream Tables as Data Sources
+----------------
+After connecting to SQream you must set up the SQream tables as data sources.
+
+**To set up SQream tables as data sources:**
+	
+1. From the **Table** menu, select the desired database and schema.
+
+   SQream's default schema is **public**.
+   
+    ::
+	
+#. Drag the desired tables into the main area (labeled **Drag tables here**).
+
+   This area is also used for specifying joins and data source filters.
+   
+    ::
+	
+#. Open a new sheet to analyze data. 
+
+Tableau Best Practices and Troubleshooting
+---------------
+This section describes the following best practices and troubleshooting procedures when connecting to SQream using Tableau:
+
+.. contents::
+   :local:
+
+Using Tableau's Table Query Syntax
+~~~~~~~~~~~~~~~~~~~
+Dragging your desired tables into the main area in Tableau builds queries based on its own syntax. This helps ensure increased performance, while using views or custom SQL may degrade performance. In addition, SQream recommends using the :ref:`create_view` to create pre-optimized views, which your datasources point to. 
+
+Creating a Separate Service for Tableau
+~~~~~~~~~~~~~~~~~~~
+SQream recommends creating a separate service for Tableau with the DWLM. This reduces the impact that Tableau has on other applications and processes, such as ETL. In addition, this works in conjunction with the load balancer to ensure good performance.
+
+Troubleshooting Workbook Performance Before Deploying to the Tableau Server
+~~~~~~~~~~~~~~~~~~~
+Tableau has a built-in `performance recorder `_ that shows how time is being spent. If you're seeing slow performance, this could be the result of a misconfiguration such as setting concurrency too low.
+
+Use the Tableau Performance Recorder for viewing the performance of queries run by Tableau. You can use this information to identify queries that can be optimized by using views.
+
+Troubleshooting Error Codes
+~~~~~~~~~~~~~~~~~~~
+Tableau may be unable to locate the SQream JDBC driver. The following message is displayed when Tableau cannot locate the driver:
+
+.. code-block:: console
+     
+   Error Code: 37CE01A3, No suitable driver installed or the URL is incorrect
+   
+**To troubleshoot error codes:**
+
+If Tableau cannot locate the SQream JDBC driver, do the following:
+
+ 1. Verify that the JDBC driver is located in the correct directory:
+ 
+   * **Tableau Desktop on Windows:** *C:\Program Files\Tableau\Drivers*
+   
+      ::
+	  
+   * **Tableau Desktop on MacOS:** *~/Library/Tableau/Drivers*
+   
+      ::
+	  
+   * **Tableau on Linux**: */opt/tableau/tableau_driver/jdbc*
+   
+ 2. Find the file path for the JDBC driver and add it to the Java classpath:
+   
+   * **For Linux** - ``export CLASSPATH=;$CLASSPATH``
+
+        ::
+		
+   * **For Windows** - add an environment variable for the classpath:  
+
+	.. image:: /_static/images/Third_Party_Connectors/tableau/envrionment_variable_for_classpath.png
+
+If you experience issues after restarting Tableau, see the `SQream support portal `_.
\ No newline at end of file
diff --git a/connecting_to_sqream/client_platforms/talend.rst b/connecting_to_sqream/client_platforms/talend.rst
new file mode 100644
index 000000000..1d0fb51b9
--- /dev/null
+++ b/connecting_to_sqream/client_platforms/talend.rst
@@ -0,0 +1,123 @@
+.. _talend:
+
+*************************
+Connecting to SQream Using Talend
+*************************
+
+Overview
+================= 
+This page describes how to use Talend to interact with a SQream cluster. The Talend connector is used for reading data from a SQream cluster and loading data into SQream. In addition, this page provides a viability report on Talend's comptability with SQream for stakeholders.
+
+The **Connecting to SQream Using Talend** describes the following:
+
+.. contents::
+   :local:
+   :depth: 1
+
+Creating a New Metadata JDBC DB Connection
+----------------
+**To create a new metadata JDBC DB connection:**
+
+1. In the **Repository** panel, nagivate to **Metadata** and right-click **Db connections**.
+
+    ::
+	
+2. Select **Create connection**.
+
+    ::
+	
+3. In the **Name** field, type a name.
+
+    ::
+
+   Note that the name cannot contain spaces.
+
+4. In the **Purpose** field, type a purpose and click **Next**.
+
+   Note that you cannot continue to the next step until you define both a Name and a Purpose.
+
+    ::
+
+5. In the **DB Type** field, select **JDBC**.
+
+    ::
+
+6. In the **JDBC URL** field, type the relevant connection string.
+
+   For connection string examples, see `Connection Strings `_.
+   
+7. In the **Drivers** field, click the **Add** button.
+
+   The **"newLine"** entry is added.
+
+8. One the **"newLine'** entry, click the ellipsis.
+
+   The **Module** window is displayed.
+
+9. From the Module window, select **Artifact repository(local m2/nexus)** and select **Install a new module**.
+
+    ::
+
+10. Click the ellipsis.
+
+    Your hard drive is displayed.	
+
+11. Navigate to a **JDBC jar file** (such as **sqream-jdbc-4.5.3.jar**)and click **Open**.
+
+     ::
+
+12. Click **Detect the module install status**.
+
+     ::
+
+13. Click **OK**.
+
+    The JDBC that you selected is displayed in the **Driver** field.
+
+14. Click **Select class name**.
+
+     ::
+
+15. Click **Test connection**.
+
+    If a driver class is not found (for example, you didn't select a JDBC jar file), the following error message is displayed:
+
+    After creating a new metadata JDBC DB connection, you can do the following:
+
+    * Use your new metadata connection.
+	
+	   ::
+	   
+    * Drag it to the **job** screen.
+	
+	   ::
+	   
+    * Build Talend components.
+ 
+    For more information on loading data from JSON files to the Talend Open Studio, see `How to Load Data from JSON Files in Talend `_.
+
+Supported SQream Drivers
+----------------
+The following list shows the supported SQream drivers and versions:
+
+* **JDBC** - Version 4.3.3 and higher.
+
+   ::
+   
+* **ODBC** - Version 4.0.0. This version requires a Bridge to connect. For more information on the required Bridge, see `Connecting Talend on Windows to an ODBC Database `_.
+
+Supported Data Sources
+----------------
+Talend Cloud connectors let you create reusable connections with a wide variety of systems and environments, such as those shown below. This lets you access and read records of a range of diverse data.
+
+* **Connections:** Connections are environments or systems for storing datasets, including databases, file systems, distributed systems and platforms. Because these systems are reusable, you only need to establish connectivity with them once.
+
+   ::
+
+* **Datasets:** Datasets include database tables, file names, topics (Kafka), queues (JMS) and file paths (HDFS). For more information on the complete list of connectors and datasets that Talend supports, see `Introducing Talend Connectors `_.
+
+Known Issues
+----------------
+As of 6/1/2021 schemas were not displayed for tables with identical names.
+
+If you experience issues using Talend, see the `SQream support portal `_.
\ No newline at end of file
diff --git a/guides/third_party_tools/test.php b/connecting_to_sqream/client_platforms/test.php
similarity index 100%
rename from guides/third_party_tools/test.php
rename to connecting_to_sqream/client_platforms/test.php
diff --git a/connecting_to_sqream/client_platforms/tibco_spotfire.rst b/connecting_to_sqream/client_platforms/tibco_spotfire.rst
new file mode 100644
index 000000000..b98ae0825
--- /dev/null
+++ b/connecting_to_sqream/client_platforms/tibco_spotfire.rst
@@ -0,0 +1,387 @@
+.. _tibco_spotfire:
+
+
+*************************
+Connecting to SQream Using TIBCO Spotfire
+*************************
+Overview
+=========
+The **TIBCO Spotfire** software is an analytics solution that enables visualizing and exploring data through dashboards and advanced analytics.
+
+This document is a Quick Start Guide that describes the following:
+   
+.. contents::
+   :local: 
+   :depth: 1
+   
+Establishing a Connection between TIBCO Spotfire and SQream
+-----------------
+TIBCO Spotfire supports the following versions:
+
+* **JDBC driver** - Version 4.5.3
+* **ODBC driver** - Version 4.1.1
+
+SQream supports TIBCO Spotfire version 7.12.0.
+
+The **Establishing a JDBC Connection between TIBCO Spotfire and SQream** section describes the following:
+
+.. contents::
+   :local: 
+   :depth: 1   
+   
+Creating a JDBC Connection
+~~~~~~~~~~~
+For TIBCO Spotfire to recognize SQream, you must add the correct JDBC jar file to Spotfire's loaded binary folder. The following is an example of a path to the Spotfire loaded binaries folder: ``C:\tibco\tss\7.12.0\tomcat\bin``.
+
+For the complete TIBCO Spotfire documentation, see `TIBCO Spotfire® JDBC Data Access Connectivity Details `_. 
+
+Creating an ODBC Connection
+~~~~~~~~~~~
+**To create an ODBC connection**
+
+1. Install and configure ODBC on Windows.
+
+   For more information, see :ref:`Install and Configure ODBC on Windows`.
+   
+#. Launch the TIBCO Spotfire application.
+
+    ::
+
+#. From the **File** menu click **Add Data Tables**.
+
+   The **Add Database Tables** window is displayed.
+
+#. Click **Add** and select **Database**.
+
+   The **Open Database** window is displayed.
+
+#. In the **Data source type** area, select **ODBC SQream** (Odbc Data Provider) and click **Configure**.
+
+   The **Configure Data Source and Connection** window is displayed.
+   
+#. Select **System or user data source** and from the drop-down menu select the DSN of your data source (SQreamDB).
+
+    ::
+
+#. Provide your database username and password and click **OK**.
+
+    ::
+
+#. In the **Open Database** window, click **OK**.
+
+   The **Specify Tables and Columns** window is displayed.
+
+#. In the **Specify Tables and Columns** window, select the checkboxes corresponding to the tables and columns that you want to include in your SQL statement.
+
+    ::
+
+#. In the **Data source name** field, set your data source name and click **OK**.
+
+   Your data source is displayed in the **Data tables** area.
+
+#. In the **Add Data Tables** dialog, click **OK** to load the data from your ODBC data source into Spotfire.
+
+.. note:: Verify that you have checked the SQL statement. 
+
+Creating the SQream Data Source Template
+~~~~~~~~~~~
+After creating a connection, you can create your SQream data source template.
+
+**To create your SQream data source template:**
+
+1. Log in to the TIBCO Spotfire Server Configuration Tool.
+
+    ::
+	
+#. From the **Configuration** tab, in the **Configuration Start** menu, click **Data Source Templates**.
+
+   The **Data Source Templates** list is displayed.
+   
+#. From the Data Source Templates list do one of the following:
+
+  * Override an existing template:
+   
+    1. In the template text field, select an existing template.
+	
+	    ::
+		
+    2. Copy and paste your data source template text.
+	 
+	     ::
+	 
+  * Create a new template:
+   
+    1. Click **New**.
+        
+       The **Add Data Source Template** window is displayed.
+	   
+       .. _creating_sqream_data_source_template:
+		
+    2. In the **Name** field, define your template name.
+	
+	    ::
+		
+    3. In the **Data Source Template** text field, copy and paste your data source template text.
+	
+       The following is an example of a data source template:		
+
+       .. code-block:: console
+	
+          
+            SQream   
+            com.sqream.jdbc.SQDriver   
+            jdbc:Sqream://<host>:<port>/database;user=sqream;password=sqream;cluster=true   
+            true   
+            true   
+            false   
+            TABLE,EXTERNAL_TABLE   
+            
+             
+                Bool   
+                Integer   
+              
+              
+                VARCHAR(2048)   
+                String   
+              
+              
+                INT   
+                Integer   
+              
+              
+                BIGINT   
+                LongInteger   
+              
+              
+                Real   
+                Real   
+              
+	           
+                Decimal   
+                Float   
+              
+               
+                Numeric   
+                Float   
+              
+              
+                Date   
+                DATE   
+              
+              
+                DateTime   
+                DateTime   
+              
+             
+               
+          			
+	
+4. Click **Save configuration**.
+
+    ::
+	
+5. Close and restart your Spotfire server.
+
+Creating a Data Source
+~~~~~~~~~~~
+After creating the SQream data source template, you can create a data source.
+
+**To create a data source:**
+
+1. Launch the TIBCO Spotfire application.
+
+    ::
+
+#. From the **Tools** menu, select **Information Designer**.
+
+   The **Information Designer** window is displayed.
+
+    ::
+	
+#. From the **New** menu, click **Data Source**.
+
+   The **Data Source** tab is displayed.
+
+    ::
+	
+#. Provide the following information:
+
+   * **Name** - define a unique name.
+   
+      ::
+	  
+   * **Type** - use the same type template name you used while configuring your template. See **Step 3** in :ref:`Creating the SQream Data Source Template`.
+   
+      ::
+	  
+   * **Connection URL** - use the standard JDBC connection string, ``:/database``.
+   
+      ::
+	  
+   * **No. of connections** - define a number between **1** and **100**. SQream recommends setting your number of connections to **100**.
+   
+      ::
+	  
+   * **Username and Password** - define your SQream username and password.   
+
+Creating an Information Link
+~~~~~~~~~~~
+After creating a data source, you can create an information link.
+
+**To create an information link**:
+
+1. From the **Tools** menu, select **Information Designer**.
+
+   The **Information Designer** window is displayed.
+
+    ::
+
+#. From the **New** menu, click **Information Link**.
+
+   The **Information link** tab is displayed.
+   
+#. From the **Elements** tab, select a column type and click **Add**.
+
+   The column type is added to the **Elements** region as a filter.
+   
+   Note the following:
+   
+   * You can select procedures from the Elements region.
+   
+      ::
+	  
+   * You can remove an element by selecting an element and clicking **Remove**.   
+
+   .. tip:: If the Elements menu is not displayed, you can display it by clicking the **Elements** tab. You can simultaneously select multiple elements by pressing **Ctrl** and making additional selections, and select a range of elements by holding **Shift** and clicking two elements.
+   
+#. If the elements you select originate from more than one data source table, specify a **Join path**.
+
+5. *Optional* - In the **Description** region, type the description of the information link.
+
+    ::
+
+#. *Optional* - To filter your data, expand the **Filters** section and do the following:
+
+    1. From the **Information Link** region, select the element you added in Step 3 above.
+	
+	    ::
+		
+    2. Click **Add**.
+	
+       The **Add Column** window is displayed.
+	   
+    3. From the drop-down list, select a column to add a hard filter to and click **OK**.
+	
+       The selected column is added to the Filters list.
+	   
+    4. Repeat steps 2 and 3 to add filters to additional columns.
+	
+	    ::
+		
+    5. For each column, from the **Filter Type** drop-down list, select **range** or **values**.
+	
+       .. note:: Filtering by range means entering the upper and lower limits of the desired range. Filtering by values means entering the exact values that you want to include in the returned data, separated by semicolon.
+
+    6. In the **Values** field type the desired values separated with semicolons, or set the upper and lower limits in the **Min Value** and **Max Value** fields. Alternatively, you can type ``?param_name`` in the Values field to use a parameter as the filter for the selected column, where ``param_name`` is the name used to identify the parameter. 
+
+       .. note:: Because limits are inclusive, setting the lower limit to **1000** includes the value **1000** in the data table.
+	   
+       .. note:: When setting upper and lower limits on **String** type columns, ``A`` precedes ``AA``, and a lone letter precedes words beginning with that latter. For example, ``S** precedes **Smith**, indicating that the name ``Smith`` will not be present when you select names from ``D`` to ``S``. The order of characters is standard ASCII.
+	   
+   For more information on adding filters, see `Adding Hard Filters `_.
+
+7. *Optional* - To add runtime filtering prompts, expand the **Prompts** section and do the following:
+
+    1. Click **Add**.
+	
+       The **Add Column** window is displayed.
+	   
+    #. From the **Select column** list, select a column to add a prompt to and click **OK**.
+	
+       The selected column is added to the Prompts list.
+	   
+    #. Repeat **Step 1** to add prompts to additional columns.
+	
+	    ::
+		
+    #. Do the following for each column:
+	
+       * Make a selection from the **Prompt Type** drop-down list.
+       * Select or clear **Mandatory**.
+       * *Optional* - Set your **Max Selections**.
+	
+   For more information on adding prompts, see `Adding Prompts `_.
+
+8. *Optional* - Expand the **Conditioning** section and specify one of the following conditions:
+
+   * None
+   * Distinct
+   * Pivot
+
+   Note that you can edit the Pivot conditioning by selecting **Pivot** and clicking **Edit**.
+   
+9. *Optional* - Expand the **Parameters** section and define your parameters.
+
+     ::
+
+10. *Optional* - Expand the **Properties** section and define your properties.
+
+     ::
+
+11. *Optional* - Expand the **Caching** section and enable or disable whether your information link can be cached.
+
+     ::
+
+12. Click **Save**.
+
+    The **Save As** window is displayed.
+
+13. In the tree, select where you want to save the information link.
+
+     ::
+
+14. In the **Name** field, type a name and description for the information link.
+
+     ::
+
+
+15. Click **Save**.
+
+    The new information link is added to the library and can be accessed by other users.
+
+.. tip:: You can test the information link directly by clicking **Open Data**. You can also view and edit the SQL belonging to the information link by clicking **SQL**.
+
+For more information on the Information Link attributes, see `Information Link Tab `_.
+
+Troubleshooting
+-------------
+The **Troubleshooting** section describes the following scenarios:
+
+.. contents::
+   :local: 
+   :depth: 1 
+
+The JDBC Driver does not Support Boolean, Decimal, or Numeric Types
+~~~~~~~~~~~
+When attempting to load data, the the Boolean, Decimal, or Numeric column types are not supported and generate the following error:
+
+.. code-block:: console
+
+   Failed to execute query: Unsupported JDBC data type in query result: Bool (HRESULT: 80131500)
+
+The error above is resolved by casting the columns as follows:
+
+* ``Bool`` columns to ``INT``.
+* ``Decimal`` and ``Numeric`` columns to ``REAL``.
+
+For more information, see the following:
+
+* **Resolving this error** - `Details on Change Data Types `_.
+
+* **Supported data types** - :ref:`Data Types`.
+
+Information Services do not Support Live Queries
+~~~~~~~~~~~
+TIBCO Spotfire data connectors support live queries, but no APIs currently exist for creating custom data connectors. This is resolved by creating a customized SQream adapter using TIBCO's **Data Virtualization (TDV)** or the **Spotfire Advanced Services (ADS)**. These can be used from the built-in TDV connector to enable live queries.
+
+This resolution applies to JDBC and ODBC drivers.
\ No newline at end of file
diff --git a/connecting_to_sqream/connectivity_ecosystem.jpg b/connecting_to_sqream/connectivity_ecosystem.jpg
new file mode 100644
index 000000000..8bcb05369
Binary files /dev/null and b/connecting_to_sqream/connectivity_ecosystem.jpg differ
diff --git a/connecting_to_sqream/index.rst b/connecting_to_sqream/index.rst
new file mode 100644
index 000000000..ecb9e4715
--- /dev/null
+++ b/connecting_to_sqream/index.rst
@@ -0,0 +1,18 @@
+.. _connecting_to_sqream:
+
+*************************
+Connecting to SQream
+*************************
+SQream supports the most common database tools and interfaces, giving you direct access through a variety of drivers, connectors, and visualiztion tools and utilities. The tools described on this page have been tested and approved for use with SQream. Most third party tools that work through JDBC, ODBC, and Python should work.
+
+This section provides information about the following third party tools:
+
+.. toctree::
+   :maxdepth: 2
+   :glob:
+   :titlesonly:
+   
+   client_platforms/index
+   client_drivers/index
+
+If you need a tool that SQream does not support, contact SQream Support or your SQream account manager for more information.
\ No newline at end of file
diff --git a/guides/migration/csv.rst b/data_ingestion/csv.rst
similarity index 96%
rename from guides/migration/csv.rst
rename to data_ingestion/csv.rst
index 431ec259f..ac4ba1741 100644
--- a/guides/migration/csv.rst
+++ b/data_ingestion/csv.rst
@@ -1,10 +1,10 @@
 .. _csv:
 
 **********************
-Insert from CSV
+Inserting Data from a CSV File
 **********************
 
-This guide covers inserting data from CSV files into SQream DB using the :ref:`copy_from` method.
+This guide covers inserting data from CSV files into SQream DB using the :ref:`copy_from` method. 
 
 
 .. contents:: In this topic:
@@ -83,15 +83,15 @@ We will make note of the file structure to create a matching ``CREATE TABLE`` st
    
    CREATE TABLE nba
    (
-      Name varchar(40),
-      Team varchar(40),
-      Number tinyint,
-      Position varchar(2),
-      Age tinyint,
-      Height varchar(4),
-      Weight real,
-      College varchar(40),
-      Salary float
+      "Name" varchar(40),
+      "Team" varchar(40),
+      "Number" tinyint,
+      "Position" varchar(2),
+      "Age" tinyint,
+      "Height" varchar(4),
+      "Weight" real,
+      "College" varchar(40),
+      "Salary" float
     );
 
 
diff --git a/data_ingestion/index.rst b/data_ingestion/index.rst
new file mode 100644
index 000000000..8ebb646c1
--- /dev/null
+++ b/data_ingestion/index.rst
@@ -0,0 +1,18 @@
+.. _data_ingestion:
+
+*************************
+Data Ingestion Sources
+*************************
+The **Data Ingestion Sources** provides information about the following:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+   
+   inserting_data
+   csv
+   parquet
+   orc
+   oracle
+
+For information about database tools and interfaces that SQream supports, see `Third Party Tools `_.
\ No newline at end of file
diff --git a/data_ingestion/inserting_data.rst b/data_ingestion/inserting_data.rst
new file mode 100644
index 000000000..e20e37920
--- /dev/null
+++ b/data_ingestion/inserting_data.rst
@@ -0,0 +1,481 @@
+.. _inserting_data:
+
+***************************
+Inserting Data Overview
+***************************
+The **Inserting Data Overview** page provides basic information useful when ingesting data into SQream from a variety of sources and locations, and describes the following:
+
+.. contents::
+   :local:
+   :depth: 1
+   
+Getting Started
+================================
+SQream supports ingesting data using the following methods:
+
+* Executing the ``INSERT`` statement using a client driver.
+
+   ::
+   
+* Executing the ``COPY FROM`` statement or ingesting data from foreign tables:
+
+  * Local filesystem and locally mounted network filesystems
+  * Inserting Data using the Amazon S3 object storage service
+  * Inserting Data using an HDFS data storage system
+
+SQream supports loading files from the following formats:
+
+* Text - CSV, TSV, and PSV
+* Parquet
+* ORC
+
+For more information, see the following:
+
+* Using the ``INSERT`` statement - :ref:`insert`
+
+* Using client drivers - :ref:`Client drivers`
+
+* Using the ``COPY FROM`` statement - :ref:`copy_from`
+
+* Using the Amazon S3 object storage service - :ref:`s3`
+
+* Using the HDFS data storage system - :ref:`hdfs`
+
+* Loading data from foreign tables - :ref:`foreign_tables`
+
+Data Loading Considerations
+================================
+The **Data Loading Considerations** section describes the following:
+
+.. contents:: 
+   :local:
+   :depth: 1
+   
+Verifying Data and Performance after Loading
+-----------------------------------------
+Like many RDBMSs, SQream recommends its own set of best practices for table design and query optimization. When using SQream, verify the following:
+
+* That your data is structured as you expect (row counts, data types, formatting, content).
+
+* That your query performance is adequate.
+
+* That you followed the table design best practices (:ref:`Optimization and Best Practices`).
+
+* That you've tested and verified that your applications work (such as :ref:`Tableau`).
+
+* That your data types have not been not over-provisioned.
+
+File Soure Location when Loading
+--------------------------------
+While you are loading data, you can use the ``COPY FROM`` command to let statements run on any worker. If you are running multiple nodes, verify that all nodes can see the source the same. Loading data from a local file that is only on one node and not on shared storage may cause it to fail. If required, you can also control which node a statement runs on using the Workload Manager).
+
+For more information, see the following:
+
+* :ref:`copy_from`
+
+* :ref:`workload_manager`
+
+Supported Load Methods
+-------------------------------
+You can use the ``COPY FROM`` syntax to load CSV files.
+
+.. note:: The ``COPY FROM`` cannot be used for loading data from Parquet and ORC files.
+
+You can use foreign tables to load text files, Parquet, and ORC files, and to transform your data before generating a full table, as described in the following table:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   :stub-columns: 1
+   
+   * - Method/File Type
+     - Text (CSV)
+     - Parquet
+     - ORC
+     - Streaming Data
+   * - COPY FROM
+     - Supported
+     - Not supported
+     - Not supported
+     - Not supported
+   * - Foreign tables
+     - Supported
+     - Supported
+     - Supported
+     - Not supported
+   * - INSERT
+     - Not supported
+     - Not supported
+     - Not supported
+     - Supported (Python, JDBC, Node.JS)
+	 
+For more information, see the following:
+
+* :ref:`COPY FROM`
+
+* :ref:`Foreign tables`
+
+* :ref:`INSERT`
+
+Unsupported Data Types
+-----------------------------
+SQream does not support certain features that are supported by other databases, such as ``ARRAY``, ``BLOB``, ``ENUM``, and ``SET``. You must convert these data types before loading them. For example, you can store ``ENUM`` as ``TEXT``.
+
+Handing Extended Errors
+----------------------------
+While you can use foreign tables to load CSVs, the ``COPY FROM`` statement provides more fine-grained error handling options and extended support for non-standard CSVs with multi-character delimiters, alternate timestamp formats, and more.
+
+For more information, see :ref:`foreign tables`.
+
+Best Practices for CSV
+------------------------------
+Text files, such as CSV, rarely conform to `RFC 4180 `_ , so you may need to make the following modifications:
+
+* Use ``OFFSET 2`` for files containing header rows.
+
+* You can capture failed rows in a log file for later analysis, or skip them. See :ref:`capturing_rejected_rows` for information on skipping rejected rows.
+
+* You can modify record delimiters (new lines) using the :ref:`RECORD DELIMITER` syntax.
+
+* If the date formats deviate from ISO 8601, refer to the :ref:`copy_date_parsers` section for overriding the default parsing.
+
+* *(Optional)* You can quote fields in a CSV using double-quotes (``"``).
+
+.. note:: You must quote any field containing a new line or another double-quote character.
+
+* If a field is quoted, you must double quote any double quote, similar to the **string literals quoting rules**. For example, to encode ``What are "birds"?``, the field should appear as ``"What are ""birds""?"``. For more information, see :ref:`string literals quoting rules`.
+
+* Field delimiters do not have to be a displayable ASCII character. For all supported field delimiters, see :ref:`field_delimiters`.
+
+Best Practices for Parquet
+--------------------------------
+The following list shows the best practices when inserting data from Parquet files:
+
+* You must load Parquet files through :ref:`foreign_tables`. Note that the destination table structure must be identical to the number of columns between the source files.
+
+* Parquet files support **predicate pushdown**. When a query is issued over Parquet files, SQream uses row-group metadata to determine which row-groups in a file must be read for a particular query and the row indexes can narrow the search to a particular set of rows.
+
+Supported Types and Behavior Notes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Unlike the ORC format, the column types should match the data types exactly, as shown in the table below:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   :stub-columns: 1
+   
+   * -   SQream DB type →
+   
+         Parquet source
+     - ``BOOL``
+     - ``TINYINT``
+     - ``SMALLINT``
+     - ``INT``
+     - ``BIGINT``
+     - ``REAL``
+     - ``DOUBLE``
+     - Text [#f0]_
+     - ``DATE``
+     - ``DATETIME``
+   * - ``BOOLEAN``
+     - Supported 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``INT16``
+     - 
+     - 
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``INT32``
+     - 
+     - 
+     - 
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``INT64``
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``FLOAT``
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - 
+     - 
+     - 
+     - 
+   * - ``DOUBLE``
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - 
+     - 
+     - 
+   * - ``BYTE_ARRAY`` [#f2]_
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - 
+     - 
+   * - ``INT96`` [#f3]_
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported [#f4]_
+
+If a Parquet file has an unsupported type, such as ``enum``, ``uuid``, ``time``, ``json``, ``bson``, ``lists``, ``maps``, but the table does not reference this data (i.e., the data does not appear in the :ref:`SELECT` query), the statement will succeed. If the table **does** reference a column, an error will be displayed explaining that the type is not supported, but the column may be omitted.
+
+Best Practices for ORC
+--------------------------------
+The following list shows the best practices when inserting data from ORC files:
+
+* You must load ORC files through :ref:`foreign_tables`. Note that the destination table structure must be identical to the number of columns between the source files.
+
+* ORC files support **predicate pushdown**. When a query is issued over ORC files, SQream uses ORC metadata to determine which stripes in a file need to be read for a particular query and the row indexes can narrow the search to a particular set of 10,000 rows.
+
+Type Support and Behavior Notes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+You must load ORC files through foreign table. Note that the destination table structure must be identical to the number of columns between the source files.
+
+For more information, see :ref:`foreign_tables`.
+
+The types should match to some extent within the same "class", as shown in the following table:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   :stub-columns: 1
+   
+   * -   SQream DB Type →
+   
+         ORC Source
+     - ``BOOL``
+     - ``TINYINT``
+     - ``SMALLINT``
+     - ``INT``
+     - ``BIGINT``
+     - ``REAL``
+     - ``DOUBLE``
+     - Text [#f0]_
+     - ``DATE``
+     - ``DATETIME``
+   * - ``boolean``
+     - Supported 
+     - Supported [#f5]_
+     - Supported [#f5]_
+     - Supported [#f5]_
+     - Supported [#f5]_
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``tinyint``
+     - ○ [#f6]_
+     - Supported
+     - Supported
+     - Supported
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``smallint``
+     - ○ [#f6]_
+     - ○ [#f7]_
+     - Supported
+     - Supported
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``int``
+     - ○ [#f6]_
+     - ○ [#f7]_
+     - ○ [#f7]_
+     - Supported
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``bigint``
+     - ○ [#f6]_
+     - ○ [#f7]_
+     - ○ [#f7]_
+     - ○ [#f7]_
+     - Supported
+     - 
+     - 
+     - 
+     - 
+     - 
+   * - ``float``
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - Supported
+     - 
+     - 
+     - 
+   * - ``double``
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - Supported
+     - 
+     - 
+     - 
+   * - ``string`` / ``char`` / ``varchar``
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - 
+     - 
+   * - ``date``
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+     - Supported
+   * - ``timestamp``, ``timestamp`` with timezone
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - Supported
+
+* If an ORC file has an unsupported type like ``binary``, ``list``, ``map``, and ``union``, but the data is not referenced in the table (it does not appear in the :ref:`SELECT` query), the statement will succeed. If the column is referenced, an error will be thrown to the user, explaining that the type is not supported, but the column may be ommited.
+
+
+
+..
+   insert
+
+   example
+
+   are there some variations to highlight?:
+
+   create table as
+
+   sequences, default values
+
+   insert select
+
+   make distinction between an insert command, and a parameterized/bulk
+   insert "over the network"
+
+
+   copy
+
+
+   best practices for insert
+
+   chunks and extents, and storage reorganisation
+
+   copy:
+
+   give an example
+
+   supports csv and parquet
+
+   what else do we have right now? any other formats? have the s3 and
+   hdfs url support also
+
+   error handling
+
+   best practices
+
+   try to combine sensibly with the external table stuff
+
+Further Reading and Migration Guides
+=======================================
+For more information, see the following:
+
+* :ref:`copy_from`
+* :ref:`insert`
+* :ref:`foreign_tables`
+
+.. rubric:: Footnotes
+
+.. [#f0] Text values include ``TEXT``, ``VARCHAR``, and ``NVARCHAR``
+
+.. [#f2] With UTF8 annotation
+
+.. [#f3] With ``TIMESTAMP_NANOS`` or ``TIMESTAMP_MILLIS`` annotation
+
+.. [#f4] Any microseconds will be rounded down to milliseconds.
+
+.. [#f5] Boolean values are cast to 0, 1
+
+.. [#f6] Will succeed if all values are 0, 1
+
+.. [#f7] Will succeed if all values fit the destination type
\ No newline at end of file
diff --git a/data_ingestion/nba-t10.csv b/data_ingestion/nba-t10.csv
new file mode 100644
index 000000000..c3d41e729
--- /dev/null
+++ b/data_ingestion/nba-t10.csv
@@ -0,0 +1,9 @@
+Avery Bradley,Boston Celtics,0,PG,25,2-Jun,180,Texas,7730337
+Jae Crowder,Boston Celtics,99,SF,25,6-Jun,235,Marquette,6796117
+John Holland,Boston Celtics,30,SG,27,5-Jun,205,Boston University,
+R.J. Hunter,Boston Celtics,28,SG,22,5-Jun,185,Georgia State,1148640
+Jonas Jerebko,Boston Celtics,8,PF,29,10-Jun,231,,5000000
+Amir Johnson,Boston Celtics,90,PF,29,9-Jun,240,,12000000
+Jordan Mickey,Boston Celtics,55,PF,21,8-Jun,235,LSU,1170960
+Kelly Olynyk,Boston Celtics,41,C,25,Jul-00,238,Gonzaga,2165160
+Terry Rozier,Boston Celtics,12,PG,22,2-Jun,190,Louisville,1824360
diff --git a/guides/migration/oracle.rst b/data_ingestion/oracle.rst
similarity index 99%
rename from guides/migration/oracle.rst
rename to data_ingestion/oracle.rst
index 86be41517..0b0e6d5c8 100644
--- a/guides/migration/oracle.rst
+++ b/data_ingestion/oracle.rst
@@ -1,10 +1,10 @@
 .. _oracle:
 
 **********************
-Migrating from Oracle
+Migrating Data from Oracle
 **********************
 
-This guide covers actions required for migrating from Oracle to SQream DB with CSV files.
+This guide covers actions required for migrating from Oracle to SQream DB with CSV files. 
 
 .. contents:: In this topic:
    :local:
diff --git a/guides/migration/orc.rst b/data_ingestion/orc.rst
similarity index 84%
rename from guides/migration/orc.rst
rename to data_ingestion/orc.rst
index 8f660bf51..effaa1e32 100644
--- a/guides/migration/orc.rst
+++ b/data_ingestion/orc.rst
@@ -1,10 +1,10 @@
 .. _orc:
 
 **********************
-Insert from ORC
+Inserting Data from an ORC File
 **********************
 
-This guide covers inserting data from ORC files into SQream DB using :ref:`FOREIGN TABLE`.
+This guide covers inserting data from ORC files into SQream DB using :ref:`FOREIGN TABLE`. 
 
 
 1. Prepare the files
@@ -31,11 +31,11 @@ Prepare the source ORC files, with the following requirements:
      - ``DATE``
      - ``DATETIME``
    * - ``boolean``
-     - ✓ 
-     - ✓ [#f5]_
-     - ✓ [#f5]_
-     - ✓ [#f5]_
-     - ✓ [#f5]_
+     - Supported 
+     - Supported [#f5]_
+     - Supported [#f5]_
+     - Supported [#f5]_
+     - Supported [#f5]_
      - 
      - 
      - 
@@ -43,10 +43,10 @@ Prepare the source ORC files, with the following requirements:
      - 
    * - ``tinyint``
      - ○ [#f6]_
-     - ✓
-     - ✓
-     - ✓
-     - ✓
+     - Supported
+     - Supported
+     - Supported
+     - Supported
      - 
      - 
      - 
@@ -55,9 +55,9 @@ Prepare the source ORC files, with the following requirements:
    * - ``smallint``
      - ○ [#f6]_
      - ○ [#f7]_
-     - ✓
-     - ✓
-     - ✓
+     - Supported
+     - Supported
+     - Supported
      - 
      - 
      - 
@@ -67,8 +67,8 @@ Prepare the source ORC files, with the following requirements:
      - ○ [#f6]_
      - ○ [#f7]_
      - ○ [#f7]_
-     - ✓
-     - ✓
+     - Supported
+     - Supported
      - 
      - 
      - 
@@ -79,7 +79,7 @@ Prepare the source ORC files, with the following requirements:
      - ○ [#f7]_
      - ○ [#f7]_
      - ○ [#f7]_
-     - ✓
+     - Supported
      - 
      - 
      - 
@@ -91,8 +91,8 @@ Prepare the source ORC files, with the following requirements:
      - 
      - 
      - 
-     - ✓
-     - ✓
+     - Supported
+     - Supported
      - 
      - 
      - 
@@ -102,12 +102,12 @@ Prepare the source ORC files, with the following requirements:
      - 
      - 
      - 
-     - ✓
-     - ✓
+     - Supported
+     - Supported
      - 
      - 
      - 
-   * - ``string`` / ``char`` / ``varchar``
+   * - ``string`` / ``char`` / ``text``
      - 
      - 
      - 
@@ -115,7 +115,7 @@ Prepare the source ORC files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
    * - ``date``
@@ -127,8 +127,8 @@ Prepare the source ORC files, with the following requirements:
      - 
      - 
      - 
-     - ✓
-     - ✓
+     - Supported
+     - Supported
    * - ``timestamp``, ``timestamp`` with timezone
      - 
      - 
@@ -139,19 +139,19 @@ Prepare the source ORC files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
 
 * If an ORC file has an unsupported type like ``binary``, ``list``, ``map``, and ``union``, but the data is not referenced in the table (it does not appear in the :ref:`SELECT` query), the statement will succeed. If the column is referenced, an error will be thrown to the user, explaining that the type is not supported, but the column may be ommited. This can be worked around. See more information in the examples.
 
 .. rubric:: Footnotes
 
-.. [#f0] Text values include ``TEXT``, ``VARCHAR``, and ``NVARCHAR``
+.. [#f0] Text values include ``TEXT``.
 
-.. [#f5] Boolean values are cast to 0, 1
+.. [#f5] Boolean values are cast to 0, 1.
 
-.. [#f6] Will succeed if all values are 0, 1
+.. [#f6] Will succeed if all values are 0, 1.
 
-.. [#f7] Will succeed if all values fit the destination type
+.. [#f7] Will succeed if all values fit the destination type.
 
 2. Place ORC files where SQream DB workers can access them
 ================================================================
@@ -186,15 +186,15 @@ We will make note of the file structure to create a matching ``CREATE FOREIGN TA
    
    CREATE FOREIGN TABLE ext_nba
    (
-        Name       VARCHAR(40),
-        Team       VARCHAR(40),
-        Number     BIGINT,
-        Position   VARCHAR(2),
-        Age        BIGINT,
-        Height     VARCHAR(4),
-        Weight     BIGINT,
-        College    VARCHAR(40),
-        Salary     FLOAT
+      "Name" varchar(40),
+      "Team" varchar(40),
+      "Number" tinyint,
+      "Position" varchar(2),
+      "Age" tinyint,
+      "Height" varchar(4),
+      "Weight" real,
+      "College" varchar(40),
+      "Salary" float
     )
       WRAPPER orc_fdw
       OPTIONS
@@ -212,7 +212,7 @@ We will make note of the file structure to create a matching ``CREATE FOREIGN TA
 4. Verify table contents
 ====================================
 
-External tables do not verify file integrity or structure, so verify that the table definition matches up and contains the correct data.
+Foreign tables do not verify file integrity or structure, so verify that the table definition matches up and contains the correct data.
 
 .. code-block:: psql
    
@@ -247,7 +247,7 @@ Working around unsupported column types
 
 Suppose you only want to load some of the columns - for example, if one of the columns isn't supported.
 
-By ommitting unsupported columns from queries that access the ``EXTERNAL TABLE``, they will never be called, and will not cause a "type mismatch" error.
+By ommitting unsupported columns from queries that access the ``FOREIGN TABLE``, they will never be called, and will not cause a "type mismatch" error.
 
 For this example, assume that the ``Position`` column isn't supported because of its type.
 
@@ -288,7 +288,7 @@ Loading a table from a directory of ORC files on HDFS
 .. code-block:: postgres
 
    CREATE FOREIGN TABLE ext_users
-     (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL)  
+     (id INT NOT NULL, name TEXT(30) NOT NULL, email TEXT(50) NOT NULL)  
    WRAPPER orc_fdw
      OPTIONS
        ( 
@@ -302,8 +302,8 @@ Loading a table from a bucket of files on S3
 
 .. code-block:: postgres
 
-   CREATE FOREIGN TABLE ext_users
-     (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL)  
+   CREATE FOREIGN TABLE TEXT
+     (id INT NOT NULL, name TEXT NOT NULL, email TEXT(50) NOT NULL)  
    WRAPPER orc_fdw
    OPTIONS
      (  LOCATION = 's3://pp-secret-bucket/users/*.ORC',
diff --git a/guides/migration/parquet.rst b/data_ingestion/parquet.rst
similarity index 92%
rename from guides/migration/parquet.rst
rename to data_ingestion/parquet.rst
index 7f80ab206..c7064a001 100644
--- a/guides/migration/parquet.rst
+++ b/data_ingestion/parquet.rst
@@ -1,10 +1,10 @@
 .. _parquet:
 
 **********************
-Insert from Parquet
+Inserting Data from a Parquet File
 **********************
 
-This guide covers inserting data from Parquet files into SQream DB using :ref:`FOREIGN TABLE`.
+This guide covers inserting data from Parquet files into SQream DB using :ref:`FOREIGN TABLE`. 
 
 .. contents:: In this topic:
    :local:
@@ -33,7 +33,7 @@ Prepare the source Parquet files, with the following requirements:
      - ``DATE``
      - ``DATETIME``
    * - ``BOOLEAN``
-     - ✓ 
+     - Supported 
      - 
      - 
      - 
@@ -46,7 +46,7 @@ Prepare the source Parquet files, with the following requirements:
    * - ``INT16``
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
      - 
@@ -58,7 +58,7 @@ Prepare the source Parquet files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
      - 
@@ -70,7 +70,7 @@ Prepare the source Parquet files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
      - 
@@ -82,7 +82,7 @@ Prepare the source Parquet files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
      - 
@@ -94,7 +94,7 @@ Prepare the source Parquet files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
      - 
@@ -106,7 +106,7 @@ Prepare the source Parquet files, with the following requirements:
      - 
      - 
      - 
-     - ✓
+     - Supported
      - 
      - 
    * - ``INT96`` [#f3]_
@@ -119,17 +119,17 @@ Prepare the source Parquet files, with the following requirements:
      - 
      - 
      - 
-     - ✓ [#f4]_
+     - Supported [#f4]_
 
 * If a Parquet file has an unsupported type like ``enum``, ``uuid``, ``time``, ``json``, ``bson``, ``lists``, ``maps``, but the data is not referenced in the table (it does not appear in the :ref:`SELECT` query), the statement will succeed. If the column is referenced, an error will be thrown to the user, explaining that the type is not supported, but the column may be ommited. This can be worked around. See more information in the examples.
 
 .. rubric:: Footnotes
 
-.. [#f0] Text values include ``TEXT``, ``VARCHAR``, and ``NVARCHAR``
+.. [#f0] Text values include ``TEXT``.
 
-.. [#f2] With UTF8 annotation
+.. [#f2] With UTF8 annotation.
 
-.. [#f3] With ``TIMESTAMP_NANOS`` or ``TIMESTAMP_MILLIS`` annotation
+.. [#f3] With ``TIMESTAMP_NANOS`` or ``TIMESTAMP_MILLIS`` annotation.
 
 .. [#f4] Any microseconds will be rounded down to milliseconds.
 
@@ -161,20 +161,20 @@ For example, to import the data from ``nba.parquet``, we will first look at the
 * The file is stored on S3, at ``s3://sqream-demo-data/nba.parquet``.
 
 
-We will make note of the file structure to create a matching ``CREATE EXTERNAL TABLE`` statement.
+We will make note of the file structure to create a matching ``CREATE FOREIGN TABLE`` statement.
 
 .. code-block:: postgres
    
    CREATE FOREIGN TABLE ext_nba
    (
-        Name       VARCHAR(40),
-        Team       VARCHAR(40),
+        Name       TEXT,
+        Team       TEXT,
         Number     BIGINT,
-        Position   VARCHAR(2),
+        Position   TEXT,
         Age        BIGINT,
-        Height     VARCHAR(4),
+        Height     TEXT,
         Weight     BIGINT,
-        College    VARCHAR(40),
+        College    TEXT,
         Salary     FLOAT
     )
     WRAPPER parquet_fdw
@@ -269,7 +269,7 @@ Loading a table from a directory of Parquet files on HDFS
 .. code-block:: postgres
 
    CREATE FOREIGN TABLE ext_users
-     (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL)  
+     (id INT NOT NULL, name TEXT NOT NULL, email TEXT NOT NULL)  
    WRAPPER parquet_fdw
    OPTIONS
      (
@@ -284,7 +284,7 @@ Loading a table from a bucket of files on S3
 .. code-block:: postgres
 
    CREATE FOREIGN TABLE ext_users
-     (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL)  
+     (id INT NOT NULL, name TEXT NOT NULL, email TEXT NOT NULL)  
    WRAPPER parquet_fdw
    OPTIONS
      ( LOCATION = 's3://pp-secret-bucket/users/*.parquet',
diff --git a/guides/migration/to_csv.sql b/data_ingestion/to_csv.sql
similarity index 100%
rename from guides/migration/to_csv.sql
rename to data_ingestion/to_csv.sql
diff --git a/data_type_guides/converting_and_casting_types.rst b/data_type_guides/converting_and_casting_types.rst
new file mode 100644
index 000000000..bb6475646
--- /dev/null
+++ b/data_type_guides/converting_and_casting_types.rst
@@ -0,0 +1,28 @@
+.. _converting_and_casting_types:
+
+*************************
+Converting and Casting Types
+*************************
+SQream supports explicit and implicit casting and type conversion. The system may automatically add implicit casts when combining different data types in the same expression. In many cases, while the details related to this are not important, they can affect the query results of a query. When necessary, an explicit cast can be used to override the automatic cast added by SQream DB.
+
+For example, the ANSI standard defines a ``SUM()`` aggregation over an ``INT`` column as an ``INT``. However, when dealing with large amounts of data this could cause an overflow. 
+
+You can rectify this by casting the value to a larger data type, as shown below:
+
+.. code-block:: postgres
+
+   SUM(some_int_column :: BIGINT)
+
+SQream supports the following three data conversion types:
+
+* ``CAST( TO )``, to convert a value from one type to another. For example, ``CAST('1997-01-01' TO DATE)``, ``CAST(3.45 TO SMALLINT)``, ``CAST(some_column TO VARCHAR(30))``.
+
+   ::
+  
+* `` :: ``, a shorthand for the ``CAST`` syntax. For example, ``'1997-01-01' :: DATE``, ``3.45 :: SMALLINT``, ``(3+5) :: BIGINT``.
+
+   ::
+  
+* See the :ref:`SQL functions reference ` for additional functions that convert from a specific value which is not an SQL type, such as :ref:`from_unixts`, etc.
+
+.. note:: SQream interprets integer constants exceeding the maximum bigint value as float constants, which may cause precision loss.
\ No newline at end of file
diff --git a/data_type_guides/index.rst b/data_type_guides/index.rst
new file mode 100644
index 000000000..6afec60cf
--- /dev/null
+++ b/data_type_guides/index.rst
@@ -0,0 +1,14 @@
+.. _data_type_guides:
+
+*************************
+Data Type Guides
+*************************
+This section describes the following:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   converting_and_casting_types
+   supported_data_types
+   supported_casts
\ No newline at end of file
diff --git a/data_type_guides/sql_data_types_boolean.rst b/data_type_guides/sql_data_types_boolean.rst
new file mode 100644
index 000000000..84b7c14ce
--- /dev/null
+++ b/data_type_guides/sql_data_types_boolean.rst
@@ -0,0 +1,53 @@
+.. _sql_data_types_boolean:
+
+*************************
+Boolean
+*************************
+The following table describes the Boolean data type.
+
+.. list-table::
+   :widths: 30 30 30
+   :header-rows: 1
+   
+   * - Values
+     - Syntax
+     - Data Size (Not Null, Uncompressed)	 
+   * - ``true``, ``false`` (case sensitive)
+     - When loading from CSV, ``BOOL`` columns can accept ``0`` as ``false`` and ``1`` as ``true``.
+     - 1 byte, but resulting average data sizes may be lower after compression.
+	 
+Boolean Examples
+^^^^^^^^^^
+The following is an example of the Boolean syntax:
+
+.. code-block:: postgres
+   
+   CREATE TABLE animals (name TEXT, is_angry BOOL);
+   
+   INSERT INTO animals VALUES ('fox',true), ('cat',true), ('kiwi',false);
+   
+   SELECT name, CASE WHEN is_angry THEN 'Is really angry!' else 'Is not angry' END FROM animals;
+   
+The following is an example of the correct output:
+
+.. code-block:: text
+
+   "fox","Is really angry!"
+   "cat","Is really angry!"
+   "kiwi","Is not angry"
+
+Boolean Casts and Conversions
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The following table shows the possible Boolean value conversions:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Type
+     - Details
+   * - ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT``
+     - ``true`` → ``1``, ``false`` → ``0``
+   * - ``REAL``, ``DOUBLE``
+     - ``true`` → ``1.0``, ``false`` → ``0.0``
\ No newline at end of file
diff --git a/data_type_guides/sql_data_types_date.rst b/data_type_guides/sql_data_types_date.rst
new file mode 100644
index 000000000..da83f80cc
--- /dev/null
+++ b/data_type_guides/sql_data_types_date.rst
@@ -0,0 +1,112 @@
+.. _sql_data_types_date:
+
+*************************
+Date
+*************************
+``DATE`` is a type designed for storing year, month, and day. ``DATETIME`` is a type designed for storing year, month, day, hour, minute, seconds, and milliseconds in UTC with 1 millisecond precision.
+
+
+Date Types
+^^^^^^^^^^^^^^^^^^^^^^
+The following table describes the Date types:
+
+.. list-table:: Date Types
+   :widths: auto
+   :header-rows: 1
+   
+   * - Name
+     - Details
+     - Data Size (Not Null, Uncompressed)
+     - Example
+   * - ``DATE``
+     - Date
+     - 4 bytes
+     - ``'1955-11-05'``
+   * - ``DATETIME``
+     - Date and time pairing in UTC
+     - 8 bytes
+     - ``'1955-11-05 01:24:00.000'``
+
+Aliases
+^^^^^^^^^^
+
+``DATETIME`` is also known as ``TIMESTAMP`` or ``DATETIME2``.
+
+
+Syntax
+^^^^^^^^
+``DATE`` values are formatted as string literals. 
+
+The following is an example of the DATETIME syntax:
+
+.. code-block:: console
+     
+   '1955-11-05'
+
+.. code-block:: console
+     
+   date '1955-11-05'
+
+``DATETIME`` values are formatted as string literals conforming to `ISO 8601 `_.
+
+The following is an example of the DATETIME syntax:
+
+
+.. code-block:: console
+     
+   '1955-11-05 01:26:00'
+
+SQream attempts to guess if the string literal is a date or datetime based on context, for example when used in date-specific functions.
+
+Size
+^^^^^^
+A ``DATE`` column is 4 bytes in length, while a ``DATETIME`` column is 8 bytes in length.
+
+However, the size of these values is compressed by SQream DB.
+
+Date Examples
+^^^^^^^^^^
+The following is an example of the Date syntax:
+
+.. code-block:: postgres
+   
+   CREATE TABLE important_dates (a DATE, b DATETIME);
+
+   INSERT INTO important_dates VALUES ('1997-01-01', '1955-11-05 01:24');
+
+   SELECT * FROM important_dates;
+   
+The following is an example of the correct output:
+
+.. code-block:: text
+
+   1997-01-01,1955-11-05 01:24:00.0
+   
+The following is an example of the Datetime syntax:
+
+.. code-block:: postgres
+   
+   SELECT a :: DATETIME, b :: DATE FROM important_dates;
+   
+The following is an example of the correct output:
+
+.. code-block:: text
+
+   1997-01-01 00:00:00.0,1955-11-05
+   
+
+.. warning:: Some client applications may alter the ``DATETIME`` value by modifying the timezone.
+
+Date Casts and Conversions
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The following table shows the possible ``DATE`` and ``DATETIME`` value conversions:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Type
+     - Details
+   * - ``VARCHAR(n)``
+     - ``'1997-01-01'`` → ``'1997-01-01'``, ``'1955-11-05 01:24'`` → ``'1955-11-05 01:24:00.000'``
\ No newline at end of file
diff --git a/data_type_guides/sql_data_types_floating_point.rst b/data_type_guides/sql_data_types_floating_point.rst
new file mode 100644
index 000000000..18227140c
--- /dev/null
+++ b/data_type_guides/sql_data_types_floating_point.rst
@@ -0,0 +1,80 @@
+.. _sql_data_types_floating_point:
+
+*************************
+Floating Point
+*************************
+The **Floating Point** data types (``REAL`` and ``DOUBLE``) store extremely close value approximations, and are therefore recommended for values that tend to be inexact, such as Scientific Notation. While Floating Point generally runs faster than Numeric, it has a lower precision of ``9`` (``REAL``) or ``17`` (``DOUBLE``) compared to Numeric's ``38``. For operations that require a higher level of precision, using :ref:`Numeric ` is recommended.
+
+The floating point representation is based on `IEEE 754 `_.
+
+Floating Point Types
+^^^^^^^^^^^^^^^^^^^^^^
+The following table describes the Floating Point data types.
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Name
+     - Details
+     - Data Size (Not Null, Uncompressed)
+     - Example
+   * - ``REAL``
+     - Single precision floating point (inexact)
+     - 4 bytes
+     - ``3.141``
+   * - ``DOUBLE``
+     - Double precision floating point (inexact)
+     - 8 bytes
+     - ``0.000003``
+	 
+The following table shows information relevant to the Floating Point data types.
+
+.. list-table::
+   :widths: 30 30 30
+   :header-rows: 1
+   
+   * - Aliases
+     - Syntax
+     - Data Size (Not Null, Uncompressed)	 
+   * - ``DOUBLE`` is also known as ``FLOAT``.
+     - A double precision floating point can be entered as a regular literal, such as ``3.14``, ``2.718``, ``.34``, or ``2.71e-45``. To enter a ``REAL`` floating point number, cast the value. For example, ``(3.14 :: REAL)``.
+     - Floating point types are either 4 or 8 bytes, but size could be lower after compression.
+
+Floating Point Examples
+^^^^^^^^^^
+The following are examples of the Floating Point syntax:
+
+.. code-block:: postgres
+   
+   CREATE TABLE cool_numbers (a REAL NOT NULL, b DOUBLE);
+   
+   INSERT INTO cool_numbers VALUES (1,2), (3.14159265358979, 2.718281828459);
+   
+   SELECT * FROM cool_numbers;
+
+.. code-block:: text
+
+   1.0,2.0
+   3.1415927,2.718281828459
+
+.. note:: Most SQL clients control display precision of floating point numbers, and values may appear differently in some clients.
+
+Floating Point Casts and Conversions
+^^^^^^^^^^^^^^^^^^^^^^^
+The following table shows the possible Floating Point value conversions:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Type
+     - Details
+   * - ``BOOL``
+     - ``1.0`` → ``true``, ``0.0`` → ``false``
+   * - ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT``
+     - ``2.0`` → ``2``, ``3.14159265358979`` → ``3``, ``2.718281828459`` → ``2``, ``0.5`` → ``0``, ``1.5`` → ``1``
+   * - ``VARCHAR(n)`` (n > 6 recommended)
+     - ``1`` → ``'1.0000'``, ``3.14159265358979`` → ``'3.1416'``
+
+.. note:: As shown in the above examples, casting ``real`` to ``int`` rounds down.
\ No newline at end of file
diff --git a/data_type_guides/sql_data_types_integer.rst b/data_type_guides/sql_data_types_integer.rst
new file mode 100644
index 000000000..9d4210731
--- /dev/null
+++ b/data_type_guides/sql_data_types_integer.rst
@@ -0,0 +1,83 @@
+.. _sql_data_types_integer:
+
+*************************
+Integer
+*************************
+Integer data types are designed to store whole numbers.
+
+For more information about identity sequences (sometimes called auto-increment or auto-numbers), see :ref:`identity`.
+
+Integer Types
+^^^^^^^^^^^^^^^^^^^
+The following table describes the Integer types.
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Name
+     - Details
+     - Data Size (Not Null, Uncompressed)
+     - Example
+   * - ``TINYINT``
+     - Unsigned integer (0 - 255)
+     - 1 byte
+     - ``5``
+   * - ``SMALLINT``
+     - Integer (-32,768 - 32,767)
+     - 2 bytes
+     - ``-155``
+   * - ``INT``
+     - Integer (-2,147,483,648 - 2,147,483,647)
+     - 4 bytes
+     - ``1648813``
+   * - ``BIGINT``
+     - Integer (-9,223,372,036,854,775,808 - 9,223,372,036,854,775,807)
+     - 8 bytes
+     - ``36124441255243``	 
+	 
+The following table describes the Integer data type.
+	 
+.. list-table::
+   :widths: 25 25
+   :header-rows: 1
+   
+   * - Syntax
+     - Data Size (Not Null, Uncompressed)	 
+   * - An integer can be entered as a regular literal, such as ``12``, ``-365``.
+     - Integer types range between 1, 2, 4, and 8 bytes - but resulting average data sizes could be lower after compression.
+
+Integer Examples
+^^^^^^^^^^
+The following is an example of the Integer syntax:
+
+.. code-block:: postgres
+   
+   CREATE TABLE cool_numbers (a INT NOT NULL, b TINYINT, c SMALLINT, d BIGINT);
+   
+   INSERT INTO cool_numbers VALUES (1,2,3,4), (-5, 127, 32000, 45000000000);
+   
+   SELECT * FROM cool_numbers;
+   
+The following is an example of the correct output:
+
+.. code-block:: text
+
+   1,2,3,4
+   -5,127,32000,45000000000
+
+Integer Casts and Conversions
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The following table shows the possible Integer value conversions:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Type
+     - Details
+   * - ``REAL``, ``DOUBLE``
+     - ``1`` → ``1.0``, ``-32`` → ``-32.0``
+   * - ``VARCHAR(n)`` (All numberic values must fit in the string length)
+     - ``1`` → ``'1'``, ``2451`` → ``'2451'``
\ No newline at end of file
diff --git a/data_type_guides/sql_data_types_numeric.rst b/data_type_guides/sql_data_types_numeric.rst
new file mode 100644
index 000000000..8a78e19fa
--- /dev/null
+++ b/data_type_guides/sql_data_types_numeric.rst
@@ -0,0 +1,38 @@
+.. _sql_data_types_numeric:
+
+*************************
+Numeric
+*************************
+The **Numeric** data type (also known as **Decimal**) is recommended for values that tend to occur as exact decimals, such as in Finance. While Numeric has a fixed precision of ``38``, higher than ``REAL`` (``9``) or ``DOUBLE`` (``17``), it runs calculations more slowly. For operations that require faster performance, using :ref:`Floating Point ` is recommended.
+
+The correct syntax for Numeric is ``numeric(p, s)``), where ``p`` is the total number of digits (``38`` maximum), and ``s`` is the total number of decimal digits.
+
+Numeric Examples
+^^^^^^^^^^
+
+The following is an example of the Numeric syntax:
+
+.. code-block:: postgres
+
+   $ create or replace table t(x numeric(20, 10), y numeric(38, 38));
+   $ insert into t values(1234567890.1234567890, 0.123245678901234567890123456789012345678);
+   $ select x + y from t;
+   
+The following table shows information relevant to the Numeric data type:
+
+.. list-table::
+   :widths: 30 30 30
+   :header-rows: 1
+   
+   * - Description
+     - Data Size (Not Null, Uncompressed)
+     - Example	 
+   * - 38 digits
+     - 16 bytes
+     - ``0.123245678901234567890123456789012345678``
+
+Numeric supports the following operations:
+
+   * All join types.
+   * All aggregation types (not including Window functions).
+   * Scalar functions (not including some trigonometric and logarithmic functions).
\ No newline at end of file
diff --git a/data_type_guides/sql_data_types_string.rst b/data_type_guides/sql_data_types_string.rst
new file mode 100644
index 000000000..539198fe6
--- /dev/null
+++ b/data_type_guides/sql_data_types_string.rst
@@ -0,0 +1,86 @@
+.. _sql_data_types_string:
+
+*************************
+String
+*************************
+The ``TEXT`` type is designed for storing text or strings of characters.
+
+.. note:: The data type ``NVARCHAR`` has been deprecated by ``TEXT`` as of version 2020.1.
+
+String Types
+^^^^^^^^^^^^^^^^^^^^^^
+The following table describes the String types:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Name
+     - Details
+     - Data Size (Not Null, Uncompressed)
+     - Example
+   * - ``TEXT [(n)]``
+     - Variable length string - UTF-8 unicode. ``NVARCHAR`` is synonymous with ``TEXT``.
+     - Up to ``4*n`` bytes
+     - ``'キウイは楽しい鳥です'``
+
+Length
+^^^^^^^^^
+When using ``TEXT``, specifying a size is optional. If not specified, the text field carries no constraints. To limit the size of the input, use ``VARCHAR(n)`` or ``TEXT(n)``, where ``n`` is the permitted number of characters.
+
+The following apply to setting the String type length:
+
+* If the data exceeds the column length limit on ``INSERT`` or ``COPY`` operations, SQream DB will return an error.
+* When casting or converting, the string has to fit in the target. For example, ``'Kiwis are weird birds' :: TEXT`` will return an error. Use ``SUBSTRING`` to truncate the length of the string.
+* ``TEXT`` strings are padded with spaces.
+
+Syntax
+^^^^^^^^
+String types can be written with standard SQL string literals, which are enclosed with single quotes, such as
+``'Kiwi bird'``. To include a single quote in the string, use double quotations, such as ``'Kiwi bird''s wings are tiny'``. String literals can also be dollar-quoted with the dollar sign ``$``, such as ``$$Kiwi bird's wings are tiny$$`` is the same as ``'Kiwi bird''s wings are tiny'``.
+
+Size
+^^^^^^
+``TEXT(n)`` can occupy up to *4*n* bytes. However, the size of strings is variable and is compressed by SQream.
+
+String Examples
+^^^^^^^^^^
+The following is an example of the String syntax: 
+
+.. code-block:: postgres
+   
+   CREATE TABLE cool_strings (a TEXT NOT NULL, b TEXT);
+   
+   INSERT INTO cool_strings VALUES ('hello world', 'Hello to kiwi birds specifically');
+   
+   INSERT INTO cool_strings VALUES ('This is ASCII only', 'But this column can contain 中文文字');
+
+   SELECT * FROM cool_strings;
+   
+The following is an example of the correct output:
+
+.. code-block:: text
+
+   hello world	,Hello to kiwi birds specifically
+   This is ASCII only,But this column can contain 中文文字
+
+.. note:: Most clients control the display precision of floating point numbers, and values may appear differently in some clients.
+
+String Casts and Conversions
+^^^^^^^^^^^^^^^^^^^^^^^
+The following table shows the possible String value conversions:
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Type
+     - Details
+   * - ``BOOL``
+     - ``'true'`` → ``true``, ``'false'`` → ``false``
+   * - ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT``
+     - ``'2'`` → ``2``, ``'-128'`` → ``-128``
+   * - ``REAL``, ``DOUBLE``
+     - ``'2.0'`` → ``2.0``, ``'3.141592'`` → ``3.141592``
+   * - ``DATE``, ``DATETIME``
+     - Requires a supported format, such as ``'1955-11-05`` → ``date '1955-11-05'``, ``'1955-11-05 01:24:00.000'`` → ``'1955-11-05 01:24:00.000'``
\ No newline at end of file
diff --git a/data_type_guides/supported_casts.rst b/data_type_guides/supported_casts.rst
new file mode 100644
index 000000000..34053b647
--- /dev/null
+++ b/data_type_guides/supported_casts.rst
@@ -0,0 +1,17 @@
+.. _supported_casts:
+
+*************************
+Supported Casts
+*************************
+The **Supported Casts** section describes supported casts for the following types:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   sql_data_types_numeric
+   sql_data_types_boolean
+   sql_data_types_integer
+   sql_data_types_floating_point
+   sql_data_types_string
+   sql_data_types_date
\ No newline at end of file
diff --git a/data_type_guides/supported_data_types.rst b/data_type_guides/supported_data_types.rst
new file mode 100644
index 000000000..2743b054b
--- /dev/null
+++ b/data_type_guides/supported_data_types.rst
@@ -0,0 +1,80 @@
+.. _supported_data_types:
+
+*************************
+Supported Data Types
+*************************
+The **Supported Data Types** page describes SQream's supported data types:
+
+The following table shows the supported data types.
+
+.. list-table::
+   :widths: 20 15 20 30 20
+   :header-rows: 1
+   
+   * - Name
+     - Description
+     - Data Size (Not Null, Uncompressed)
+     - Example
+     - Alias
+   * - ``BOOL``
+     - Boolean values (``true``, ``false``)
+     - 1 byte
+     - ``true``
+     - ``BIT``
+   * - ``TINYINT``
+     - Unsigned integer (0 - 255)
+     - 1 byte
+     - ``5``
+     - NA
+   * - ``SMALLINT``
+     - Integer (-32,768 - 32,767)
+     - 2 bytes
+     - ``-155``
+     - NA
+   * - ``INT``
+     - Integer (-2,147,483,648 - 2,147,483,647)
+     - 4 bytes
+     - ``1648813``
+     - ``INTEGER``
+   * - ``BIGINT``
+     - Integer (-9,223,372,036,854,775,808 - 9,223,372,036,854,775,807)
+     - 8 bytes
+     - ``36124441255243``
+     - ``NUMBER``
+   * - ``REAL``
+     - Floating point (inexact)
+     - 4 bytes
+     - ``3.141``
+     - NA
+   * - ``DOUBLE``
+     - Floating point (inexact)
+     - 8 bytes
+     - ``0.000003``
+     - ``FLOAT``/``DOUBLE PRECISION``
+   * - ``TEXT [(n)]``, ``NVARCHAR (n)``
+     - Variable length string - UTF-8 unicode
+     - Up to ``4*n`` bytes
+     - ``'キウイは楽しい鳥です'``
+     - ``CHAR VARYING``, ``CHAR``, ``CHARACTER VARYING``, ``CHARACTER``, ``NATIONAL CHARACTER VARYING``, ``NATIONAL CHARACTER``, ``NCHAR VARYING``, ``NCHAR``, ``NVARCHAR``
+   * - ``NUMERIC``
+     -  38 digits
+     - 16 bytes
+     - ``0.123245678901234567890123456789012345678``
+     - ``DECIMAL``
+   * - ``VARCHAR (n)``
+     - Variable length string - ASCII only
+     - ``n`` bytes
+     - ``'Kiwis have tiny wings, but cannot fly.'``
+     - ``SQL VARIANT``
+   * - ``DATE``
+     - Date
+     - 4 bytes
+     - ``'1955-11-05'``
+     - NA
+   * - ``DATETIME``
+     - Date and time pairing in UTC
+     - 8 bytes
+     - ``'1955-11-05 01:24:00.000'``
+     -  ``TIMESTAMP``, ``DATETIME2``
+
+.. note:: SQream compresses all columns and types. The data size noted is the maximum data size allocation for uncompressed data.
\ No newline at end of file
diff --git a/external_storage_platforms/hdfs.rst b/external_storage_platforms/hdfs.rst
new file mode 100644
index 000000000..a6ffe0ead
--- /dev/null
+++ b/external_storage_platforms/hdfs.rst
@@ -0,0 +1,252 @@
+.. _hdfs:
+
+.. _back_to_top_hdfs:
+
+Using SQream in an HDFS Environment
+=======================================
+
+.. _configuring_an_hdfs_environment_for_the_user_sqream:
+
+Configuring an HDFS Environment for the User **sqream**
+----------------------------------------------------------
+
+This section describes how to configure an HDFS environment for the user **sqream** and is only relevant for users with an HDFS environment.
+
+**To configure an HDFS environment for the user sqream:**
+
+1. Open your **bash_profile** configuration file for editing:
+
+   .. code-block:: console
+     
+       $ vim /home/sqream/.bash_profile
+       
+..
+   Comment: - see below; do we want to be a bit more specific on what changes we're talking about?
+
+   .. code-block:: console
+     
+      $ #PATH=$PATH:$HOME/.local/bin:$HOME/bin
+
+      $ #export PATH
+
+      $ # PS1
+      $ #MYIP=$(curl -s -XGET "http://ip-api.com/json" | python -c 'import json,sys; jstr=json.load(sys.stdin); print jstr["query"]')
+      $ #PS1="\[\e[01;32m\]\D{%F %T} \[\e[01;33m\]\u@\[\e[01;36m\]$MYIP \[\e[01;31m\]\w\[\e[37;36m\]\$ \[\e[1;37m\]"
+
+      $ SQREAM_HOME=/usr/local/sqream
+      $ export SQREAM_HOME
+
+      $ export JAVA_HOME=${SQREAM_HOME}/hdfs/jdk
+      $ export HADOOP_INSTALL=${SQREAM_HOME}/hdfs/hadoop
+      $ export CLASSPATH=`${HADOOP_INSTALL}/bin/hadoop classpath --glob`
+      $ export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_INSTALL}/lib/native
+      $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${SQREAM_HOME}/lib:$HADOOP_COMMON_LIB_NATIVE_DIR
+
+
+      $ PATH=$PATH:$HOME/.local/bin:$HOME/bin:${SQREAM_HOME}/bin/:${JAVA_HOME}/bin:$HADOOP_INSTALL/bin
+      $ export PATH
+
+2. Verify that the edits have been made:
+
+   .. code-block:: console
+     
+      source /home/sqream/.bash_profile
+       
+3. Check if you can access Hadoop from your machine:       
+       
+  .. code-block:: console
+     
+     $ hadoop fs -ls hdfs://:8020/
+      
+..
+   Comment: - 
+   **NOTICE:** If you cannot access Hadoop from your machine because it uses Kerberos, see `Connecting a SQream Server to Cloudera Hadoop with Kerberos `_
+
+
+4. Verify that an HDFS environment exists for SQream services:
+
+   .. code-block:: console
+     
+      $ ls -l /etc/sqream/sqream_env.sh
+	  
+.. _step_6:
+
+      
+5. If an HDFS environment does not exist for SQream services, create one (sqream_env.sh):
+   
+   .. code-block:: console
+     
+      $ #!/bin/bash
+
+      $ SQREAM_HOME=/usr/local/sqream
+      $ export SQREAM_HOME
+
+      $ export JAVA_HOME=${SQREAM_HOME}/hdfs/jdk
+      $ export HADOOP_INSTALL=${SQREAM_HOME}/hdfs/hadoop
+      $ export CLASSPATH=`${HADOOP_INSTALL}/bin/hadoop classpath --glob`
+      $ export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_INSTALL}/lib/native
+      $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${SQREAM_HOME}/lib:$HADOOP_COMMON_LIB_NATIVE_DIR
+
+
+      $ PATH=$PATH:$HOME/.local/bin:$HOME/bin:${SQREAM_HOME}/bin/:${JAVA_HOME}/bin:$HADOOP_INSTALL/bin
+      $ export PATH
+	  
+:ref:`Back to top `
+
+	  
+.. _authenticate_hadoop_servers_that_require_kerberos:
+
+Authenticating Hadoop Servers that Require Kerberos
+---------------------------------------------------
+
+If your Hadoop server requires Kerberos authentication, do the following:
+
+1. Create a principal for the user **sqream**.
+
+   .. code-block:: console
+   
+      $ kadmin -p root/admin@SQ.COM
+      $ addprinc sqream@SQ.COM
+      
+2. If you do not know yor Kerberos root credentials, connect to the Kerberos server as a root user with ssh and run **kadmin.local**:
+
+   .. code-block:: console
+   
+      $ kadmin.local
+      
+   Running **kadmin.local** does not require a password.
+
+3. If a password is not required, change your password to **sqream@SQ.COM**.
+
+   .. code-block:: console
+   
+      $ change_password sqream@SQ.COM
+
+4. Connect to the hadoop name node using ssh:
+
+   .. code-block:: console
+   
+      $ cd /var/run/cloudera-scm-agent/process
+
+5. Check the most recently modified content of the directory above:
+
+   .. code-block:: console
+   
+      $ ls -lrt
+
+6. Look for a recently updated folder containing the text **hdfs**.
+
+The following is an example of the correct folder name:
+
+   .. code-block:: console
+   
+      cd -hdfs-
+	  
+   This folder should contain a file named **hdfs.keytab** or another similar .keytab file.
+   
+
+ 
+..
+   Comment: - Does "something" need to be replaced with "file name"
+   
+
+7. Copy the .keytab file to user **sqream's** Home directory on the remote machines that you are planning to use Hadoop on.
+
+8. Copy the following files to the **sqream sqream@server:/hdfs/hadoop/etc/hadoop:** directory:
+
+   * core-site.xml
+   * hdfs-site.xml
+
+9. Connect to the sqream server and verify that the .keytab file's owner is a user sqream and is granted the correct permissions:
+
+   .. code-block:: console
+   
+      $ sudo chown sqream:sqream /home/sqream/hdfs.keytab
+      $ sudo chmod 600 /home/sqream/hdfs.keytab
+
+10. Log into the sqream server.
+
+11. Log in as the user **sqream**.
+
+12. Navigate to the Home directory and check the name of a Kerberos principal represented by the following .keytab file:
+
+   .. code-block:: console
+   
+      $ klist -kt hdfs.keytab
+
+   The following is an example of the correct output:
+
+   .. code-block:: console
+   
+      $ sqream@Host-121 ~ $ klist -kt hdfs.keytab
+      $ Keytab name: FILE:hdfs.keytab
+      $ KVNO Timestamp           Principal
+      $ ---- ------------------- ------------------------------------------------------
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 HTTP/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+      $    5 09/15/2020 18:03:05 hdfs/nn1@SQ.COM
+
+13. Verify that the hdfs service named **hdfs/nn1@SQ.COM** is shown in the generated output above.
+
+14. Run the following:
+
+   .. code-block:: console
+   
+      $ kinit -kt hdfs.keytab hdfs/nn1@SQ.COM
+
+15. Check the output:
+  
+   .. code-block:: console
+   
+      $ klist
+      
+   The following is an example of the correct output:
+
+   .. code-block:: console
+   
+      $ Ticket cache: FILE:/tmp/krb5cc_1000
+      $ Default principal: sqream@SQ.COM
+      $ 
+      $ Valid starting       Expires              Service principal
+      $ 09/16/2020 13:44:18  09/17/2020 13:44:18  krbtgt/SQ.COM@SQ.COM
+
+16. List the files located at the defined server name or IP address:
+
+   .. code-block:: console
+   
+      $ hadoop fs -ls hdfs://:8020/
+
+17. Do one of the following:
+
+    * If the list below is output, continue with Step 16.
+    * If the list is not output, verify that your environment has been set up correctly.
+	
+If any of the following are empty, verify that you followed :ref:`Step 6 ` in the **Configuring an HDFS Environment for the User sqream** section above correctly:
+
+  .. code-block:: console
+   
+      $ echo $JAVA_HOME
+      $ echo $SQREAM_HOME
+      $ echo $CLASSPATH
+      $ echo $HADOOP_COMMON_LIB_NATIVE_DIR
+      $ echo $LD_LIBRARY_PATH
+      $ echo $PATH
+
+18. Verify that you copied the correct keytab file.
+
+19. Review this procedure to verify that you have followed each step.
+
+:ref:`Back to top `
\ No newline at end of file
diff --git a/external_storage_platforms/index.rst b/external_storage_platforms/index.rst
new file mode 100644
index 000000000..92c35ee63
--- /dev/null
+++ b/external_storage_platforms/index.rst
@@ -0,0 +1,25 @@
+.. _external_storage_platforms:
+
+***********************
+External Storage Platforms
+***********************
+SQream supports the following external storage platforms:
+
+.. toctree::
+   :maxdepth: 1
+   :titlesonly:
+
+   s3
+   hdfs
+   
+For more information, see the following:
+
+* :ref:`foreign_tables`
+
+   ::
+   
+* :ref:`copy_from`
+
+   ::
+   
+* :ref:`copy_to`
diff --git a/guides/features/external_data/s3.rst b/external_storage_platforms/s3.rst
similarity index 53%
rename from guides/features/external_data/s3.rst
rename to external_storage_platforms/s3.rst
index 2a1d9f628..f8bb2dab6 100644
--- a/guides/features/external_data/s3.rst
+++ b/external_storage_platforms/s3.rst
@@ -1,62 +1,56 @@
 .. _s3:
 
 ***********************
-S3
+Inserting Data Using Amazon S3
 ***********************
+SQream uses a native S3 connector for inserting data. The ``s3://`` URI specifies an external file path on an S3 bucket. File names may contain wildcard characters, and the files can be in CSV or columnar format, such as Parquet and ORC.
 
-SQream DB has a native S3 connector for inserting data. The ``s3://`` URI specifies an external file path on an S3 bucket.
+The **Amazon S3** describes the following topics:
 
-File names may contain wildcard characters and the files can be a CSV or columnar format like Parquet and ORC.
-
-
-.. contents:: In this topic:
+.. contents::
    :local:
    
-S3 configuration
+S3 Configuration
 ==============================
+Any database host with access to S3 endpoints can access S3 without any configuration. To read files from an S3 bucket, the database must have listable files.
 
-Any SQream DB host that has access to S3 endpoints can access S3 without any configuration.
-
-To read files from an S3 bucket, it must have listable files.
-
-S3 URI format
+S3 URI Format
 ===============
-
 With S3, specify a location for a file (or files) when using :ref:`copy_from` or :ref:`external_tables`.
 
-The general S3 syntax is:
-``s3://bucket_name/path``
+The following is an example of the general S3 syntax:
+
+.. code-block:: console
+ 
+   s3://bucket_name/path
 
 Authentication
 =================
-
-SQream DB supports ``AWS ID`` and ``AWS SECRET`` authentication.
-These should be specified when executing a statement.
+SQream supports ``AWS ID`` and ``AWS SECRET`` authentication. These should be specified when executing a statement.
 
 Examples
 ==========
+Use a foreign table to stage data from S3 before loading from CSV, Parquet, or ORC files.
 
-Use an external table to stage data from S3 before loading from CSV, Parquet or ORC files.
+The **Examples** section includes the following examples:
 
-Planning for data staging
+.. contents::
+   :local:
+   :depth: 1
+Planning for Data Staging
 --------------------------------
-
-For the following examples, we will want to interact with a CSV file. Here's a peek at the table contents:
+The examples in this section are based on a CSV file, as shown in the following table:
 
 .. csv-table:: nba.csv
    :file: ../nba-t10.csv
    :widths: auto
    :header-rows: 1 
 
-The file is stored on :ref:`s3`, at ``s3://sqream-demo-data/nba_players.csv``.
-This bucket is public and listable.
+The file is stored on Amazon S3, and this bucket is public and listable. To create a matching ``CREATE FOREIGN TABLE`` statement you can make note of the file structure.
 
-We will make note of the file structure, to create a matching ``CREATE FOREIGN TABLE`` statement.
-
-Creating the external table
+Creating a Foreign Table
 -----------------------------
-
-Based on the source file structure, we we :ref:`create a foreign table` with the appropriate structure, and point it to the file.
+Based on the source file's structure, you can create a foreign table with the appropriate structure, and point it to your file as shown in the following example:
 
 .. code-block:: postgres
    
@@ -76,18 +70,20 @@ Based on the source file structure, we we :ref:`create a foreign table`.
+* **Using SQream in an HDFS environment** - see :ref:`hdfs`.
 
-Let's peek at the data from the external table:
+Querying Foreign Tables
+------------------------------
+The following shows the data in the foreign table:
 
 .. code-block:: psql
    
@@ -105,19 +101,23 @@ Let's peek at the data from the external table:
    Terry Rozier  | Boston Celtics |     12 | PG       |  22 | 6-2    |    190 | Louisville        |  1824360
    Marcus Smart  | Boston Celtics |     36 | PG       |  22 | 6-4    |    220 | Oklahoma State    |  3431040
    
-Bulk loading a file from a public S3 bucket
+Bulk Loading a File from a Public S3 Bucket
 ----------------------------------------------
+The ``COPY FROM`` command can also be used to load data without staging it first.
 
-The :ref:`copy_from` command can also be used to load data without staging it first.
+.. note:: The bucket must be publicly available and objects can be listed.
 
-.. note:: The bucket must be publicly available and objects can be listed
+The following is an example of bulk loading a file from a public S3 bucket:
 
 .. code-block:: postgres
 
    COPY nba FROM 's3://sqream-demo-data/nba.csv' WITH OFFSET 2 RECORD DELIMITER '\r\n';
+   
+For more information on the ``COPY FROM`` command, see :ref:`copy_from`.
 
-Loading files from an authenticated S3 bucket
+Loading Files from an Authenticated S3 Bucket
 ---------------------------------------------------
+The following is an example of loading fles from an authenticated S3 bucket:
 
 .. code-block:: postgres
 
diff --git a/guides/features/compression.rst b/feature_guides/compression.rst
similarity index 99%
rename from guides/features/compression.rst
rename to feature_guides/compression.rst
index 662eda044..8e15c5686 100644
--- a/guides/features/compression.rst
+++ b/feature_guides/compression.rst
@@ -15,7 +15,7 @@ When data is stored in a columnar format, it is often in a common format. This i
 
 Because encoding uses specific data formats and encodings, it increases performance and reduces data size. 
 
-SQream DB encodes data in several ways depending on the data type. For example, a date is stored as an integer, with March 1st 1CE as the start. This is a lot more efficient than encoding the date as a string, and offers a wider range than storing it relative to the Unix Epoch.
+SQream DB encodes data in several ways depending on the data type. For example, a date is stored as an integer, with March 1st 1CE as the start. This is a lot more efficient than encoding the date as a string, and offers a wider range than storing it relative to the Unix Epoch. 
 
 Compression
 ==============
@@ -148,6 +148,7 @@ For example:
       id BIGINT NOT NULL CHECK('CS "sequence"'),
       y VARCHAR(110) CHECK('CS "lz4"'), -- General purpose text compression
       z VARCHAR(80) CHECK('CS "dict"'), -- Low cardinality column
+
       
    );
 
diff --git a/guides/features/concurrency_and_locks.rst b/feature_guides/concurrency_and_locks.rst
similarity index 73%
rename from guides/features/concurrency_and_locks.rst
rename to feature_guides/concurrency_and_locks.rst
index f917bb128..2a85d2642 100644
--- a/guides/features/concurrency_and_locks.rst
+++ b/feature_guides/concurrency_and_locks.rst
@@ -1,16 +1,16 @@
 .. _concurrency_and_locks:
 
 ***********************
-Concurrency and locks
+Concurrency and Locks
 ***********************
 
-Locks are used in SQream DB to provide consistency when there are multiple concurrent transactions updating the database.
+Locks are used in SQream DB to provide consistency when there are multiple concurrent transactions updating the database. 
 
 Read only transactions are never blocked, and never block anything. Even if you drop a database while concurrently running a query on it, both will succeed correctly (as long as the query starts running before the drop database commits).
 
 .. _locking_modes:
 
-Locking modes
+Locking Modes
 ================
 
 SQream DB has two kinds of locks:
@@ -27,7 +27,7 @@ SQream DB has two kinds of locks:
    
    This lock allows other statements to insert or delete data from a table, but they'll have to wait in order to run DDL.
 
-When are locks obtained?
+When are Locks Obtained?
 ============================
 
 .. list-table::
@@ -64,23 +64,7 @@ When are locks obtained?
 
 Statements that wait will exit with an error if they hit the lock timeout. The default timeout is 3 seconds, see ``statementLockTimeout``.
 
-Global locks
-----------------
-
-Some operations require exclusive global locks at the cluster level. These usually short-lived locks will be obtained for the following operations:
-
-   * :ref:`create_database`
-   * :ref:`create_role`
-   * :ref:`create_table`
-   * :ref:`alter_role`
-   * :ref:`alter_table`
-   * :ref:`drop_database`
-   * :ref:`drop_role`
-   * :ref:`drop_table`
-   * :ref:`grant`
-   * :ref:`revoke`
-
-Monitoring locks
+Monitoring Locks
 ===================
 
 Monitoring locks across the cluster can be useful when transaction contention takes place, and statements appear "stuck" while waiting for a previous statement to release locks.
@@ -101,34 +85,4 @@ In this example, we create a table based on results (:ref:`create_table_as`), bu
    287          | CREATE OR REPLACE TABLE nba2 AS SELECT "Name" FROM nba WHERE REGEXP_COUNT("Name", '( )+', 8)>1; | sqream   | 192.168.1.91 | 5000 | table$t$public$nba2$Insert      | Exclusive | 2019-12-26 00:03:30  | 2019-12-26 00:03:30
    287          | CREATE OR REPLACE TABLE nba2 AS SELECT "Name" FROM nba WHERE REGEXP_COUNT("Name", '( )+', 8)>1; | sqream   | 192.168.1.91 | 5000 | table$t$public$nba2$Update      | Exclusive | 2019-12-26 00:03:30  | 2019-12-26 00:03:30
 
-
-Troubleshooting locks
-==========================
-
-Sometimes, a rare situation can occur where a lock is never freed. 
-
-The workflow for troubleshooting locks is:
-
-#. Identify which statement has obtained locks
-#. Understand if the statement is itself stuck, or waiting for another statement
-#. Try to abort the offending statement
-#. Force the stale locks to be removed
-
-Example
------------
-
-We will assume that the statement from the previous example is stuck (statement #\ ``287``). We can attempt to abort it using :ref:`stop_statement`:
-
-.. code-block:: psql
-
-   t=> SELECT STOP_STATEMENT(287);
-   executed
-
-If the locks still appear in the :ref:`show_locks` utility, we can force remove the stale locks:
-
-.. code-block:: psql
-
-   t=> SELECT RELEASE_DEFUNCT_LOCKS();
-   executed
-
-.. warning:: This operation can cause some statements to fail on the specific worker on which they are queued. This is intended as a "last resort" to solve stale locks.
+For more information on troubleshooting lock related issues, see :ref:`lock_related_issues`.
\ No newline at end of file
diff --git a/guides/features/concurrency_and_scaling_in_sqream.rst b/feature_guides/concurrency_and_scaling_in_sqream.rst
similarity index 94%
rename from guides/features/concurrency_and_scaling_in_sqream.rst
rename to feature_guides/concurrency_and_scaling_in_sqream.rst
index aafdda2b5..0370913fa 100644
--- a/guides/features/concurrency_and_scaling_in_sqream.rst
+++ b/feature_guides/concurrency_and_scaling_in_sqream.rst
@@ -1,10 +1,10 @@
 .. _concurrency_and_scaling_in_sqream:
 
 ***************************************
-Concurrency and scaling in SQream DB
+Concurrency and Scaling in SQream DB
 ***************************************
 
-A SQream DB cluster can concurrently run one regular statement per worker process. A number of small statements will execute alongside these statements without waiting or blocking anything. 
+A SQream DB cluster can concurrently run one regular statement per worker process. A number of small statements will execute alongside these statements without waiting or blocking anything.
 
 SQream DB supports ``n`` concurrent statements by having ``n`` workers in a cluster. Each worker uses a fixed slice of a GPU's memory, with usual values are around 8-16GB of GPU memory per worker. This size is ideal for queries running on large data with potentially large row sizes.
 
diff --git a/feature_guides/index.rst b/feature_guides/index.rst
new file mode 100644
index 000000000..1639ea378
--- /dev/null
+++ b/feature_guides/index.rst
@@ -0,0 +1,22 @@
+.. _feature_guides:
+
+***********************
+Feature Guides
+***********************
+The **Feature Guides** section describes background processes that SQream uses to manage several areas of operation, such as data ingestion, load balancing, and access control. 
+
+This section describes the following features: 
+
+.. toctree::
+   :maxdepth: 1
+   :titlesonly:
+   
+
+   compression
+   python_functions
+   saved_queries
+   viewing_system_objects_as_ddl
+   workload_manager
+   transactions
+   concurrency_and_locks
+   concurrency_and_scaling_in_sqream
\ No newline at end of file
diff --git a/guides/features/python_functions.rst b/feature_guides/python_functions.rst
similarity index 98%
rename from guides/features/python_functions.rst
rename to feature_guides/python_functions.rst
index 3d3c6f00f..3717cdcd8 100644
--- a/guides/features/python_functions.rst
+++ b/feature_guides/python_functions.rst
@@ -1,10 +1,10 @@
 .. _python_functions:
 
 *************************************
-Python UDF (user-defined functions)
+Python UDF (User-Defined Functions)
 *************************************
 
-User-defined functions (UDFs) are a feature that extends SQream DB's built in SQL functionality. SQream DB's Python UDFs allow developers to create new functionality in SQL by writing the lower-level language implementation in Python.
+User-defined functions (UDFs) are a feature that extends SQream DB's built in SQL functionality. SQream DB's Python UDFs allow developers to create new functionality in SQL by writing the lower-level language implementation in Python. 
 
 .. contents:: In this topic:
    :local:
diff --git a/feature_guides/sqream_installer_cli_ref_admin.rst b/feature_guides/sqream_installer_cli_ref_admin.rst
new file mode 100644
index 000000000..fad8107b9
--- /dev/null
+++ b/feature_guides/sqream_installer_cli_ref_admin.rst
@@ -0,0 +1,142 @@
+.. _sqream_installer_cli_ref_admin:
+
+*********************************
+SQream Installer
+*********************************
+``sqream-installer`` is an application that prepares and configures a dockerized SQream DB installation.
+
+This page serves as a reference for the options and parameters. 
+
+.. contents:: In this topic:
+   :local:
+
+
+Operations and flag reference
+===============================
+
+Command line flags
+-----------------------
+
+.. list-table:: 
+   :widths: auto
+   :header-rows: 1
+   
+   * - Flag
+     - Description
+   * - ``-i``
+     - Loads the docker images for installation
+   * - ``-k``
+     - Load new licenses from the ``license`` subdirectory
+   * - ``-K``
+     - Validate licenses
+   * - ``-f``
+     - Force overwrite any existing installation **and data directories currently in use**
+   * - ``-c ``
+     - Specifies a path to read and store configuration files in. Defaults to ``/etc/sqream``.
+   * - ``-v ``
+     - Specifies a path to the storage cluster. The path is created if it does not exist.
+   * - ``-l ``
+     - Specifies a path to store system startup logs. Defaults to ``/var/log/sqream``
+   * - ``-d ``
+     - Specifies a path to expose to SQream DB workers. To expose several paths, repeat the usage of this flag.
+   * - ``-s``
+     - Shows system settings
+   * - ``-r``
+     - Reset the system configuration. This flag can't be combined with other flags.
+
+Usage
+=============
+
+Install SQream DB for the first time
+----------------------------------------
+
+Assuming license package tarball has been placed in the ``license`` subfolder.
+
+* The path where SQream DB will store data is ``/home/rhendricks/sqream_storage``.
+
+* Logs will be stored in /var/log/sqream
+
+* Source CSV, Parquet, and ORC files can be accessed from ``/home/rhendricks/source_data``. All other directory paths are hidden from the Docker container.
+
+.. code-block:: console
+   
+   # ./sqream-install -i -k -v /home/rhendricks/sqream_storage -l /var/log/sqream -c /etc/sqream -d /home/rhendricks/source_data
+
+.. note:: Installation commands should be run with ``sudo`` or root access.
+
+Modify exposed directories
+-------------------------------
+
+To expose more directory paths for SQream DB to read and write data from, re-run the installer with additional directory flags.
+
+.. code-block:: console
+   
+   # ./sqream-install -d /home/rhendricks/more_source_data
+
+There is no need to specify the initial installation flags - only the modified exposed directory paths flag.
+
+
+Install a new license package
+----------------------------------
+
+Assuming license package tarball has been placed in the ``license`` subfolder.
+
+.. code-block:: console
+   
+   # ./sqream-install -k
+
+View system settings
+----------------------------
+
+This information may be useful to identify problems accessing directory paths, or locating where data is stored.
+
+.. code-block:: console
+   
+   # ./sqream-install -s
+   SQREAM_CONSOLE_TAG=1.7.4
+   SQREAM_TAG=2020.1
+   SQREAM_EDITOR_TAG=3.1.0
+   license_worker_0=[...]
+   license_worker_1=[...]
+   license_worker_2=[...]
+   license_worker_3=[...]
+   SQREAM_VOLUME=/home/rhendricks/sqream_storage
+   SQREAM_DATA_INGEST=/home/rhendricks/source_data
+   SQREAM_CONFIG_DIR=/etc/sqream/
+   LICENSE_VALID=true
+   SQREAM_LOG_DIR=/var/log/sqream/
+   SQREAM_USER=sqream
+   SQREAM_HOME=/home/sqream
+   SQREAM_ENV_PATH=/home/sqream/.sqream/env_file
+   PROCESSOR=x86_64
+   METADATA_PORT=3105
+   PICKER_PORT=3108
+   NUM_OF_GPUS=8
+   CUDA_VERSION=10.1
+   NVIDIA_SMI_PATH=/usr/bin/nvidia-smi
+   DOCKER_PATH=/usr/bin/docker
+   NVIDIA_DRIVER=418
+   SQREAM_MODE=single_host
+
+
+.. _upgrade_with_docker:
+
+Upgrading to a new version of SQream DB
+----------------------------------------------
+
+When upgrading to a new version with Docker, most settings don't need to be modified.
+
+The upgrade process replaces the existing docker images with new ones.
+
+#. Obtain the new tarball, and untar it to an accessible location. Enter the newly extracted directory.
+
+#. 
+   Install the new images
+   
+   .. code-block:: console
+   
+      # ./sqream-install -i
+
+#. The upgrade process will check for running SQream DB processes. If any are found running, the installer will ask to stop them in order to continue the upgrade process. Once all services are stopped, the new version will be loaded. 
+
+#. After the upgrade, open :ref:`sqream_console_cli_reference` and restart the desired services.
\ No newline at end of file
diff --git a/guides/features/transactions.rst b/feature_guides/transactions.rst
similarity index 97%
rename from guides/features/transactions.rst
rename to feature_guides/transactions.rst
index f63c3a9c6..675957d51 100644
--- a/guides/features/transactions.rst
+++ b/feature_guides/transactions.rst
@@ -4,7 +4,7 @@
 Transactions
 ***********************
 
-SQream DB supports serializable transactions. This is also called 'ACID compliance'.
+SQream DB supports serializable transactions. This is also called 'ACID compliance'. 
 
 The implementation of transactions means that commit, rollback and recovery are all extremely fast.
 
diff --git a/guides/features/workload_manager.rst b/feature_guides/workload_manager.rst
similarity index 58%
rename from guides/features/workload_manager.rst
rename to feature_guides/workload_manager.rst
index 03f99d093..ca545a2ca 100644
--- a/guides/features/workload_manager.rst
+++ b/feature_guides/workload_manager.rst
@@ -1,15 +1,15 @@
 .. _workload_manager:
 
 ***********************
-Workload manager
+Workload Manager
 ***********************
 
-The workload manager (WLM) allows SQream DB workers to identify their availability to clients with specific service names. The load balancer will then use that information to route statements to specific workers.
+The **Workload Manager** allows SQream DB workers to identify their availability to clients with specific service names. The load balancer uses that information to route statements to specific workers. 
 
-Why use the workload manager?
+Overview
 ===============================
 
-The workload manager allows a system engineer or database administrator to allocate specific workers and compute resoucres for various tasks.
+The Workload Manager allows a system engineer or database administrator to allocate specific workers and compute resources for various tasks.
 
 For example:
 
@@ -17,19 +17,18 @@ For example:
 
 #. Creating a service for the company's leadership during working hours for dedicated access, and disabling this service at night to allow maintenance operations to use the available compute.
 
-Setting up service queues
+Setting Up Service Queues
 ==========================
 
 By default, every worker subscribes to the ``sqream`` service queue.
 
 Additional service names are configured in the configuration file for every worker, but can also be :ref:`set on a per-session basis`.
 
-Example - Allocating resources for ETL
+Example - Allocating ETL Resources
 ========================================
+Allocating ETL resources ensures high quality service without requiring management users to wait.
 
-We want to allocate resources for ETL to ensure a good quality of service, but we also have management users who don't like waiting.
-
-The configuration in this example allocates resources as follows:
+The configuration in this example allocates resources as shown below:
 
 * 1 worker for ETL work
 * 3 workers for general queries
@@ -61,14 +60,14 @@ The configuration in this example allocates resources as follows:
      - ✓
      - ✓
 
-This configuration gives the ETL queue dedicated access to two workers, one of which can't be used by regular queries.
+This configuration gives the ETL queue dedicated access to one worker, which cannot be used.
 
-Queries from management will use any available worker.
+Queries from management uses any available worker.
 
-Creating this configuration
+Creating the Configuration
 -----------------------------------
 
-The persistent configuration for this setup is listed in these four configuration files.
+The persistent configuration for this set-up is listed in the four configuration files shown below.
 
 Each worker gets a comma-separated list of service queues that it subscribes to. These services are specified in the ``initialSubscribedServices`` attribute.
 
@@ -112,12 +111,17 @@ Each worker gets a comma-separated list of service queues that it subscribes to.
        }
    }
 
-.. tip:: This configuration can be created temporarily (for the current session only) by using the :ref:`subscribe_service` and :ref:`unsubscribe_service` statements.
+.. tip:: You can create this configuration temporarily (for the current session only) by using the :ref:`subscribe_service` and :ref:`unsubscribe_service` statements.
 
-Verifying the configuration
+Verifying the Configuration
 -----------------------------------
 
-Use :ref:`show_subscribed_instances` to view service subscriptions for each worker. Use ref:`show_server_status` to see the statement queues.
+Use :ref:`show_subscribed_instances` to view service subscriptions for each worker. Use `SHOW_SERVER_STATUS `_ to see the statement queues.
+
+
+
+
+
 
 .. code-block:: psql
    
@@ -133,13 +137,30 @@ Use :ref:`show_subscribed_instances` to view service subscriptions for each work
    query      | node_9551  | 192.168.1.91  |       5000
    management | node_9551  | 192.168.1.91  |       5000
 
-Configuring a client to connect to a specific service
+Configuring a Client Connection to a Specific Service
 ===========================================================
+You can configure a client connection to a specific service in one of the following ways:
 
-Using :ref:`sqream_sql_cli_reference`
+.. contents::
+   :local:
+
+Using SQream Studio
 --------------------------------------------
+When using **SQream Studio**, you can configure a client connection to a specific service from the SQream Studio, as shown below:
+
+.. image:: /_static/images/TPD_33.png
+
+
+
+For more information, in Studio, see `Executing Statements from the Toolbar `_.
+
+
+
+
 
-Add ``--service=`` to the command line.
+Using the SQream SQL CLI Reference
+--------------------------------------------
+When using the **SQream SQL CLI Reference**, you can configure a client connection to a specific service by adding ``--service=`` to the command line, as shown below:
 
 .. code-block:: psql
 
@@ -150,21 +171,27 @@ Add ``--service=`` to the command line.
    To quit, use ^D or \q.
    
    master=>_
+   
+For more information, see the :ref:`sqream_sql_cli_reference`.
+
 
-Using :ref:`JDBC`
---------------------------------------------
 
-Add ``--service=`` to the command line.
+Using a JDBC Client Driver
+--------------------------------------------
+When using a **JDBC client driver**, you can configure a client connection to a specific service by adding ``--service=`` to the command line, as shown below:
 
 .. code-block:: none
-   :caption: JDBC connection string
+   :caption: JDBC Connection String
    
    jdbc:Sqream://127.0.0.1:3108/raviga;user=rhendricks;password=Tr0ub4dor&3;service=etl;cluster=true;ssl=false;
+   
 
-Using :ref:`odbc`
---------------------------------------------
+For more information, see the `JDBC Client Driver `_.
 
-On Linux, modify the :ref:`DSN parameters` in ``odbc.ini``.
+
+Using an ODBC Client Driver
+--------------------------------------------
+When using an **ODBC client driver**, you can configure a client connection to a specific service on Linux by modifying the :ref:`DSN parameters` in ``odbc.ini``.
 
 For example, ``Service="etl"``:
 
@@ -186,10 +213,12 @@ For example, ``Service="etl"``:
 
 On Windows, change the parameter in the :ref:`DSN editing window`.
 
-Using :ref:`pysqream`
---------------------------------------------
+For more information, see the `ODBC Client Driver `_.
 
-In Python, set the ``service`` parameter in the connection command:
+
+Using a Python Client Driver
+--------------------------------------------
+When using a **Python client driver**, you can configure a client connection to a specific service by setting the ``service`` parameter in the connection command, as shown below:
 
 .. code-block:: python
    :caption: Python
@@ -198,14 +227,16 @@ In Python, set the ``service`` parameter in the connection command:
    con = pysqream.connect(host='127.0.0.1', port=3108, database='raviga'
                           , username='rhendricks', password='Tr0ub4dor&3'
                           , clustered=True, use_ssl = False, service='etl')
+						  
+For more information, see the `Python (pysqream) connector `_.
 
-Using :ref:`nodejs`
---------------------------------------------
 
-Add the service to the connection settings:
+Using a Node.js Client Driver
+--------------------------------------------
+When using a **Node.js client driver**, you can configure a client connection to a specific service by adding the service to the connection settings, as shown below:
 
 .. code-block:: javascript
-   :caption: Node.JS
+   :caption: Node.js
    :emphasize-lines: 5
    
    const Connection = require('sqreamdb');
@@ -218,3 +249,5 @@ Add the service to the connection settings:
       cluster: 'true',
       service: 'etl'
    };
+
+For more information, see the `Node.js Client Driver `_.
\ No newline at end of file
diff --git a/features_tour.rst b/features_tour.rst
deleted file mode 100644
index b300ff68e..000000000
--- a/features_tour.rst
+++ /dev/null
@@ -1,189 +0,0 @@
-.. _features_tour:
-
-*****************************
-Tour of the SQream DB product
-*****************************
-
-Overview
-=========
-
-SQream DB is a columnar analytic SQL database management system. SQream DB uses GPUs to load and analyze large amounts of data.
-
-SQL Features
-============
-
-Types
------
-
-bool
-tinyint, smallint, int, bigint
-real, double
-varchar, nvarchar
-date
-timestamp
-
-
-Queries
--------
-
-select lists
-from
-where
-tablerefs, joins
-order by
-limit
-set operators (only union all)
-distinct
-aggregates, having
-what are all the aggregate fns?
-also show agg distinct
-
-window functions
-what are all the window fns
-show aggs in windwos
-partition, order
-
-
-
-nested queries
-ctes
-
-query external table
-
-
-..
-  not enough interesting to say about scalar expressions here specifically
-  here are some things to try to work into the other examples:
-
-  'string literal'
-  'something with ''quotes'' and stuff'
-  1
-  0.2
-  3e-4
-  null
-  true
-  false
-  a + b
-    and, or, comparisons,
-    ||, like, rlike
-  not true
-  is null/ is not null
-  a in (1,2,3)
-  between
-  extract
-  coalesce
-  nullif
-  case 2 variations
-  *
-  function app
-
-  cast(a as b)
-  a :: b
- 
-
-
-    
-
-Tables
-------
-
-create table
-columns + types
-schema
-defaults
-null, not null
-
-
-alter table: add column, drop column, rename column, rename table
-
-create table as
-
-Inserting data
---------------
-
-insert
-
-external tables
-select insert
-copy
-
-
-delete
------------
-
-delete from t;
-truncate t;
-delete from t where ...
-
-views
------
-
-create and drop
-
-saved queries
--------------
-
-udfs
-------
-
-access control
---------------
-
-roles as users/logins:
-create, alter, drop role
-rename role
-
-permissions
-grant, revoke
-
-groups
-
-default permissions
-  
-notes
-=============
-
-figure out how to put these in sections
-maybe some of it should go before the sql features and some after
-
-
-
-5TB to 500TB+
-
-columnar
-external algos
-scaling
-gpus
-
-catalog
-
-wlm
-
-cluster/concurrency/ha system
-
-docker
-
-metadata system
-
-
-can support 30+ concurrent users
-
-has high availability
-
-runs on prem or on the cloud
-
-we see customers able to go from 3 months to 12 years data
-
-extremely fast data loading speed
-
-robust serializable transactions and concurrency control
-
-wide range of client drivers
-
-integrates with a wide range of third party components
-
-performance
-cost/ tco
-
-highly responsive team, including new feature development
-
diff --git a/first_steps.rst b/first_steps.rst
deleted file mode 100644
index 160d6f11a..000000000
--- a/first_steps.rst
+++ /dev/null
@@ -1,329 +0,0 @@
-.. _first_steps:
-
-****************************
-First steps with SQream DB
-****************************
-
-.. TODO: part one of the tutorial is how to connect with clientcmd,
-.. and create some tables, insert data and query it
-.. part 2: expand on this
-.. how to run sqream/check it's running
-.. how to add a database and do roles and permissions (to get started)
-.. options for inserting data - overview of all of them
-.. deleting/managing data overview (including storage reorg)
-
-This tutorial takes you through a few basic operations in SQream DB.
-
-.. contents:: In this topic:
-   :local:
-
-Preparing for this tutorial
-===============================
-
-This tutorial assumes you already have a SQream DB cluster running and the SQream command line client installed on the machine you are on.
-
-.. rubric:: If you haven't already:
-
-* :ref:`Set up a SQream DB cluster`
-
-* :ref:`Install SQream SQL CLI`
-
-
-Run the SQream SQL client like this. It will interactively ask for the password.
-
-.. code-block:: psql
-
-   $ sqream sql --port=5000 --username=rhendricks -d master
-   Password:
-   
-   Interactive client mode
-   To quit, use ^D or \q.
-   
-   master=> _
-
-You should use a username and password that you have set up or your DBA has given you.
-  
-.. tip::
-   * To exit the shell, type ``\q``  or :kbd:`Ctrl-d`. 
-   * A new SQream DB cluster contains a database named `master`. We will start with this database.
-
-Create a new database for playing around in
-=======================================================
-
-To create a database, we will use the :ref:`create_database` syntax.
-
-.. code-block:: psql
-
-   master=> CREATE DATABASE test;
-   executed
-
-Now, reconnect to the newly created database.
-
-First, exit the client by typing ``\q`` and hitting enter.
-
-From the Linux shell, restart the client with the new database name:
-
-.. code-block:: psql
-
-   $ sqream sql --port=5000 --username=rhendricks -d test
-   Password:
-   
-   Interactive client mode
-   To quit, use ^D or \q.
-   
-   test=> _
-
-The new database name appears in the prompt. This lets you know which database you're connected to.
-
-Creating your first table
-============================
-
-To create a table, we will use the :ref:`create_table` syntax, with a table name and some column specifications.
-
-For example,
-
-.. code-block:: postgres
-
-   CREATE TABLE cool_animals (
-      id INT NOT NULL,
-      name VARCHAR(20),
-      weight INT
-   );
-
-If the table already exists and you want to drop the current table and create a new one, you can add ``OR REPLACE`` after the ``CREATE`` keyword.
-
-.. code-block:: postgres
-
-   CREATE OR REPLACE TABLE cool_animals (
-      id INT NOT NULL,
-      name VARCHAR(20),
-      weight INT
-   );
-
-.. TODO: cool_animals? I think we need to decide if the customer for this documenation is 12 years old, or an IT professional.
-   
-
-You can ask SQream DB to list the full, verbose ``CREATE TABLE`` statement for any table, by using the :ref:`get_ddl` function, with the table name.
-
-.. code-block:: psql
-
-   test=> SELECT GET_DDL('cool_animals');
-   create table "public"."cool_animals" (
-   "id" int not null,
-   "name" varchar(20),
-   "weight" int
-   );
-
-.. note:
-   * SQream DB identifier names such as table names and column names are not case sensitive. SQream DB lowercases all identifiers bu default. If you want to maintain case, enclose the identifiers with double-quotes.
-   * SQream DB places all tables in the `public` schema, unless another schema is created and specified as part of the table name.
-
-If you are done with this table, you can use :ref:`drop_table` to remove the table and all of its data.
-
-.. code-block:: psql
-
-   test=> DROP TABLE cool_animals;
-   
-   executed
-
-Listing tables
-=================
-
-To see the tables in the current database, we will query the catalog
-
-.. code-block:: psql
-
-   test=> SELECT table_name FROM sqream_catalog.tables;
-   cool_animals
-   
-   1 rows
-
-
-Inserting rows
-=================
-
-Inserting rows into a table can be performed with the :ref:`insert` statement.
-The statement includes the table name, an optional list of column names, and column values listed in the same order as the column names:
-
-.. code-block:: psql
-
-   test=> INSERT INTO cool_animals VALUES (1, 'Dog', 7);
-   
-   executed
-
-To change the order of values, specify the column order:
-
-.. code-block:: psql
-
-   test=> INSERT INTO cool_animals(weight, id, name) VALUES (3, 2, 'Possum');
-   
-   executed
-
-You can use ``INSERT`` to insert multiple rows too. Here, you use sets of parentheses separated by commas:
-
-.. code-block:: psql
-
-   test=> INSERT INTO cool_animals VALUES
-         (3, 'Cat', 5) ,
-         (4, 'Elephant', 6500) ,
-         (5, 'Rhinoceros', 2100);
-   
-   executed
-
-.. note:: To load big data sets, use bulk loading methods instead. See our :ref:`inserting_data` guide for more information.
-
-When you leave out columns that have a :ref:`default value` (including default ``NULL`` value) the default value is used.
-
-.. code-block:: psql
-
-   test=> INSERT INTO cool_animals (id) VALUES (6);
-   
-   executed
-
-.. code-block:: psql
-
-   test=> INSERT INTO cool_animals (id) VALUES (6);
-   
-   executed
-   test=> SELECT * FROM cool_animals;
-   1,Dog                 ,7
-   2,Possum              ,3
-   3,Cat                 ,5
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-   6,\N,\N
-   
-   6 rows
-
-.. note:: Null row values are represented as ``\N``
-
-Queries
-=========
-
-For querying, use the :ref:`select` keyword, followed by a list of columns and values to be returned, and the table to get the data from.
-
-.. code-block:: psql
-
-   test=> SELECT id, name, weight FROM cool_animals;
-   1,Dog                 ,7
-   2,Possum              ,3
-   3,Cat                 ,5
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-   6,\N,\N
-   
-   6 rows
-
-To get all columns without specifying them, use the star operator ``*``:
-
-.. code-block:: psql
-
-   test=> SELECT * FROM cool_animals;
-   1,Dog                 ,7
-   2,Possum              ,3
-   3,Cat                 ,5
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-   6,\N,\N
-   
-   6 rows
-
-To get the number of values in a table without getting the full result set, use :ref:`COUNT(*)`:
-
-.. code-block:: psql
-
-   test=> SELECT COUNT(*) FROM cool_animals;
-   6
-   
-   1 row
-
-
-Filter results by adding a :ref:`WHERE` clause and specifying the filter condition:
-
-.. code-block:: psql
-
-   test=> SELECT id, name, weight FROM cool_animals WHERE weight > 1000;
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-   
-   2 rows
-
-Sort the results by adding an :ref:`ORDER BY` clause, and specifying ascending (``ASC``) or descending (``DESC``) order:
-
-.. code-block:: psql
-
-   test=> SELECT * FROM cool_animals ORDER BY weight DESC;
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-   1,Dog                 ,7
-   3,Cat                 ,5
-   2,Possum              ,3
-   6,\N,\N
-
-   6 rows
-
-Filter null rows by adding a filter :ref:`IS NOT NULL`:
-
-.. code-block:: psql
-
-   test=> SELECT * FROM cool_animals WHERE weight IS NOT NULL ORDER BY weight DESC;
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-   1,Dog                 ,7
-   3,Cat                 ,5
-   2,Possum              ,3
-
-   5 rows
-
-Deleting rows
-==============
-
-To delete rows in a table selectively, use the :ref:`DELETE` command, with a table name and a `WHERE` clause to specify which rows are to be deleted:
-
-.. code-block:: psql
-
-   test=> DELETE FROM cool_animals WHERE weight is null;
-   
-   executed
-   master=> SELECT  * FROM cool_animals;
-   1,Dog                 ,7
-   2,Possum              ,3
-   3,Cat                 ,5
-   4,Elephant            ,6500
-   5,Rhinoceros          ,2100
-
-   5 rows
-
-To delete all rows in a table, use the :ref:`TRUNCATE` command followed by the table name:
-
-.. code-block:: psql
-
-   test=> TRUNCATE TABLE cool_animals;
-   
-   executed
-
-.. note:: While :ref:`truncate` deletes data from disk immediately, :ref:`delete` does not physically remove the deleted rows. For more information on removing the rows from disk, see :ref:`delete`.
-
-Saving query results to a CSV or PSV file
-==============================================
-
-The command line client :ref:`sqream sql` can be used to save query results to a CSV or other delimited file format.
-
-.. code-block:: console
-
-   $ sqream sql --username=mjordan --database=nba --host=localhost --port=5000 -c "SELECT * FROM nba LIMIT 5" --results-only --delimiter='|' > nba.psv
-   $ cat nba.psv
-   Avery Bradley           |Boston Celtics        |0|PG|25|6-2 |180|Texas                |7730337
-   Jae Crowder             |Boston Celtics        |99|SF|25|6-6 |235|Marquette            |6796117
-   John Holland            |Boston Celtics        |30|SG|27|6-5 |205|Boston University    |\N
-   R.J. Hunter             |Boston Celtics        |28|SG|22|6-5 |185|Georgia State        |1148640
-   Jonas Jerebko           |Boston Celtics        |8|PF|29|6-10|231|\N|5000000
-
-
-See the :ref:`Controlling the output of the client` section of the reference for more options.
-
-.. rubric:: What's next?
-
-* Explore all of SQream DB's :ref:`SQL Syntax `
-* See the full :ref:`SQream SQL CLI reference `
-* Connect a :ref:`third party tool to SQream DB ` and start analyzing data
diff --git a/getting_started/connecting_clients_to_sqream.rst b/getting_started/connecting_clients_to_sqream.rst
new file mode 100644
index 000000000..9a376cc3d
--- /dev/null
+++ b/getting_started/connecting_clients_to_sqream.rst
@@ -0,0 +1,26 @@
+.. _connecting_clients_to_sqream:
+
+****************************
+Connecting Clients to SQream
+****************************
+SQream supports the most common database tools and interfaces, giving you direct access through a variety of drivers, connectors, and visualiztion tools and utilities.
+
+SQream supports the following client platforms:
+
+* Connect to SQream Using SQL Workbench
+* Connecting to SQream Using Tableau
+* Connect to SQream Using Pentaho Data Integration
+* Connect to SQream Using MicroStrategy
+* Connect to SQream Using R
+* Connect to SQream Using PHP
+* Connect to SQream Using SAS Viya
+
+SQream supports the following client drivers:
+
+* JDBC
+* Python (pysqream)
+* Node.JS
+* ODBC
+* C++ Driver
+
+For more information, see `Third Party Tools`_.
\ No newline at end of file
diff --git a/getting_started/creating_your_first_table.rst b/getting_started/creating_your_first_table.rst
new file mode 100644
index 000000000..db2e96964
--- /dev/null
+++ b/getting_started/creating_your_first_table.rst
@@ -0,0 +1,79 @@
+.. _creating_your_first_table:
+
+****************************
+Creating Your First Table
+****************************
+The **Creating Your First Table** section describes the following:
+
+* :ref:`Creating a table`
+* :ref:`Replacing a table`
+* :ref:`Listing a CREATE TABLE statement`
+* :ref:`Dropping a table`
+
+
+.. _creating_a_table:
+
+**Creating a Table**
+
+The ``CREATE TABLE`` syntax is used to create your first table. This table includes a table name and column specifications, as shown in the following example:
+
+.. code-block:: postgres
+
+   CREATE TABLE cool_animals (
+      id INT NOT NULL,
+      name VARCHAR(20),
+      weight INT
+   );
+
+For more information on creating a table, see :ref:`create_table`.
+
+.. _replacing_a_table:
+
+**Replacing a Table**
+
+You can drop an existing table and create a new one by adding the ``OR REPLACE`` parameter after the ``CREATE`` keyword, as shown in the following example:
+
+.. code-block:: postgres
+
+   CREATE OR REPLACE TABLE cool_animals (
+      id INT NOT NULL,
+      name VARCHAR(20),
+      weight INT
+   );
+
+.. TODO: cool_animals? I think we need to decide if the customer for this documenation is 12 years old, or an IT professional.
+.. _listing_a_create_table_statement:
+
+**Listing a CREATE TABLE Statement**
+
+You can list the full, verbose ``CREATE TABLE`` statement for a table by using the **GET DDL** function with the table name as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT GET_DDL('cool_animals');
+   create table "public"."cool_animals" (
+   "id" int not null,
+   "name" varchar(20),
+   "weight" int
+   );
+
+.. note:: 
+
+   * SQream DB identifier names such as table names and column names are not case sensitive. SQream DB lowercases all identifiers bu default. If you want to maintain case, enclose the identifiers with double-quotes.
+   * SQream DB places all tables in the `public` schema, unless another schema is created and specified as part of the table name.
+
+For information on listing a ``CREATE TABLE`` statement, see :ref:`get_ddl`.
+
+.. _dropping_a_table:
+
+**Dropping a Table**
+
+When you have finished working with your table, you can drop the table to remove it table and its content, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> DROP TABLE cool_animals;
+
+   executed
+   
+For more information on dropping tables, see :ref:`drop_table`.
\ No newline at end of file
diff --git a/getting_started/deleting_rows.rst b/getting_started/deleting_rows.rst
new file mode 100644
index 000000000..2ff33b8dc
--- /dev/null
+++ b/getting_started/deleting_rows.rst
@@ -0,0 +1,48 @@
+.. _deleting_rows:
+
+****************************
+Deleting Rows
+****************************
+The **Deleting Rows** section describes the following:
+
+* :ref:`Deleting selected rows`
+* :ref:`Deleting all rows`
+
+.. _deleting_selected_rows:
+
+**Deleting Selected Rows**
+
+You can delete rows in a table selectively using the ``DELETE`` command. You must include a table name and `WHERE` clause to specify the rows to delete, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> DELETE FROM cool_animals WHERE weight is null;
+   
+   executed
+   master=> SELECT  * FROM cool_animals;
+   1,Dog                 ,7
+   2,Possum              ,3
+   3,Cat                 ,5
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+
+   5 rows
+
+.. _deleting_all_rows:
+
+**Deleting All Rows**
+
+You can delete all rows in a table using the ``TRUNCATE`` command followed by the table name, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> TRUNCATE TABLE cool_animals;
+   
+   executed
+
+.. note:: While :ref:`truncate` deletes data from disk immediately, :ref:`delete` does not physically remove the deleted rows.
+
+For more information, see the following:
+
+* Deleting selected rows - :ref:`DELETE`
+* Deleting all rows - :ref:`TRUNCATE`
\ No newline at end of file
diff --git a/getting_started/executing_statements_in_sqream.rst b/getting_started/executing_statements_in_sqream.rst
new file mode 100644
index 000000000..f02260e3f
--- /dev/null
+++ b/getting_started/executing_statements_in_sqream.rst
@@ -0,0 +1,10 @@
+.. _executing_statements_in_sqream:
+
+****************************
+Executing Statements in SQream
+****************************
+You can execute statements in SQream using one of the following tools:
+
+* `SQream SQL CLI `_ - a command line interface
+* `SQream Acceleration Studio `_ - an intuitive and easy-to-use interface.
+
diff --git a/getting_started/hardware_guide.rst b/getting_started/hardware_guide.rst
new file mode 100644
index 000000000..5337db9cd
--- /dev/null
+++ b/getting_started/hardware_guide.rst
@@ -0,0 +1,228 @@
+.. _hardware_guide:
+
+***********************
+Hardware Guide
+***********************
+The **Hardware Guide** describes the SQream reference architecture, emphasizing the benefits to the technical audience, and provides guidance for end-users on selecting the right configuration for a SQream installation.
+
+.. rubric:: Need help?
+
+This page is intended as a "reference" to suggested hardware. However, different workloads require different solution sizes. SQream's experienced customer support has the experience to advise on these matters to ensure the best experience.
+
+Visit `SQream's support portal `_ for additional support.
+
+A SQream Cluster
+============================
+SQream recommends rackmount servers by server manufacturers Dell, Lenovo, HP, Cisco, Supermicro, IBM, and others.
+
+A typical SQream cluster includes one or more nodes, consisting of
+
+* Two-socket enterprise processors, like the Intel® Xeon® Gold processor family or an IBM® POWER9 processors, providing the high performance required for compute-bound database workloads.
+
+   ::
+
+* NVIDIA Tesla GPU accelerators, with up to 5,120 CUDA and Tensor cores, running on PCIe or fast NVLINK busses, delivering high core count, and high-throughput performance on massive datasets.
+
+   ::
+
+* High density chassis design, offering between 2 and 4 GPUs in a 1U, 2U, or 3U package, for best-in-class performance per cm\ :sup:`2`.
+
+Single-Node Cluster Example
+-----------------------------------
+A single-node SQream cluster can handle between 1 and 8 concurrent users, with up to 1PB of data storage (when connected via NAS).
+
+An average single-node cluster can be a rackmount server or workstation, containing the following components:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+   
+   * - Component
+     - Type
+   * - Server
+     - Dell R750, Dell R940xa, HP ProLiant DL380 Gen10 or similar (Intel only)
+   * - Processor
+     - 2x Intel Xeon Gold 6240 (18C/36HT) 2.6GHz or similar
+   * - RAM
+     - 1.5 TB
+   * - Onboard storage
+     - 
+         * 2x 960GB SSD 2.5in hot plug for OS, RAID1
+         * 2x 2TB SSD or NVMe, for temporary spooling, RAID1
+         * 10x 3.84TB SSD 2.5in Hot plug for storage, RAID6
+
+   * - GPU
+     - 2x A100 NVIDIA
+   * - Operating System
+     - Red Hat Enterprise Linux v7.x or CentOS v7.x or Amazon Linux
+
+.. note:: If you are using internal storage, your volumes must be formatted as xfs.
+
+In this system configuration, SQream can store about 200TB of raw data (assuming average compression ratio and ~50TB of usable raw storage).
+
+If a NAS is used, the 14x SSD drives can be omitted, but SQream recommends 2TB of local spool space on SSD or NVMe drives.
+
+Multi-Node Cluster Examples
+-----------------------------------
+Multi-node clusters can handle any number of concurrent users. A typical SQream cluster relies on a minimum of two GPU-enabled servers and shared storage connected over a network fabric, such as InfiniBand EDR, 40GbE, or 100GbE.
+
+The **Multi-Node Cluster Examples** section describes the following specifications: 
+
+.. contents:: 
+   :local:
+   :depth: 1
+   
+Hardware Specifications
+~~~~~~~~~~~~~~~~~~~~~~~~~
+The following table shows SQream's recommended hardware specifications:
+
+.. list-table::
+   :widths: 15 65
+   :header-rows: 1
+   
+   * - Component
+     - Type
+   * - Server
+     - Dell R750, Dell R940xa, HP ProLiant DL380 Gen10 or similar (Intel only)
+   * - Processor
+     - 2x Intel Xeon Gold 6240 (18C/36HT) 2.6GHz or similar
+   * - RAM
+     - 2 TB
+   * - Onboard storage
+     -   
+         * 2x 960GB SSD 2.5in hot plug for OS, RAID1
+         * 2x 2TB SSD or NVMe, for temporary spooling, RAID1
+   * - External Storage
+     -   
+         * Mellanox Connectx5/6 100G NVIDIA Network Card (if applicable) or other high speed network card minimum 40G compatible to customer’s infrastructure
+         * 50 TB (NAS connected over GPFS, Lustre, or NFS) GPFS recommended
+   * - GPU
+     - 2x A100 NVIDIA
+   * - Operating System
+     - Red Hat Enterprise Linux v7.x or CentOS v7.x or Amazon Linux
+	 
+Metadata Specifications
+~~~~~~~~~~~~~~~~~~~~~~~~~	 
+The following table shows SQream's recommended metadata server specifications:
+
+.. list-table::
+   :widths: 15 90
+   :header-rows: 1
+   
+   * - Component
+     - Type
+   * - Processors
+     - Two Intel Xeon Gold 6342 2.8 Ghz 24C processors or similar
+   * - RAM
+     - 512GB DDR4 RAM 8x64GB RDIMM or similar
+   * - Discs
+     - Two 960 GB MVMe SSD drives in RAID 1 or similar
+   * - Network Card (Storage)
+     - Two Mellanox ConnectX-6 Single Port HDR VPI InfiniBand Adapter cards at 100GbE or similar.
+   * - Network Card (Corporate)
+     - Two 1 GbE cards or similar
+   * - Power sources
+     - Two Power Supplies - 800W AC 50/60Hz 100~240Vac/9.2-4.7A, 3139 BTU/hr
+   * - Operating System
+     - Red Hat Enterprise Linux v7.x or CentOS v7.x or Amazon Linux
+
+.. note:: With a NAS connected over GPFS, Lustre, or NFS, each SQream worker can read data at up to 5GB/s.
+
+SQream Studio Server Example
+-----------------------------------
+The following table shows SQream's recommended Studio server specifications:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+   
+   * - Component
+     - Type
+   * - Server
+     - Physical or virtual machine
+   * - Processor
+     - 1x Intel Core i7
+   * - RAM
+     - 16 GB
+   * - Onboard storage
+     - 50 GB SSD 2.5in Hot plug for OS, RAID1
+   * - Operating System
+     - Red Hat Enterprise Linux v7.x or CentOS v7.x
+
+Cluster Design Considerations
+====================================
+This section describes the following cluster design considerations:
+
+* In a SQream installation, the storage and compute are logically separated. While they may reside on the same machine in a standalone installation, they may also reside on different hosts, providing additional flexibility and scalability.
+
+ ::
+
+* SQream uses all resources in a machine, including CPU, RAM, and GPU to deliver the best performance. At least 256GB of RAM per physical GPU is recommended.
+
+ ::
+
+* Local disk space is required for good temporary spooling performance, particularly when performing intensive operations exceeding the available RAM, such as sorting. SQream recommends an SSD or NVMe drive in RAID 1 configuration with about twice the RAM size available for temporary storage. This can be shared with the operating system drive if necessary.
+
+ ::
+
+* When using SAN or NAS devices, SQream recommends approximately 5GB/s of burst throughput from storage per GPU.
+
+Balancing Cost and Performance
+--------------------------------
+Prior to designing and deploying a SQream cluster, a number of important factors must be considered. 
+
+The **Balancing Cost and Performance** section provides a breakdown of deployment details to ensure that this installation exceeds or meets the stated requirements. The rationale provided includes the necessary information for modifying configurations to suit the customer use-case scenario, as shown in the following table:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+   
+   * - Component
+     - Value
+   * - Compute - CPU
+     - Balance price and performance
+   * - Compute – GPU
+     - Balance price with performance and concurrency
+   * - Memory – GPU RAM
+     - Balance price with concurrency and performance.
+   * - Memory - RAM
+     - Balance price and performance
+   * - Operating System
+     - Availability, reliability, and familiarity
+   * - Storage
+     - Balance price with capacity and performance
+   * - Network
+     - Balance price and performance
+
+CPU Compute
+-------------
+SQream relies on multi-core Intel Gold Xeon processors or IBM POWER9 processors, and recommends a dual-socket machine populated with CPUs with 18C/36HT or better. While a higher core count may not necessarily affect query performance, more cores will enable higher concurrency and better load performance.
+
+GPU Compute and RAM
+-------------------------
+The NVIDIA Tesla range of high-throughput GPU accelerators provides the best performance for enterprise environments. Most cards have ECC memory, which is crucial for delivering correct results every time. SQream recommends the NVIDIA Tesla V100 32GB or NVIDIA Tesla A100 40GB GPU for best performance and highest concurrent user support.
+
+GPU RAM, sometimes called GRAM or VRAM, is used for processing queries. It is possible to select GPUs with less RAM, like the NVIDIA Tesla V100 16GB or P100 16GB, or T4 16GB. However, the smaller GPU RAM results in reduced concurrency, as the GPU RAM is used extensively in operations like JOINs, ORDER BY, GROUP BY, and all SQL transforms.
+
+RAM
+--------
+SQream requires using **Error-Correcting Code memory (ECC)**, standard on most enterprise servers. Large amounts of memory are required for improved performance for heavy external operations, such as sorting and joining.
+
+SQream recommends at least 256GB of RAM per GPU on your machine. 
+
+Operating System
+---------------------
+SQream can run on the following 64-bit Linux operating systems:
+
+   * Red Hat Enterprise Linux (RHEL) v7
+   * CentOS v7
+   * Amazon Linux 2018.03
+   * Other Linux distributions may be supported via nvidia-docker
+
+Storage
+-----------
+For clustered scale-out installations, SQream relies on NAS/SAN storage. For stand-alone installations, SQream relies on redundant disk configurations, such as RAID 5, 6, or 10. These RAID configurations replicate blocks of data between disks to avoid data loss or system unavailability. 
+
+SQream recommends using enterprise-grade SAS SSD or NVMe drives. For a 32-user configuration, the number of GPUs should roughly match the number of users. SQream recommends 1 Tesla V100 or A100 GPU per 2 users, for full, uninterrupted dedicated access.
+
+Download the full `SQream Reference Architecture `_ document.
\ No newline at end of file
diff --git a/getting_started/index.rst b/getting_started/index.rst
new file mode 100644
index 000000000..860c7a7a7
--- /dev/null
+++ b/getting_started/index.rst
@@ -0,0 +1,17 @@
+.. _getting_started:
+
+*************************
+Getting Started
+*************************
+The **Getting Started** page describes the following things you need to start using SQream:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   preparing_your_machine_to_install_sqream
+   installing_sqream
+   creating_a_database
+   executing_statements_in_sqream
+   performing_basic_sqream_operations
+   hardware_guide
\ No newline at end of file
diff --git a/getting_started/ingesting_data.rst b/getting_started/ingesting_data.rst
new file mode 100644
index 000000000..8d57a8019
--- /dev/null
+++ b/getting_started/ingesting_data.rst
@@ -0,0 +1,8 @@
+.. _ingesting_data:
+
+****************************
+Ingesting Data
+****************************
+After creating a database you can begin ingesting data into SQream.
+
+For more information about ingesting data, see `Data Ingestion Guides `_.
\ No newline at end of file
diff --git a/getting_started/inserting_rows.rst b/getting_started/inserting_rows.rst
new file mode 100644
index 000000000..890befebf
--- /dev/null
+++ b/getting_started/inserting_rows.rst
@@ -0,0 +1,86 @@
+.. _inserting_rows:
+
+****************************
+Inserting Rows
+****************************
+The **Inserting Rows** section describes the following:
+
+* :ref:`Inserting basic rows`
+* :ref:`Changing value order`
+* :ref:`Inserting multiple rows`
+* :ref:`Omitting columns`
+
+
+.. _inserting_basic_rows:
+
+**Inserting Basic Rows**
+
+You can insert basic rows into a table using the ``INSERT`` statement. The inserted statement includes the table name, an optional list of column names, and column values listed in the same order as the column names, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> INSERT INTO cool_animals VALUES (1, 'Dog', 7);
+   
+   executed
+
+.. _changing_value_order:
+
+**Changing Value Order**
+
+You can change the order of values by specifying the column order, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> INSERT INTO cool_animals(weight, id, name) VALUES (3, 2, 'Possum');
+   
+   executed
+
+.. _inserting_multiple_rows:
+
+**Inserting Multiple Rows**
+
+You can insert multiple rows using the ``INSERT`` statement by using sets of parentheses separated by commas, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> INSERT INTO cool_animals VALUES
+         (3, 'Cat', 5) ,
+         (4, 'Elephant', 6500) ,
+         (5, 'Rhinoceros', 2100);
+   
+   executed
+
+.. note:: You can load large data sets using bulk loading methods instead. For more information, see :ref:`inserting_data`.
+
+.. _omitting_columns:
+
+**Omitting Columns**
+
+Omitting columns that have a default values (including default ``NULL`` values) uses the default value, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> INSERT INTO cool_animals (id) VALUES (6);
+   
+   executed
+
+.. code-block:: psql
+
+   test=> INSERT INTO cool_animals (id) VALUES (6);
+   
+   executed
+   test=> SELECT * FROM cool_animals;
+   1,Dog                 ,7
+   2,Possum              ,3
+   3,Cat                 ,5
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+   6,\N,\N
+   
+   6 rows
+
+.. note:: Null row values are represented as ``\N``
+
+For more information on inserting rows, see :ref:`insert`.
+
+For more information on default values, see :ref:`default value`.
\ No newline at end of file
diff --git a/getting_started/installing_sqream.rst b/getting_started/installing_sqream.rst
new file mode 100644
index 000000000..3c8b202c1
--- /dev/null
+++ b/getting_started/installing_sqream.rst
@@ -0,0 +1,11 @@
+.. _installing_sqream:
+
+****************************
+Installing SQream
+****************************
+
+The **Installing SQream** section includes the following SQream installation methods:
+
+* `Installing SQream natively `_ - Describes installing SQream using binary packages provided by SQream.
+* `Installing SQream with Kubernetes `_ - Describes installing SQream using the Kubernetes open source platform.
+* `Installing and running SQream in a Docker container `_ - Describes how to run SQream in a Docker container.
\ No newline at end of file
diff --git a/getting_started/listing_tables.rst b/getting_started/listing_tables.rst
new file mode 100644
index 000000000..d376cb7ee
--- /dev/null
+++ b/getting_started/listing_tables.rst
@@ -0,0 +1,13 @@
+.. _listing_tables:
+
+****************************
+Listing Tables
+****************************
+To see the tables in the current database you can query the catalog, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT table_name FROM sqream_catalog.tables;
+   cool_animals
+   
+   1 rows
\ No newline at end of file
diff --git a/getting_started/performing_basic_sqream_operations.rst b/getting_started/performing_basic_sqream_operations.rst
new file mode 100644
index 000000000..b46bba95a
--- /dev/null
+++ b/getting_started/performing_basic_sqream_operations.rst
@@ -0,0 +1,23 @@
+.. _performing_basic_sqream_operations:
+
+****************************
+Performing Basic SQream Operations
+****************************
+After installing SQream you can perform the operations described on this page:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   running_the_sqream_sql_client
+   creating_your_first_table
+   listing_tables
+   inserting_rows
+   running_queries
+   deleting_rows
+   saving_query_results_to_a_csv_or_psv_file
+
+For more information on other basic SQream operations, see the following:
+
+* `Creating a Database `_
+* :ref:`data_ingestion`
\ No newline at end of file
diff --git a/getting_started/preparing_your_machine_to_install_sqream.rst b/getting_started/preparing_your_machine_to_install_sqream.rst
new file mode 100644
index 000000000..74af66820
--- /dev/null
+++ b/getting_started/preparing_your_machine_to_install_sqream.rst
@@ -0,0 +1,35 @@
+.. _preparing_your_machine_to_install_sqream:
+
+****************************
+Preparing Your Machine to Install SQream
+****************************
+To prepare your machine to install SQream, do the following:
+
+ * Set up your local machine according to SQream's recommended pre-installation configurations.
+ 
+    ::
+   
+ * Verify you have an NVIDIA-capable server, either on-premise or on supported cloud platforms: 
+
+   * Red Hat Enterprise Linux v7.x   
+ 
+   * CentOS v7.x
+ 
+   * Amazon Linux 7
+	 
+ * Verify that you have the following:
+ 
+   * An NVIDIA GPU - SQream recommends using a Tesla GPU.
+ 
+
+   * An SSH connection to your server.
+ 
+
+   * SUDO permissions for installation and configuration purposes.
+ 
+ 
+   * A SQream license - Contact support@sqream.com or your SQream account manager for your license key.
+For more information, see the following:
+
+* :ref:`recommended_pre-installation_configurations`
+* `Hardware Guide `_
diff --git a/getting_started/querying_data.rst b/getting_started/querying_data.rst
new file mode 100644
index 000000000..36bf9e78b
--- /dev/null
+++ b/getting_started/querying_data.rst
@@ -0,0 +1,51 @@
+.. _querying_data:
+
+****************************
+Querying Data
+****************************
+One of the most basic operations when using SQream is querying data.
+
+To begin familiarizing yourself with querying data, you can create the following table using the ``CREATE TABLE`` statement:
+
+.. code-block:: postgres
+   
+   CREATE TABLE nba
+   (
+      Name varchar(40),
+      Team varchar(40),
+      Number tinyint,
+      Position varchar(2),
+      Age tinyint,
+      Height varchar(4),
+      Weight real,
+      College varchar(40),
+      Salary float
+    );
+
+
+You can down download the above (:download:`nba.csv table `) if needed, shown below:
+
+.. csv-table:: nba.csv
+   :file: nba-t10.csv
+   :widths: auto
+   :header-rows: 1
+
+The above query gets the following from the table above, limited to showing the first ten results:
+
+* Name
+* Team name
+* Age
+
+.. code-block:: psql
+   
+   nba=> SELECT Name, Team, Age FROM nba LIMIT 10;
+   Avery Bradley,Boston Celtics,25
+   Jae Crowder,Boston Celtics,25
+   John Holland,Boston Celtics,27
+   R.J. Hunter,Boston Celtics,22
+   Jonas Jerebko,Boston Celtics,29
+   Amir Johnson,Boston Celtics,29
+   Jordan Mickey,Boston Celtics,21
+   Kelly Olynyk,Boston Celtics,25
+   Terry Rozier,Boston Celtics,22
+   Marcus Smart,Boston Celtics,22
\ No newline at end of file
diff --git a/getting_started/running_queries.rst b/getting_started/running_queries.rst
new file mode 100644
index 000000000..57f6811d7
--- /dev/null
+++ b/getting_started/running_queries.rst
@@ -0,0 +1,121 @@
+.. _running_queries:
+
+****************************
+Running Queries
+****************************
+The **Running Queries** section describes the following:
+
+* :ref:`Running basic queries`
+* :ref:`Outputting all columns`
+* :ref:`Outputting shorthand table values`
+* :ref:`Filtering results`
+* :ref:`Sorting results`
+* :ref:`Filtering null rows`
+
+
+.. _running_basic_queries:
+
+**Running Basic Queries**
+
+You can run a basic query using the ``SELECT`` keyword, followed by a list of columns and values to be returned, and the table to get the data from, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT id, name, weight FROM cool_animals;
+   1,Dog                 ,7
+   2,Possum              ,3
+   3,Cat                 ,5
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+   6,\N,\N
+   
+   6 rows
+   
+For more information on the ``SELECT`` keyword, see :ref:`select`.
+
+.. _outputting_all_columns:
+
+**To Output All Columns**
+
+You can output all columns without specifying them using the star operator ``*``, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT * FROM cool_animals;
+   1,Dog                 ,7
+   2,Possum              ,3
+   3,Cat                 ,5
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+   6,\N,\N
+   
+   6 rows
+
+.. _outputting_shorthand_table_values:
+
+**Outputting Shorthand Table Values**
+
+You can output the number of values in a table without getting the full result set by using the ``COUNT`` statement:
+
+.. code-block:: psql
+
+   test=> SELECT COUNT(*) FROM cool_animals;
+   6
+   
+   1 row
+
+.. _filtering_results:
+
+**Filtering Results**
+
+You can filter results by adding a ``WHERE`` clause and specifying the filter condition, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT id, name, weight FROM cool_animals WHERE weight > 1000;
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+   
+   2 rows
+
+.. _sorting_results:
+
+**Sorting Results**
+
+You can sort results by adding an ``ORDER BY`` clause and specifying ascending (``ASC``) or descending (``DESC``) order, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT * FROM cool_animals ORDER BY weight DESC;
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+   1,Dog                 ,7
+   3,Cat                 ,5
+   2,Possum              ,3
+   6,\N,\N
+
+   6 rows
+
+.. _filtering_null_rows:
+
+**Filtering Null Rows**
+
+You can filter null rows by adding an ``IS NOT NULL`` filter, as shown in the following example:
+
+.. code-block:: psql
+
+   test=> SELECT * FROM cool_animals WHERE weight IS NOT NULL ORDER BY weight DESC;
+   4,Elephant            ,6500
+   5,Rhinoceros          ,2100
+   1,Dog                 ,7
+   3,Cat                 ,5
+   2,Possum              ,3
+
+   5 rows
+   
+For more information, see the following:
+
+* Outputting the number of values in a table without getting the full result set - :ref:`COUNT(*)`.
+* Filtering results - :ref:`WHERE`
+* Sorting results - :ref:`ORDER BY`
+* Filtering rows - :ref:`IS NOT NULL`
\ No newline at end of file
diff --git a/getting_started/running_the_sqream_sql_client.rst b/getting_started/running_the_sqream_sql_client.rst
new file mode 100644
index 000000000..faba6494d
--- /dev/null
+++ b/getting_started/running_the_sqream_sql_client.rst
@@ -0,0 +1,22 @@
+.. _running_the_sqream_sql_client:
+
+****************************
+Running the SQream SQL Client
+****************************
+The following example shows how to run the SQream SQL client:
+
+.. code-block:: psql
+
+   $ sqream sql --port=5000 --username=rhendricks -d master
+   Password:
+   
+   Interactive client mode
+   To quit, use ^D or \q.
+   
+   master=> _
+
+Running the SQream SQL client prompts you to provide your password. Use the username and password that you have set up, or your DBA has provided.
+  
+.. tip::
+   * You can exit the shell by typing ``\q``  or :kbd:`Ctrl-d`. 
+   * A new SQream cluster contains a database named `master,` which is the database used in the examples on this page.
\ No newline at end of file
diff --git a/getting_started/saving_query_results_to_a_csv_or_psv_file.rst b/getting_started/saving_query_results_to_a_csv_or_psv_file.rst
new file mode 100644
index 000000000..9a40ad440
--- /dev/null
+++ b/getting_started/saving_query_results_to_a_csv_or_psv_file.rst
@@ -0,0 +1,24 @@
+.. _saving_query_results_to_a_csv_or_psv_file:
+
+****************************
+Saving Query Results to a CSV or PSV File
+****************************
+You can save query results to a CSV or PSV file using the ``sqream sql`` command from a CLI client. This saves your query results to the selected delimited file format, as shown in the following example:
+
+.. code-block:: console
+
+   $ sqream sql --username=mjordan --database=nba --host=localhost --port=5000 -c "SELECT * FROM nba LIMIT 5" --results-only --delimiter='|' > nba.psv
+   $ cat nba.psv
+   Avery Bradley           |Boston Celtics        |0|PG|25|6-2 |180|Texas                |7730337
+   Jae Crowder             |Boston Celtics        |99|SF|25|6-6 |235|Marquette            |6796117
+   John Holland            |Boston Celtics        |30|SG|27|6-5 |205|Boston University    |\N
+   R.J. Hunter             |Boston Celtics        |28|SG|22|6-5 |185|Georgia State        |1148640
+   Jonas Jerebko           |Boston Celtics        |8|PF|29|6-10|231|\N|5000000
+
+For more output options, see :ref:`Controlling the Client Output`.
+
+.. rubric:: What's next?
+
+* Explore all of SQream DB's :ref:`SQL Syntax `.
+* See the full :ref:`SQream SQL CLI reference `.
+* Connect a :ref:`third party tool ` to start analyzing data.
\ No newline at end of file
diff --git a/glossary.rst b/glossary.rst
index 5a4ee4cb1..546476e16 100644
--- a/glossary.rst
+++ b/glossary.rst
@@ -3,37 +3,33 @@
 Glossary
 =====================================
 
-.. glossary::
 
-   Cluster
-      A SQream DB deployment containing several :term:`workers` running on one or more :term:`nodes`.
-
-   Node
-      A machine running SQream DB :term:`workers`.
-
-   Worker
-      A SQream DB application that can respond to statements. Several workers can run on one or more :term:`nodes` to form a cluster.
-
-   Metadata
-      SQream DB's internal storage which contains details about database objects.
-
-   Authentication
-      Authentication is the process that identifies a user or :term:`role` to verify an identity - to make sure the user is who they say they are. This is done with a username and password. 
-
-   Authorization
-      Authorization defines the set of actions that an authenticaed :term:`role` can perform after gaining access to the system, protecting from threats that :term:`authentication` controls alone are not enough against.
-   
-   Role
-      A role is a group or a user, depending on the permissions granted. A role groups together a set of permissions.
-
-   Catalog
-      A set of views that contains information (metadata) about the objects in a database (e.g. tables, columns, chunks, etc...).
-   
-   Storage cluster
-      The storage cluster is the directory in which SQream DB stores data, including database objects, metadata database, and logs.
-   
-   UDF
-   User-defined function
-      A feature that extends SQream DB's built in SQL functionality with user-written Python code.
+The following table shows the **Glossary** descriptions: 
    
-   
\ No newline at end of file
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| **Term**        | **Description**                                                                                                                                                                                              |
++=================+==============================================================================================================================================================================================================+
+| Authentication  | The process of verifying identity by validating a user or role identity using a username and password.                                                                                                       |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Authorization   | Defines the set of actions that an authenticaed role can perform after gaining access to the system.                                                                                                         |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Catalog         | A set of views containing metadata information about objects in a database.                                                                                                                                  |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Cluster         | A SQream deployment containing several workers running on one or more nodes.                                                                                                                                 |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Custom connector| When SQream is integrated with Power BI, used for running direct queries.                                                                                                                                    |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Direct query    | A Power BI data extraction method that retrieves data from a remote source instead of from a local repository.                                                                                               |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Import          | A Power BI data extraction method that retrieves data to local repository to be visualized at a later point.                                                                                                 |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Metadata        | SQream's internal storage containing details about database objects.                                                                                                                                         |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Node            | A machine used to run SQream workers.                                                                                                                                                                        |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Role            | A group or a user. For more information see `SQream Studio `_.                        |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Storage cluster | The directory where SQream stores data.                                                                                                                                                                      |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Worker          | A SQream application that responds to statements. Several workers running on one or more nodes form a cluster.                                                                                               |
++-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
\ No newline at end of file
diff --git a/guides/client_drivers/index.rst b/guides/client_drivers/index.rst
deleted file mode 100644
index e3ece94e4..000000000
--- a/guides/client_drivers/index.rst
+++ /dev/null
@@ -1,101 +0,0 @@
-.. _client_drivers:
-
-************************************
-Client drivers for |latest_version|
-************************************
-
-These guides explain how to use the SQream DB client drivers, and how to use client applications with SQream DB.
-
-Client driver downloads
-=============================
-
-All operating systems
----------------------------
-
-* 
-   **JDBC** - `sqream-jdbc v4.3.1 (.jar) `_
-   
-   :ref:`java_jdbc` Driver
-   
-   (SQream recommends installing via ``mvn``)
-
-* 
-   **Python** - `pysqream v3.1.0 (.tar.gz) `_
-   
-   :ref:`pysqream` - Python driver
-   
-   (SQream recommends installing via ``pip``)
-
-* 
-   **Node.JS** - `sqream-v4.2.0 (.tar.gz) `_
-   
-   :ref:`nodejs` - Node.JS driver
-   
-   (SQream recommends installing via ``npm``)
-
-* 
-   **Tableau Connector** - `SQreamDB (.taco) `_
-   
-   :ref:`Tableau connector` - Tableau connector for manual installation
-
-Windows
---------------
-
-* 
-   **JDBC installer** - `SQream_JDBC_Driver_v2020.2.0.exe `_ 
-   
-   Windows installer for JDBC driver, with Tableau connector.
-
-* 
-   **ODBC installer** - ``SQream Drivers v2020.2.0``
-   
-   :ref:`Windows installer for ODBC and JDBC`, with Tableau customizations. Please contact your SQream representative to get this installer.
-
-* **.Net driver** - `SQream .Net driver v2.0.0 `_
-   
-   
-
-Linux
---------------
-
-* 
-   **SQream SQL** (x86) - `sqream-sql-v2020.1.1_stable.x86_64.tar.gz `_ 
-   
-   :ref:`sqream sql` - Interactive command-line SQL client for Intel-based machines
-   
-* 
-   **SQream SQL** (IBM POWER9) - `sqream-sql-v2020.1.1_stable.ppc64le.tar.gz `_ 
-   
-   :ref:`sqream sql` - Interactive command-line SQL client for IBM POWER9-based machines
-   
-* 
-   **ODBC installer** - ``sqream_odbc_4.0.0_x86_64_linux.tar.gz``
-   
-   :ref:`Linux installer for ODBC`. Please contact your SQream representative to get this installer.
-
-* 
-   **C++ connector** - `libsqream-4.0 `_ 
-   
-   :ref:`C++ shared object` library
-
-
-.. toctree::
-   :maxdepth: 4
-   :caption: Client driver documentation:
-   :titlesonly:
-   
-   python/index
-   cpp/index
-   jdbc/index
-   odbc/index
-   nodejs/index
-
-
-
-.. rubric:: Need help?
-
-If you couldn't find what you're looking for, we're always happy to help. Visit `SQream's support portal `_ for additional support.
-
-.. rubric:: Looking for older drivers?
-
-If you're looking for an older version of SQream DB drivers, versions 1.10 through 2019.2.1 are available at https://sqream.com/product/client-drivers/ .
diff --git a/guides/client_drivers/jdbc/index.rst b/guides/client_drivers/jdbc/index.rst
deleted file mode 100644
index b5fcd4bca..000000000
--- a/guides/client_drivers/jdbc/index.rst
+++ /dev/null
@@ -1,162 +0,0 @@
-.. _java_jdbc:
-
-*************************
-JDBC
-*************************
-
-The SQream DB JDBC driver allows many Java applications and tools connect to SQream DB.
-This tutorial shows how to write a Java application using the JDBC interface.
-
-The JDBC driver requires Java 1.8 or newer.
-
-.. contents:: In this topic:
-   :local:
-
-Installing the JDBC driver
-==================================
-
-Prerequisites
-----------------
-
-The SQream DB JDBC driver requires Java 1.8 or newer. We recommend either Oracle Java or OpenJDK.
-
-**Oracle Java**
-
-Download and install Java 8 from Oracle for your platform
-
-https://www.java.com/en/download/manual.jsp
-
-**OpenJDK**
-
-For Linux and BSD, see https://openjdk.java.net/install/
-
-For Windows, SQream recommends Zulu 8 https://www.azul.com/downloads/zulu-community/?&version=java-8-lts&architecture=x86-64-bit&package=jdk
-
-.. _get_jdbc_jar:
-
-Getting the JAR file
----------------------
-
-The JDBC driver is provided as a zipped JAR file, available for download from the :ref:`client drivers download page`. This JAR file can integrate into your Java-based applications or projects.
-
-
-Extract the zip archive
--------------------------
-
-Extract the JAR file from the zip archive
-
-.. code-block:: console
-
-   $ unzip sqream-jdbc-4.3.0.zip
-
-Setting up the Class Path
-----------------------------
-
-To use the driver, the JAR named ``sqream-jdbc-.jar`` (for example, ``sqream-jdbc-4.3.0.jar``) needs to be included in the class path, either by putting it in the ``CLASSPATH`` environment variable, or by using flags on the relevant Java command line.
-
-For example, if the JDBC driver has been unzipped to ``/home/sqream/sqream-jdbc-4.3.0.jar``, the application should be run as follows:
-
-.. code-block:: console
-
-   $ export CLASSPATH=/home/sqream/sqream-jdbc-4.3.0.jar:$CLASSPATH
-   $ java my_java_app
-
-An alternative method is to pass ``-classpath`` to the Java executable:
-
-.. code-block:: console
-
-   $ java -classpath .:/home/sqream/sqream-jdbc-4.3.0.jar my_java_app
-
-
-Connect to SQream DB with a JDBC application
-==============================================
-
-Driver class
---------------
-
-Use ``com.sqream.jdbc.SQDriver`` as the driver class in the JDBC application.
-
-
-.. _connection_string:
-
-Connection string
---------------------
-
-JDBC drivers rely on a connection string. Use the following syntax for SQream DB
-
-.. code-block:: text
-
-   jdbc:Sqream:///;user=;password=sqream;[; ...]
-
-Connection parameters
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-   
-   * - Item
-     - Optional
-     - Default
-     - Description
-   * - ````
-     - ✗
-     - None
-     - Hostname and port of the SQream DB worker. For example, ``127.0.0.1:5000``, ``sqream.mynetwork.co:3108``
-   * - ````
-     - ✗
-     - None
-     - Database name to connect to. For example, ``master``
-   * - ``username=``
-     - ✗
-     - None
-     - Username of a role to use for connection. For example, ``username=rhendricks``
-   * - ``password=``
-     - ✗
-     - None
-     - Specifies the password of the selected role. For example, ``password=Tr0ub4dor&3``
-   * - ``service=``
-     - ✓
-     - ``sqream``
-     - Specifices service queue to use. For example, ``service=etl``
-   * - ````
-     - ✓
-     - ``false``
-     - Specifies SSL for this connection. For example, ``ssl=true``
-   * - ````
-     - ✓
-     - ``false``
-     - Connect via load balancer (use only if exists, and check port). For example, ``cluster=true``
-
-Connection string examples
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-For a SQream DB cluster with load balancer and no service queues, with SSL
-
-.. code-block:: text
-
-   jdbc:Sqream://sqream.mynetwork.co:3108/master;user=rhendricks;password=Tr0ub4dor&3;ssl=true;cluster=true
-
-Minimal example for a local, standalone SQream DB
-
-.. code-block:: text 
-
-   jdbc:Sqream://127.0.0.1:5000/master;user=rhendricks;password=Tr0ub4dor&3
-
-For a SQream DB cluster with load balancer and a specific service queue named ``etl``, to the database named ``raviga``
-
-.. code-block:: text
-
-   jdbc:Sqream://sqream.mynetwork.co:3108/raviga;user=rhendricks;password=Tr0ub4dor&3;cluster=true;service=etl
-
-
-Sample Java program
---------------------
-
-Download this file by right clicking and saving to your computer :download:`sample.java `.
-
-.. literalinclude:: sample.java
-    :language: java
-    :caption: JDBC application sample
-    :linenos:
-
diff --git a/guides/client_drivers/python/api-reference.rst b/guides/client_drivers/python/api-reference.rst
deleted file mode 100644
index 36da33c39..000000000
--- a/guides/client_drivers/python/api-reference.rst
+++ /dev/null
@@ -1,191 +0,0 @@
-.. _pysqream_api_reference:
-
-*************************
-pysqream API reference
-*************************
-
-The SQream Python connector allows Python programs to connect to SQream DB.
-
-pysqream conforms to Python DB-API specifications `PEP-249 `_
-
-
-The main module is pysqream, which contains the :py:meth:`Connection` class.
-
-.. method:: connect(host, port, database, username, password, clustered = False, use_ssl = False, service='sqream', reconnect_attempts=3, reconnect_interval=10)
-   
-   Creates a new :py:meth:`Connection` object and connects to SQream DB.
-   
-   host
-      SQream DB hostname or IP
-
-   port
-      SQream DB port 
-
-   database
-      database name
-
-   username
-      Username to use for connection
-
-   password
-      Password for ``username``
-
-   clustered
-      Connect through load balancer, or direct to worker (Default: false - direct to worker)
-
-   use_ssl
-      use SSL connection (default: false)
-
-   service
-      Optional service queue (default: 'sqream')
-
-   reconnect_attempts
-      Number of reconnection attempts to attempt before closing the connection
-
-   reconnect_interval
-      Time in seconds between each reconnection attempt
-
-.. class:: Connection
-   
-   .. attribute:: arraysize
-   
-      Specifies the number of rows to fetch at a time with :py:meth:`~Connection.fetchmany`. Defaults to 1 - one row at a time.
-
-   .. attribute:: rowcount
-   
-      Unused, always returns -1.
-   
-   .. attribute:: description
-      
-      Read-only attribute that contains result set metadata.
-      
-      This attribute is populated after a statement is executed.
-      
-      .. list-table:: 
-         :widths: auto
-         :header-rows: 1
-         
-         * - Value
-           - Description
-         * - ``name``
-           - Column name
-         * - ``type_code``
-           - Internal type code
-         * - ``display_size``
-           - Not used - same as ``internal_size``
-         * - ``internal_size``
-           - Data size in bytes
-         * - ``precision``
-           - Precision of numeric data (not used)
-         * - ``scale``
-           - Scale for numeric data (not used)
-         * - ``null_ok``
-           - Specifies if ``NULL`` values are allowed for this column
-
-   .. method:: execute(self, query, params=None)
-      
-      Execute a statement.
-      
-      Parameters are not supported
-      
-      self
-         :py:meth:`Connection`
-
-      query
-         statement or query text
-      
-      params
-         Unused
-      
-   .. method:: executemany(self, query, rows_or_cols=None, data_as='rows', amount=None)
-      
-      Prepares a statement and executes it against all parameter sequences found in ``rows_or_cols``.
-
-      self
-         :py:meth:`Connection`
-
-      query
-         INSERT statement
-         
-      rows_or_cols
-         Data buffer to insert. This should be a sequence of lists or tuples.
-      
-      data_as
-         (Optional) Read data as rows or columns
-      
-      amount
-         (Optional) count of rows to insert
-   
-   .. method:: close(self)
-      
-      Close a statement and connection.
-      After a statement is closed, it must be reopened by creating a new cursor.
-            
-      self
-         :py:meth:`Connection`
-
-   .. method:: cursor(self)
-      
-      Create a new :py:meth:`Connection` cursor.
-      
-      We recommend creating a new cursor for every statement.
-      
-      self
-         :py:meth:`Connection`
-
-   .. method:: fetchall(self, data_as='rows')
-      
-         Fetch all remaining records from the result set.
-         
-         An empty sequence is returned when no more rows are available.
-      
-      self
-         :py:meth:`Connection`
-
-      data_as
-         (Optional) Read data as rows or columns
-
-   .. method:: fetchone(self, data_as='rows')
-      
-      Fetch one record from the result set.
-      
-      An empty sequence is returned when no more rows are available.
-      
-      self
-         :py:meth:`Connection`
-
-      data_as
-         (Optional) Read data as rows or columns
-
-
-   .. method:: fetchmany(self, size=[Connection.arraysize], data_as='rows')
-      
-         Fetches the next several rows of a query result set.
-
-         An empty sequence is returned when no more rows are available.
-
-      self
-         :py:meth:`Connection`
-
-      size
-         Number of records to fetch. If not set, fetches :py:obj:`Connection.arraysize` (1 by default) records
-
-      data_as
-         (Optional) Read data as rows or columns
-
-   .. method:: __iter__()
-
-         Makes the cursor iterable.
-
-
-.. attribute:: apilevel = '2.0'
-   
-   String constant stating the supported API level. The connector supports API "2.0".
-
-.. attribute:: threadsafety = 1
-      
-   Level of thread safety the interface supports. pysqream currently supports level 1, which states that threads can share the module, but not connections.
-
-.. attribute:: paramstyle = 'qmark'
-   
-   The placeholder marker. Set to ``qmark``, which is a question mark (``?``).
diff --git a/guides/client_drivers/python/index.rst b/guides/client_drivers/python/index.rst
deleted file mode 100644
index c2f5911b3..000000000
--- a/guides/client_drivers/python/index.rst
+++ /dev/null
@@ -1,502 +0,0 @@
-.. _pysqream:
-
-*************************
-Python (pysqream)
-*************************
-
-The SQream Python connector is a set of packages that allows Python programs to connect to SQream DB.
-
-* ``pysqream`` is a pure Python connector. It can be installed with ``pip`` on any operating system, including Linux, Windows, and macOS.
-
-* ``pysqream-sqlalchemy`` is a SQLAlchemy dialect for ``pysqream``
-
-The connector supports Python 3.6.5 and newer.
-
-The base ``pysqream`` package conforms to Python DB-API specifications `PEP-249 `_.
-
-.. contents:: In this topic:
-   :local:
-
-Installing the Python connector
-==================================
-
-Prerequisites
-----------------
-
-1. Python
-^^^^^^^^^^^^
-
-The connector requires Python 3.6.5 or newer. To verify your version of Python:
-
-.. code-block:: console
-
-   $ python --version
-   Python 3.7.3
-   
-
-.. note:: If both Python 2.x and 3.x are installed, you can run ``python3`` and ``pip3`` instead of ``python`` and ``pip`` respectively for the rest of this guide
-
-.. warning:: If you're running on an older version, ``pip`` will fetch an older version of ``pysqream``, with version <3.0.0. This version is currently not supported.
-
-2. PIP
-^^^^^^^^^^^^
-The Python connector is installed via ``pip``, the Python package manager and installer.
-
-We recommend upgrading to the latest version of ``pip`` before installing. To verify that you are on the latest version, run the following command:
-
-.. code-block:: console
-
-   $ python -m pip install --upgrade pip
-   Collecting pip
-      Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
-        |████████████████████████████████| 1.4MB 1.6MB/s
-   Installing collected packages: pip
-     Found existing installation: pip 19.1.1
-       Uninstalling pip-19.1.1:
-         Successfully uninstalled pip-19.1.1
-   Successfully installed pip-19.3.1
-
-.. note:: 
-   * On macOS, you may want to use virtualenv to install Python and the connector, to ensure compatibility with the built-in Python environment
-   *  If you encounter an error including ``SSLError`` or ``WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.`` - please be sure to reinstall Python with SSL enabled, or use virtualenv or Anaconda.
-
-3. OpenSSL for Linux
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Some distributions of Python do not include OpenSSL. The Python connector relies on OpenSSL for secure connections to SQream DB.
-
-* To install OpenSSL on RHEL/CentOS
-
-   .. code-block:: console
-   
-      $ sudo yum install -y libffi-devel openssl-devel
-
-* To install OpenSSL on Ubuntu
-
-   .. code-block:: console
-   
-      $ sudo apt-get install libssl-dev libffi-dev -y
-
-4. Cython (optional)
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Optional but highly recommended is Cython, which improves performance of Python applications.
-
-   .. code-block:: console
-   
-      $ pip install cython
-
-Install via pip
------------------
-
-The Python connector is available via `PyPi `_.
-
-Install the connector with ``pip``:
-
-.. code-block:: console
-   
-   $ pip install pysqream pysqream-sqlalchemy
-
-``pip`` will automatically install all necessary libraries and modules.
-
-Upgrading an existing installation
---------------------------------------
-
-The Python drivers are updated periodically.
-To upgrade an existing pysqream installation, use pip's ``-U`` flag.
-
-.. code-block:: console
-   
-   $ pip install pysqream pysqream-sqlalchemy -U
-
-
-Validate the installation
------------------------------
-
-Create a file called ``test.py``, containing the following:
-
-.. literalinclude:: test.py
-    :language: python
-    :caption: pysqream Validation Script
-    :linenos:
-
-Make sure to replace the parameters in the connection with the respective parameters for your SQream DB installation.
-
-Run the test file to verify that you can connect to SQream DB:
-
-.. code-block:: console
-   
-   $ python test.py
-   Version: v2020.1
-
-If all went well, you are now ready to build an application using the SQream DB Python connector!
-
-If any connection error appears, verify that you have access to a running SQream DB and that the connection parameters are correct.
-
-SQLAlchemy examples
-========================
-
-SQLAlchemy is an ORM for Python.
-
-When you install the SQream DB dialect (``pysqream-sqlalchemy``) you can use frameworks like Pandas, TensorFlow, and Alembic to query SQream DB directly.
-
-A simple connection example
----------------------------------
-
-.. code-block:: python
-
-   import sqlalchemy as sa
-   from sqlalchemy.engine.url import URL
-
-   engine_url = URL('sqream'
-                 , username='rhendricks'
-                 , password='secret_passwor"
-                 , host='localhost'
-                 , port=5000
-                 , database='raviga'
-                 , query={'use_ssl': False})
-
-   engine = sa.create_engine(engine_url)
-
-   res = engine.execute('create table test (ints int)')
-   res = engine.execute('insert into test values (5), (6)')
-   res = engine.execute('select * from test')
-
-Pulling a table into Pandas
----------------------------------
-
-In this example, we use the URL method to create the connection string.
-
-.. code-block:: python
-
-   import sqlalchemy as sa
-   import pandas as pd
-   from sqlalchemy.engine.url import URL
-
-
-   engine_url = URL('sqream'
-                 , username='rhendricks'
-                 , password='secret_passwor"
-                 , host='localhost'
-                 , port=5000
-                 , database='raviga'
-                 , query={'use_ssl': False})
-
-   engine = sa.create_engine(engine_url)
-   
-   table_df = pd.read_sql("select * from nba", con=engine)
-
-
-API Examples
-===============
-
-Explaining the connection example
----------------------------------------
-
-First, import the package and create a connection
-
-.. code-block:: python
-   
-   # Import pysqream package
-   
-   import pysqream
-
-   """
-   Connection parameters include:
-   * IP/Hostname
-   * Port
-   * database name
-   * username
-   * password 
-   * Connect through load balancer, or direct to worker (Default: false - direct to worker)
-   * use SSL connection (default: false)
-   * Optional service queue (default: 'sqream')
-   """
-   
-   # Create a connection object
-   
-   con = pysqream.connect(host='127.0.0.1', port=3108, database='raviga'
-                      , username='rhendricks', password='Tr0ub4dor&3'
-                      , clustered=True)
-
-Then, run a query and fetch the results
-
-.. code-block:: python
-
-   cur = con.cursor()  # Create a new cursor
-   # Prepare and execute a query
-   cur.execute('select show_version()')
-   
-   result = cur.fetchall() # `fetchall` gets the entire data set
-   
-   print (f"Version: {result[0][0]}")
-
-This should print the SQream DB version. For example ``v2020.1``.
-
-Finally, we will close the connection
-
-.. code-block:: python
-   
-   con.close()
-
-Using the cursor
---------------------------------------------
-
-The DB-API specification includes several methods for fetching results from the cursor.
-
-We will use the ``nba`` example. Here's a peek at the table contents:
-
-.. csv-table:: nba
-   :file: nba-t10.csv
-   :widths: auto
-   :header-rows: 1 
-
-Like before, we will import the library and create a :py:meth:`~Connection`, followed by :py:meth:`~Connection.execute` on a simple ``SELECT *`` query.
-
-.. code-block:: python
-   
-   import pysqream
-   con = pysqream.connect(host='127.0.0.1', port=3108, database='master'
-                      , username='rhendricks', password='Tr0ub4dor&3'
-                      , clustered=True)
-
-   cur = con.cursor() # Create a new cursor
-   # The select statement:
-   statement = 'SELECT * FROM nba'
-   cur.execute(statement)
-
-After executing the statement, we have a :py:meth:`Connection` cursor object waiting. A cursor is iterable, meaning that everytime we fetch, it advances the cursor to the next row.
-
-Use :py:meth:`~Connection.fetchone` to get one record at a time:
-
-.. code-block:: python
-   
-   first_row = cur.fetchone() # Fetch one row at a time (first row)
-   
-   second_row = cur.fetchone() # Fetch one row at a time (second row)
-
-To get several rows at a time, use :py:meth:`~Connection.fetchmany`:
-
-.. code-block:: python
-   
-   # executing `fetchone` twice is equivalent to this form:
-   third_and_fourth_rows = cur.fetchmany(2)
-
-To get all rows at once, use :py:meth:`~Connection.fetchall`:
-
-.. code-block:: python
-   
-   # To get all rows at once, use `fetchall`
-   remaining_rows = cur.fetchall()
-
-   # Close the connection when done
-   con.close()
-
-Here are the contents of the row variables we used:
-
-.. code-block:: pycon
-   
-   >>> print(first_row)
-   ('Avery Bradley', 'Boston Celtics', 0, 'PG', 25, '6-2', 180, 'Texas', 7730337)
-   >>> print(second_row)
-   ('Jae Crowder', 'Boston Celtics', 99, 'SF', 25, '6-6', 235, 'Marquette', 6796117)
-   >>> print(third_and_fourth_rows)
-   [('John Holland', 'Boston Celtics', 30, 'SG', 27, '6-5', 205, 'Boston University', None), ('R.J. Hunter', 'Boston Celtics', 28, 'SG', 22, '6-5', 185, 'Georgia State', 1148640)]
-   >>> print(remaining_rows)
-   [('Jonas Jerebko', 'Boston Celtics', 8, 'PF', 29, '6-10', 231, None, 5000000), ('Amir Johnson', 'Boston Celtics', 90, 'PF', 29, '6-9', 240, None, 12000000), ('Jordan Mickey', 'Boston Celtics', 55, 'PF', 21, '6-8', 235, 'LSU', 1170960), ('Kelly Olynyk', 'Boston Celtics', 41, 'C', 25, '7-0', 238, 'Gonzaga', 2165160),
-   [...]
-
-.. note:: Calling a fetch command after all rows have been fetched will return an empty array (``[]``).
-
-Reading result metadata
-----------------------------
-
-When executing a statement, the connection object also contains metadata about the result set (e.g.column names, types, etc).
-
-The metadata is stored in the :py:attr:`Connection.description` object of the cursor.
-
-.. code-block:: pycon
-   
-   >>> import pysqream
-   >>> con = pysqream.connect(host='127.0.0.1', port=3108, database='master'
-   ...                , username='rhendricks', password='Tr0ub4dor&3'
-   ...                , clustered=True)
-   >>> cur = con.cursor()
-   >>> statement = 'SELECT * FROM nba'
-   >>> cur.execute(statement)
-   
-   >>> print(cur.description)
-   [('Name', 'STRING', 24, 24, None, None, True), ('Team', 'STRING', 22, 22, None, None, True), ('Number', 'NUMBER', 1, 1, None, None, True), ('Position', 'STRING', 2, 2, None, None, True), ('Age (as of 2018)', 'NUMBER', 1, 1, None, None, True), ('Height', 'STRING', 4, 4, None, None, True), ('Weight', 'NUMBER', 2, 2, None, None, True), ('College', 'STRING', 21, 21, None, None, True), ('Salary', 'NUMBER', 4, 4, None, None, True)]
-
-To get a list of column names, iterate over the ``description`` list:
-   
-.. code-block:: pycon
-   
-   >>> [ i[0] for i in cur.description ]
-   ['Name', 'Team', 'Number', 'Position', 'Age (as of 2018)', 'Height', 'Weight', 'College', 'Salary']
-
-Loading data into a table
----------------------------
-
-This example loads 10,000 rows of dummy data to a SQream DB instance
-
-.. code-block:: python
-   
-   import pysqream
-   from datetime import date, datetime
-   from time import time
-
-   con = pysqream.connect(host='127.0.0.1', port=3108, database='master'
-                      , username='rhendricks', password='Tr0ub4dor&3'
-                      , clustered=True)
-   
-   # Create a table for loading
-   create = 'create or replace table perf (b bool, t tinyint, sm smallint, i int, bi bigint, f real, d double, s varchar(12), ss text, dt date, dtt datetime)'
-   con.execute(create)
-
-   # After creating the table, we can load data into it with the INSERT command
-
-   # Create dummy data which matches the table we created
-   data = (False, 2, 12, 145, 84124234, 3.141, -4.3, "Marty McFly" , u"キウイは楽しい鳥です" , date(2019, 12, 17), datetime(1955, 11, 4, 1, 23, 0, 0))
-   
-   
-   row_count = 10**4
-
-   # Get a new cursor
-   cur = con.cursor()
-   insert = 'insert into perf values (?,?,?,?,?,?,?,?,?,?,?)'
-   start = time()
-   cur.executemany(insert, [data] * row_count)
-   print (f"Total insert time for {row_count} rows: {time() - start} seconds")
-
-   # Close this cursor
-   cur.close()
-   
-   # Verify that the data was inserted correctly
-   # Get a new cursor
-   cur = con.cursor()
-   cur.execute('select count(*) from perf')
-   result = cur.fetchall() # `fetchall` collects the entire data set
-   print (f"Count of inserted rows: {result[0][0]}")
-
-   # When done, close the cursor
-   cur.close()
-   
-   # Close the connection
-   con.close()
-
-Reading data from a CSV file for load into a table
-----------------------------------------------------------
-
-We will write a helper function to create an :ref:`insert` statement, by reading an existing table's metadata.
-
-.. code-block:: python
-   
-   import pysqream
-   import datetime
-
-   def insert_from_csv(cur, table_name, csv_filename, field_delimiter = ',', null_markers = []):
-      """
-      We will first ask SQream DB for some table information.
-      This is important for understanding the number of columns, and will help
-      to create a matching INSERT statement
-      """
-
-      column_info = cur.execute(f"SELECT * FROM {table_name} LIMIT 0").description
-
-
-      def parse_datetime(v):
-         try:
-               return datetime.datetime.strptime(row[i], '%Y-%m-%d %H:%M:%S.%f')
-         except ValueError:
-               try:
-                  return datetime.datetime.strptime(row[i], '%Y-%m-%d %H:%M:%S')
-               except ValueError:
-                  return datetime.datetime.strptime(row[i], '%Y-%m-%d')
-
-      # Create enough placeholders (`?`) for the INSERT query string
-      qstring = ','.join(['?'] * len(column_info))
-      insert_statement = f"insert into {table_name} values ({qstring})"
-
-      # Open the CSV file
-      with open(csv_filename, mode='r') as csv_file:
-         csv_reader = csv.reader(csv_file, delimiter=field_delimiter)
-
-      # Execute the INSERT statement with the CSV data
-      cur.executemany(insert_statement, [row for row in csv_reader])
-
-
-   con = pysqream.connect(host='127.0.0.1', port=3108, database='master'
-                      , username='rhendricks', password='Tr0ub4dor&3'
-                      , clustered=True)
-   
-   cur = con.cursor()
-   insert_from_csv(cur, 'nba', 'nba.csv', field_delimiter = ',', null_markers = [])
-   
-   con.close()
-
-
-Using SQLAlchemy ORM to create tables and fill them with data
------------------------------------------------------------------------
-
-You can also use the ORM to create tables and insert data to them from Python objects.
-
-For example:
-
-.. code-block:: python
-   
-   import sqlalchemy as sa
-   import pandas as pd
-   from sqlalchemy.engine.url import URL
-
-
-   engine_url = URL('sqream'
-                 , username='rhendricks'
-                 , password='secret_passwor"
-                 , host='localhost'
-                 , port=5000
-                 , database='raviga'
-                 , query={'use_ssl': False})
-
-   engine = sa.create_engine(engine_url)
-   
-   # Build a metadata object and bind it
-   
-   metadata = sa.MetaData()
-   metadata.bind = engine
-   
-   # Create a table in the local metadata
-   
-   employees = sa.Table(
-   'employees'
-   , metadata 
-   , sa.Column('id', sa.Integer)
-   , sa.Column('name', sa.VARCHAR(32))
-   , sa.Column('lastname', sa.VARCHAR(32))
-   , sa.Column('salary', sa.Float)
-   )
-
-   # The create_all() function uses the SQream DB engine object
-   # to create all the defined table objects.
-
-   metadata.create_all(engine)
-   
-   # Now that the table exists, we can insert data into it.
-   
-   # Build the data rows
-   insert_data = [ {'id': 1, 'name': 'Richard','lastname': 'Hendricks',   'salary': 12000.75}
-                  ,{'id': 3,  'name': 'Bertram', 'lastname': 'Gilfoyle', 'salary': 8400.0}
-                  ,{'id': 8,  'name': 'Donald', 'lastname': 'Dunn', 'salary': 6500.40}
-                 ]
-
-   # Build the insert command
-   ins = employees.insert(insert_data)
-   
-   # Execute the command
-   result = engine.execute(ins)
-
-.. toctree::
-   :maxdepth: 8
-   :caption: Further information
-   
-   api-reference
diff --git a/guides/client_drivers/python/nba-t10.csv b/guides/client_drivers/python/nba-t10.csv
deleted file mode 100644
index 024530355..000000000
--- a/guides/client_drivers/python/nba-t10.csv
+++ /dev/null
@@ -1,10 +0,0 @@
-Name,Team,Number,Position,Age,Height,Weight,College,Salary
-Avery Bradley,Boston Celtics,0.0,PG,25.0,6-2,180.0,Texas,7730337.0
-Jae Crowder,Boston Celtics,99.0,SF,25.0,6-6,235.0,Marquette,6796117.0
-John Holland,Boston Celtics,30.0,SG,27.0,6-5,205.0,Boston University,
-R.J. Hunter,Boston Celtics,28.0,SG,22.0,6-5,185.0,Georgia State,1148640.0
-Jonas Jerebko,Boston Celtics,8.0,PF,29.0,6-10,231.0,,5000000.0
-Amir Johnson,Boston Celtics,90.0,PF,29.0,6-9,240.0,,12000000.0
-Jordan Mickey,Boston Celtics,55.0,PF,21.0,6-8,235.0,LSU,1170960.0
-Kelly Olynyk,Boston Celtics,41.0,C,25.0,7-0,238.0,Gonzaga,2165160.0
-Terry Rozier,Boston Celtics,12.0,PG,22.0,6-2,190.0,Louisville,1824360.0
diff --git a/guides/features/access_control.rst b/guides/features/access_control.rst
deleted file mode 100644
index 9850fafda..000000000
--- a/guides/features/access_control.rst
+++ /dev/null
@@ -1,599 +0,0 @@
-.. _access_control:
-
-**************
-Access control
-**************
-
-.. contents:: In this topic:
-   :local:
-
-Overview
-==========
-
-Access control provides authentication and authorization in SQream DB.
-
-SQream DB manages authentication and authorization using a role-based access control system (RBAC), like ANSI SQL and other SQL products.
-
-SQream DB has a default permissions system which is inspired by Postgres, but with more power. In most cases, this allows an administrator to set things up so that every object gets permissions set automatically.
-
-In SQream DB, users log in from any worker which verifies their roles and permissions from the metadata server. Each statement issues commands as the currently logged in role.
-
-Roles are defined at the cluster level, meaning they are valid for all databases in the cluster.
-
-To bootstrap SQream DB, a new install will always have one ``SUPERUSER`` role, typically named ``sqream``. To create more roles, you should first connect as this role.
-
-
-Terminology
-================
-
-Roles
-----------
-
-:term:`Role` : a role can be a user, a group, or both.
-
-Roles can own database objects (e.g. tables), and can assign permissions on those objects to other roles.
-
-Roles can be members of other roles, meaning a user role can inherit permissions from its parent role.
-
-Authentication
---------------------
-
-:term:`Authentication` : verifying the identity of the role. User roles have usernames (:term:`role names`) and passwords.
-
-
-Authorization
-----------------
-
-:term:`Authorization` : checking the role has permissions to do a particular thing. The :ref:`grant` command is used for this.
-
-
-Roles
-=====
-
-Roles are used for both users and groups.
-
-Roles are global across all databases in the SQream DB cluster.
-
-To use a ``ROLE`` as a user, it should have a password, the login permission, and connect permissions to the relevant databases.
-
-Creating new roles (users)
-------------------------------
-
-A user role can log in to the database, so it should have ``LOGIN`` permissions, as well as a password.
-
-For example:
-
-.. code-block:: postgres
-                
-   CREATE ROLE role_name ;
-   GRANT LOGIN to role_name ;
-   GRANT PASSWORD 'new_password' to role_name ;
-   GRANT CONNECT ON DATABASE database_name to role_name ;
-
-Examples:
-
-.. code-block:: postgres
-
-   CREATE  ROLE  new_role_name  ;  
-   GRANT  LOGIN  TO  new_role_name;  
-   GRANT  PASSWORD  'my_password'  TO  new_role_name;  
-   GRANT  CONNECT  ON  DATABASE  master  TO  new_role_name;
-
-A database role may have a number of permissions that define what tasks it can perform. These are assigned using the :ref:`grant` command.
-
-Dropping a user
----------------
-
-.. code-block:: postgres
-
-   DROP ROLE role_name ;
-
-Examples:
-
-.. code-block:: postgres
-
-   DROP ROLE  admin_role ;
-
-Altering a user name
-------------------------
-
-Renaming a user's role:
-
-.. code-block:: postgres
-
-   ALTER ROLE role_name RENAME TO new_role_name ;
-
-Examples:
-
-.. code-block:: postgres
-
-   ALTER ROLE  admin_role  RENAME  TO  copy_role ;
-
-.. _change_password:
-
-Changing user passwords
---------------------------
-
-To change a user role's password, grant the user a new password.
-
-.. code-block:: postgres
-
-   GRANT  PASSWORD  'new_password'  TO  rhendricks;  
-
-.. note:: Granting a new password overrides any previous password. Changing the password while the role has an active running statement does not affect that statement, but will affect subsequent statements.
-
-Public Role
------------
-
-There is a public role which always exists. Each role is granted to the ``PUBLIC`` role (i.e. is a member of the public group), and this cannot be revoked. You can alter the permissions granted to the public role.
-
-The ``PUBLIC`` role has ``USAGE`` and ``CREATE`` permissions on ``PUBLIC`` schema by default, therefore, new users can create, :ref:`insert`, :ref:`delete`, and :ref:`select` from objects in the ``PUBLIC`` schema.
-
-
-Role membership (groups)
--------------------------
-
-Many database administrators find it useful to group user roles together. By grouping users, permissions can be granted to, or revoked from a group with one command. In SQream DB, this is done by creating a group role, granting permissions to it, and then assigning users to that group role.
-
-To use a role purely as a group, omit granting it ``LOGIN`` and ``PASSWORD`` permissions.
-
-The ``CONNECT`` permission can be given directly to user roles, and/or to the groups they are part of.
-
-.. code-block:: postgres
-
-   CREATE ROLE my_group;
-
-Once the group role exists, you can add user roles (members) using the ``GRANT`` command. For example:
-
-.. code-block:: postgres
-
-   -- Add my_user to this group
-   GRANT my_group TO my_user;
-
-
-To manage object permissions like databases and tables, you would then grant permissions to the group-level role (see :ref:`the permissions table` below.
-
-All member roles then inherit the permissions from the group. For example:
-
-.. code-block:: postgres
-
-   -- Grant all group users connect permissions
-   GRANT  CONNECT  ON  DATABASE  a_database  TO  my_group;
-   
-   -- Grant all permissions on tables in public schema
-   GRANT  ALL  ON  all  tables  IN  schema  public  TO  my_group;
-
-Removing users and permissions can be done with the ``REVOKE`` command:
-
-.. code-block:: postgres
-
-   -- remove my_other_user from this group
-   REVOKE my_group FROM my_other_user;
-
-.. _permissions_table:
-
-Permissions
-===========
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-
-   * - Object/layer
-     - Permission
-     - Description
-
-   * - all databases
-     - ``LOGIN``
-     - use role to log into the system (the role also needs connect permission on the database it is connecting to)
-
-   * - all databases
-     - ``PASSWORD``
-     - the password used for logging into the system
-
-   * - all databases
-     - ``SUPERUSER``
-     - no permission restrictions on any activity
-
-   * - database
-     - ``SUPERUSER``
-     - no permission restrictions on any activity within that database (this does not include modifying roles or permissions)
-
-   * - database
-     - ``CONNECT``
-     - connect to the database
-
-   * - database
-     - ``CREATE``
-     - create schemas in the database
-
-   * - database
-     - ``CREATE FUNCTION``
-     - create and drop functions
-     
-   * - schema
-     - ``USAGE``
-     - allows additional permissions within the schema
-
-   * - schema
-     - ``CREATE``
-     - create tables in the schema
-
-   * - table
-     - ``SELECT``
-     - :ref:`select` from the table
-
-   * - table
-     - ``INSERT``
-     - :ref:`insert` into the table
-
-   * - table
-     - ``DELETE``
-     - :ref:`delete` and :ref:`truncate` on the table
-
-   * - table
-     - ``DDL``
-     - drop and alter on the table
-
-   * - table
-     - ``ALL``
-     - all the table permissions
-
-   * - function
-     - ``EXECUTE``
-     - use the function
-
-   * - function
-     - ``DDL``
-     - drop and alter on the function
-
-   * - function
-     - ``ALL``
-     - all function permissions
-
-GRANT
------
-
-:ref:`grant` gives permissions to a role.
-
-.. code-block:: postgres
-
-   -- Grant permissions at the instance/ storage cluster level:
-   GRANT 
-
-   { SUPERUSER
-   | LOGIN 
-   | PASSWORD '' 
-   } 
-   TO  [, ...] 
-
-   -- Grant permissions at the database level:
-        GRANT {{CREATE | CONNECT| DDL | SUPERUSER | CREATE FUNCTION} [, ...] | ALL [PERMISSIONS]}
-
-   ON DATABASE  [, ...]
-   TO  [, ...] 
-
-   -- Grant permissions at the schema level: 
-   GRANT {{ CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [ 
-   PERMISSIONS ]} 
-   ON SCHEMA  [, ...] 
-   TO  [, ...] 
-       
-   -- Grant permissions at the object level: 
-   GRANT {{SELECT | INSERT | DELETE | DDL } [, ...] | ALL [PERMISSIONS]} 
-   ON { TABLE  [, ...] | ALL TABLES IN SCHEMA  [, ...]} 
-   TO  [, ...]
-       
-   -- Grant execute function permission: 
-   GRANT {ALL | EXECUTE | DDL} ON FUNCTION function_name 
-   TO role; 
-       
-   -- Allows role2 to use permissions granted to role1
-   GRANT  [, ...] 
-   TO  
-
-    -- Also allows the role2 to grant role1 to other roles:
-   GRANT  [, ...] 
-   TO  
-   WITH ADMIN OPTION
-  
-``GRANT`` examples:
-
-.. code-block:: postgres
-
-   GRANT  LOGIN,superuser  TO  admin;
-
-   GRANT  CREATE  FUNCTION  ON  database  master  TO  admin;
-
-   GRANT  SELECT  ON  TABLE  admin.table1  TO  userA;
-
-   GRANT  EXECUTE  ON  FUNCTION  my_function  TO  userA;
-
-   GRANT  ALL  ON  FUNCTION  my_function  TO  userA;
-
-   GRANT  DDL  ON  admin.main_table  TO  userB;
-
-   GRANT  ALL  ON  all  tables  IN  schema  public  TO  userB;
-
-   GRANT  SELECT  ON  all  views  IN  schema  admin  TO  userA;
-
-   GRANT  admin  TO  userC;
-
-   GRANT  superuser  ON  schema  demo  TO  userA
-
-   GRANT  admin_role  TO  userB;
-
-REVOKE
-------
-
-:ref:`revoke` removes permissions from a role.
-
-.. code-block:: postgres
-
-   -- Revoke permissions at the instance/ storage cluster level:
-   REVOKE
-   { SUPERUSER
-   | LOGIN
-   | PASSWORD
-   }
-   FROM  [, ...]
-            
-   -- Revoke permissions at the database level:
-   REVOKE {{CREATE | CONNECT | DDL | SUPERUSER | CREATE FUNCTION}[, ...] |ALL [PERMISSIONS]}
-   ON DATABASE  [, ...]
-   FROM  [, ...]
-
-   -- Revoke permissions at the schema level:
-   REVOKE { { CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [PERMISSIONS]}
-   ON SCHEMA  [, ...]
-   FROM  [, ...]
-            
-   -- Revoke permissions at the object level:
-   REVOKE { { SELECT | INSERT | DELETE | DDL } [, ...] | ALL }
-   ON { [ TABLE ]  [, ...] | ALL TABLES IN SCHEMA
-
-          [, ...] }
-   FROM  [, ...]
-            
-   -- Removes access to permissions in role1 by role 2
-   REVOKE  [, ...] FROM  [, ...] WITH ADMIN OPTION
-
-   -- Removes permissions to grant role1 to additional roles from role2
-   REVOKE  [, ...] FROM  [, ...] WITH ADMIN OPTION
-
-
-Examples:
-
-.. code-block:: postgres
-
-   REVOKE  superuser  on  schema  demo  from  userA;
-
-   REVOKE  delete  on  admin.table1  from  userB;
-
-   REVOKE  login  from  role_test;
-
-   REVOKE  CREATE  FUNCTION  FROM  admin;
-
-Default permissions
--------------------
-
-The default permissions system (See :ref:`alter_default_permissions`) 
-can be used to automatically grant permissions to newly 
-created objects (See the departmental example below for one way it can be used).
-
-A default permissions rule looks for a schema being created, or a
-table (possibly by schema), and is table to grant any permission to
-that object to any role. This happens when the create table or create
-schema statement is run.
-
-
-.. code-block:: postgres
-
-
-   ALTER DEFAULT PERMISSIONS FOR target_role_name
-        [IN schema_name, ...]
-        FOR { TABLES | SCHEMAS }
-        { grant_clause | DROP grant_clause}
-        TO ROLE { role_name | public };
-
-   grant_clause ::=
-     GRANT
-        { CREATE FUNCTION
-        | SUPERUSER
-        | CONNECT
-        | CREATE
-        | USAGE
-        | SELECT
-        | INSERT
-        | DELETE
-        | DDL
-        | EXECUTE
-        | ALL
-        }
-
-
-Departmental Example
-=======================
-
-You work in a company with several departments.
-
-The example below shows you how to manage permissions in a database shared by multiple departments, where each department has different roles for the tables by schema. It walks you through how to set the permissions up for existing objects and how to set up default permissions rules to cover newly created objects.
-
-The concept is that you set up roles for each new schema with the correct permissions, then the existing users can use these roles. 
-
-A superuser must do new setup for each new schema which is a limitation, but superuser permissions are not needed at any other time, and neither are explicit grant statements or object ownership changes.
-
-In the example, the database is called ``my_database``, and the new or existing schema being set up to be managed in this way is called ``my_schema``.
-
-.. figure:: /_static/images/access_control_department_example.png
-   :scale: 60 %
-   
-   Our departmental example has four user group roles and seven users roles
-
-There will be a group for this schema for each of the following:
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-
-   * - Group
-     - Activities
-
-   * - database designers
-     - create, alter and drop tables
-
-   * - updaters
-     - insert and delete data
-
-   * - readers
-     - read data
-
-   * - security officers
-     - add and remove users from these groups
-
-Setting up the department permissions
-------------------------------------------
-
-As a superuser, you connect to the system and run the following:
-
-.. code-block:: postgres
-
-   -- create the groups
-
-   CREATE ROLE my_schema_security_officers;
-   CREATE ROLE my_schema_database_designers;
-   CREATE ROLE my_schema_updaters;
-   CREATE ROLE my_schema_readers;
-
-   -- grant permissions for each role
-   -- we grant permissions for existing objects here too, 
-   -- so you don't have to start with an empty schema
-
-   -- security officers
-
-   GRANT connect ON DATABASE my_database TO my_schema_security_officers;
-   GRANT usage ON SCHEMA my_schema TO my_schema_security_officers;
-
-   GRANT my_schema_database_designers TO my_schema_security_officers WITH ADMIN OPTION;
-   GRANT my_schema_updaters TO my_schema_security_officers WITH ADMIN OPTION;
-   GRANT my_schema_readers TO my_schema_security_officers WITH ADMIN OPTION;
-
-   -- database designers
-
-   GRANT connect ON DATABASE my_database TO my_schema_database_designers;
-   GRANT usage ON SCHEMA my_schema TO my_schema_database_designers;
-
-   GRANT create,ddl ON SCHEMA my_schema TO my_schema_database_designers;
-
-   -- updaters
-
-   GRANT connect ON DATABASE my_database TO my_schema_updaters;
-   GRANT usage ON SCHEMA my_schema TO my_schema_updaters;
-
-   GRANT SELECT,INSERT,DELETE ON ALL TABLES IN SCHEMA my_schema TO my_schema_updaters;
-
-   -- readers
-
-   GRANT connect ON DATABASE my_database TO my_schema_readers;
-   GRANT usage ON SCHEMA my_schema TO my_schema_readers;
-
-   GRANT SELECT ON ALL TABLES IN SCHEMA my_schema TO my_schema_readers;
-   GRANT EXECUTE ON ALL FUNCTIONS TO my_schema_readers;
-
-
-   -- create the default permissions for new objects
-
-   ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema
-    FOR TABLES GRANT SELECT,INSERT,DELETE TO my_schema_updaters;
-
-   -- For every table created by my_schema_database_designers, give access to my_schema_readers:
-   
-   ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema
-    FOR TABLES GRANT SELECT TO my_schema_readers;
-
-.. note::
-   * This process needs to be repeated by a user with ``SUPERUSER`` permissions each time a new schema is brought into this permissions management approach.
-   
-   * 
-      By default, any new object created will not be accessible by our new ``my_schema_readers`` group.
-      Running a ``GRANT SELECT ...`` only affects objects that already exist in the schema or database.
-   
-      If you're getting a ``Missing the following permissions: SELECT on table 'database.public.tablename'`` error, make sure that
-      you've altered the default permissions with the ``ALTER DEFAULT PERMISSIONS`` statement.
-
-Creating new users in the departments
------------------------------------------
-
-After the group roles have been created, you can now create user roles for each of your users.
-
-.. code-block:: postgres
-
-   -- create the new database designer users
-   
-   CREATE  ROLE  ecodd;
-   GRANT  LOGIN  TO  ecodd;
-   GRANT  PASSWORD  'ecodds_secret_password'  TO ecodd;
-   GRANT  CONNECT  ON  DATABASE  my_database  TO  ecodd;
-   GRANT my_schema_database_designers TO ecodd;
-
-   CREATE  ROLE  ebachmann;
-   GRANT  LOGIN  TO  ebachmann;
-   GRANT  PASSWORD  'another_secret_password'  TO ebachmann;
-   GRANT  CONNECT  ON  DATABASE  my_database  TO  ebachmann;
-   GRANT my_database_designers TO ebachmann;
-
-   -- If a user already exists, we can assign that user directly to the group
-   
-   GRANT my_schema_updaters TO rhendricks;
-   
-   -- Create users in the readers group
-   
-   CREATE  ROLE  jbarker;
-   GRANT  LOGIN  TO  jbarker;
-   GRANT  PASSWORD  'action_jack'  TO jbarker;
-   GRANT  CONNECT  ON  DATABASE  my_database  TO  jbarker;
-   GRANT my_schema_readers TO jbarker;
-   
-   CREATE  ROLE  lbream;
-   GRANT  LOGIN  TO  lbream;
-   GRANT  PASSWORD  'artichoke123'  TO lbream;
-   GRANT  CONNECT  ON  DATABASE  my_database  TO  lbream;
-   GRANT my_schema_readers TO lbream;
-   
-   CREATE  ROLE  pgregory;
-   GRANT  LOGIN  TO  pgregory;
-   GRANT  PASSWORD  'c1ca6a'  TO pgregory;
-   GRANT  CONNECT  ON  DATABASE  my_database  TO  pgregory;
-   GRANT my_schema_readers TO pgregory;
-
-   -- Create users in the security officers group
-
-   CREATE  ROLE  hoover;
-   GRANT  LOGIN  TO  hoover;
-   GRANT  PASSWORD  'mintchip'  TO hoover;
-   GRANT  CONNECT  ON  DATABASE  my_database  TO  hoover;
-   GRANT my_schema_security_officers TO hoover;
-
-
-.. todo:
-   create some example users
-   show that they have the right permission
-   try out the with admin option. we can't really do a security officer because
-   only superusers can create users and logins. see what can be done
-   need 1-2 users in each group, for at least 2 schemas/departments
-   this example will be very big just to show what this setup can do ...
-   example: a security officer for a department which will only have
-     read only access to a schema can only get that with admin option
-     access granted to them
-
-After this setup:
-
-* Database designers will be able to run any ddl on objects in the schema and create new objects, including ones created by other database designers
-* Updaters will be able to insert and delete to existing and new tables
-* Readers will be able to read from existing and new tables
-
-All this will happen without having to run any more ``GRANT`` statements.
-
-Any security officer will be able to add and remove users from these
-groups. Creating and dropping login users themselves must be done by a
-superuser.
diff --git a/guides/features/access_control_notes.rst b/guides/features/access_control_notes.rst
deleted file mode 100644
index 4dacb78ee..000000000
--- a/guides/features/access_control_notes.rst
+++ /dev/null
@@ -1,357 +0,0 @@
-.. _access_control_notes:
-
-**************
-Access control
-**************
-
-.. contents:: In this topic:
-   :local:
-
-The system which provides basic authentication and authorization for
-users in SQream DB.
-
-Authentication: this is how the system checks that you are who you say
-you are. This is done using the familiar usernames and passwords.
-
-Authorization: this is how the system checks that you are allowed to
-do the action that you are trying to do.
-
-Compared to ANSI SQL and other SQL products:
-
-* 
-   SQream DB has roles as users and as groups, like ANSI SQL and other SQL products a user may be familiar with
-
-* 
-   SQream DB has a default permissions system based on the system in Postgres, but with more power.
-   In most cases, this allows an administrator to set things up so that every object gets permissions set
-   automatically.
-
-* SQream DB does not have row based permissions
-
-* SQream DB does not have object ownership
-
-.. http://docs.sqream.com/latest/manual/Content/Guides/Quick_Guides/Quick_guide_to_roles_and_permissions/Quick_guide_to_roles_and_permissions.htm
-
-.. http://docs.sqream.com/latest/manual/Content/Concepts/12_1.10_Catalog_(information_schema).htm?tocpath=Concepts%7CCatalog%20(information%20schema)%7C_____0#Catalog_(information_schema)_..198
-
-.. http://docs.sqream.com/latest/manual/Content/SQL_Reference_Guide/16_2.1_Data_Definition_Language.htm?tocpath=SQream%20DB%20%20SQL%20Reference%20Guide%7CData%20Definition%20Language%7C_____0#Database_Roles_and_Permissions_..322
-
-
-
-Introductory examples
-=====================
-
-go through some basic permissions
-
-show a user trying to do something without permission
-
-then add the permission
-
-then show it succeeding
-
-show adding a new login user
-
-show adding a user to another role
-
-show adding a superuser
-
-
-Reference for all permissions
-=============================
-
-attributes for a role: name and optional password
-
-permissions:
-
-* superuser
-* login
-* connect to database 
-* create, ddl, superuser, create function 
-* create, ddl, usage, superuser 
-* select, insert, delete, ddl 
-* *  
-* execute | ddl 
-
-what happens when e.g. a database doesn't exist
-
-what if you drop it then recreate it
-
-what are the options for default permissions
-
-Permissions catalog
-===================
-
-show the permissions as data structures
-
-make sure all this appears in the catalog
-
-do the catalog reference/guide here for now
-
-establish a better catalog: write create tables statements too
-do full examples of querying the catalog
-
-sqream_catalog.roles                  
-
-sqream_catalog.role_memberships       
-
-sqream_catalog.table_permissions      
-
-sqream_catalog.database_permissions   
-
-sqream_catalog.schema_permissions     
-
-sqream_catalog.permission_types
-
-
-How the permissions work
-========================
-
-
-How to work out if a user has permission to execute a statement?
--> go through all the options
-
-the two parts are
-
-1. work out what permissions are needed from a statement
-
-2. work out what permissions a user has by role membership
-
-how do default permissions work
--------------------------------
-
-when you create an object:
-
-the system looks for default permissions statements which match the
-current role or a group the current role is in, and the object being
-created
-
-each match will behave as if a grant permissions statement was also
-run, on the created object, granted to the target role in the default
-permissions match, with the permissions in the default permissions
-match
-
-utility functions
-=================
-
-what utility functions are relevant for permissions
-
-how do they interact with permissions
-
-Syntax reference
-================
-
-Roles
------
-
-.. code-block:: postgresql
-
-     create | alter | drop role
-     grant
-     
-     -- Alter default permissions
-     
-     CREATE ROLE role_name ;
-     GRANT PASSWORD 'new_password' to role_name ;
-     
-     DROP ROLE role_name ;
-     
-     -- Alter - rename only:
-     
-     ALTER ROLE role_name RENAME TO new_role_name ;
-
-
-Granting permissions
---------------------
-
-to create a database installation wide superuser:
-
-.. code-block:: postgresql
-
-     GRANT SUPERUSER to 
-
-
-does a super user have login + connect to all databases?
-
-to allow a user to login, and to connect to a database
-
-.. code-block:: postgresql
-
-     GRANT LOGIN to role_name ;
-     GRANT CONNECT ON DATABASE database_name to role_name ;
-
-
-can a user have one and not the other?
-
-when should a user have a password
-
-.. code-block:: postgresql
-
-     GRANT 
-      { SUPERUSER
-      | LOGIN 
-      | PASSWORD '' 
-      } 
-     TO  [, ...] 
-
-     GRANT  [, ...] 
-     TO  
-     [WITH ADMIN OPTION]
-
-
-From the current docs, it's not that clear what all these mean
-
-granting permissions to objects
--------------------------------
-
-.. code-block:: postgresql
-
-     -- Grant permissions at the database level:
-     GRANT {{CREATE | CONNECT| DDL | SUPERUSER | CREATE FUNCTION} [, ...] | ALL [PERMISSIONS]}
-
-       ON DATABASE  [, ...]
-       TO  [, ...] 
-
-     -- Grant permissions at the schema level: 
-     GRANT {{ CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [ 
-       PERMISSIONS ]} 
-       ON SCHEMA  [, ...] 
-       TO  [, ...] 
-
-     -- Grant permissions at the object level: 
-     GRANT {{SELECT | INSERT | DELETE | DDL } [, ...] | ALL [PERMISSIONS]} 
-       ON { TABLE  [, ...] | ALL TABLES IN SCHEMA  [, ...]} 
-       TO  [, ...]
-
-     -- Grant execute function permission: 
-       GRANT {ALL | EXECUTE | DDL} ON FUNCTION function_name 
-       TO role; 
-
-
-alter default permissions
--------------------------
-
-.. code-block:: postgresql
-
-     ALTER DEFAULT PERMISSIONS FOR 
-       IN  FOR TABLES
-       GRANT { SELECT | INSERT | DELETE [,...] } TO ;
-
-I think you can also do it for schemas?
-
-how do you undo a default permissions - use revoke? something isn't
-quite right about that
-
-revoking permissions
---------------------
-
-.. code-block:: postgresql
-
-     -- Revoke permissions at the cluster level:
-     REVOKE
-       { SUPERUSER
-       | LOGIN
-       | PASSWORD
-       }
-     FROM  [, ...]
-
-     -- Revoke permissions at the database level:
-     REVOKE {{CREATE | CONNECT | DDL | SUPERUSER | CREATE FUNCTION}[, ...] |ALL [PERMISSIONS]}
-       ON DATABASE  [, ...]
-       FROM  [, ...]
-
-     -- Revoke permissions at the schema level:
-     REVOKE { { CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [PERMISSIONS]}
-       ON SCHEMA  [, ...]
-       FROM  [, ...]
-
-     -- Revoke permissions at the object level:
-       REVOKE { { SELECT | INSERT | DELETE | DDL } [, ...] | ALL }
-       ON { [ TABLE ]  [, ...] | ALL TABLES IN SCHEMA
-
-        [, ...] }
-       FROM  [, ...]
-
-     -- Revoke privileges from other roles by granting one role to another:
-       REVOKE  [, ...] FROM  [, ...] WITH ADMIN OPTION
-
-
-Behaviour reference
-===================
-
-show examples of every permission? Or just a subset
-
-example will have the permission fail
-
-then the add permission statement
-
-then the permission succeed
-
-can also go through something similar for default permissions
-
-Usage guides
-============
-
-minimal permission system use
------------------------------
-
-Trivial use of permissions system in sqream: use super user
-
-how to add a new superuser role
-
-what this means
-
-adding a guest user
-
-simple user, with limited read only ability
-
-
-Basic use
----------
-
-how to set up a group with permissions database wide for the following:
-
-* security officer
-* database architect
-* updater
-* reader
-* udf author
-
-how to maintain this
-
-
-Advanced use
-------------
-
-permissions/group per schema
-
-show a list of roles for a schema, how you set it up
-
-then show how to maintain this system
-
-variation: roles which cover multiple schemas
-
-* what does a superuser need to do
-* what can a division 'owner' do
-
-maintain - how to add something missing or modify:
-
-* a new schema
-
-* a new division
-
-* a new user
-
-* remove access
-
-* fix an existing schema to add permissions
-
-   * maybe a mistake
-
-   * maybe a division gets new access to an existing schema
-
-key secure things:
-
-* what can only superusers do
-* what are normal users restricted from doing
-* who else can do stuff with the permissions system
-* how are divisions protected from other divisions
diff --git a/guides/features/chunks_and_extents.rst b/guides/features/chunks_and_extents.rst
deleted file mode 100644
index 200235891..000000000
--- a/guides/features/chunks_and_extents.rst
+++ /dev/null
@@ -1,93 +0,0 @@
-.. _chunks_and_extents:
-
-***********************
-Chunks and extents
-***********************
-
-All data in SQream DB is stored in logical tables. Each table is made up of rows, spanning one or more columns.
-
-Internally however, SQream DB stores data partitioned vertically by column, and horizontally by **chunks**.
-
-This topic describes the chunk concept, which can be helpful for tuning SQream DB workloads and table structures.
-
-What are chunks? What are extents?
-======================================
-
-Chunks
------------
-
-All data in a table is automatically partitioned into columns, and each column is divided up further into chunks.
-
-A **chunk** is a contiguous number of rows from a specific column. It can be thought of as an automatic partition that spans several millions of records of one column.
-
-.. figure:: /_static/images/chunking.png
-   :scale: 80 %
-   
-   Chunks are collections of rows from a column
-
-A chunk is often between 1MB to a couple hundred megabytes uncompressed, depending on the data type (however, all data in SQream DB is stored compressed).
-
-This chunk size is suitable for filtering and deleting data from large tables, which can contain hundreds of millions or even billions of chunks.
-
-SQream DB adds :ref:`metadata` to chunks automatically. 
-
-.. note:: Chunking is automatic and transparent for all tables in SQream DB.
-
-
-Extents
-----------
-
-The next step up from the chunk, an **extent** is a specific number of contiguous chunks.
-
-Extents are designed to optimize disk access patterns, at around 20MB compressed, on-disk.
-
-An extent will therefore include between 1 and 25 chunks, based on the actual compressed chunk size.
-
-.. figure:: /_static/images/extents.png
-   :scale: 80 %
-   
-   Extents are a collection of several contiguous chunks
-
-
-Why was SQream DB built with chunks and extents?
-=======================================================
-
-Benefits of chunking
----------------------------
-
-Unlike node-partitioning (or sharding), chunking carries several benefits:
-
-* Chunks are small enough to allow multiple workers to read them concurrently
-
-* Chunks are optimized for fast insertion of data
-
-* Chunks carry metadata, which narrows down their contents for the optimizer
-
-* Chunks are ideal for data retension as they can be deleted en-masse
-
-* Chunks are optimized for reading into RAM and the GPU
-
-* Chunks are compressed individually, which improves compression and data locality
-
-Storage reorganization
---------------------------
-
-SQream DB performs some background storage reorganization to optimize I/O and read patterns.
-
-For example, when data is inserted in small batches, SQream DB will run two background processes called **rechunk** and **reextent** to reorganize the data into larger contiguous chunks and extents.
-This is also what happens when :ref:`data is deleted`.
-
-Data is never overwritten in SQream DB. Instead, new optimized chunks and extents are written to replace the old chunks and extents. Once all data has been rewritten, SQream DB will swap over to the new optimized chunks and extents and remove the old, unoptimized data.
-
-
-Metadata
-------------
-
-The chunk metadata that SQream DB collects enables effective skipping of chunks and extents when queries are executed. When a query specifies a filter (e.g. ``WHERE`` or ``JOIN`` condition) on a range of values that spans a fraction of the table values, SQream DB will optimally scan only that fraction of the table chunks.
-
-Queries that filter on fine-grained date and time ranges will be the most effective, particularly when :ref:`data is timestamped`, and when tables contain a large amount of historical data.
-
-See more in our :ref:`time_based_data_management` guide and our :ref:`metadata_system` guide.
-
-
-
diff --git a/guides/features/data_clustering.rst b/guides/features/data_clustering.rst
deleted file mode 100644
index deeedf9d8..000000000
--- a/guides/features/data_clustering.rst
+++ /dev/null
@@ -1,79 +0,0 @@
-.. _data_clustering:
-
-***********************
-Data clustering
-***********************
-
-Together with the :ref:`chunking` and :ref:`metadata_system`, SQream DB uses information to execute queries efficiently.
-
-SQream DB automatically collects metadata on incoming data. This works well when the data is naturally ordered (e.g. :ref:`with time-based data`).
-
-There are situations where you know more about the incoming data than SQream DB. If you help by defining **clustering keys**, SQream DB can automatically improve query processing. SQream DB's query optimizer typically selects the most efficient method when executing queries. If no clustering keys are available, it may have to scan tables physically.
-
-Clustered tables
-======================
-
-A table is considered "clustered" by one or more clustering keys if rows containing similar values with regard to these expressions are more likely to be located together on disk.
-
-For example, a table containing a date column whose values cover a whole month but each chunk on disk covers less than a specific day is considered clustered by this column. 
-
-Good clustering has a significant positive impact on query performance.
-
-When does clustering help?
-===================================
-
-When a table is well-clustered, the metadata collected for each chunk is much more effective (the ranges are more localized).
-
-In turn, SQream DB's query engine chooses to read fewer irrelevant chunks (or just avoid processing them).
-
-Here are some common scenarios in which data clustering is beneficial:
-
-* 
-   When a query contains a ``WHERE`` predicate of the form ``column COMPARISON value``.
-   For example, ``date_column > '2019-01-01'`` or  ``id = 107`` and the columns referenced are clustering keys/
-   
-   SQream DB will only read the portion of the data containing values matching these predicates.
-
-* When two clustered tables are joined on their respective clustering keys, SQream DB will utilize the metadata to identify the matching chunks more easily.
-
-
-Controlling data clustering
-=================================
-
-Some tables are naturally clustered. For example - a call log table containing CDRs can be naturally clustered by call date if data is inserted as it is generated, or bulk loaded in batches. Data can also be clustered by a region ID, per city, or customer type, depending on the source.
-
-
-If the incoming data is not well-clustered (by the desired key), it is possible to tell SQream DB which keys it should cluster by.
-
-This can be done upon table creation (:ref:`create_table`), or retroactively (:ref:`cluster_by`). New data will be clustered upon insert.
-
-When data is loaded to an explicitly clustered table, SQream DB partially sorts it. While this slows down the insert time, it is often beneficial for subsequent queries.
-
-.. note:: 
-
-   Some queries significantly benefit from the decision to use clustering. 
-   For example, queries that filter or join extensively on clustered columns will benefit.  
-   
-   
-   However, clustering can slow down data insertion. Some insert workloads can be up to 75% slower.
-   
-   If you are not sure whether or not a specific scenario will benefit from clustering, we recommended testing end-to-end (both insert and query performance) on a small subset of the data before commiting to permanent clustering keys.
-
-
-Examples
-==========
-
-Creating a clustered table
------------------------------
-
-Even when the table is naturally ordered by ``start_date``, we can specify a cluster key that is different. This will likely improve performance for queries that order by or rely on users' ``country``.
-
-.. code-block:: postgres
-
-   CREATE TABLE users (
-      name VARCHAR(30) NOT NULL,
-      start_date datetime not null,
-      country VARCHAR(30) DEFAULT 'Unknown' NOT NULL
-   ) CLUSTER BY country;
-
-
diff --git a/guides/features/external_data/hdfs.rst b/guides/features/external_data/hdfs.rst
deleted file mode 100644
index 71135163f..000000000
--- a/guides/features/external_data/hdfs.rst
+++ /dev/null
@@ -1,323 +0,0 @@
-.. _hdfs:
-
-***********************
-HDFS
-***********************
-
-SQream DB has a native HDFS connector for inserting data. The ``hdfs://`` URI specifies an external file path on a Hadoop Distributed File System.
-
-File names may contain wildcard characters and the files can be a CSV or columnar format like Parquet and ORC.
-
-
-.. contents:: In this topic:
-   :local:
-
-Verifying HDFS configuration
-==============================
-
-SQream DB's built-in HDFS relies on the host's Hadoop HDFS configuration. 
-
-Before you can use HDFS, you should verify that all SQream DB hosts are configured correctly.
-
-Use built-in Hadoop libraries
--------------------------------
-
-SQream DB comes with Hadoop libraries built-in. In a typical SQream DB installation, you'll find Hadoop and JDK libraries in the ``hdfs`` subdirectory of the package.
-
-If you are using the built-in libraries, it's important to note where they are.
-
-For example, if SQream DB was installed to ``/opt/sqream``, here's how to set-up the environment variables from the shell:
-
-.. _set_hadoop_classpath:
-
-.. code-block:: console
-
-   $ export JAVA_HOME=/opt/sqream/hdfs/jdk
-   $ export HADOOP_INSTALL=/opt/sqream/hdfs/hadoop
-   
-   $ export PATH=$PATH:${HADOOP_INSTALL}/bin:${HADOOP_INSTALL}/sbin
-   $ export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_INSTALL}/lib/native
-   $ export CLASSPATH=$CLASSPATH:`${HADOOP_INSTALL}/bin/hadoop classpath --glob`
-   $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_LIB_NATIVE_DIR
-
-   $ export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
-   $ export HADOOP_COMMON_HOME=$HADOOP_INSTALL
-   $ export HADOOP_HDFS_HOME=$HADOOP_INSTALL
-   $ export YARN_HOME=$HADOOP_INSTALL
-   
-   $ export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
-   $ export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
-   $ export HADOOP_HOME=$HADOOP_INSTALL
-   $ export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=${HADOOP_COMMON_LIB_NATIVE_DIR}"
-   $ export ARROW_LIBHDFS_DIR=${HADOOP_COMMON_LIB_NATIVE_DIR}
-
-
-You'll find ``core-site.xml`` and other configuration files in ``/opt/sqream/hdfs/hadoop/etc/hadoop``
-
-To persist these settings, place these variable settings in a 'run commands' file like ``.bashrc``. Test this by examining the output of ``$ echo $ARROW_LIBHDFS_DIR``.
-
-.. note:: 
-   
-   * This process needs to be repeated for every host in the SQream DB cluster, and from SQream DB's host username (often ``sqream``)
-   
-   * Restart SQream DB workers on the host after setting these parameters for them to take effect.
-
-(Optional) Overriding the Hadoop environment
-------------------------------------------------------
-
-If you have an existing Hadoop environment set-up on the host, you can override SQream DB's built-in Hadoop by setting the environment variables accordingly.
-
-For example,
-
-.. code-block:: console
-
-   $ export JAVA_HOME=/usr/local/java-1.8.0/
-   $ export HADOOP_INSTALL=/usr/local/hadoop-3.2.1
-   
-   $ export PATH=$PATH:${HADOOP_INSTALL}/bin:${HADOOP_INSTALL}/sbin
-   $ export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_INSTALL}/lib/native
-   $ export CLASSPATH=$CLASSPATH:`${HADOOP_INSTALL}/bin/hadoop classpath --glob`
-   $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_LIB_NATIVE_DIR
-
-   $ export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
-   $ export HADOOP_COMMON_HOME=$HADOOP_INSTALL
-   $ export HADOOP_HDFS_HOME=$HADOOP_INSTALL
-   $ export YARN_HOME=$HADOOP_INSTALL
-   
-   $ export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
-   $ export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
-   $ export HADOOP_HOME=$HADOOP_INSTALL
-   $ export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=${HADOOP_COMMON_LIB_NATIVE_DIR}"
-   $ export ARROW_LIBHDFS_DIR=${HADOOP_COMMON_LIB_NATIVE_DIR}
-
-
-To persist these settings, place these variable settings in a 'run commands' file like ``.bashrc``. Test this by examining the output of ``$ echo $ARROW_LIBHDFS_DIR``.
-
-.. note:: 
-   
-   * This process needs to be repeated for every host in the SQream DB cluster, and from SQream DB's host username (often ``sqream``)
-   
-   * Restart SQream DB workers on the host after setting these parameters for them to take effect.
-
-Configuring the node
-======================
-
-A Hadoop administrator will want to edit the configuration XMLs to allow access to your Hadoop cluster.
-
-If using the SQream DB Hadoop libraries, modify the following files to match your cluster settings:
-
-* ``/opt/sqream/hdfs/hadoop/etc/hadoop/core-site.xml``
-* ``/opt/sqream/hdfs/hadoop/etc/hadoop/yarn-site.xml``
-* ``/opt/sqream/hdfs/hadoop/etc/hadoop/hdfs-site.xml``
-
-If using the system Hadoop libraries, be sure to override ``JAVA_HOME``, ``CLASSPATH``, ``HADOOP_HOME``, and ``ARROW_LIBHDFS_DIR`` as described above.
-
-Verifying Hadoop configuration
-==================================
-
-To test HDFS access, try accessing files using the HDFS shell:
-
-.. code-block:: console
-
-   $ hdfs dfs -ls
-   Found 2 items
-   -rw-r--r--   3 hdfs supergroup      63446 2020-02-29 16:37 MD1.csv
-   -rw-r--r--   3 hdfs supergroup      63906 2020-02-29 16:37 MD2.csv
-   $ hdfs dfs -tail MD1.csv
-   985,Obediah,Reith,oreithrc@time.com,Male,Colombia,859.28
-   986,Lennard,Hairesnape,lhairesnaperd@merriam-webster.com,Male,North Korea,687.60
-   987,Valaree,Pieper,vpieperre@tinyurl.com,Female,Kazakhstan,1116.23
-   988,Rosemaria,Legan,rleganrf@slideshare.net,Female,Indonesia,62.19
-   989,Rafaellle,Hartill,rhartillrg@marketwatch.com,Male,Albania,1308.17
-   990,Symon,Edmett,sedmettrh@tinyurl.com,Male,China,1216.97
-   991,Hiram,Slayton,hslaytonri@amazon.de,Male,China,510.55
-   992,Sylvan,Dalgliesh,sdalglieshrj@booking.com,Male,China,1503.60
-   993,Alys,Sedgebeer,asedgebeerrk@va.gov,Female,Moldova,1947.58
-   994,Ninette,Hearl,nhearlrl@sakura.ne.jp,Female,Palau,917.66
-   995,Tommy,Atterley,tatterleyrm@homestead.com,Female,Philippines,1660.22
-   996,Sean,Mully,smullyrn@rakuten.co.jp,Female,Brunei,938.04
-   997,Gabe,Lytell,glytellro@cnn.com,Male,China,491.12
-   998,Clementius,Battison,cbattisonrp@dedecms.com,Male,Norway,1781.92
-   999,Kyle,Vala,kvalarq@paginegialle.it,Male,France,11.26
-   1000,Korrie,Odd,koddrr@bigcartel.com,Female,China,471.96
-
-If the command succeeded and the file was read correctly, you HDFS has been configured correctly and can now be used in SQream DB.
-
-If an access error occured, check your Hadoop configuration or contact SQream support.
-
-
-Configuring HDFS for Kerberos access
-========================================
-
-This section describes how to configure SQream DB to access HDFS secured with Kerberos.
-
-When a Hadoop cluster is Kerberized, SQream DB's user must be configured to to authenticate through Kerberos.
-
-Prerequisites
-----------------
-
-This section assumes you already have Java and Hadoop installed on your SQream DB hosts.
-
-* SQream DB hosts and Kerberos servers should have the same JCE (Java Cryptography Extension). You can copy the JCE files from the Kerberos server to the SQream DB hosts if needed, to the ``$JAVA_HOME/jre/lib/security`` path.
-
-* Install the Kerberos clients
-   
-   CentOS / RHEL: ``$ sudo yum install krb5-libs krb5-workstation``
-   
-   Ubuntu: ``$ sudo apt-get install krb5-user``
-
-* Configure Hadoop as per your distribution.
-
-Creating keytabs
-----------------------
-
-#. Sign into your Kerberos Key Distribution Center (KDC) as a root user
-
-#. 
-   Create a new principal for the SQream DB OS users (e.g. ``sqream`` by default):
-   
-   .. code-block:: console
-
-      # kadmin.local -q "addprinc -randkey sqream@KRLM.PIEDPIPER.COM"
-   
-   Make sure to replace the realm (``KRLM.PIEDPIPER.COM``) with your actual Kerberos realm.
-
-#. 
-   Create a Kerberos service principal for each SQream DB host in the cluster.
-   
-   In this example, three cluster hosts:
-   
-   .. code-block:: console
-   
-      # kadmin.local -q "addprinc -randkey sqream/sqreamdb-01.piedpiper.com@KRLM.PIEDPIPER.COM"
-      # kadmin.local -q "addprinc -randkey sqream/sqreamdb-02.piedpiper.com@KRLM.PIEDPIPER.COM"
-      # kadmin.local -q "addprinc -randkey sqream/sqreamdb-03.piedpiper.com@KRLM.PIEDPIPER.COM"
-   
-   The format for each principal is ``user/host@realm``, where:
-   
-   * ``user`` is the OS username
-   
-   * ``host`` is the hostname (typically the output of ``hostname -f``)
-   
-   * ``realm`` is the Kerberos realm
-
-#. Generate a keytab for each principal.
-   
-   .. code-block:: console
-      
-      # kadmin.local -q "xst -k /etc/security/keytabs/sqreamdb-01.service.keytab sqream/sqreamdb-01 sqream/sqreamdb-01.piedpiper.com@KRLM.PIEDPIPER.COM"
-      # kadmin.local -q "xst -k /etc/security/keytabs/sqreamdb-02.service.keytab sqream/sqreamdb-02 sqream/sqreamdb-02.piedpiper.com@KRLM.PIEDPIPER.COM"
-      # kadmin.local -q "xst -k /etc/security/keytabs/sqreamdb-03.service.keytab sqream/sqreamdb-03 sqream/sqreamdb-03.piedpiper.com@KRLM.PIEDPIPER.COM"
-
-   You can now exit ``kadmin``.
-   
-#. Change permissions and ownership on each keytab:
-   
-   .. code-block:: console
-      
-      # chown sqream:sqream /etc/security/keytabs/sqreamdb*
-      # chmod 440 /etc/security/keytabs/sqreamdb*
-
-#. Copy the keytab files for each service principal to its respective SQream DB host:
-   
-   .. code-block:: console
-   
-      # scp /etc/security/keytabs/sqreamdb-01.service.keytab sqreamdb-01.piedpiper.com:/home/sqream/sqreamdb-01.service.keytab
-      # scp /etc/security/keytabs/sqreamdb-02.service.keytab sqreamdb-02.piedpiper.com:/home/sqream/sqreamdb-02.service.keytab
-      # scp /etc/security/keytabs/sqreamdb-03.service.keytab sqreamdb-03.piedpiper.com:/home/sqream/sqreamdb-03.service.keytab
-
-Configuring HDFS for Kerberos
----------------------------------
-
-#. 
-   Edit the ``core-site.xml`` configuration file on each SQream DB host to enable authorization.
-
-   For example, editing ``/opt/sqream/hdfs/hadoop/etc/hadoop/core-site.xml``:
-   
-   .. code-block:: xml
-
-      
-          hadoop.security.authorization
-          true
-      
-
-#. Edit the ``yarn-site.xml`` configuration file on each SQream DB host to set the Yarn Kerberos principal
-
-   For example, editing ``/opt/sqream/hdfs/hadoop/etc/hadoop/yarn-site.xml``:
-   
-   .. code-block:: xml
-
-      
-          yarn.resourcemanager.address
-          hadoop-nn.piedpiper.com:8032
-      
-      
-          yarn.resourcemanager.principal
-          yarn/_hostname@KRLM.PIEDPIPER.COM
-      
-
-#. 
-   
-   Edit the ``hdfs-site.xml`` configuration file on each SQream DB host to set the NameNode Kerberos principals, the location of the Kerberos keytab file, and the principal:
-
-   For example, editing ``/opt/sqream/hdfs/hadoop/etc/hadoop/hdfs-site.xml`` on the first host (``sqreamdb-01``):
-   
-   .. code-block:: xml
-
-      
-          dfs.namenode.kerberos.principal
-          sqream/sqreamdb-01.piedpiper.com@KRLM.PIEDPIPER.COM
-      
-      
-          dfs.namenode.https.principal
-          sqream/sqreamdb-01.piedpiper.com@KRLM.PIEDPIPER.COM
-      
-      
-
-.. 
-      
-          security.keytab.file
-          /home/sqream/sqreamdb-01.service.keytab
-      
-      
-          security.username
-          sqream/sqreamdb-01.piedpiper.com@KRLM.PIEDPIPER.CO
-      
-
-Test the access
---------------------
-
-To confirm that Kerberized HDFS is accessible on all SQream DB hosts, run the following command to list a directory:
-
-.. code-block:: console
-
-   $ hdfs dfs -ls hdfs://hadoop-nn.piedpiper.com:8020
-
-Repeat the command on all hosts.
-If the command succeeds and you see a directory listing, Kerberized HDFS has been configured correctly and can now be used in SQream DB.
-
-If an error occured, check your configuration or contact SQream support.
-
-Testing HDFS access in SQream DB
-=====================================
-
-HDFS access from SQream DB is from :ref:`copy_from` and :ref:`external_tables`.
-
-* :ref:`Example for an HDFS-stored external table`
-
-* :ref:`Example for inserting data from a CSV on HDFS`
-
-Troubelshooting HDFS access
-==================================
-
-``class not found`` error
----------------------------------
-
-If you get a ``class not found`` error that looks like this:
-
-   java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found
-
-#. Verify that the CLASSPATH and ARROW_LIBHDFS_DIR are set correctly. Read more about :ref:`setting the environment variables` above.
-
-#. Try restarting SQream DB after setting the environment variables.
-
diff --git a/guides/features/external_data/index.rst b/guides/features/external_data/index.rst
deleted file mode 100644
index c5bbd2597..000000000
--- a/guides/features/external_data/index.rst
+++ /dev/null
@@ -1,16 +0,0 @@
-.. _external_data:
-
-**********************************
-Working with external data
-**********************************
-
-SQream DB supports external data sources for use with :ref:`external_tables`, :ref:`copy_from`, and :ref:`copy_to`.
-
-.. toctree::
-   :maxdepth: 2
-   :caption: In this section:
-   :glob:
-
-   s3
-   hdfs
-   
diff --git a/guides/features/index.rst b/guides/features/index.rst
deleted file mode 100644
index 2d7029100..000000000
--- a/guides/features/index.rst
+++ /dev/null
@@ -1,33 +0,0 @@
-.. _features_guides:
-
-**********************************
-Features guides
-**********************************
-
-.. toctree::
-   :maxdepth: 8
-   :caption: In this section:
-   :glob:
-
-   access_control
-
-   concurrency_and_locks
-   concurrency_and_scaling_in_sqream
-
-   metadata_system
-   chunks_and_extents
-   compression
-   data_clustering
-   delete
-   time_based_data_management
-
-   external_tables
-   external_data/index
-
-   transactions
-   workload_manager
-
-   python_functions
-   saved_queries
-
-   viewing_system_objects_as_ddl
diff --git a/guides/features/metadata_system.rst b/guides/features/metadata_system.rst
deleted file mode 100644
index e0b48dc17..000000000
--- a/guides/features/metadata_system.rst
+++ /dev/null
@@ -1,71 +0,0 @@
-.. _metadata_system:
-
-***********************
-Metadata system
-***********************
-
-SQream DB contains a transparent and automatic system that collects metadata describing each :ref:`chunk`.
-
-The collected metadata enables effective skipping of chunks and extents when queries are executed. 
-
-How is metadata collected?
-==============================
-
-When data is inserted into SQream DB, the load process splits data into chunks.
-
-Several parameters are collected and stored for later use, including:
-
-* Range of values for each column chunk (minimum, maximum)
-* The number of values
-* Additional information for query optimization
-
-Data is collected automatically and transparently on every column type.
-
-.. figure:: /_static/images/chunking.png
-   :scale: 80 %
-   
-   Chunks are collections of rows from a column
-
-.. figure:: /_static/images/chunking_metadata.png
-   :scale: 80 %
-   
-   Metadata is automatically added to each chunk
-
-
-How is metadata used?
-===========================
-
-Chunk metadata is collected for identifying column values and potentially skipping accessing them, to reduce unnecessary I/O operations. For example, when a query specifies a filter (e.g. ``WHERE`` or ``JOIN`` condition) on a range of values that spans a fraction of the table values, SQream DB will optimally scan only that fraction of the table chunks.
-
-Queries that filter on fine-grained date and time ranges will be the most effective, particularly when :ref:`data is timestamped`, and when tables contain a large amount of historical data.
-
-Why is metadata always on?
-=============================
-
-Metadata collection adds very little overhead to data load. WHen possible, most metadata collection is performed in the GPU.
-
-Metadata is collected for every chunk, and adds a handful of kilobytes at most per million values, and very few compute cycles.
-
-At scale, metadata collection is often negligible, resulting in a 0.005% overhead.
-
-For a 10TB dataset, the metadata storage overhead is estimated at 0.5GB.
-
-Because SQream DB's metadata collection is so light-weight and often results in effective data skipping, it is always-on.
-
-
-
-
-.. show the metadata system in action:
-.. describe a scenario
-.. show a statement which is accelerated via the metadata system
-.. do this for a bunch of variants
-
-.. * where
-.. * count
-.. * delete support
-
-.. can easily skip reading chunks and extents when running statements
-.. with the right shape
-
-.. best practice notes
-
diff --git a/guides/features/nba-t10.csv b/guides/features/nba-t10.csv
deleted file mode 100644
index 024530355..000000000
--- a/guides/features/nba-t10.csv
+++ /dev/null
@@ -1,10 +0,0 @@
-Name,Team,Number,Position,Age,Height,Weight,College,Salary
-Avery Bradley,Boston Celtics,0.0,PG,25.0,6-2,180.0,Texas,7730337.0
-Jae Crowder,Boston Celtics,99.0,SF,25.0,6-6,235.0,Marquette,6796117.0
-John Holland,Boston Celtics,30.0,SG,27.0,6-5,205.0,Boston University,
-R.J. Hunter,Boston Celtics,28.0,SG,22.0,6-5,185.0,Georgia State,1148640.0
-Jonas Jerebko,Boston Celtics,8.0,PF,29.0,6-10,231.0,,5000000.0
-Amir Johnson,Boston Celtics,90.0,PF,29.0,6-9,240.0,,12000000.0
-Jordan Mickey,Boston Celtics,55.0,PF,21.0,6-8,235.0,LSU,1170960.0
-Kelly Olynyk,Boston Celtics,41.0,C,25.0,7-0,238.0,Gonzaga,2165160.0
-Terry Rozier,Boston Celtics,12.0,PG,22.0,6-2,190.0,Louisville,1824360.0
diff --git a/guides/features/time_based_data_management.rst b/guides/features/time_based_data_management.rst
deleted file mode 100644
index 49a1f5269..000000000
--- a/guides/features/time_based_data_management.rst
+++ /dev/null
@@ -1,106 +0,0 @@
-.. _time_based_data_management:
-
-***************************
-Time based data management
-***************************
-
-SQream DB's columnar-storage system is well adapted to timestamped data.
-
-When loading data with natural ordering (sorted by a timestamp), SQream DB organizes and collects metadata in chunks of time.
-Natural ordering allows for fast retrieval when performing range queries.
-
-.. contents:: In this topic:
-   :local:
-
-
-Timestamped data
-===========================
-
-Timestamped data usually has some interesting attributes:
-
-* Data is loaded in a natural order, as it is created
-
-* Updates are infrequent or non-existent. Any updates are done by inserting a new row relating to a new timestamp
-
-* Queries on timestamped data is typically on continuous time ranges
-
-* Inserting and reading data are performed in independently, not in the operation or transaction
-
-* Timestamped data has a high data volume and accumulates faster than typical OLTP workloads
-
-Chunking
-=================
-
-Core to handling timestamped data is SQream DB's chunking and metadata system.
-
-When data is inserted, data is automatically partitioned vertically by column, and horizontally by chunk.
-
-A chunk can be thought of as an automatic partition that spans several millions of records of one column.
-Unlike node-partitioning (or sharding), chunking carries several benefits:
-
-* Chunks are small enough to allow multiple workers to read them concurrently
-
-* Chunks are optimized for fast insertion of data
-
-* Chunks carry metadata, which narrows down their contents for the optimizer
-
-* Chunks are ideal for data retension as they can be deleted en-masse
-
-* Chunks are optimized for reading into RAM and the GPU
-
-* Chunks are compressed individually, which improves compression and data locality
-
-
-Use cases
-============
-
-Consider a set of data with a timestamp column.
-
-The timestamp order matches the order of data insertion (i.e. newer data is loaded after older data).
-This is common when you insert data in small bulks - every 15 minutes, every hour, every day, etc.
-
-SQream DB's storage works by appending new data, partitioned into chunks containing millions of values.
-As new data is loaded, it is chunked and appended to a table.
-
-This is particularly useful in many scenarios:
-
-* You run analytical queries spanning specific date ranges (e.g. the sum of transactions during the summer in 2020 vs. the summer in 2019)
-
-* You :ref:`delete data` when it is older than X months old
-
-* Regulations instruct you to keep several years' worth of data, but you're not interested in querying this data all the time
-
-Best practices for time-based data
-=========================================
-
-Use date and datetime types for columns
------------------------------------------
-
-When creating tables with dates or timestamps, using the purpose-built ``DATE`` and ``DATETIME`` types over integer types or ``VARCHAR`` will bring performance and storage footprint improvements, and in many cases huge performance improvements (as well as data integrity benefits). SQream DB stores dates and datetimes very efficiently and can strongly optimize queries using these specific types.
-
-Ordering
------------
-
-Data ordering is an important factor in minimizing storage size and improving query performance.
-
-Prioritize inserting data based on timestamps. This will likely reduces the number of chunks that SQream DB reads during query execution.
-
-See our :ref:`data_clustering` guide to see how clustering keys can be defined for optimizing data order.
-
-
-Limit workload by timestamp
-------------------------------
-
-Grouping by and filtering data based on timestamps will improve performance.
-
-For example,
-
-.. code-block:: postgres
-   
-   SELECT DATEPART(HOUR, timestamp),
-          MIN(transaction_amount),
-          MAX(transaction_amount),
-          avg(transaction_amount)
-   FROM transactions
-   WHERE timestamp BETWEEN (CURRENT_TIMESTAMP AND DATEADD(MONTH,-3,CURRENT_TIMESTAMP))
-   GROUP BY 1;
diff --git a/guides/index.rst b/guides/index.rst
deleted file mode 100644
index 515f4e5c4..000000000
--- a/guides/index.rst
+++ /dev/null
@@ -1,35 +0,0 @@
-.. _guides:
-
-***********************
-Guides
-***********************
-
-This section has concept and features guides, and task focused guides.
-
-.. rubric:: Recommended guides
-
-* :ref:`Optimization and best practices in SQream DB`
-
-* :ref:`Using third party tools`
-
-* :ref:`Client drivers for SQream DB`
-
-Full list of guides
-=========================
-
-.. toctree::
-   :maxdepth: 3
-   :glob:
-   :titlesonly:
-
-   xxqueries
-   inserting_data
-
-   client_drivers/index
-   third_party_tools/index
-   
-   operations/index
-
-   features/index
-
-   architecture/index
diff --git a/guides/inserting_data.rst b/guides/inserting_data.rst
deleted file mode 100644
index 2affcd795..000000000
--- a/guides/inserting_data.rst
+++ /dev/null
@@ -1,474 +0,0 @@
-.. _inserting_data:
-
-***************************
-Inserting data
-***************************
-
-This guide covers inserting data into SQream DB, with subguides on inserting data from a variety of sources and locations.
-
-.. contents:: In this topic:
-   :local:
-
-
-Data loading overview
-================================
-
-SQream DB supports importing data from the following sources:
-
-* Using :ref:`insert` with :ref:`a client driver`
-* Using :ref:`copy_from`:
-
-   - Local filesystem and locally mounted network filesystems
-   - :ref:`s3`
-   - :ref:`hdfs`
-
-* Using :ref:`external_tables`:
-
-   - Local filesystem and locally mounted network filesystems
-   - :ref:`s3`
-   - :ref:`hdfs`
-
-
-SQream DB supports loading files in the following formats:
-
-* Text - CSV, TSV, PSV
-* Parquet
-* ORC
-
-Data loading considerations
-================================
-
-Verify data and performance after load
------------------------------------------
-
-Like other RDBMSs, SQream DB has its own set of best practcies for table design and query optimization.
-
-SQream therefore recommends:
-
-* Verify that the data is as you expect it (e.g. row counts, data types, formatting, content)
-
-* The performance of your queries is adequate
-
-* :ref:`Best practices` were followed for table design
-
-* Applications such as :ref:`Tableau` and others have been tested, and work
-
-* Data types were not over-provisioned (e.g. don't use VARCHAR(2000) to store a short string)
-
-File source location for loading
---------------------------------
-
-During loading using :ref:`copy_from`, the statement can run on any worker. If you are running multiple nodes, make sure that all nodes can see the source the same. If you load from a local file which is only on 1 node and not on shared storage, it will fail some of the time. (If you need to, you can also control which node a statement runs on using the :ref:`workload_manager`).
-
-Supported load methods
--------------------------------
-
-SQream DB's :ref:`COPY FROM` syntax can be used to load CSV files, but can't be used for Parquet and ORC.
-
-:ref:`FOREIGN TABLE` can be used to load text files, Parquet, and ORC files, and can also transform the data prior to materialization as a full table.
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-   :stub-columns: 1
-   
-   * - Method / File type
-     - Text (CSV)
-     - Parquet
-     - ORC
-     - Streaming data
-   * - :ref:`copy_from`
-     - ✓
-     - ✗
-     - ✗
-     - ✗
-   * - :ref:`external_tables`
-     - ✓
-     - ✓
-     - ✓
-     - ✗
-   * - :ref:`insert`
-     - ✗
-     - ✗
-     - ✗
-     - ✓ (Python, JDBC, Node.JS)
-
-Unsupported data types
------------------------------
-
-SQream DB doesn't support the entire set of features that some other database systems may have, such as ``ARRAY``, ``BLOB``, ``ENUM``, ``SET``, etc.
-
-These data types will have to be converted before load. For example, ``ENUM`` can often be stored as a ``VARCHAR``.
-
-Extended error handling
-----------------------------
-
-While :ref:`external tables` can be used to load CSVs, the ``COPY FROM`` statement provides more fine-grained error handling options, as well as extended support for non-standard CSVs with multi-character delimiters, alternate timestamp formats, and more.
-
-Best practices for CSV
-------------------------------
-
-Text files like CSV rarely conform to `RFC 4180 `_ , so alterations may be required:
-
-* Use ``OFFSET 2`` for files containing header rows
-
-* Failed rows can be captured in a log file for later analysis, or just to skip them. See :ref:`capturing_rejected_rows` for information on skipping rejected rows.
-
-* Record delimiters (new lines) can be modified with the :ref:`RECORD DELIMITER` syntax.
-
-* If the date formats differ from ISO 8601, refer to the :ref:`copy_date_parsers` section to see how to override default parsing.
-
-* 
-   Fields in a CSV can be optionally quoted with double-quotes (``"``). However, any field containing a newline or another double-quote character must be quoted.
-
-   If a field is quoted, any double quote that appears must be double-quoted (similar to the :ref:`string literals quoting rules`. For example, to encode ``What are "birds"?``, the field should appear as ``"What are ""birds""?"``.
-
-* Field delimiters don't have a to be a displayable ASCII character. See :ref:`field_delimiters` for all options.
-
-
-Best practices for Parquet
---------------------------------
-
-* Parquet files are loaded through :ref:`external_tables`. The destination table structure has to match in number of columns between the source files.
-
-* Parquet files support predicate pushdown. When a query is issued over Parquet files, SQream DB uses row-group metadata to determine which row-groups in a file need to be read for a particular query and the row indexes can narrow the search to a particular set of rows.
-
-Type support and behavior notes
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* Unlike ORC, the column types should match the data types exactly (see table below).
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-   :stub-columns: 1
-   
-   * -   SQream DB type →
-   
-         Parquet source
-     - ``BOOL``
-     - ``TINYINT``
-     - ``SMALLINT``
-     - ``INT``
-     - ``BIGINT``
-     - ``REAL``
-     - ``DOUBLE``
-     - Text [#f0]_
-     - ``DATE``
-     - ``DATETIME``
-   * - ``BOOLEAN``
-     - ✓ 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``INT16``
-     - 
-     - 
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``INT32``
-     - 
-     - 
-     - 
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``INT64``
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``FLOAT``
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - 
-     - 
-     - 
-     - 
-   * - ``DOUBLE``
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - 
-     - 
-     - 
-   * - ``BYTE_ARRAY`` [#f2]_
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - 
-     - 
-   * - ``INT96`` [#f3]_
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓ [#f4]_
-
-* If a Parquet file has an unsupported type like ``enum``, ``uuid``, ``time``, ``json``, ``bson``, ``lists``, ``maps``, but the data is not referenced in the table (it does not appear in the :ref:`SELECT` query), the statement will succeed. If the column is referenced, an error will be thrown to the user, explaining that the type is not supported, but the column may be ommited.
-
-Best practices for ORC
---------------------------------
-
-* ORC files are loaded through :ref:`external_tables`. The destination table structure has to match in number of columns between the source files.
-
-* ORC files support predicate pushdown. When a query is issued over ORC files, SQream DB uses ORC metadata to determine which stripes in a file need to be read for a particular query and the row indexes can narrow the search to a particular set of 10,000 rows.
-
-Type support and behavior notes
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* ORC files are loaded through :ref:`external_tables`. The destination table structure has to match in number of columns between the source files.
-
-* The types should match to some extent within the same "class" (see table below).
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-   :stub-columns: 1
-   
-   * -   SQream DB type →
-   
-         ORC source
-     - ``BOOL``
-     - ``TINYINT``
-     - ``SMALLINT``
-     - ``INT``
-     - ``BIGINT``
-     - ``REAL``
-     - ``DOUBLE``
-     - Text [#f0]_
-     - ``DATE``
-     - ``DATETIME``
-   * - ``boolean``
-     - ✓ 
-     - ✓ [#f5]_
-     - ✓ [#f5]_
-     - ✓ [#f5]_
-     - ✓ [#f5]_
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``tinyint``
-     - ○ [#f6]_
-     - ✓
-     - ✓
-     - ✓
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``smallint``
-     - ○ [#f6]_
-     - ○ [#f7]_
-     - ✓
-     - ✓
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``int``
-     - ○ [#f6]_
-     - ○ [#f7]_
-     - ○ [#f7]_
-     - ✓
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``bigint``
-     - ○ [#f6]_
-     - ○ [#f7]_
-     - ○ [#f7]_
-     - ○ [#f7]_
-     - ✓
-     - 
-     - 
-     - 
-     - 
-     - 
-   * - ``float``
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - ✓
-     - 
-     - 
-     - 
-   * - ``double``
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - ✓
-     - 
-     - 
-     - 
-   * - ``string`` / ``char`` / ``varchar``
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - 
-     - 
-   * - ``date``
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-     - ✓
-   * - ``timestamp``, ``timestamp`` with timezone
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - 
-     - ✓
-
-* If an ORC file has an unsupported type like ``binary``, ``list``, ``map``, and ``union``, but the data is not referenced in the table (it does not appear in the :ref:`SELECT` query), the statement will succeed. If the column is referenced, an error will be thrown to the user, explaining that the type is not supported, but the column may be ommited.
-
-
-
-..
-   insert
-
-   example
-
-   are there some variations to highlight?:
-
-   create table as
-
-   sequences, default values
-
-   insert select
-
-   make distinction between an insert command, and a parameterized/bulk
-   insert "over the network"
-
-
-   copy
-
-
-   best practices for insert
-
-   chunks and extents, and storage reorganisation
-
-   copy:
-
-   give an example
-
-   supports csv and parquet
-
-   what else do we have right now? any other formats? have the s3 and
-   hdfs url support also
-
-   error handling
-
-   best practices
-
-   try to combine sensibly with the external table stuff
-
-Further reading and migration guides
-=======================================
-
-.. toctree::
-   :caption: Data loading guides
-   :titlesonly:
-   
-   migration/csv
-   migration/parquet
-   migration/orc
-
-.. toctree::
-   :caption: Migration guides
-   :titlesonly:
-   
-   migration/oracle
-
-
-.. rubric:: See also:
-
-* :ref:`copy_from`
-* :ref:`insert`
-* :ref:`external_tables`
-
-.. rubric:: Footnotes
-
-.. [#f0] Text values include ``TEXT``, ``VARCHAR``, and ``NVARCHAR``
-
-.. [#f2] With UTF8 annotation
-
-.. [#f3] With ``TIMESTAMP_NANOS`` or ``TIMESTAMP_MILLIS`` annotation
-
-.. [#f4] Any microseconds will be rounded down to milliseconds.
-
-.. [#f5] Boolean values are cast to 0, 1
-
-.. [#f6] Will succeed if all values are 0, 1
-
-.. [#f7] Will succeed if all values fit the destination type
diff --git a/guides/migration/nba-t10.csv b/guides/migration/nba-t10.csv
deleted file mode 100644
index 024530355..000000000
--- a/guides/migration/nba-t10.csv
+++ /dev/null
@@ -1,10 +0,0 @@
-Name,Team,Number,Position,Age,Height,Weight,College,Salary
-Avery Bradley,Boston Celtics,0.0,PG,25.0,6-2,180.0,Texas,7730337.0
-Jae Crowder,Boston Celtics,99.0,SF,25.0,6-6,235.0,Marquette,6796117.0
-John Holland,Boston Celtics,30.0,SG,27.0,6-5,205.0,Boston University,
-R.J. Hunter,Boston Celtics,28.0,SG,22.0,6-5,185.0,Georgia State,1148640.0
-Jonas Jerebko,Boston Celtics,8.0,PF,29.0,6-10,231.0,,5000000.0
-Amir Johnson,Boston Celtics,90.0,PF,29.0,6-9,240.0,,12000000.0
-Jordan Mickey,Boston Celtics,55.0,PF,21.0,6-8,235.0,LSU,1170960.0
-Kelly Olynyk,Boston Celtics,41.0,C,25.0,7-0,238.0,Gonzaga,2165160.0
-Terry Rozier,Boston Celtics,12.0,PG,22.0,6-2,190.0,Louisville,1824360.0
diff --git a/guides/operations/hardware_guide.rst b/guides/operations/hardware_guide.rst
deleted file mode 100644
index 24ada0c4f..000000000
--- a/guides/operations/hardware_guide.rst
+++ /dev/null
@@ -1,215 +0,0 @@
-.. _hardware_guide:
-
-***********************
-Hardware Guide
-***********************
-
-This guide describes the SQream DB reference architecture, emphasizing the benefits to the technical audience, and provides guidance for end-users on selecting the right configuration for a SQream DB installation.
-
-
-.. rubric:: Need help?
-
-This page is intended as a "reference" to suggested hardware. However, different workloads require different solution sizes. SQream's experienced customer support has the experience to advise on these matters to ensure the best experience.
-
-Visit `SQream's support portal `_ for additional support.
-
-A SQream DB cluster
-============================
-
-SQream recommends rackmount servers by server manufacturers Dell, Lenovo, HP, Cisco, Supermicro, IBM, and others.
-
-A typical SQream DB cluster includes one or more nodes, consisting of
-
-* Two-socket enterprise processors, like the Intel® Xeon® Gold processor family or an IBM® POWER9 processors, providing the high performance required for compute-bound database workloads. 
-
-* NVIDIA Tesla GPU accelerators, with up to 5,120 CUDA and Tensor cores, running on PCIe or fast NVLINK busses, delivering high core count, and high-throughput performance on massive datasets
-
-* High density chassis design, offering between 2 and 4 GPUs in a 1U, 2U, or 3U package, for best-in-class performance per cm\ :sup:`2`.
-
-Example for a single-node cluster
------------------------------------
-
-A single-node SQream DB cluster can handle between 1 and 8 concurrent users, with up to 1PB of data storage (when connected via NAS).
-
-An average single-node cluster can be a rackmount server or workstation, containing the following components:
-
-.. list-table::
-   :widths: auto
-   :header-rows: 1
-   
-   * - Component
-     - Type
-   * - Server
-     - Rackmount or workstation, like the Dell T640, Dell R740, Dell R940xa, HP ProLiant DL380 Gen10
-   * - Processor
-     - 2x Intel Xeon Gold 6240 (18C/36HT) 2.6GHz
-   * - RAM
-     - 384 GB LRDIMM 2666MT/s, ECC registered
-   * - Onboard storage
-     - 
-         * 2x 960GB SSD 2.5in Hot-plug for OS, RAID1
-         * 14x 3.84TB SSD 2.5in Hot-plug for storage, RAID10
-   * - Networking
-     - Intel X710 DP 10Gb DA/SFP+
-   * - GPU
-     - 2x or 4x NVIDIA Tesla T4 or P100
-
-In this system configuration, SQream DB can store about 200TB of raw data (assuming average compression ratio and ~50TB of usable raw storage).
-
-If a NAS is used, the 14x SSD drives can be omitted, but SQream recommends 2TB of local spool space on SSD or NVMe drives.
-
-Example for multi-node clusters
------------------------------------
-
-Multi-node clusters can handle any number of concurrent users.
-
-A typical SQream DB cluster relies on a shared storage connected over a network fabric like InfiniBand EDR, 40GbE, or 100GbE
-
-An example of a cluster node providing the best performance:
-
-.. list-table::
-   :widths: auto
-   :header-rows: 1
-   
-   * - Component
-     - Type
-   * - Server
-     - High-density GPU-capable rackmount server, like Dell C4140, IBM AC922, Lenovo SR650.
-   * - Processor
-     - 2x Intel Platinum 8180M (28C/56HT) 3.8GHz or 2x IBM POWER9
-   * - RAM
-     - 1024 GB RDIMM 2933T/s, ECC registered
-   * - Onboard storage
-     -   
-         * 2x 960GB SSD 2.5in, for OS, RAID1
-         * 2x 2TB SSD or NVMe, for temporary spooling, RAID1
-   * - Networking
-     - 
-         Intel X710 DP 10Gb DA/SFP+ for BI fabric
-      
-         Mellanox ConnectX-4, EDR for storage fabric
-   * - GPU
-     - 4x NVIDIA Tesla V100 32GB
-
-.. note:: With a NAS connected over GPFS, Lustre, or NFS - each SQream DB worker can read data at up to 5GB/s.
-
-
-Considerations in cluster design
-====================================
-
-* In a SQream DB installation, the storage and compute are logically separated. While they may reside on the same machine in a standalone installation, they may also reside on different hosts, providing additional flexibility and scalability.
-
-* SQream DB uses all resources in a machine, including CPU, RAM, and GPU to deliver the best performance. 256GB of RAM per physical GPU is recommended, but not required.
-
-* Local disk space is required for good performance temporary spooling - particularly when performing intensive larger-than-RAM operations like sorting. SQream recommends an SSD or NVMe drive, in mirrored RAID 1 configuration, with about 2x the RAM size available for temporary storage. This can be shared with the operating system drive if necessary.
-
-* When using SAN or NAS devices, SQream recommends around 5GB/s of burst throughput from storage, per GPU.
-
-Balancing cost and performance
---------------------------------
-
-Prior to designing and deploying a SQream DB cluster, there are a number of important factors to consider. 
-
-This section provides a breakdown of deployment details intended to help ensure that this installation exceeds or meets the stated requirements. The rationale provided includes the necessary information for modifying configurations to suit the customer use-case scenario.
-
-.. list-table::
-   :widths: auto
-   :header-rows: 1
-   
-   * - Component
-     - Value
-   * - Compute - CPU
-     - Balance price and performance
-   * - Compute – GPU
-     - Balance price with performance and concurrency
-   * - Memory – GPU RAM
-     - Balance price with concurrency and performance.
-   * - Memory - RAM
-     - Balance price and performance
-   * - Operating System
-     - Availability, reliability, and familiarity
-   * - Storage
-     - Balance price with capacity and performance
-   * - Network
-     - Balance price and performance
-
-CPU compute
--------------
-
-SQream DB relies on multi-core Intel® Xeon® processors or IBM® POWER9 processors.
-
-SQream recommends a dual-socket machine populated with CPUs with 18C/36HT or better.
-
-While a higher core count may not necessarily affect query performance, more cores will enable higher concurrency and better load performance.
-
-GPU compute and RAM
--------------------------
-
-The NVIDIA Tesla range of high-throughput GPU accelerators provides the best performance for enterprise environments. Most cards have ECC memory, which is crucial for delivering correct results every time.
-
-SQream recommends the NVIDIA Tesla V100 32GB GPU for best performance and highest concurrent user support.
-
-GPU RAM, sometimes called GRAM or VRAM is used for processing queries. It is possible to select GPUs with less RAM, like the NVIDIA Tesla V100 16GB or P100 16GB. However, the smaller GPU RAM available will result in reduced concurrency, as the GPU RAM is used extensively in operations like JOINs, ORDER BY, GROUP BY, and all SQL transforms.
-
-RAM
---------
-
-Use of error-correcting code memory (ECC) is a practical requirement for SQream DB and is standard on most enterprise server. SQream DB benefits from having large amounts of memory for improved performance on large 'external' operations like sorting and joining.
-
-Although SQream DB can function with less, we recommend a key of 256GB of RAM per GPU in the machine. 
-
-Operating system
----------------------
-
-SQream DB can run on 64-bit Linux operating systems:
-
-   * Red Hat Enterprise Linux (RHEL) v7
-   * CentOS v7
-   * Amazon Linux 2018.03
-   * Ubuntu v16.04 LTS, v18.04 LTS
-   * Other Linux distributions may be supported via nvidia-docker
-
-Storage
------------
-
-For clustered scale-out installations, SQream DB relies on NAS/SAN storage. These devices have extremely high reliability and durability, with five 9s of up-time.
-
-For stand-alone installations, SQream DB relies on redundant disk configurations, like RAID 10/50. These RAID configurations ensure that blocks of data are replicated between disks, so that failure of a number of disks will not result in data loss or availability of the system. 
-
-Because storage reliability is important, SQream recommends enterprise-grade SAS SSD drives. However, as with other components – there is a tradeoff for cost/performance. When performance and reliability are important, SQream recommends SAS SSD or NVMe drives. 
-
-SQream DB functions well with more cost-effective SATA drives and even large spinning-disk arrays.
-
-
-Example cluster supporting 32 concurrent active users
-==========================================================
-
-For a 32-user configuration, the number of GPUs should roughly match the number of users. SQream DB recommends 1 Tesla V100 GPU per 2 users, for full, uninterrupted dedicated access.
-
-Each of these servers can support about 8 users on average. The actual number of concurrent users can be higher, depending on the workload.
-
-A SQream DB cluster for 32 users consists of the following components:
-
-#. 
-   4 high-density GPU-enabled servers, like the Dell C4140 (Configuration C) with 4x NVIDIA Tesla V100 32GB PCIe GPUs.
-   
-   Each server is equipped with dual Intel ® Xeon ® Gold 6240 CPU, and 1,024GB of RAM.
-   
-#. 
-   NAS/SAN storage, capable of delivering 1 GB/s per GPU.
-   
-   For the system above, with 4x4 NVIDIA Tesla V100 GPUs, this results in 16GB/s, over multiple bonded, 40GigE or InfiniBand links via a fabric switch.
-
-#. Top-of-Rack (ToR) 10GigE ethernet switch for the BI fabric
-
-#. 40GigE or InfiniBand switches for the storage fabric
-
-#.	At least 1 PDU
-
-.. figure:: /_static/images/reference_architecture_30u.png
-   :alt: A 4-node system, supporting up to 30 users with 1PB of storage
-
-
-.. rubric:: Read more
-
-Download the full `SQream DB Reference Architecture `_ document.
diff --git a/guides/operations/index.rst b/guides/operations/index.rst
deleted file mode 100644
index 55afe6f92..000000000
--- a/guides/operations/index.rst
+++ /dev/null
@@ -1,28 +0,0 @@
-.. _operations:
-
-**********************************
-Operations
-**********************************
-
-The guides in this section include information about best practices, monitoring, logging, troubleshooting, and maintaining a SQream DB cluster.
-
-.. toctree::
-   :maxdepth: 8
-   :caption: In this section:
-   :glob:
-
-   optimization_best_practices
-   xxmonitoring
-   monitoring_query_performance
-   logging
-   configuration
-   troubleshooting
-   information_for_support
-   creating_or_cloning_a_storage_cluster
-   xxstarting_and_stopping_the_system
-   sqream_studio
-   statement_editor
-   hardware_guide
-   security
-   setup/index
-
diff --git a/guides/operations/monitoring.rst b/guides/operations/monitoring.rst
deleted file mode 100644
index ab734a61e..000000000
--- a/guides/operations/monitoring.rst
+++ /dev/null
@@ -1,87 +0,0 @@
-.. _monitoring:
-
-***********************
-Monitoring
-***********************
-
-monitoring sqream:
-
-monitoring alerts
-watching processes
-
-GPU temperature  
-nvidia-smi  - Temp 
-Temp > 85 
-GPU usage 
-nvidia-smi -  GPU-Util 
-For Information only 
-Nice to put on graph to track system usage 
-Processes - sqreamd 
-ps aux | grep sqreamd 
-Alert if one or more sqreamd processes are down 
-Process - metadata 
-ps aux | grep metadata 
-Alert if down 
-Process – server picker 
-ps aux | grep server_picker 
-Alert if down 
-Disk Space – Storage 
-df -h 
-Alert if storage usage > 75% 
-Disk Space – Cluster 
-du -sh /SQream/sqream_cluster/sqreamdb 
-For Information only 
-Nice to put on graph to track full DB growth 
-Disk Space – leveldb 
-du -sh 
-/SQream/sqream_cluster/leveldb
-For Information only 
-Nice to put on graph to track system usage 
-Disk Space – DB usage 
-du -sh 
-/SQream/sqream_cluster/sqreamdb/datab ases/ 
-For Information only 
-Nice to put on graph to track specific platform db growth 
-Server Disk Space 
-df -h 
-Alert if disk usage > 75% 
-Memory Consumption 
-free –g 
-Alert if Mem usage > 95% 
-
-
-monitoring a running statement
-select show_node_info(statement_id);
-try to explain how to actually use it (if this is even possible)
-
-monitoring the activity of the system
-
-
-show_cluster_nodes
-
-show_conf
-
-backup_storage?
-
-show_server_status()
-
-stop_statement()
-
-
-show_locks
-
-how do you release locks
-
-sqream_version
-
-what else?
-
-viewing the usage history - link to the logs
-
-
-
-.. toctree::
-   :maxdepth: 2
-   :caption: In this section:
-   :glob:
-
diff --git a/guides/operations/setup/before_you_begin.rst b/guides/operations/setup/before_you_begin.rst
deleted file mode 100644
index 9c307ce8b..000000000
--- a/guides/operations/setup/before_you_begin.rst
+++ /dev/null
@@ -1,49 +0,0 @@
-.. _before_you_begin:
-
-***********************
-Before you begin
-***********************
-
-Before you install or deploy SQream DB, please verify you have the following:
-
-* An NVIDIA-capable server, either on-premise or on supported cloud platforms. 
-
-   * Supported operating systems: Red Hat Enterprise Linux v7.x / CentOS v7.x / Ubuntu 18.04 / Amazon Linux
-   * NVIDIA GPU. A Tesla GPU is highly recommended. See more information in our :ref:`hardware_guide`.
-   * A privileged SSH connection to the server (sudo)
-
-* A SQream DB license (contact support@sqream.com or your SQream account manager for your license key)
-
-* A 3rd party tool that can connect to SQream DB via JDBC, ODBC, or Python
-
-If you do not have a SQream DB license, visit our website and `sign up for SQream DB`_ today!
-
-Refer to our :ref:`hardware_guide` for more information about supported and recommended configurations.
-
-What does the installation process look like?
-----------------------------------------------
-
-To install SQream DB, we will go through the following steps:
-
-#. Prepare the host OS for NVIDIA driver installation
-
-#. Install the NVIDIA driver
-
-#. Install Docker CE
-
-#. Install nvidia-docker2
-
-#. Prepare disk space for SQream DB
-
-#. Install SQream DB
-
-#. Additional system configuration for performance and stability
-
-
-.. rubric:: What's next?
-
-* When ready, start :ref:`installing SQream DB `
-
-.. _`sign up for SQream DB`: https://sqream.com/try-sqream-db
-
-.. TODO Links for if the user has already installed SQream DB but wants to configure, upgrade, or scale out the system further
\ No newline at end of file
diff --git a/guides/operations/setup/index.rst b/guides/operations/setup/index.rst
deleted file mode 100644
index b422d9c5b..000000000
--- a/guides/operations/setup/index.rst
+++ /dev/null
@@ -1,19 +0,0 @@
-.. _setup:
-
-***********************
-Setting up SQream DB
-***********************
-
-The guides below cover installing SQream DB.
-
-.. toctree::
-   :maxdepth: 4
-   :caption: In this section:
-   :titlesonly:
-   :glob:
-   
-   before_you_begin
-   local_docker
-   recommended_configuration
-
-
diff --git a/guides/operations/setup/local_docker.rst b/guides/operations/setup/local_docker.rst
deleted file mode 100644
index a6f0d9adc..000000000
--- a/guides/operations/setup/local_docker.rst
+++ /dev/null
@@ -1,529 +0,0 @@
-.. _installing_sqream_db_docker:
-
-*********************************************
-Start a local SQream DB cluster with Docker
-*********************************************
-
-See :ref:`Release Notes ` to learn about what's new in the latest release of SQream DB. To upgrade to this release, see :ref:`Upgrading SQream DB with Docker`.
-
-SQream DB is installed on your hosts with NVIDIA Docker. There are several preparation steps to ensure before installing SQream DB, so follow these instructions carefully.
-
-.. note:: Installing SQream DB requires a license key. Go to `SQream Support `_ or contact your SQream account manager for your license key.
-
-.. contents:: In this topic:
-   :local:
-
-Preparing your machine for NVIDIA Docker
-=========================================
-To install NVIDIA Docker, we must first install the NVIDIA driver.
-
-.. note:: SQream DB works best on NVIDIA Tesla series GPUs, which provide better reliability, performance, and stability. The instructions below are written for NVIDIA Tesla GPUs, but other NVIDIA GPUs may work.
-
-.. contents:: Follow the instructions for your OS and architecture:
-   :local:
-
-CentOS 7 / RHEL 7 / Amazon Linux
----------------------------------------------------------------
-
-.. admonition:: Recommended
-
-   Follow the installation instructions on `NVIDIA's CUDA Installation Guide`_ for full instructions suitable for your platform. The information listed below is a summary of the necessary steps, and does not cover the full range of options available.
-
-#. Enable EPEL
-
-   EPEL provides additional open-source and free packages from the RHEL ecosystem. The NVIDIA driver depends on packages such as DKMS and libvdpau which are only available on third-party repositories, such as EPEL.
-
-   .. code-block:: console
-      
-      $ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
-
-.. There are other ways of installing EPEL: $ sudo yum -y install epel-release
-.. Do we need DKMS? $ sudo yum -y install dkms
-
-#. Install the kernel headers and development tools necessary to compile the NVIDIA driver
-
-   .. code-block:: console
-      
-      $ sudo yum -y install kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc
-
-#. Install the CUDA repository and install the latest display driver
-
-   .. code-block:: console
-      
-      $ sudo rpm -Uvh https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.1.243-1.x86_64.rpm
-      $ sudo yum install -y nvidia-driver-latest
-
-.. note:: If Linux is running with X, switch to text-only mode before installing the display driver.
-   
-   .. code-block:: console
-   
-      $ sudo systemctl set-default multi-user.target
-
-   This change permanently disables X. If you need X, change ``set-default`` to ``isolate``. This will re-enable X on the next reboot.
-   
-#. Restart your machine
-
-   .. code-block:: console
-      
-      ``sudo reboot``
-
-
-#. Verify the installation completed correctly, by asking ``nvidia-smi``, NVIDIA's system management interface application, to list the available GPUs.
-   
-   .. code-block:: console
-      
-      $ nvidia-smi -L
-      GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
-      GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
-
-#. Enable NVIDIA's persistence daemon. This is mandatory for IBM POWER, but is recommended for other platforms as well.
-      
-      .. code-block:: console
-         
-         $ sudo systemctl enable nvidia-persistenced && sudo systemctl start nvidia-persistenced
-
-     .. Important:: On POWER9 systems only, disable the udev rule for hot-pluggable memory probing.
-
-         For Red Hat 7 this rule can be found in ``/lib/udev/rules.d/40-redhat.rules``
-         
-         For Ubuntu, this rule can be found in in ``/lib/udev/rules.d/40-vm-hotadd.rules``
-         The rule generally takes a form where it detects the addition of a memory block and changes the 'state' attribute to online. For example, in RHEL7, the rule looks like this:
-         
-         ``SUBSYSTEM=="memory", ACTION=="add", PROGRAM="/bin/uname -p", RESULT!="s390*", ATTR{state}=="offline", ATTR{state}="online"``
-
-         This rule must be disabled by copying the file to ``/etc/udev/rules.d`` and commenting out, removing, or changing the hot-pluggable memory rule in the ``/etc`` copy so that it does not apply to NVIDIA devices on POWER9. 
-         
-         * On RHEL 7.5 or earlier versions:
-         
-            .. code-block:: console
-               
-               $ sudo cp /lib/udev/rules.d/40-redhat.rules /etc/udev/rules.d
-               $ sudo sed -i '/SUBSYSTEM=="memory", ACTION=="add"/d' /etc/udev/rules.d/40-redhat.rules
-
-         * On RHEL 7.6 and later versions:
-            
-            .. code-block:: console
-               
-               $ sudo cp /lib/udev/rules.d/40-redhat.rules /etc/udev/rules.d 
-               $ sudo sed -i 's/SUBSYSTEM!="memory", ACTION!="add", GOTO="memory_hotplug_end"/SUBSYSTEM=="*", GOTO="memory_hotplug_end"/' /etc/udev/rules.d/40-redhat.rules
-      
-      *Reboot the system to initialize the above changes*
-
-#. Continue to :ref:`installing NVIDIA Docker for RHEL `
-
-Ubuntu 18.04
----------------------------------------------------------------
-
-.. admonition:: Recommended
-
-   Follow the installation instructions on `NVIDIA's CUDA Installation Guide`_ for full instructions suitable for your platform. The information listed below is a summary of the necessary steps, and does not cover the full range of options available.
-
-#. Install the kernel headers and development tools necessary
-
-   .. code-block:: console
-      
-      $ sudo apt-get update
-      $ sudo apt-get install linux-headers-$(uname -r) gcc
-
-#. Install the CUDA repository and driver on Ubuntu
-
-   .. code-block:: console
-      
-      $ curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
-      $ sudo dpkg -i cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
-      $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
-      $ sudo apt-get update && sudo apt-get install -y nvidia-driver-418
-
-#. Restart your machine
-
-   ``sudo reboot``
-
-#. Verify the installation completed correctly, by asking ``nvidia-smi``, NVIDIA's system management interface application, to list the available GPUs.
-   
-   .. code-block:: console
-      
-      $ nvidia-smi -L
-      GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
-      GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
-
-#. Enable NVIDIA's persistence daemon. This is mandatory for IBM POWER, but is recommended for other platforms as well.
-      
-      .. code-block:: console
-         
-         $ sudo systemctl enable nvidia-persistenced
-
-#. Continue to :ref:`installing NVIDIA Docker for Ubuntu `
-
-Install Docker CE and NVIDIA docker
-====================================
-
-.. contents:: Follow the instructions for your OS and architecture:
-   :local:
-
-.. _docker_rhel:
-
-CentOS 7 / RHEL 7 / Amazon Linux (x64)
---------------------------------------
-
-.. note:: For IBM POWER9, see the next section :ref:`installing NVIDIA Docker for IBM POWER `
-
-#. Follow the instructions for Docker CE for your platform at `Get Docker Engine - Community for CentOS`_
-
-#. Tell Docker to start after a reboot
-
-   .. code-block:: console
-   
-      $ sudo systemctl enable docker && sudo systemctl start docker
-
-#. Verify that docker is running
-
-   .. code-block:: console
-      :emphasize-lines: 4
-      
-      $ sudo systemctl status docker
-      ● docker.service - Docker Application Container Engine
-      Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
-      Active: active (running) since Mon 2019-08-12 08:22:30 IDT; 1 months 27 days ago
-        Docs: https://docs.docker.com
-        Main PID: 65794 (dockerd)
-        Tasks: 76
-      Memory: 124.5M
-      CGroup: /system.slice/docker.service
-              └─65794 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-
-#. Let your current user manage Docker, without requiring `sudo`
-
-   .. code-block:: console
-   
-      $ sudo usermod -aG docker $USER
-
-   Then, log out and log back in:
-   
-   .. code-block:: console
-   
-      $ exit
-
-#. Install nvidia-docker
-
-   .. code-block:: console
-   
-      $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
-      $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
-      $ sudo yum install -y nvidia-container-toolkit
-      $ sudo yum install nvidia-docker2
-      $ sudo pkill -SIGHUP dockerd
-      $ sudo systemctl restart docker
-
-   .. note::
-      
-      Occasionally, there may be a signature verification error while obtaining ``nvidia-docker``.
-      The error looks something like this:
-      ``[Errno -1] repomd.xml signature could not be verified for nvidia-docker``
-      
-      Run the following commands to update the repository keys:
-      
-      .. code-block:: console
-         
-         $ DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
-         $ DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
-         $ sudo rpm -e gpg-pubkey-f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/*/gpgdir --delete-key f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-docker/gpgdir --delete-key f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-container-runtime/gpgdir --delete-key f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/libnvidia-container/gpgdir --delete-key f796ecb0
-
-#. Verify the NVIDIA docker installation
-
-   .. code-block:: console
-   
-      $ sudo docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi -L
-      GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
-      GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
-      
-#. Continue to :ref:`Installing the SQream DB Docker container `
-
-.. _docker_power:
-
-CentOS 7.6 / RHEL 7.6 (IBM POWER)
-------------------------------------
-
-On POWER9, SQream DB is supported only on RHEL 7.6.
-
-#. Install Docker for IBM POWER
-
-   .. code-block:: console
-      
-      $ wget http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/container-selinux-2.9-4.el7.noarch.rpm
-      $ wget http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/docker-ce-18.03.1.ce-1.el7.centos.ppc64le.rpm
-      $ yum install -y container-selinux-2.9-4.el7.noarch.rpm docker-ce-18.03.1.ce-1.el7.centos.ppc64le.rpm
-
-#. Tell Docker to start after a reboot
-
-   .. code-block:: console
-   
-      $ sudo systemctl enable docker && sudo systemctl start docker
-
-#. Verify that docker is running
-
-   .. code-block:: console
-      :linenos:
-      :emphasize-lines: 4
-      
-      $ sudo systemctl status docker
-      ● docker.service - Docker Application Container Engine
-      Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
-      Active: active (running) since Mon 2019-08-12 08:22:30 IDT; 1 months 27 days ago
-        Docs: https://docs.docker.com
-        Main PID: 65794 (dockerd)
-        Tasks: 76
-      Memory: 124.5M
-      CGroup: /system.slice/docker.service
-              └─65794 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-
-#. Let your current user manage Docker, without requiring `sudo`
-
-   .. code-block:: console
-   
-      $ sudo usermod -aG docker $USER
-
-   .. Note:: Log out and log back in again after this action
-
-#. Install nvidia-docker
-
-   * Install the NVIDIA container and container runtime packages from NVIDIA's repository:
-      
-      .. code-block:: console
-      
-         $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
-         $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
-         
-         $ sudo yum install -y libnvidia-container* nvidia-container-runtime*
-
-   * Add the NVIDIA runtime to the Docker daemon and restart docker:
-      
-      .. code-block:: console
-      
-         $ sudo mkdir -p /etc/systemd/system/docker.service.d/
-         $ echo -e "[Service]\nExecStart\nExecStart=/usr/bin/dockerd --add-runtime=nvidia=/usr/bin/nvidia-container-runtime" | sudo tee /etc/systemd/system/docker.service.d/override.conf
-
-         $ sudo systemctl daemon-reload && sudo systemctl restart docker
-
-   .. note::
-      
-      Occasionally, there may be a signature verification error while obtaining ``nvidia-docker``.
-      The error looks something like this:
-      ``[Errno -1] repomd.xml signature could not be verified for nvidia-docker``
-      
-      Run the following commands to update the repository keys:
-      
-      .. code-block:: console
-         
-         $ DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
-         $ DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
-         $ sudo rpm -e gpg-pubkey-f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/*/gpgdir --delete-key f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-docker/gpgdir --delete-key f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-container-runtime/gpgdir --delete-key f796ecb0
-         $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/libnvidia-container/gpgdir --delete-key f796ecb0
-
-#. Verify the NVIDIA docker installation succeeded
-
-   .. code-block:: console
-   
-      $ docker run --runtime=nvidia --rm nvidia/cuda-ppc64le:10.1-base nvidia-smi -L
-      GPU 0: Tesla V100-SXM2-16GB (UUID: GPU-...)
-      GPU 1: Tesla V100-SXM2-16GB (UUID: GPU-...)
-
-#. Continue to :ref:`Installing the SQream DB Docker container`
-
-.. _docker_ubuntu:
-
-Ubuntu 18.04 (x64)
------------------------------------
-
-#. Follow the instructions for Docker CE for your platform at `Get Docker Engine - Community for CentOS`_
-
-#. Tell Docker to start after a reboot
-
-   .. code-block:: console
-   
-      $ sudo systemctl enable docker && sudo systemctl start docker
-
-#. Verify that docker is running
-
-   .. code-block:: console
-      :linenos:
-      :emphasize-lines: 4
-      
-      $ sudo systemctl status docker
-      ● docker.service - Docker Application Container Engine
-      Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
-      Active: active (running) since Mon 2019-08-12 08:22:30 IDT; 1 months 27 days ago
-        Docs: https://docs.docker.com
-        Main PID: 65794 (dockerd)
-        Tasks: 76
-      Memory: 124.5M
-      CGroup: /system.slice/docker.service
-              └─65794 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-
-#. Let your current user manage Docker, without requiring `sudo`
-
-   .. code-block:: console
-   
-      $ sudo usermod -aG docker $USER
-
-   .. Note:: Log out and log back in again after this action
-
-#. Install nvidia-docker
-
-   .. code-block:: console
-   
-      $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
-      $ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
-      $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
-
-      $ sudo apt-get update
-      
-      $ sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit nvidia-docker2
-      $ sudo pkill -SIGHUP dockerd
-      $ sudo systemctl restart docker
-
-
-#. Verify the NVIDIA docker installation
-
-   .. code-block:: console
-   
-      $ sudo docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi -L
-      GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
-      GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
-
-#. Continue to :ref:`Installing the SQream DB Docker container `
-
-
-.. _preparing_mounts:
-
-Preparing directories and mounts for SQream DB
-===============================================
-
-SQream DB contains several directories that need to be defined
-
-.. list-table:: Directories and paths
-   :widths: 40 60
-   :header-rows: 1
-   
-   * - Path name
-     - Definition
-   * - ``storage``
-     - The location where SQream DB stores data, metadata, and logs
-   * - ``exposed path``
-     - A location that SQream DB can read and write to. Used for allowing access to shared raw files like CSVs on local or NFS drives
-   * - ``logs``
-     - Optional location for debug logs
-
-.. note:: By default, SQream DB can't access any OS path. You must explicitly allow it.
-
-
-.. _installing_sqream_db_container:
-
-Install the SQream DB Docker container
-=========================================
-
-#. Download the SQream DB tarball and license package
-
-   In the e-mail from your account manager at SQream, you have received a download link for the SQream DB installer and a license package.
-   Download the SQream DB tarball to the user home directory. For example:
-
-   .. code-block:: console
-   
-      $ cd ~
-      $ curl -O {download URL}
-
-#. Extract the tarball into your home directory
-
-   .. code-block:: console
-   
-      $ tar xf sqream_installer-2.0.7-DB2019.2.1.4-CO1.7.5-ED3.0.1-x86_64.tar.gz
-
-#. Copy the license package
-
-   Copy the license package from your home directory to the license subdirectory which is located in the newly created SQream installer directory.
-   
-   For example, if the licence package is titled ``license_package.tar.gz``:
-   
-   .. code-block:: console
-      
-      $ cp ~/license_package.tar.gz sqream_installer-2.0.7-DB2019.2.1.4-CO1.7.5-ED3.0.1-x86_64/license
-
-#. Enter the installer directory
-
-   .. code-block:: console
-   
-      $ cd sqream_installer-2.0.7-DB2019.2.1.4-CO1.7.5-ED3.0.1-x86_64
-
-#. Install SQream DB
-   
-   In most cases, the installation command will look like this:
-   
-   .. code-block:: console
-   
-      $ ./sqream-install -i -k -v  -d  -l 
-   
-   For example, if the main storage path for SQream DB is ``/mnt/largedrive`` and the desired shared access path is ``/mnt/nfs/source_files``, the command will look like:
-   
-   .. code-block:: console
-   
-      $ ./sqream-install -i -k -v /mnt/largedrive -d /mnt/nfs/source_files
-   
-   For a full list of options and commands, see the :ref:`sqream-installer reference `
-
-#. SQream DB is now successfully installed, but not yet running.
-
-
-.. _start_local_cluster:
-
-Starting your first local cluster
-=========================================
-
-#. Enter the :ref:`sqream console utility`, which helps coordinate SQream DB components
-   
-   .. code-block:: console
-   
-      $ ./sqream-console
-
-#. Start the master components:
-   
-   .. code-block:: console
-   
-      sqream-console>sqream master --start
-        starting master server in single_host mode ...
-        sqream_single_host_master is up and listening on ports:   3105,3108
-
-
-#. Start workers to join the cluster:
-   
-   .. code-block:: console
-   
-      sqream-console>sqream worker --start 2
-        started sqream_single_host_worker_0 on port 5000, allocated gpu: 0
-        started sqream_single_host_worker_1 on port 5001, allocated gpu: 1
-
-   .. note:: By default, each worker is allocated a full GPU. To launch more workers than available GPUs, see the :ref:`sqream console reference `
-
-#. SQream DB is now running! (Exit the console by typing ``exit`` when done.)
-
-.. rubric:: What's next?
-
-* :ref:`Test your SQream DB installation by creating your first table`
-
-* :ref:`Connect an external tool to SQream DB `
-
-* :ref:`Additional system configuration for performance and stability `
-
-
-.. Some replacements:
-
-.. _`Latest Tesla driver for Linux x64`: https://www.nvidia.com/Download/driverResults.aspx/152242/en-us
-.. _`Latest Tesla driver for Linux x64 POWER LE`: https://www.nvidia.com/Download/driverResults.aspx/152241/en-us
-.. _`NVIDIA's CUDA Installation Guide`: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions
-.. _`Get Docker Engine - Community for CentOS`: https://docs.docker.com/install/linux/docker-ce/centos/
-.. _`Get Docker Engine - Community for Ubuntu`: https://docs.docker.com/install/linux/docker-ce/ubuntu/
diff --git a/guides/operations/setup/recommended_configuration.rst b/guides/operations/setup/recommended_configuration.rst
deleted file mode 100644
index d56048a57..000000000
--- a/guides/operations/setup/recommended_configuration.rst
+++ /dev/null
@@ -1,271 +0,0 @@
-.. _recommended_configuration:
-
-*********************************************
-Recommended post-installation configuration
-*********************************************
-
-Once you've :ref:`installed SQream DB`, you can and should tune your system for better performance and stability.
-
-This page provides recommendations for production deployments of SQream DB.
-
-.. contents:: In this topic:
-   :local:
-
-Recommended BIOS settings
-==========================
-
-The BIOS settings may have a variety of names, or may not exist on your system. Each system vendor has a different set of settings and variables. 
-
-It is safe to skip any and all of the configuration steps, but this may impact performance.
-
-If any doubt arises, consult the documentation for your server or your hardware vendor for the correct way to apply the settings.
-
-.. list-table:: 
-   :widths: auto
-   :header-rows: 1
-   
-   * - Item
-     - Setting
-     - Rationale
-   * - **Fan speed** or **Thermal Configuration**
-     - **High**, **Maximum**, or **Increased cooling**
-     - NVIDIA Tesla GPUs are passively cooled and require high amounts of airflow to function properly
-   * - **System Profile**, **Power Profile**, or **Performance Profile**
-     - **High Performance**
-     - The Performance profile provides potentially increased performance by maximizing processor frequency, and the disabling certain power saving features such as C-states. Use this setting for environments that are not sensitive to power consumption.
-   * - **Power Cap Policy** or **Dynamic power capping**
-     - **Disabled**
-     - Other power profiles (like "balanced") throttle CPU performance which diminish performance. This setting may appear together with the above (Power profile or Power regulator). This option allows for disabling System ROM Power Calibration during the boot process.
-   * - **Intel Turbo Boost**
-     - **Enable**
-     - Intel Turbo Boost allows the processor to be overclocked under high load, which may improve performance in CPU-bound operations, but at the risk of computational jitter if the processor changes its turbo frequency. When that happens, processing stops for a small period of time, introducing uncertainty in application processing time. Turbo operation is a function of power consumption, processor temperature, and the number of active cores.
-   * - **Hyperthreading** or **Logical processor**
-     - **Enable**
-     - Hyperthreading doubles the amount of logical processors, which may improve performance generally by ~5-10% in CPU-bound operations
-   * - **Intel Virtualization Technology** and **VT-d**
-     - **Disable**
-     - Unless you are running VMs, disabling this setting boosts performance by up to 10%.
-   * - **C-States** or **Minimum processor idle power core state**
-     - **Disable** 
-     - Processor C-States allows the server to reduce power when the system is idle. This causes slower 'cold-starts' when the system is transitioning from idle to load, and could reduce query performance by up to 15%.
-   * - **Energy/Performance bias**
-     - **Maximum performance**
-     - Configures processor subsystems for high-performance/low-latency. Other power profiles (like "balanced" or "power savings") throttle CPU performance, which diminish performance. Use this setting for environments that are not sensitive to power consumption.
-   * - **DIMM voltage**
-     - **Optimized for Performance**
-     - Setting a higher voltage for DIMMs can increases performance
-   * - **Memory Operating Mode**
-     - **Optimizer Mode**, **Disable Node Interleaving**, **Auto Memory Operating Voltage**
-     - Memory Operating Mode is tuned for performance in Optimizer mode. Other modes may improve reliability but reduce performance. Enabling Node Interleaving means the memory is interleaved between memory nodes, which harms NUMA-aware applications like SQream DB, so leave disabled.
-   * - **Memory power savings mode**
-     - **Maximum performance**
-     - This option configures several memory parameters to optimize the memory subsystems performance and is configured to Balanced by default.
-   * - **ACPI SLIT**
-     - **Enabled**
-     - ACPI SLIT describes the relative access times between processors, memory subsystems, and I/O subsystems. Operating systems that support the SLIT can use this information to improve performance by allocating resources and workloads more efficiently.
-   * - **QPI Snoop**
-     - **Cluster on Die** or **Home Snoop**
-     - This option allows for the configurations of different snoop modes that impact the QPI interconnect. Changing this option may improve performance in certain workloads. Home Snoop provides high memory bandwidth in an average NUMA environment (default setting). Cluster on Die may provide increased memory bandwidth in highly optimized NUMA workloads. Early Snoop may decrease memory latency but may also result in lower overall bandwidth as compared to other modes.
-
-Use a dedicated SQream DB administration account
-===================================================
-
-Create a user for SQream DB, and optionally assign it to the ``wheel`` group for ``sudo`` access.
-
-.. code-block:: console
-
-   $ useradd -m -u 1132 -U sqream
-   $ passwd sqream 
-   $ usermod -aG wheel sqream 
-
-.. note::
-   * The UID (1132 in the example above) is set to ensure all shared files are accessible by all workers.
-
-Configure the OS locale and timezone
-=====================================
-
-#. Set your OS to use UTF-8, which SQream DB uses for non-English language support.
-   
-   .. code-block:: console
-   
-      $ sudo localectl set-locale LANG=en_US.UTF-8
-
-#. Set the correct timezone for your server.
-   Refer to `the list of available timezones `_ to find a timezone that matches your location.
-   
-   .. code-block:: console
-   
-      $ sudo timedatectl set-timezone America/New_York
-
-
-Configure NTP for clock synchronization
-=========================================
-
-SQream DB clusters rely on clock synchronization to function correctly.
-
-
-   .. code-block:: console
-   
-      $ sudo yum install -y ntp ntpdate
-      $ sudo systemctl enable ntpd
-      $ sudo systemctl start ntpd
-
-If your organization has an NTP server, configure it by adding records to ``/etc/ntpd.conf``, reloading the service, and checking that synchronization is enabled:
-
-   .. code-block:: console
-   
-      $ echo -e "\nserver \n" | sudo tee -a /etc/ntp.conf
-      $ sudo systemctl restart ntpd
-      $ sudo timedatectl
-               Local time: Sat 2019-10-12 17:26:13 EDT
-           Universal time: Sat 2019-10-12 21:26:13 UTC
-                 RTC time: Sat 2019-10-12 21:26:13
-                Time zone: America/New_York (EDT, -0400)
-              NTP enabled: yes
-         NTP synchronized: yes
-          RTC in local TZ: no
-               DST active: yes
-          Last DST change: DST began at
-                           Sun 2019-03-10 01:59:59 EST
-                           Sun 2019-03-10 03:00:00 EDT
-          Next DST change: DST ends (the clock jumps one hour backwards) at
-                           Sun 2019-11-03 01:59:59 EDT
-                           Sun 2019-11-03 01:00:00 EST
-
-
-
-Install recommended utilities
-===============================
-
-The following packages contain tools that are recommended but not required for using SQream DB.
-
-   .. code-block:: console
-   
-      $ sudo yum install -y bash-completion.noarch vim-enhanced.x86_64 vim-common.x86_64 net-tools iotop htop psmisc screen xfsprogs wget yum-utils deltarpm dos2unix tuned  pciutils
-
-
-Tuning OS parameters for performance and stability
-===================================================
-
-SQream DB requires certain OS parameters to be set on all hosts in your cluster.
-
-These settings affect:
-
-* Shared memory - Most OS installations may try to limit high throughput software like SQream DB.
-* Network - On high throughput operations like ingest, optimizing network connection parameters can boost performance
-* User limits - SQream DB may open a large amount of files. The default OS settings may cause some statements to fail if the system runs out of file descriptors.
-* Core dump creation rules
-
-#. Create a directory for core dumps
-
-   In this step, you will create a directory for writing core dumps - which you will configure in the next step.
-   
-   .. code-block:: console
-      
-      $ sudo mkdir /tmp/core_dumps
-
-   .. note::
-      Core dumps can be large - up to the size of the system memory (i.e. for a machine with 512GB of RAM, the size of the core dump will be 512GB).
-
-      Make sure the directory has enough space for writing a core dump.
-
-
-#. Set ``sysctl`` overrides to tune system performance
-   
-   .. code-block:: console
-      :linenos:
-      
-      $ sudo tee /etc/sysctl.d/sqreamdb.conf > /dev/null < /dev/null <`
-
-* :ref:`Connect an external tool to SQream DB `
-
diff --git a/guides/operations/sqream_studio.rst b/guides/operations/sqream_studio.rst
deleted file mode 100644
index 0dd41aa34..000000000
--- a/guides/operations/sqream_studio.rst
+++ /dev/null
@@ -1,578 +0,0 @@
-.. _sqream_studio:
-
-****************************
-SQream Acceleration Studio
-****************************
-
-The studio is a web-based client for use with SQream DB. 
-
-It can be used to run statements, manage roles and permissions, and manage a SQream DB cluster.
-
-.. contents:: In this topic:
-   :local:
-
-
-Setting up and starting the studio
-====================================================
-
-The studio is included with all :ref:`dockerized installations of SQream DB`.
-
-.. todo: How to start it?
-.. You can start the studio using :ref:`sqream-console`:
-.. .. code-block:: console
-.. 
-..    $ ./sqream-console
-..   sqream-console> sqream editor --start
-..    access sqream statement editor through Chrome http://192.168.0.100:3000
-
-When starting the studio, it listens on the local machine, on port 8080.
-
-Logging in
-===================
-
-Open a browser to the host, on port 8080. (e.g. If the machine is ``196.168.0.100``, navigate to http://192.168.0.100:8080) .
-
-Fill in your login details for SQream DB. These are the same credentials you might use when using :ref:`sqream sql` or JDBC.
-
-.. image:: /_static/images/studio_login.png
-
-Your user level in SQream DB changes what you see.
-
-* ``SUPERUSER`` levels have full access to the studio, including the :ref:`Dashboard`.
-
-* All users have access to the :ref:`Editor`, and can only see databases they have permissions for.
-
-.. _studio_editor:
-
-Statement editor
-=================
-
-Familiarizing yourself with the editor
------------------------------------------
-
-The editor is built up of main panes.
-
-.. image:: /_static/images/studio_editor_familiarize.png
-
-* :ref:`Toolbar` - used to select the active database you want to work on, limit the number of rows, save query, etc.
-
-* :ref:`Statement area` - The statement area is a multi-tab text editor where you write SQL statements. Each tab can connect to a different database.
-
-* :ref:`Results` - Results from a query will populate here. This is where you can copy or save query results, or show query execution details.
-
-* :ref:`Database tree` - contains a heirarchy tree of databases, views, tables, and columns. Can be used to navigate and perform some table operations.
-
-See more about each pane below:
-
-Navigation and user information
--------------------------------------
-
-The user information menu is located on the bottom left portion of the screen |icon-user|.
-
-.. image:: /_static/images/studio_user_info.png
-
-
-The menu contains information about the currently signed-in user, as well as version information.
-
-You can sign out of the current user at any point, by selecting :kbd:`Logout` in the user menu.
-
-
-.. _studio_editor_toolbar:
-
-Toolbar
--------------
-
-In the toolbar, you can perform the folllowing operations (from left to right):
-
-.. image:: /_static/images/studio_editor_toolbar.png
-
-* Database dropdown - Select the database you want to the statements to run on.
-
-* Queue - specify which service queue the statement should run in
-
-* :kbd:`⯈ Execute` / :kbd:`STOP` - Use the :kbd:`⯈ EXECUTE` button to execute the statement in the Editor pane. When a statement is running, the button changes to :kbd:`STOP`, and can be used to :ref:`stop the active statement`.
-
-* :kbd:`Format SQL` - Reformats and reindents the statement
-
-* :kbd:`Download query` - save query text to your computer
-
-* :kbd:`Open query` - load query text from your computer
-
-* Max. Rows - By default, the editor will only fetch the first 1000 rows. Click the number to edit. Click outside the number area to save. Setting a higher limit can slow down your browser if the result set is very large. This number is limited to 100000 results (To see more results, consider saving the results to a file or a table with :ref:`create_table_as`).
-
-.. _studio_editor_statement_area:
-
-Statement area
-----------------
-
-The multi-tabbed statement area is where you write queries and statements.
-
-.. image:: /_static/images/studio_editor_statement.png
-
-
-Select the database you wish to use in the toolbar, and then write and execute statements.
-
-A new tab can be opened for each statement. Tabs can be used to separate statements to different databases. Clicking the |icon-plus| will open a new tab with a default name of SQL + a running number.
-
-Multiple statements can be written in the same tab, separated by semicolons (``;``).
-
-If too many tabs are open, pagination controls will appear. Click |icon-left| or |icon-right| to scroll through the tab listings.
-Rename a tab by double clicking it's name.
-
-Close a tab by clicking |icon-close|
-
-To close all tabs, click :kbd:`Close all`, to the right of the tabs.
-
-
-.. tip:: If this is your first time with SQream DB, see our :ref:`first steps guide`.
-
-.. Keyboard shortcuts
-.. ^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. :kbd:`Ctrl` +: kbd:`Enter` - Execute all queries in the statement area, or just the highlighted part of the query.
-
-.. :kbd:`Ctrl` + :kbd:`Space` - Auto-complete the current keyword
-
-.. :kbd:`Ctrl` + :kbd:`↑` - Switch to next tab.
-
-.. :kbd:`Ctrl` + :kbd:`↓` - Switch to previous tab
-
-.. _studio_editor_results:
-
-
-Formatting your SQL
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The |icon-format-sql| button can be used to automatically indent and reformat your SQL statements.
-
-Saving statements
-^^^^^^^^^^^^^^^^^^^^^
-
-The |icon-download-query| saves the tab contents to your computer.
-
-Loading SQL to a tab
-^^^^^^^^^^^^^^^^^^^^^^^
-
-The |icon-open-query| button loads a local file from your computer into a new editor tab.
-
-Executing SQL statements
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking |icon-execute| will execute statements from the active tab.
-
-The button has three modes, which can be selected with the dropdown arrow :kbd:`ᐯ`
-
-* Execute statements – executes the statements where the cursor is located.
-* Execute selected – executes the exact highlighted text. This mode is good for executing a subquery or other part of a large query (as long as it is a valid SQL).
-* Execute all – executes all statements in the active tab, regardless of any selection
-
-When a statement is running, the button changes to :kbd:`STOP`, and can be used to :ref:`stop the active statement`.
-
-Results
--------------
-
-The results pane shows query results and execution information. By default, only the first 10000 results are returned (modify via the :ref:`studio_editor_toolbar`).
-
-.. image:: /_static/images/studio_editor_results.png
-
-By default, executing several statements together will open a separate results tab for each statement.
-
-Statements will be executed serially. Any failed statement will cancel subsequent statements.
-
-If the |keep-tabs| switch is on, new statements will create new tabs. When off, existing result will be cleared.
-
-If too many result tabs are open, pagination controls will appear. Click |icon-left| or |icon-right| to scroll through the tab listings.
-
-Close a tab by clicking |icon-close|
-
-To close all tabs, click :kbd:`Close all`, to the right of the tabs.
-
-.. contents:: In this topic:
-   :local:
-
-
-Sorting results
-^^^^^^^^^^^^^^^^^^^^^^
-
-After the results have appeared, sort them by clicking the column name.
-
-Viewing execution information
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-During query execution the time elapsed is tracked in seconds.
-
-The :kbd:`Show Execution Details` button opens the query's :ref:`execution plan`, for monitoring purposes.
-
-Saving results to a file or clipboard
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Query results can be saved to a clipboard (for pasting into another text editor) or a local file.
-
-.. _studio_editor_db_tree:
-
-Database tree
----------------
-
-The database tree shows the database objects (e.g. tables, columns, views), as well as some metadata like row counts.
-
-It also contains a few predefined catalog queries for execution.
-
-.. image:: /_static/images/studio_editor_db_tree.png
-
-Each level contains a context menu relevant to that object, accessible via a right-click.
-
-.. contents:: In this topic:
-   :local:
-
-System Queries
-^^^^^^^^^^^^^^^^^^^^^^^
-
-The studio editor comes with several predefined catalog queries that are useful for analysis of table compression rates, users and permissions, etc.
-
-Clicking on the :kbd:`System queries` tab in the Tree section will show a list of pre-defined system queries.
-
-Clicking on an item will paste the query into the editing area.
-
-
-Filtering (searching) for objects
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the |icon-filter| filter icon by columns or tables opens an editable field that can be used for searching.
-
-To remove the filter, click the icon again or select ❌.
-
-Copying object names
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the |icon-copy| icon will copy the object name
-
-Generating SELECT statements
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the |icon-select| icon will generate a :ref:`select` query for the selected table in the editing area.
-
-Generating an INSERT statement
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Insert statement` option under the :kbd:`⋮` menu generates an :ref:`insert` statement for the selected table in the editing area.
-
-Generating a DELETE statement
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Delete statement` option under the :kbd:`⋮` menu generates a :ref:`delete` statement for the selected table in the editing area.
-
-Generating a CREATE TABLE AS statement
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Create table as` option under the :kbd:`⋮` menu generates a :ref:`create_table_as` statement for the selected table in the editing area.
-
-Renaming a table
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Rename table` option under the :kbd:`⋮` menu generates an :ref:`alter_table` statement for renaming the selected table in the editing area.
-
-
-Adding columns to table
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Add column` option under the :kbd:`⋮` menu generates an :ref:`alter_table` statement for adding columns to the selected table in the editing area.
-
-Truncate a table
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Truncate table` option under the :kbd:`⋮` menu generates a :ref:`truncate` statement for the selected table in the 
-editing area.
-
-
-Dropping an object
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Drop table`, :kbd:`Drop view`, or :kbd:`Drop function` option under the :kbd:`⋮` menu generates a ``DROP`` statement for the selected object in the editing area.
-
-
-Generating DDL statements
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Clicking the :kbd:`Table DDL`, :kbd:`View DDL`, or :kbd:`Function function` option under the :kbd:`⋮` menu generates a DDL  statement for the selected object in the editing area.
-
-To get the entire database DDL, click the |icon-ddl-edit| icon next to the database name in the tree root.
-See also :ref:`seeing_system_objects_as_sql`.
-
-The DDL optimizer
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The DDL optimizer tab analyzes database tables and recommends possible optimizations, per the :ref:`sql_best_practices` guide.
-
-Using the DDL optimizer
----------------------------
-
-Navigate to the DDL optimizer module by selecting :kbd:`DDL Optimizer` from the :kbd:`⋮` ("More") menu.
-
-.. image:: /_static/images/studio_ddl_optimizer.png
-
-* ``Rows`` - number of rows to scan for analysis. Defaults to 1,000,000
-
-* ``Buffer Size`` - overhead threshold to use when analyzing ``VARCHAR`` fields. Defaults to 10%.
-
-* ``Optimize NULLs`` - attempt to figure out field nullability.
-
-Click :kbd:`Run Optimizer` to start the optimization process.
-
-Analyzing the results
-----------------------------
-
-When results are produced, a :kbd:`Generate CREATE statement` button will appear.
-Clicking the button creates a new tab with an optimized :ref:`create_table` statement, and an :ref:`insert` statement to copy the data to the new table.
-
-
-.. _studio_dashboard:
-
-Administration Dashboard
-==============================
-
-If you signed in with a ``SUPERUSER`` role, you can enter the administration dashboard.
-
-Enter the administration dashboard by clicking the |icon-dashboard| icon in the navigation bar.
-
-
-
-Familiarizing yourself with the dashboard
----------------------------------------------
-
-.. image:: /_static/images/studio_dashboard_familiarize.png
-
-The main dashboard screen contains two main panes:
-
-* 
-   :ref:`Data storage pane` - monitor the SQream DB cluster's storage
-
-   - can be expanded to :ref:`drill down into database storage`
-
-* 
-   :ref:`Worker pane` - monitor system health
-   
-   - the worker pane used to monitor workers and :ref:`service queues` in the cluster.
-
-.. _administration_storage_pane:
-
-Data storage pane
------------------------
-
-The left section of the Admin Dashboard shows you the status of your system's storage as a donut.
-
-.. image:: /_static/images/studio_dashboard_storage.png
-
-Storage is displayed broken up into four components:
-
-* Data – Storage occupied by databases in SQream DB
-
-* Free – Free storage space
-
-* 
-   Deleted – Storage that is temporarily occupied but hasn't been reclaimed (see our :ref:`delete guide` to understand how data deletion works). 
-   
-   (This value is estimated and may not be accurate)
-   
-* Other – Storage used by other applications. On a dedicated SQream DB cluster, this should be close to zero.
-
-.. _administration_storage_database:
-
-Database storage
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Expanding the storage pane (|icon-expand|) will show a breakdown of how much storage is used by each database in the cluster.
-
-.. image:: /_static/images/studio_dashboard_storage_breakdown.png
-
-This can be used to drill down into each database's storage footprint.
-
-Databases are displayed in a table, containing the following information:
-* Database name
-* Raw storage size – the estimated size of raw data (uncompressed) in the database
-* Storage size – the physical size of the compressed data
-* Ratio – effective compression ratio
-* Deleted data – storage that is temporarily occupied but hasn't been reclaimed (see our :ref:`delete guide` to understand how data deletion works). (This value is estimated and may not be accurate)
-
-Below the table, a graph shows the database storage trends.
-
-By default, the graph shows the total storage for all databases. Clicking a database in the table will filter to show just that database.
-
-The scale of the presented information can be controlled by changing the timeframe in the scale dropdown (|icon-scale|).
-
-.. _administration_worker_pane:
-
-Service and workers pane
---------------------------
-
-This pane shows the cluster status in workers and their :ref:`service queues`.
-
-.. _administration_services:
-
-Services
-^^^^^^^^^^^
-
-The services bar shows the defined :ref:`service queues`.
-
-Services are used to divide workers and associate (subscribe) workers to services.
-
-Each service queue contains the following details:
-* Service name
-* A graph of load over time (statements in that queue)
-* Currently processed queries of the Service / total queries for that service in the system (including queued queries)
-
-Creating new service queues
-********************************
-
-Click the |icon-add| button above the service list. Type the service queue name and associate new workers to the service queue.
-
-.. note:: if you choose not to associate a worker with the new service, it will not be created.
-
-Associating a worker with an existing service
-**********************************************
-
-Clicking on the |icon-add-worker| icon on a service name is used to attach workers to a service.
-
-Clicking on a service queue in the services bar will display the list of workers in the main pane.
-
-.. image:: /_static/images/studio_dashboard_services.png
-
-In this mode, the :kbd:`⋮` icon (more menu) can be used to detach a worker from a service.
-
-You can select a Worker from the list that is available to process queries of the relevant Service and by clicking on the  button of that Worker that Worker will be associated with the Service. After that the page will go back to its normal layout and you will be able to click the Service and see the Worker associated with the Service.
-Other Services associated with that Worker will remain associated to it.
-
-
-.. _administration_workers:
-
-Workers
-^^^^^^^^^^^^^
-
-The worker pane shows each worker (``sqreamd``) running in the cluster. 
-
-Each worker has a status bar that represents the status over time. The scale of the presented information can be controlled by changing the timeframe in the scale dropdown (|icon-scale|).
-
-The status bar is divided into 20 equal sections, showing the most dominant activity in that slice. Hover over the status bar sections to see the activity:
-
-* Idle – worker is idle and available for statements
-* Compiling – Compiling a statement, in preparation for execution
-* Executing – executing a statement after compilation
-* Stopped – worker was stopped (either deliberately or due to a fault)
-* Waiting – worker was waiting on an object locked by another worker
-
-Show host resources
-*****************************
-
-Clicking the |icon-expand-down| button below each host will expand to show the host resource utilization.
-
-.. image:: /_static/images/studio_worker_activity.png
-
-The host resource utilization includes information about:
-
-* CPU utilization
-* Memory utilization
-* GPU utilization
-
-Graphs show resource utilization over time. Current values are shown on the right.
-
-Hover over the graph line to see the activity at a given time.
-
-
-Active queries
-******************
-
-Clicking the |icon-expand-down| button on a worker will expand to show the active statements running.
-
-Each statement has a statement ID, status, service queue, elapsed time, execution time, and estimated completion status.
-
-Each statement can be stopped or expanded to show its execution plan and progress (:ref:`show_node_info`).
-
-.. include:: /reference/sql/sql_statements/monitoring_commands/show_server_status.rst
-   :start-line: 67
-   :end-line: 84
-
-Control worker status (start, stop, restart)
-****************************************************
-
-In some cases, it may be useful to stop or restart workers for maintenance.
-
-Each Worker line has a :kbd:`⋮` menu (more menu). This menu allows stopping, starting, or restarting workers.
-
-When a worker is stopped, it has a gray background and its status is "Stopped". 
-
-
-
-
-.. |icon-user| image:: /_static/images/studio_icon_user.png
-   :align: middle
-
-.. |icon-editor| image:: /_static/images/studio_icon_editor.png
-   :align: middle
-
-.. |icon-copy| image:: /_static/images/studio_icon_copy.png
-   :align: middle
-
-.. |icon-select| image:: /_static/images/studio_icon_select.png
-   :align: middle
-
-.. |icon-dots| image:: /_static/images/studio_icon_dots.png
-   :align: middle
-
-.. |icon-filter| image:: /_static/images/studio_icon_filter.png
-   :align: middle
-
-.. |icon-ddl-edit| image:: /_static/images/studio_icon_ddl_edit.png
-   :align: middle
-
-.. |icon-run-optimizer| image:: /_static/images/studio_icon_run_optimizer.png
-   :align: middle
-
-.. |icon-generate-create-statement| image:: /_static/images/studio_icon_generate_create_statement.png
-   :align: middle
-
-.. |icon-plus| image:: /_static/images/studio_icon_plus.png
-   :align: middle
-
-.. |icon-close| image:: /_static/images/studio_icon_close.png
-   :align: middle
-
-.. |icon-left| image:: /_static/images/studio_icon_left.png
-   :align: middle
-
-.. |icon-right| image:: /_static/images/studio_icon_right.png
-   :align: middle
-
-.. |icon-format-sql| image:: /_static/images/studio_icon_format.png
-   :align: middle
-
-.. |icon-download-query| image:: /_static/images/studio_icon_download_query.png
-   :align: middle
-
-.. |icon-open-query| image:: /_static/images/studio_icon_open_query.png
-   :align: middle
-
-.. |icon-execute| image:: /_static/images/studio_icon_execute.png
-   :align: middle
-
-.. |icon-stop| image:: /_static/images/studio_icon_stop.png
-   :align: middle
-
-.. |icon-dashboard| image:: /_static/images/studio_icon_dashboard.png
-   :align: middle
-
-.. |icon-expand| image:: /_static/images/studio_icon_expand.png
-   :align: middle
-
-.. |icon-scale| image:: /_static/images/studio_icon_scale.png
-   :align: middle
-
-.. |icon-expand-down| image:: /_static/images/studio_icon_expand_down.png
-   :align: middle
-
-.. |icon-add| image:: /_static/images/studio_icon_add.png
-   :align: middle
-
-.. |icon-add-worker| image:: /_static/images/studio_icon_add_worker.png
-   :align: middle
-
-.. |keep-tabs| image:: /_static/images/studio_keep_tabs.png
-   :align: middle
\ No newline at end of file
diff --git a/guides/operations/starting_and_stopping_the_system.rst b/guides/operations/starting_and_stopping_the_system.rst
deleted file mode 100644
index 62dbb6d72..000000000
--- a/guides/operations/starting_and_stopping_the_system.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-.. _running_and_stopping_the_system:
-
-****************************************
-running_and_stopping_the_system
-****************************************
-
-how do you run sqream with the various methods:
-
-monit
-
-pacemaker
-
-docker
-
-k8s
-
-from command line
-
-how do you check it's running/view it's status/check you can use it
-
-how do you stop it
diff --git a/guides/operations/statement_editor.rst b/guides/operations/statement_editor.rst
deleted file mode 100644
index 55c29def1..000000000
--- a/guides/operations/statement_editor.rst
+++ /dev/null
@@ -1,290 +0,0 @@
-.. _statement_editor:
-
-*****************************************
-Using the statement editor (deprecated)
-*****************************************
-
-The statement editor is a web-based client for use with SQream DB. It can be used to run statements, and manage roles and permissions.
-
-.. note:: The statement editor is deprecated from SQream DB v2020.2. It is replaced by :ref:`SQream Studio`
-
-.. contents:: In this topic:
-   :local:
-
-Setting up and starting the statement editor
-====================================================
-
-.. note:: We recommend using Google Chrome or Chrome-based browsers to access the statement editor.
-
-The statement editor is included with all :ref:`dockerized installations of SQream DB`.
-
-You can start the editor using :ref:`sqream-console`:
-
-
-.. code-block:: console
-
-   $ ./sqream-console
-   sqream-console> sqream editor --start
-   access sqream statement editor through Chrome http://192.168.0.100:3000
-
-When starting the editor, it listens on the local machine, on port 3000.
-
-Logging in
-===================
-
-Open a browser to the host, on port 3000. If the machine is ``196.168.0.100``, navigate to http://192.168.0.100:3000 .
-
-Fill in your login details for SQream DB. These are the same credentials you might use when using :ref:`sqream sql` or JDBC.
-
-.. image:: /_static/images/statement_editor_login.png
-
-.. tip:: 
-   * If using a :ref:`load balancer`, select the ``Server picker`` box and make sure to use the correct port (usually ``3108``). If connecting directly to a worker, make sure to untick the box and use the correct port, usually ``5000`` and up.
-   * If this is your first time using SQream DB, use database name ``master``.
-   * When using SQream DB on AWS, the initial password is the EC2 instance ID.
-
-Familiarizing yourself with the editor
-==============================================
-
-The editor is built up of main panes.
-
-.. image:: /_static/images/statement_editor_main.png
-
-* :ref:`Toolbar` - used to select the active database you want to work on, limit the number of rows, save query results, control tab behavior, and more.
-
-* :ref:`Statement area` - The statement area is a multi-tab text editor where you write SQL statements. Each tab can connect to a different database.
-
-* :ref:`Results` - Results from a query will populate here. 
-
-* :ref:`Database tree` - contains a heirarchy tree of databases, views, tables, and columns. Can be used to navigate and perform some table operations.
-
-See more about each pane below:
-
-.. _editor_toolbar:
-
-Toolbar
--------------
-
-.. image:: /_static/images/statement_editor_toolbar.png
-
-In the toolbar, you can perform the folllowing operations (from left to right):
-
-* Toggle Database Tree - Click •••​ to show or hide the Database Tree pane.
-
-* Database dropdown - Select the database you want to the statements to run on.
-
-* :kbd:`⯈ RUN` / :kbd:`◼ STOP` - Use the :kbd:`⯈ RUN` button to execute the statement in the Editor pane. When a statement is running, the button changes to :kbd:`◼ STOP`, and can be used to :ref:`stop the active statement`.
-
-* :kbd:`SQL` - Reformats and reindents the statement
-
-* Max. Rows - By default, the editor will only fetch the first 1000 rows. Click the number to edit. Click outside the number area to save. Setting a higher limit can slow down your browser if the result set is very large.
-
-* 💾 (Save) - Save the query text to a file.
-
-* 📃 (Load) - Load query text from a file.
-
-* ⋮ (more) - 
-   
-   * Append new results - When checked, every statement executed will open a new Results tab. If unchecked, the Results tab is reused and overwritten with every new statement.
-
-.. _editor_statement_area:
-
-Statement area
-----------------
-
-.. image:: /_static/images/statement_editor_editor.png
-
-The multi-tabbed statement area is where you write queries and statements.
-
-Select the database you wish to use in the toolbar, and then write and execute statements.
-
-* A new tab can be opened for each statement. Tabs can be used to separate statements to different databases.
-
-* Multiple statements can be written in the same tab, separated by semicolons. 
-
-* When multiple statements exist in the tab, clicking :kbd:`Run` executes all statements in the tab, or only the selected statements.
-
-.. tip:: If this is your first time with SQream DB, see our :ref:`first steps guide`.
-
-Keyboard shortcuts
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-:kbd:`Ctrl` +: kbd:`Enter` - Execute all queries in the statement area, or just the highlighted part of the query.
-
-:kbd:`Ctrl` + :kbd:`Space` - Auto-complete the current keyword
-
-:kbd:`Ctrl` + :kbd:`↑` - Switch to next tab.
-
-:kbd:`Ctrl` + :kbd:`↓` - Switch to previous tab
-
-.. _editor_results:
-
-Results
--------------
-
-The results pane shows query results and execution information. By default, only the first 1000 results are returned (modify via the :ref:`editor_toolbar`).
-
-A context menu, accessible via a right click on the results tab, enables:
-
-* Renaming the tab name
-* Show the SQL query text
-* Reload results
-* Close the current tab
-* Close all result tabs
-
-Sorting results
-^^^^^^^^^^^^^^^^^^^^^^
-
-After the results have appeared, sort them by clicking the column name.
-
-Viewing execution information
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-During query execution the time elapsed is tracked in seconds.
-
-.. image:: /_static/images/statement_editor_statistics.png
-
-The :kbd:`SHOW STATISTICS` button opens the query's :ref:`execution plan`, for monitoring purposes.
-
-Saving results to a file or clipboard
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. image:: /_static/images/statement_editor_save.png
-
-Query results can be saved to a clipboard (for pasting into another text editor) or a local file.
-
-.. _editor_db_tree:
-
-Database tree
----------------
-
-The database tree shows the database objects (e.g. tables, columns, views), as well as some metadata like row counts.
-
-It also contains a few predefined catalog queries for execution.
-
-.. image:: /_static/images/statement_editor_db_tree.png
-
-Each level contains a context menu relevant to that object, accessible via a right-click.
-
-Database
-^^^^^^^^^^^^^
-
-* :ref:`Copy the database DDL` to the clipboard
-
-Schema
-^^^^^^^^^^
-
-* Drop the schema (copies statement to the clipboard)
-
-Table
-^^^^^^^^^^
-
-* Show row count in the database tree
-
-* :ref:`Copy the create table script` to the clipboard
-
-* Copy :ref:`select` to clipboard
-
-* Copy :ref:`insert` to clipboard
-
-* Copy :ref:`delete` to clipboard
-
-* Rename table - Copy :ref:`rename_table` to clipboard
-
-* Create table LIKE - Copy :ref:`create_table_as` to clipboard
-
-* Add column - Copy :ref:`add_column` to clipboard
-
-* Truncate table - Copy :ref:`truncate` to clipboard
-
-* Drop table - Copy :ref:`drop_table` to pclipboard
-
-* Create a table - Add a new table by running a statement, or alternatively use the **Add new** link near the **TABLES** group. 
-
-Create a table
-^^^^^^^^^^^^^^^^^^^^
-
-Creating a new table is also possible using the wizard which can guide you with creating a table.
-
-Refer to the :ref:`create_table` reference for information about creating a table (e.g. able parameters like default values, identity, etc.).
-
-.. image:: /_static/images/statement_editor_add_table.png
-
-Fill in the table name, and add a new row for each table column.
-
-If a table by the same name exists, check **Create or Replace table** to overwrite it.
-
-Click :kbd:`EXEC` to create the table.
-
-Catalog views
-^^^^^^^^^^^^^^^^^^^
-
-To see :ref:`catalog views`, click the catalog view name in the tree. The editor will run a query on that view.
-
-.. image:: /_static/images/statement_editor_view_catalog.png
-
-Predefined queries
-^^^^^^^^^^^^^^^^^^^^^^^
-
-The editor comes with several predefined catalog queries that are useful for analysis of table compression rates, users and permissions, etc.
-
-.. image:: /_static/images/statement_editor_predefined_queries.png
-
-Notifications
-===================
-
-Desktop notificaitons lets you receive a notification when a statement is completed. 
-
-You can minimize the browser or switch to other tabs, and still recieve a notification when the query is done.
-
-.. image:: /_static/images/statement_editor_notifications.png
-
-Enable the desktop notification through the **Allow Desktop Notification** from the menu options.
-
-DDL optimizer
-==================
-
-The DDL optimizer tab analyzes database tables and recommends possible optimizations, per the :ref:`sql_best_practices` guide.
-
-Using the DDL optimizer
----------------------------
-
-Navigate to the DDL optimizer module by selecting it from the :kbd:`⋮` ("More") menu.
-
-.. image:: /_static/images/statement_editor_ddl_optimizer.png
-
-* ``Database`` and ``Table`` - select the database and desired table to optimize
-* ``Rows`` is the number to scan for analysis. Defaults to 1,000,000
-
-* ``Buffer Size`` - overhead threshold to use when analyzing ``VARCHAR`` fields. Defaults to 10%.
-
-* ``Optimize NULLs`` - attempt to figure out field nullability.
-
-Click ``EXECUTE`` to start the optimization process.
-
-Analyzing the results
-----------------------------
-
-The analysis process shows results for each row.
-
-.. image:: /_static/images/statement_editor_ddl_optimizer_results.png
-
-The results are displayed in two tabs:
-
-* **OPTIMIZED COLUMNS** - review the system recommendation to:
-
-   #. decrease the length of ``VARCHAR`` fields
-
-   #. remove the ``NULL`` option
-
-* **OPTIMIZED DDL** - The recommended :ref:`create_table` statement
-
-Analyzing the DDL culminates in four possible actions:
-
-* :kbd:`COPY DDL TO CLIPBOARD` - Copies the optimized :ref:`create_table` to the clipboard
-
-* :kbd:`CREATE A NEW TABLE` - Creates the new table structure with ``_new`` appended to the table name. No data is populated
-
-* :kbd:`CREATE AND INSERT INTO EXISTING DATA` - Create a new table in same database and schema as the original table and populates the data
-
-* **Back** - go back to the statement editor and abandon any recommendations
diff --git a/guides/operations/troubleshooting.rst b/guides/operations/troubleshooting.rst
deleted file mode 100644
index 7eed4c757..000000000
--- a/guides/operations/troubleshooting.rst
+++ /dev/null
@@ -1,163 +0,0 @@
-.. _troubleshooting:
-
-***********************
-Troubleshooting
-***********************
-
-.. contents:: In this topic:
-   :local:
-
-Follow this checklist if you find that the performance is slower than you expect.
-
-.. list-table:: Troubleshooting checklist
-   :widths: auto
-   :header-rows: 1
-   
-   * - Step
-     - Description
-     - Results
-   * - 1
-     - A single query is slow
-     - 
-         If a query isn't performing as you expect, follow the :ref:`Query best practices` part of the :ref:`sql_best_practices` guide.
-         
-         If all queries are slow, continue to step 2.
-   * - 2
-     - All queries on a specific table are slow
-     - 
-         #. If all queries on a specific table aren't performing as you expect, follow the :ref:`Table design best practices` part of the :ref:`sql_best_practices` guide.
-         #. Check for active delete predicates in the table. Consult the :ref:`delete_guide` guide for more information.
-         
-         If the problem spans all tables, continue to step 3.
-   * - 3
-     - Check that all workers are up
-     - 
-         Use ``SELECT show_cluster_nodes();`` to list the active cluster workers.
-         
-         If the worker list is incomplete, follow the :ref:`cluster troubleshooting` section below.
-         
-         If all workers are up, continue to step 4.
-   * - 4
-     - Check that all workers are performing well
-     - 
-         #. Identify if a specific worker is slower than others by running the same query on different workers. (e.g. by connecting directly to the worker or through a service queue)
-         #. If a specific worker is slower than others, investigate performance issues on the host using standard monitoring tools (e.g. ``top``).
-         #. Restart SQream DB workers on the problematic host.
-         
-         If all workers are performing well, continue to step 5.
-   * - 5 
-     - Check if the workload is balanced across all workers
-     - 
-         #. Run the same query several times and check that it appears across multiple workers (use ``SELECT show_server_status()`` to monitor)
-         #. If some workers have a heavier workload, check the service queue usage. Refer to the :ref:`workload_manager` guide.
-         
-         If the workload is balanced, continue to step 6.
-   * - 6
-     - Check if there are long running statements
-     - 
-         #. Identify any currently running statements (use ``SELECT show_server_status()`` to monitor)
-         #. If there are more statements than available resources, some statements may be in an ``In queue`` mode.
-         #. If there is a statement that has been running for too long and is blocking the queue, consider stopping it (use ``SELECT stop_statement()``).
-         
-         If the statement does not stop correctly, contact SQream support.
-         
-         If there are no long running statements or this does not help, continue to step 7.
-   * - 7
-     - Check if there are active locks
-     - 
-         #. Use ``SELECT show_locks()`` to list any outstanding locks.
-         #. If a statement is locking some objects, consider waiting for that statement to end or stop it.
-         #. If after a statement is completed the locks don't free up, refer to the :ref:`concurrency_and_locks` guide.
-         
-         If performance does not improve after the locks are released, continue to step 8.
-   * - 8
-     - Check free memory across hosts
-     - 
-         #. Check free memory across the hosts by running ``$ free -th`` from the terminal.
-         #. If the machine has less than 5% free memory, consider **lowering** the ``limitQueryMemoryGB`` and ``spoolMemoryGB`` settings. Refer to the :ref:`configuration` guide.
-         #. If the machine has a lot of free memory, consider **increasing** the ``limitQueryMemoryGB`` and ``spoolMemoryGB`` settings.
-         
-         If performance does not improve, contact SQream support for more help.
-
-
-
-Troubleshooting common issues
-======================================
-
-.. _cluster_troubleshooting:
-
-Troubleshoot cluster setup and configuration
------------------------------------------------------
-
-#. Note any errors - Make a note of any error you see, or check the :ref:`logs` for errors you might have missed.
-
-#. If SQream DB can't start, start SQream DB on a new storage cluster, with default settings. If it still can't start, there could be a driver or hardware issue. :ref:`Contact SQream support`.
-
-#. Reproduce the issue with a standalone SQream DB - starting up a temporary, standalone SQream DB can isolate the issue to a configuration issue, network issue, or similar.
-
-#. Reproduce on a minimal example - Start a standalone SQream DB on a clean storage cluster and try to replicate the issue if possible.
-
-
-Troubleshoot connectivity issues
------------------------------------
-
-#. Verify the correct login credentials - username, password, and database name.
-
-#. Verify the host name and port
-
-#. Try connecting directly to a SQream DB worker, rather than via the load balancer
-
-#. Verify that the driver version you're using is supported by the SQream DB version. Driver versions often get updated together with major SQream DB releases.
-
-#. Try connecting directly with :ref:`the built in SQL client`. If you can connect with the local SQL client, check network availability and firewall settings.
-
-Troubleshoot query performance
-------------------------------------
-
-#. Use :ref:`show_node_info` to examine which building blocks consume time in a statement. If the query has finished, but the results are not yet materialized in the client, it could point to a problem in the application's data buffering or a network throughput issue..
-
-#. If a problem occurs through a 3\ :sup:`rd` party client, try reproducing it directly with :ref:`the built in SQL client`. If the performance is better in the local client, it could point to a problem in the application or network connection.
-
-#. Consult the :ref:`sql_best_practices` guide to learn how to optimize queries and table structures.
-
-
-Troubleshoot query behavior
----------------------------------
-
-#. Consult the :ref:`sql` reference to verify if a statement or syntax behaves correctly. SQream DB may have some differences in behavior when compared to other databases.
-
-#. If a problem occurs through a 3\ :sup:`rd` party client, try reproducing it directly with :ref:`the built in SQL client`. If the problem still occurs, file an issue with SQream support.
-
-File an issue with SQream support
-------------------------------------
-
-To file an issue, follow our :ref:`information_for_support` guide.
-
-Examining logs
-========================
-
-See the :ref:`collecting_logs` section of the :ref:`information_for_support` guide for information about collecting logs for support.
-
-
-Start a temporary SQream DB for testing
-===============================================
-
-Starting a SQream DB temporarily (not as part of a cluster, with default settings) can be helpful in identifying configuration issues.
-
-Example:
-
-.. code-block:: console
-
-   $ sqreamd /home/rhendricks/raviga_database 0 5000 /home/sqream/.sqream/license.enc
-
-.. tip:: 
-   
-   * Using ``nohup`` and ``&`` sends SQream DB to run in the background.
-   
-   * 
-      It is safe to stop SQream DB at any time using ``kill``. No partial data or data corruption should occur when using this method to stop the process.
-      
-      .. code-block:: console
-      
-         $ kill -9 $SQREAM_PID
-
diff --git a/guides/queries.rst b/guides/queries.rst
deleted file mode 100644
index 24a4e589d..000000000
--- a/guides/queries.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-.. _queries:
-
-**********************************
-Queries
-**********************************
-
-go through all the query features in sqream and give examples and
-state what we do that's interesting, or less usual, and maybe what we
-don't do that's common in other products/ limitations
-
diff --git a/guides/third_party_tools/php.rst b/guides/third_party_tools/php.rst
deleted file mode 100644
index a04aa981a..000000000
--- a/guides/third_party_tools/php.rst
+++ /dev/null
@@ -1,46 +0,0 @@
-.. _php:
-
-*****************************
-Connecting with PHP
-*****************************
-
-You can use PHP to interact with a SQream DB cluster.
-
-This tutorial is a guide that will show you how to connect a PHP application to SQream DB.
-
-.. contents:: In this topic:
-   :local:
-
-Prerequisites
-===============
-
-#. Install the :ref:`SQream DB ODBC driver for Linux` and create a DSN.
-
-#. 
-   Install the `uODBC `_ extension for your PHP installation.
-   To configure PHP to enable uODBC, configure it with ``./configure --with-pdo-odbc=unixODBC,/usr/local`` when compiling php or install ``php-odbc`` and ``php-pdo`` along with php (version 7.1 minimum for best results) using your distribution package manager.
-
-Testing the connection
-===========================
-
-#. 
-   Create a test connection file. Be sure to use the correct parameters for your SQream DB installation.
-
-   Download this :download:`PHP example connection file ` .
-
-   .. literalinclude:: test.php
-      :language: php
-      :emphasize-lines: 4
-      :linenos:
-
-   .. tip::
-      An example of a valid DSN line is:
-      
-      .. code:: php
-         
-         $dsn = "odbc:Driver={SqreamODBCDriver};Server=192.168.0.5;Port=5000;Database=master;User=rhendricks;Password=super_secret;Service=sqream";
-      
-      For more information about supported DSN parameters, see :ref:`dsn_params`.
-
-#. Run the PHP file either directly with PHP (``php test.php``) or through a browser.
-
diff --git a/guides/third_party_tools/tableau.rst b/guides/third_party_tools/tableau.rst
deleted file mode 100644
index 3ac0bc549..000000000
--- a/guides/third_party_tools/tableau.rst
+++ /dev/null
@@ -1,276 +0,0 @@
-.. _connect_to_tableau:
-
-*************************
-Connecting to Tableau
-*************************
-
-You can use Tableau to connect to a SQream DB cluster. This tutorial is a guide that will show you how to connect to Tableau, as well as provide some guidelines and best practices for exploring data with Tableau and SQream DB.
-
-SQream DB supports both Tableau Desktop and Tableau Server on Windows, MacOS, and Linux distributions.
-
-.. contents:: In this topic:
-   :local:
-
-Installing Tableau Desktop
-============================
-
-SQream DB has been tested with versions 9.2 and newer.
-If you do not already have Tableau Desktop installed, download and install Tabelau Desktop. https://www.tableau.com/products/trial
-
-Tableau offers a time-limited trial version.
-
-Installing the JDBC driver and connector
-=================================================
-
-Starting from Tableau v2019.4, SQream DB recommends using the JDBC driver instead of the previously recommended ODBC driver.
-
-If you have Tableau Desktop on Windows, we recommend using the :ref:`JDBC installer method`. 
-
-If you have Tableau Server or Tableau on MacOS and Linux follow the instructions for a :ref:`manual installation`.
-
-.. _tableau_jdbc_installer:
-
-Installing with the Windows installer
------------------------------------------
-
-1. Close Tableau Desktop
-
-2. Download the JDBC installer :ref:`from the client drivers page`.
-
-3. 
-   Start the installer, and ensure that the "**Tableau Desktop Connector**" item is selected, as in the image below.
-   
-   .. image:: /_static/images/jdbc_windows_installer_screen.png
-
-4. Restart Tableau Desktop, continue to :ref:`connecting to SQream DB`.
-
-.. _tableau_manual_installation:
-
-Installing the JDBC driver manually (MacOS, Linux, Tableau Server)
------------------------------------------------------------------------
-
-Download the JDBC and Tableau Connector (taco)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-1. Download the JDBC installer :ref:`from the client drivers page`.
-
-2. Download the SQream DB Tableau connector (.taco) :ref:`from the client drivers page`.
-
-Install JDBC driver 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Unzip the JDBC driver into a Tableau drivers directory. Based on your installation of Tableau, this may be at:
-
-* Tableau Desktop on Windows: ``c:\Program Files\Tableau\Drivers``
-
-* Tableau Desktop on MacOS: ``~/Library/Tableau/Drivers``
-
-* Tableau on Linux: ``/opt/tableau/tableau_driver/jdbc``
-
-Install taco
-^^^^^^^^^^^^^^^^^^
-
-Place the ``SQreamDB.taco`` file in the Tableau connectors directory. Based on your installation of Tableau, this may be at:
-
-* Tableau Desktop on Windows: ``C:\Users\\My Tableau Repository\Connectors``
-
-* Tableau Desktop on MacOS: ``~/My Tableau Repository/Connectors``
-
-* Tableau Server:
-   
-   1. 
-      Create a directory for Tableau connectors. For example: ``C:\tableau_connectors``
-      
-      This directory needs to exist on all Tableau servers
-   
-   2. Copy the ``SQreamDB.taco`` file into the directory you created on all servers
-   
-   3. Set the ``native_api.connect_plugins_path`` option with ``tsm``. For example:
-      
-      ``tsm configuration set -k native_api.connect_plugins_path -v C:/tableau_connectors``
-      
-      If you get a configuration error during this step, add the ``--force-keys`` option to the end of the command.
-
-      Then, apply the pending configuration changes with ``tsm pending-changes apply``
-      
-      .. warning:: This restarts the server.
-
- 
-
-You can now restart Tableau Desktop or Server to begin using the SQream DB driver. Continue to :ref:`connecting to SQream DB`.
-
-Legacy method - ODBC for Tableau versions before v2019.3
-==================================================================
-
-Installing the ODBC driver and customizations
---------------------------------------------------
-
-If you've already installed the SQream DB ODBC driver, we recommend that you :ref:`re-run the ODBC driver installer ` after installing Tableau, and select the Tableau customizations checkbox, as in the image below:
-
-.. image:: /_static/images/odbc_windows_installer_tableau.png
-
-This is necessary because by default, Tableau has a tendency to create temporary tables and run lots of discovery queries which could impact performance.
-The ODBC driver installer installs customizations for Tableau automatically.
-
-If you want to perform this step manually, follow the instructions in the next section.
-
-The TDC file
-^^^^^^^^^^^^^^^^^^^
-
-The TDC file (Tableau Datasource Customization) helps Tableau make full use of SQream DB's features and capabilities.
-
-Before you start, check which version of Tableau is used. The version needs to be placed in the TDC file.
-
-#. Download the TDC file to your computer :download:`odbc-sqream.tdc `.
-   
-   Alternatively, copy the text below to a text editor.
-   
-   .. literalinclude:: odbc-sqream.tdc
-      :language: xml
-      :caption: SQream DB ODBC TDC
-      :emphasize-lines: 2
-
-
-#. Change the highlighted line to match your major Tableau version. For example, if you're on Tableau ``2019.2.1``, writing ``2019.2`` is enough.
-
-#. 
-   * For **Tableau Desktop** - save the TDC file to ``C:\Users\\Documents\My Tableau Repository\Datasources``, where ```` is the Windows username Tableau is installed in.
-   
-   * For **Tableau Server** - save the TDC file to ``C:\ProgramData\Tableau\Tableau Server\data\tabsvc\vizqlserver\Datasources``.
-
-Configure the ODBC connection (DSN)
-------------------------------------------
-
-Create an ODBC DSN before connecting Tableau with SQream DB. See the section titled :ref:`create_windows_odbc_dsn` for information about creating an ODBC DSN in Windows.
-
-Remember to test the connectivity before saving the DSN.
-
-Connecting Tableau to SQream DB
----------------------------------------
-
-#. Start Tableau Desktop and select "Other Database (ODBC)", by navigating :menuselection:`Connect --> To a server --> More --> Other Database (ODBC)`
-   
-   .. image:: /_static/images/tableau_more_servers.png
-   
-#. In the DSN selection window, select the DSN that you created earlier and select :menuselection:`Connect --> OK`. 
-   
-   If prompted by Tableau, you may need to specify the user name and password again after clicking Connect.
-   
-   .. image:: /_static/images/tableau_choose_dsn_and_connect.png
-   
-
-.. _tableau_connect_to_sqream_db:
-
-Connecting to SQream DB
-===========================
-
-#. Start Tableau Desktop.
-
-#. Select "More", by navigating :menuselection:`Connect --> To a server --> More`
-   
-   .. image:: /_static/images/tableau_more_servers_2.png
-
-#. Select "SQream DB by SQream Technologies"
-   
-   .. image:: /_static/images/tableau_more_servers_3.png
-
-#. Fill in the details for your SQream DB installation and click :menuselection:`Sign In`.
-   
-   .. image:: /_static/images/tableau_new_connection.png
-   
-
-.. list-table:: Connection parameters reference
-   :widths: auto
-   :header-rows: 1
-   
-   * - Item
-     - Description
-   * - Server
-     - Hostname of the SQream DB worker. For example, ``127.0.0.1`` or ``sqream.mynetwork.co``
-   * - Port
-     - TCP port of the SQream DB worker. For example, ``3108`` when using a load balancer or ``5100`` when connecting directly to a worker with SSL
-   * - Database
-     - Specifies the database name to connect to. For example, ``master``
-   * - Cluster
-     - Connect via load balancer. Accepts ``true`` and ``false``. Double check the connection port when setting this.
-   * - Username
-     - Username of a role to use for connection. For example, ``rhendricks``
-   * - Password
-     - Specifies the password of the selected role. For example, ``Tr0ub4dor&3``
-   * - Require SSL
-     - Specifies SSL for this connection
-
-
-Setting up SQream DB tables as data sources
-======================================================
-Once connected, you are taken to the data source page.
-
-The left side of the screen contains a database and schema drop-down. Select the database name and schema name you wish to use (``public`` is the default schema in SQream DB).
-
-   .. image:: /_static/images/tableau_data_sources.png
-
-Drag tables you wish to use to the main area, marked as **Drag tables here**. This is also where you specify joins and data source filters.
-
-When data source setup is completed, navigate to a new sheet to start analyzing data.
-
-.. tip:: 
-   * Read more about configuring data sources, joining, filtering, and more on `Tableau's Set Up Data Sources `_ tutorials.
-   * Rename the connection with a descriptive name for other users to understand. Alternatively, Tableau will generate a default name based on the DSN and tables.
-
-Tableau best practices and troubleshooting
-=================================================
-
-Cut out what you don't need
------------------------------
-
-* Bring only the data sources you need into Tableau. As a best practice, do not bring in tables that you don't intend to explore.
-
-* Add filters before exploring. Every change you make while exploring data will query SQream DB, sometimes several times. Add filters to the datasource before exploring, so that the queries sent to SQream DB run faster.
-
-Let Tableau create the queries
---------------------------------
-
-Create pre-optimized views (see :ref:`create_view`) and point the datasource at these views.
-
-In some cases, using views or custom SQL as a datasource can actually degrade performance. 
-
-We recommend testing performance of custom SQL and views, and compare with Tableau's generated SQL.
-
-Create a separate service for Tableau
----------------------------------------
-
-SQream recommends that Tableau get a separate service with the DWLM. This will reduce the impact of Tableau on other applications and processes, such as ETL.
-This works in conjunction with the load balancer to ensure good performance.
-
-
-Troubleshoot workbook performance before deploying to Tableau Server
------------------------------------------------------------------------
-
-Tableau has a built in `performance recorder `_ that shows how time is being spent. If you're seeing slow performance, this could be the result of a misconfiguration such as setting concurrency too low.
-
-Use the Tableau Performance Recorder to view the performance of the queries that Tableau runs. Using this information, you can identify queries that can be optimized with the use of views.
-
-Troubleshooting ``Error Code: 37CE01A3``, ``No suitable driver installed or the URL is incorrect``
---------------------------------------------------------------------------------------------------------
-
-In some cases, Tableau may have trouble finding the SQream DB JDBC driver. This message explains that the driver can't be found.
-
-To solve this issue, try two things:
-
-1. Verify that the JDBC driver was placed in the correct directory:
-
-   * Tableau Desktop on Windows: ``c:\Program Files\Tableau\Drivers``
-
-   * Tableau Desktop on MacOS: ``~/Library/Tableau/Drivers``
-
-   * Tableau on Linux: ``/opt/tableau/tableau_driver/jdbc``
-
-2. Find the file path for the JDBC driver and add it to the Java classpath:
-   
-   * On Linux, ``export CLASSPATH=;$CLASSPATH``
-   
-   * On Windows, add an envrionment variable for the classpath:
-   
-         .. image:: /_static/images/set_java_classpath.png
-
-If you're still experiencing issues after restarting Tableau, we're always happy to help. Visit `SQream's support portal `_ for additional support.
diff --git a/guides/third_party_tools/talend.rst b/guides/third_party_tools/talend.rst
deleted file mode 100644
index 2b678d1a1..000000000
--- a/guides/third_party_tools/talend.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. _talend:
-
-*************************
-Connecting to Talend
-*************************
-
-You can use Talend to read data from a SQream DB cluster, and load data into SQream DB. This tutorial is a guide that will show you how to connect Talend to SQream DB.
-
-
-.. contents:: In this topic:
-   :local:
diff --git a/index.rst b/index.rst
index c4499a443..e6cdb5be6 100644
--- a/index.rst
+++ b/index.rst
@@ -1,16 +1,16 @@
 .. _index:
 
 *************************
-SQream DB documentation
+SQream DB Documentation
 *************************
 
-For SQream DB |latest_version|.
+For SQream Version 2020.3.2.1.
 
 .. only:: html
 
    .. tip::
       Want to read this offline?
-      `Download the documentation as a single PDF `_ .
+      `Download the documentation as a single PDF `_ .
 
 .. only:: pdf or latex
    
@@ -31,34 +31,32 @@ SQream DB easily plugs in to third-party tools like :ref:`Tableau`
      - 
-         :ref:`SQL reference`
+         :ref:`SQL Reference`
          
          :ref:`sql_statements`
          
          :ref:`sql_functions`
      - 
-         :ref:`setup`
+         `Getting Set Up `_
          
          :ref:`Best practices`
          
-         :ref:`connect_to_tableau`
+         `Connecting to Tableau `_
 
    * - **Releases**
-     - **Driver and deployment**
-     - **Help & Support**
-   * -
-         :ref:`2020.3.1<2020.3.1>`
-
+     - **Driver and Deployment**
+     - **Help and Support**
+   * -      
          :ref:`2020.3<2020.3>`
 
          :ref:`2020.2<2020.2>`
@@ -70,13 +68,14 @@ SQream DB easily plugs in to third-party tools like :ref:`Tableau`
 
-         :ref:`Third party tools integration`
+         `Connecting to SQream `_
 
-         :ref:`connect_to_tableau`
      - 
-         :ref:`troubleshooting` guide
+         `Troubleshooting Page `_
          
-         :ref:`information_for_support`
+         `Gathering Information for SQream Support `_
+
+
 
 
 
@@ -87,7 +86,7 @@ If you couldn't find what you're looking for, we're always happy to help. Visit
 
 .. rubric:: Looking for older versions?
 
-This version of the documentation is for SQream DB |latest_version|.
+This version of the documentation is for SQream version 2020.3.2.1.
 
 If you're looking for an older version of the documentation, versions 1.10 through 2019.2.1 are available at http://previous.sqream.com .
 
@@ -98,11 +97,21 @@ If you're looking for an older version of the documentation, versions 1.10 throu
    :titlesonly:
    :hidden:
 
-   first_steps
-   xxfeatures_tour
-   guides/index
+   getting_started/index
+   installation_guides/index
+   data_ingestion/index
+   connecting_to_sqream/index
+   external_storage_platforms/index
+   loading_and_unloading_data/index
+   feature_guides/index
+   operational_guides/index
+   sqream_studio_5.4.3/index
+   architecture/index
+   configuration_guides/index
    reference/index
+   data_type_guides/index
    releases/index
+   troubleshooting/index
    glossary
 
 ..
diff --git a/installation_guides/index.rst b/installation_guides/index.rst
new file mode 100644
index 000000000..47f89c219
--- /dev/null
+++ b/installation_guides/index.rst
@@ -0,0 +1,15 @@
+.. _installation_guides:
+
+*************************
+Installation Guides
+*************************
+Before you get started using SQream, consider your business needs and available resources. SQream was designed to run in a number of environments, and to be installed using different methods depending on your requirements. This determines which installation method to use.
+
+The **Installation Guides** section describes the following installation guide sets:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   installing_and_launching_sqream
+   sqream_studio_installation
\ No newline at end of file
diff --git a/installation_guides/installing_and_launching_sqream.rst b/installation_guides/installing_and_launching_sqream.rst
new file mode 100644
index 000000000..7953f7cc3
--- /dev/null
+++ b/installation_guides/installing_and_launching_sqream.rst
@@ -0,0 +1,19 @@
+.. _installing_and_launching_sqream:
+
+*************************
+Installing and Launching SQream
+*************************
+The **Installing SQream Studio** page incudes the following installation guides:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   recommended_pre-installation_configurations
+   installing_sqream_with_binary
+   running_sqream_in_a_docker_container
+   installing_sqream_with_kubernetes
+   installing_monit
+
+
+
diff --git a/installation_guides/installing_dashboard_data_collector.rst b/installation_guides/installing_dashboard_data_collector.rst
new file mode 100644
index 000000000..04efc81fe
--- /dev/null
+++ b/installation_guides/installing_dashboard_data_collector.rst
@@ -0,0 +1,161 @@
+.. _installing_dashboard_data_collector:
+
+
+
+***********************
+Installing the Dashboard Data Collector
+***********************
+
+Installing the Dashboard Data Collector
+^^^^^^^^^^^^^^^
+After accessing the Prometheus user interface, you can install the **Dashboard Data Collector**. You must install the Dashboard Data Collector to enable the Dashboard in Studio.
+
+.. note:: Before installing the Dashboard Data collector, verify that Prometheus has been installed and configured for the cluster.
+
+**To install the Dashboard Data Collector:**
+
+1. Store the Data Collector Package obtained from `SQream Artifactory `_.
+
+  ::
+
+2. Extract and rename the package:
+
+   .. code-block:: console
+   
+      $ tar -xvf dashboard-data-collector-0.5.2.tar.gz 
+      $ mv package dashboard-data-collector
+	  
+3. Change your directory to the location of the package folder: 
+
+   .. code-block:: console
+   
+      $ cd dashboard-data-collector
+
+4. Set up the data collection by modifying the SQream and Data Collector IPs, ports, user name, and password according to the cluster:
+
+   .. code-block:: console
+   
+      $ npm run setup -- \
+      $ 	--host=127.0.0.1 \
+      $ 	--port=3108 \
+      $ 	--database=master \
+      $ 	--is-cluster=true \
+      $ 	--service=sqream \
+      $ 	--dashboard-user=sqream \
+      $ 	--dashboard-password=sqream \
+      $ 	--prometheus-url=http://127.0.0.1:9090/api/v1/query
+
+5. Debug the Data Collector:
+
+   .. code-block:: console
+   
+      $ npm start
+
+   A json file is generated in the log, as shown below:   
+
+   .. code-block:: console
+   
+      $ {
+      $   "machines": [
+      $     {
+      $       "machineId": "dd4af489615",
+      $       "name": "Server 0",
+      $       "location": "192.168.4.94",
+      $       "totalMemory": 31.19140625,
+      $       "gpus": [
+      $         {
+      $           "gpuId": "GPU-b17575ec-eeba-3e0e-99cd-963967e5ee3f",
+      $           "machineId": "dd4af489615",
+      $           "name": "GPU 0",
+      $           "totalMemory": 3.9453125
+      $         }
+      $       ],
+      $       "workers": [
+      $         {
+      $           "workerId": "sqream_01",
+      $           "gpuId": "",
+      $           "name": "sqream_01"
+      $         }
+      $       ],
+      $       "storageWrite": 0,
+      $       "storageRead": 0,
+      $       "freeStorage": 0
+      $     },
+      $     {
+      $       "machineId": "704ec607174",
+      $       "name": "Server 1",
+      $       "location": "192.168.4.95",
+      $       "totalMemory": 31.19140625,
+      $       "gpus": [
+      $         {
+      $           "gpuId": "GPU-8777c14f-7611-517a-e9c7-f42eeb21700b",
+      $           "machineId": "704ec607174",
+      $           "name": "GPU 0",
+      $           "totalMemory": 3.9453125
+      $         }
+      $       ],
+      $       "workers": [
+      $         {
+      $           "workerId": "sqream_02",
+      $           "gpuId": "",
+      $           "name": "sqream_02"
+      $         }
+      $       ],
+      $       "storageWrite": 0,
+      $       "storageRead": 0,
+      $       "freeStorage": 0
+      $     }
+      $   ],
+      $   "clusterStatus": true,
+      $   "storageStatus": {
+      $     "dataStorage": 49.9755859375,
+      $     "totalDiskUsage": 52.49829018075231,
+      $     "storageDetails": {
+      $       "data": 0,
+      $       "freeData": 23.7392578125,
+      $       "tempData": 0,
+      $       "deletedData": 0,
+      $       "other": 26.236328125
+      $     },
+      $     "avgThroughput": {
+      $       "read": 0,
+      $       "write": 0
+      $     },
+      $     "location": "/"
+      $   },
+      $   "queues": [
+      $     {
+      $       "queueId": "sqream",
+      $       "name": "sqream",
+      $       "workerIds": [
+      $         "sqream_01",
+      $         "sqream_02"
+      $       ]
+      $     }
+      $   ],
+      $   "queries": [],
+      $   "collected": true,
+      $   "lastCollect": "2021-11-17T12:46:31.601Z"
+      $ }
+	  
+.. note:: Verify that all machines and workers are correctly registered.
+
+
+6. Press **CTRL + C** to stop ``npm start``.
+
+  ::
+
+
+7. Start the Data Collector with the ``pm2`` service:
+
+   .. code-block:: console
+   
+      $ pm2 start ./index.js --name=dashboard-data-collector
+	  
+8. Add the following parameter to the SQream Studio setup defined in :ref:`Step 4` in **Installing Studio** below.
+
+   .. code-block:: console
+   
+      --data-collector-url=http://127.0.0.1:8100/api/dashboard/data
+
+Back to :ref:`Installing Studio on a Stand-Alone Server`
diff --git a/installation_guides/installing_monit.rst b/installation_guides/installing_monit.rst
new file mode 100644
index 000000000..b27800cce
--- /dev/null
+++ b/installation_guides/installing_monit.rst
@@ -0,0 +1,319 @@
+.. _installing_monit:
+
+*********************************************
+Installing Monit
+*********************************************
+
+Getting Started
+==============================
+
+Before installing SQream with Monit, verify that you have followed the required :ref:`recommended pre-installation configurations `.
+
+The procedures in the **Installing Monit** guide must be performed on each SQream cluster node.
+
+
+
+
+
+.. _back_to_top:
+
+Overview
+==============================
+
+
+Monit is a free open source supervision utility for managing and monitoring Unix and Linux. Monit lets you view system status directly from the command line or from a native HTTP web server. Monit can be used to conduct automatic maintenance and repair, such as executing meaningful causal actions in error situations.
+
+SQream uses Monit as a watchdog utility, but you can use any other utility that provides the same or similar functionality.
+
+The **Installing Monit** procedures describes how to install, configure, and start Monit.
+
+You can install Monit in one of the following ways:
+
+* :ref:`Installing Monit on CentOS `
+* :ref:`Installing Monit on CentOS offline `
+* :ref:`Installing Monit on Ubuntu `
+* :ref:`Installing Monit on Ubuntu offline `
+ 
+ 
+ 
+
+
+
+
+.. _installing-monit-on-centos:
+
+Installing Monit on CentOS:
+------------------------------------
+
+
+
+**To install Monit on CentOS:**   
+   
+1. Install Monit as a superuser on CentOS:
+ 
+    .. code-block:: console
+     
+       $ sudo yum install monit  
+       
+       
+.. _installing-monit-on-centos-offline:
+
+
+	   
+Installing Monit on CentOS Offline:
+------------------------------------
+
+
+Installing Monit on CentOS offline can be done in either of the following ways:
+
+* :ref:`Building Monit from Source Code `
+* :ref:`Building Monit from Pre-Built Binaries `
+
+ 
+ 
+ 
+.. _building_monit_from_source_code:
+
+Building Monit from Source Code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+
+**To build Monit from source code:**
+
+1. Copy the Monit package for the current version:
+       
+   .. code-block:: console
+     
+      $ tar zxvf monit-.tar.gz
+       
+ The value ``x.y.z`` denotes the version numbers.
+       
+2. Navigate to the directory where you want to store the package:
+
+   .. code-block:: console
+     
+      $ cd monit-x.y.z
+ 
+3. Configure the files in the package:
+
+   .. code-block:: console
+     
+      $ ./configure (use ./configure --help to view available options)
+ 
+4. Build and install the package:
+
+   .. code-block:: console
+     
+      $ make && make install
+      
+The following are the default storage directories:
+
+* The Monit package: **/usr/local/bin/**
+* The **monit.1 man-file**: **/usr/local/man/man1/**
+
+5. **Optional** - To change the above default location(s), use the **--prefix** option to ./configure.
+
+..
+  _**Comment - I took this line directly from the external online documentation. Is the "prefix option" referrin gto the "--help" in Step 3? URL: https://mmonit.com/wiki/Monit/Installation**
+
+6. **Optional** - Create an RPM package for CentOS directly from the source code:
+
+   .. code-block:: console
+     
+      $ rpmbuild -tb monit-x.y.z.tar.gz
+      
+..
+  _**Comment - Is this an optional or mandatory step?**
+
+ 
+
+
+.. _building_monit_from_pre_built_binaries:   
+
+Building Monit from Pre-Built Binaries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**To build Monit from pre-built binaries:**
+
+1. Copy the Monit package for the current version:
+       
+   .. code-block:: console
+
+      $ tar zxvf monit-x.y.z-linux-x64.tar.gz
+      
+   The value ``x.y.z`` denotes the version numbers.
+
+2. Navigate to the directory where you want to store the package:
+
+   .. code-block:: console$ cd monit-x.y.z
+   
+3. Copy the **bin/monit** and **/usr/local/bin/** directories:
+ 
+    .. code-block:: console
+
+      $ cp bin/monit /usr/local/bin/
+ 
+4. Copy the **conf/monitrc** and **/etc/** directories:
+ 
+    .. code-block:: console
+
+      $ cp conf/monitrc /etc/
+       
+..
+  _**Comment - please review this procedure.**
+
+For examples of pre-built Monit binarties, see :ref:`Download Precompiled Binaries`.
+
+:ref:`Back to top `
+
+
+
+.. _installing-monit-on-ubuntu:
+
+
+      
+Installing Monit on Ubuntu:
+------------------------------------
+
+
+**To install Monit on Ubuntu:**   
+   
+1. Install Monit as a superuser on Ubuntu:
+
+    .. code-block:: console
+     
+       $ sudo apt-get install monit
+	   
+:ref:`Back to top `
+
+
+	   
+.. _installing-monit-on-ubuntu-offline:
+
+
+Installing Monit on Ubuntu Offline:
+-------------------------------------
+
+
+You can install Monit on Ubuntu when you do not have an internet connection.
+
+**To install Monit on Ubuntu offline:**   
+   
+1. Compress the required file:
+
+   .. code-block:: console
+     
+      $ tar zxvf monit--linux-x64.tar.gz
+      
+   **NOTICE:** ** denotes the version number.
+
+2. Navigate to the directory where you want to save the file:
+   
+   .. code-block:: console
+     
+      $ cd monit-x.y.z
+       
+3. Copy the **bin/monit** directory into the **/usr/local/bin/** directory:
+
+   .. code-block:: console
+     
+      $ cp bin/monit /usr/local/bin/
+       
+4. Copy the **conf/monitrc** directory into the **/etc/** directory:
+       
+   .. code-block:: console
+     
+      $ cp conf/monitrc /etc/
+	  
+:ref:`Back to top `
+
+       
+Configuring Monit
+====================================
+
+When the installation is complete, you can configure Monit. You configure Monit by modifying the Monit configuration file, called **monitrc**. This file contains blocks for each service that you want to monitor.
+
+The following is an example of a service block:
+
+    .. code-block:: console
+     
+       $ #SQREAM1-START
+       $ check process sqream1 with pidfile /var/run/sqream1.pid
+       $ start program = "/usr/bin/systemctl start sqream1"
+       $ stop program = "/usr/bin/systemctl stop sqream1"
+       $ #SQREAM1-END
+
+For example, if you have 16 services, you can configure this block by copying the entire block 15 times and modifying all service names as required, as shown below:
+
+    .. code-block:: console
+     
+       $ #SQREAM2-START
+       $ check process sqream2 with pidfile /var/run/sqream2.pid
+       $ start program = "/usr/bin/systemctl start sqream2"
+       $ stop program = "/usr/bin/systemctl stop sqream2"
+       $ #SQREAM2-END
+       
+For servers that don't run the **metadataserver** and **serverpicker** commands, you can use the block example above, but comment out the related commands, as shown below:
+
+    .. code-block:: console
+     
+       $ #METADATASERVER-START
+       $ #check process metadataserver with pidfile /var/run/metadataserver.pid
+       $ #start program = "/usr/bin/systemctl start metadataserver"
+       $ #stop program = "/usr/bin/systemctl stop metadataserver"
+       $ #METADATASERVER-END
+
+**To configure Monit:**   
+   
+1. Copy the required block for each required service.
+2. Modify all service names in the block.
+3. Copy the configured **monitrc** file to the **/etc/monit.d/** directory:
+
+   .. code-block:: console
+     
+      $ cp monitrc /etc/monit.d/
+       
+4. Set file permissions to **600** (full read and write access):
+ 
+    .. code-block:: console
+
+       $ sudo chmod 600 /etc/monit.d/monitrc
+       
+5. Reload the system to activate the current configurations:
+ 
+    .. code-block:: console
+     
+       $ sudo systemctl daemon-reload
+ 
+6. **Optional** - Navigate to the **/etc/sqream** directory and create a symbolic link to the **monitrc** file:
+ 
+    .. code-block:: console
+     
+      $ cd /etc/sqream
+      $ sudo ln -s /etc/monit.d/monitrc monitrc    
+         
+Starting Monit
+====================================  
+
+After configuring Monit, you can start it.
+
+**To start Monit:**
+
+1. Start Monit as a super user:
+
+   .. code-block:: console
+     
+      $ sudo systemctl start monit   
+ 
+2. View Monit's service status:
+
+   .. code-block:: console
+     
+      $ sudo systemctl status monit
+
+3. If Monit is functioning correctly, enable the Monit service to start on boot:
+    
+   .. code-block:: console
+     
+      $ sudo systemctl enable monit
diff --git a/installation_guides/installing_nginx_proxy_over_secure_connection.rst b/installation_guides/installing_nginx_proxy_over_secure_connection.rst
new file mode 100644
index 000000000..5aef5eaff
--- /dev/null
+++ b/installation_guides/installing_nginx_proxy_over_secure_connection.rst
@@ -0,0 +1,403 @@
+.. _installing_nginx_proxy_over_secure_connection:
+
+*************************
+Installing an NGINX Proxy Over a Secure Connection
+*************************
+Configuring your NGINX server to use a strong encryption for client connections provides you with secure servers requests, preventing outside parties from gaining access to your traffic.
+
+The **Installing an NGINX Proxy Over a Secure Connection** page describes the following:
+
+.. contents::
+   :local:
+   :depth: 1
+
+Overview
+==============
+The Node.js platform that SQream uses with our Studio user interface is susceptible to web exposure. This page describes how to implement HTTPS access on your proxy server to establish a secure connection.
+
+**TLS (Transport Layer Security)**, and its predecessor **SSL (Secure Sockets Layer)**, are standard web protocols used for wrapping normal traffic in a protected, encrypted wrapper. This technology prevents the interception of server-client traffic. It also uses a certificate system for helping users verify the identity of sites they visit. The **Installing an NGINX Proxy Over a Secure Connection** guide describes how to set up a self-signed SSL certificate for use with an NGINX web server on a CentOS 7 server.
+
+.. note:: A self-signed certificate encrypts communication between your server and any clients. However, because it is not signed by trusted certificate authorities included with web browsers, you cannot use the certificate to automatically validate the identity of your server.
+
+A self-signed certificate may be appropriate if your domain name is not associated with your server, and in cases where your encrypted web interface is not user-facing. If you do have a domain name, using a CA-signed certificate is generally preferrable.
+
+For more information on setting up a free trusted certificate, see `How To Secure Nginx with Let's Encrypt on CentOS 7 `_.
+
+Prerequisites
+==============
+The following prerequisites are required for installing an NGINX proxy over a secure connection:
+
+* Super user privileges
+
+   ::
+   
+* A domain name to create a certificate for
+
+Installing NGINX and Adjusting the Firewall
+==============
+After verifying that you have the above preriquisites, you must verify that the NGINX web server has been installed on your machine.
+
+Though NGINX is not available in the default CentOS repositories, it is available from the **EPEL (Extra Packages for Enterprise Linux)** repository.
+
+**To install NGINX and adjust the firewall:**
+
+1. Enable the EPEL repository to enable server access to the NGINX package:
+
+   .. code-block:: console
+
+      $ sudo yum install epel-release
+
+2. Install NGINX:
+
+   .. code-block:: console
+
+      $ sudo yum install nginx
+ 
+3. Start the NGINX service:
+
+   .. code-block:: console
+
+      $ sudo systemctl start nginx
+ 
+4. Verify that the service is running:
+
+   .. code-block:: console
+
+      $ systemctl status nginx
+
+   The following is an example of the correct output:
+
+   .. code-block:: console
+
+      Output● nginx.service - The nginx HTTP and reverse proxy server
+         Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
+         Active: active (running) since Fri 2017-01-06 17:27:50 UTC; 28s ago
+
+      . . .
+
+      Jan 06 17:27:50 centos-512mb-nyc3-01 systemd[1]: Started The nginx HTTP and reverse proxy server.
+
+5. Enable NGINX to start when your server boots up:
+
+   .. code-block:: console
+
+      $ sudo systemctl enable nginx
+ 
+6. Verify that access to **ports 80 and 443** are not blocked by a firewall.
+
+    ::
+	
+7. Do one of the following:
+
+   * If you are not using a firewall, skip to :ref:`Creating Your SSL Certificate`.
+
+      ::
+	  
+   * If you have a running firewall, open ports 80 and 443:
+
+     .. code-block:: console
+
+        $ sudo firewall-cmd --add-service=http
+        $ sudo firewall-cmd --add-service=https
+        $ sudo firewall-cmd --runtime-to-permanent 
+
+8. If you have a running **iptables firewall**, for a basic rule set, add HTTP and HTTPS access:
+
+   .. code-block:: console
+
+      $ sudo iptables -I INPUT -p tcp -m tcp --dport 80 -j ACCEPT
+      $ sudo iptables -I INPUT -p tcp -m tcp --dport 443 -j ACCEPT
+
+   .. note:: The commands in Step 8 above are highly dependent on your current rule set.
+
+9. Verify that you can access the default NGINX page from a web browser.
+
+.. _creating_your_ssl_certificate:
+
+Creating Your SSL Certificate
+==============
+After installing NGINX and adjusting your firewall, you must create your SSL certificate.
+
+TLS/SSL combines public certificates with private keys. The SSL key, kept private on your server, is used to encrypt content sent to clients, while the SSL certificate is publicly shared with anyone requesting content. In addition, the SSL certificate can be used to decrypt the content signed by the associated SSL key. Your public certificate is located in the **/etc/ssl/certs** directory on your server.
+
+This section describes how to create your **/etc/ssl/private directory**, used for storing your private key file. Because the privacy of this key is essential for security, the permissions must be locked down to prevent unauthorized access:
+
+**To create your SSL certificate:**
+
+1. Set the following permissions to **private**:
+
+   .. code-block:: console
+
+      $ sudo mkdir /etc/ssl/private
+      $ sudo chmod 700 /etc/ssl/private
+ 
+2. Create a self-signed key and certificate pair with OpenSSL with the following command:
+
+   .. code-block:: console
+
+      $ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt
+ 
+   The following list describes the elements in the command above:
+   
+   * **openssl** - The basic command line tool used for creating and managing OpenSSL certificates, keys, and other files.
+   
+    ::
+
+   * **req** - A subcommand for using the X.509 **Certificate Signing Request (CSR)** management. A public key infrastructure standard, SSL and TLS adhere X.509 key and certificate management regulations.
+
+    ::
+
+   * **-x509** - Used for modifying the previous subcommand by overriding the default functionality of generating a certificate signing request with making a self-signed certificate.
+
+    ::
+
+   * **-nodes** - Sets **OpenSSL** to skip the option of securing our certificate with a passphrase, letting NGINX read the file without user intervention when the server is activated. If you don't use **-nodes** you must enter your passphrase after every restart.
+
+    ::
+
+   * **-days 365** - Sets the certificate's validation duration to one year.
+
+    ::
+
+   * **-newkey rsa:2048** - Simultaneously generates a new certificate and new key. Because the key required to sign the certificate was not created in the previous step, it must be created along with the certificate. The **rsa:2048** generates an RSA 2048 bits long.
+
+    ::
+
+   * **-keyout** - Determines the location of the generated private key file.
+
+    ::
+
+   * **-out** - Determines the location of the certificate.
+
+  After creating a self-signed key and certificate pair with OpenSSL, a series of prompts about your server is presented to correctly embed the information you provided in the certificate.
+
+3. Provide the information requested by the prompts.
+
+   The most important piece of information is the **Common Name**, which is either the server **FQDN** or **your** name. You must enter the domain name associated with your server or your server’s public IP address.
+
+   The following is an example of a filled out set of prompts:
+
+   .. code-block:: console
+
+      OutputCountry Name (2 letter code) [AU]:US
+      State or Province Name (full name) [Some-State]:New York
+      Locality Name (eg, city) []:New York City
+      Organization Name (eg, company) [Internet Widgits Pty Ltd]:Bouncy Castles, Inc.
+      Organizational Unit Name (eg, section) []:Ministry of Water Slides
+      Common Name (e.g. server FQDN or YOUR name) []:server_IP_address
+      Email Address []:admin@your_domain.com
+
+   Both files you create are stored in their own subdirectories of the **/etc/ssl** directory.
+
+   Although SQream uses OpenSSL, in addition we recommend creating a strong **Diffie-Hellman** group, used for negotiating **Perfect Forward Secrecy** with clients.
+   
+4. Create a strong Diffie-Hellman group:
+
+   .. code-block:: console
+
+      $ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
+ 
+   Creating a Diffie-Hellman group takes a few minutes, which is stored as the **dhparam.pem** file in the **/etc/ssl/certs** directory. This file can use in the configuration.
+   
+Configuring NGINX to use SSL
+==============
+After creating your SSL certificate, you must configure NGINX to use SSL.
+
+The default CentOS NGINX configuration is fairly unstructured, with the default HTTP server block located in the main configuration file. NGINX checks for files ending in **.conf** in the **/etc/nginx/conf.d** directory for additional configuration.
+
+SQream creates a new file in the **/etc/nginx/conf.d** directory to configure a server block. This block serves content using the certificate files we generated. In addition, the default server block can be optionally configured to redirect HTTP requests to HTTPS.
+
+.. note:: The example on this page uses the IP address **127.0.0.1**, which you should replace with your machine's IP address.
+
+**To configure NGINX to use SSL:**
+
+1. Create and open a file called **ssl.conf** in the **/etc/nginx/conf.d** directory:
+
+   .. code-block:: console
+
+      $ sudo vi /etc/nginx/conf.d/ssl.conf
+
+2. In the file you created in Step 1 above, open a server block:
+
+   1. Listen to **port 443**, which is the TLS/SSL default port.
+   
+       ::
+   
+   2. Set the ``server_name`` to the server’s domain name or IP address you used as the Common Name when generating your certificate.
+   
+       ::
+	   
+   3. Use the ``ssl_certificate``, ``ssl_certificate_key``, and ``ssl_dhparam`` directives to set the location of the SSL files you generated, as shown in the **/etc/nginx/conf.d/ssl.conf** file below:
+   
+   .. code-block:: console
+
+          upstream ui {
+              server 127.0.0.1:8080;
+          }
+      server {
+          listen 443 http2 ssl;
+          listen [::]:443 http2 ssl;
+
+          server_name nginx.sq.l;
+
+          ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
+          ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
+          ssl_dhparam /etc/ssl/certs/dhparam.pem;
+
+      root /usr/share/nginx/html;
+
+      #    location / {
+      #    }
+
+        location / {
+              proxy_pass http://ui;
+              proxy_set_header           X-Forwarded-Proto https;
+              proxy_set_header           X-Forwarded-For $proxy_add_x_forwarded_for;
+              proxy_set_header           X-Real-IP       $remote_addr;
+              proxy_set_header           Host $host;
+                      add_header                 Front-End-Https   on;
+              add_header                 X-Cache-Status $upstream_cache_status;
+              proxy_cache                off;
+              proxy_cache_revalidate     off;
+              proxy_cache_min_uses       1;
+              proxy_cache_valid          200 302 1h;
+              proxy_cache_valid          404 3s;
+              proxy_cache_use_stale      error timeout invalid_header updating http_500 http_502 http_503 http_504;
+              proxy_no_cache             $cookie_nocache $arg_nocache $arg_comment $http_pragma $http_authorization;
+              proxy_redirect             default;
+              proxy_max_temp_file_size   0;
+              proxy_connect_timeout      90;
+              proxy_send_timeout         90;
+              proxy_read_timeout         90;
+              proxy_buffer_size          4k;
+              proxy_buffering            on;
+              proxy_buffers              4 32k;
+              proxy_busy_buffers_size    64k;
+              proxy_temp_file_write_size 64k;
+              proxy_intercept_errors     on;
+
+              proxy_set_header           Upgrade $http_upgrade;
+              proxy_set_header           Connection "upgrade";
+          }
+
+          error_page 404 /404.html;
+          location = /404.html {
+          }
+
+          error_page 500 502 503 504 /50x.html;
+          location = /50x.html {
+          }
+      }
+ 
+4. Open and modify the **nginx.conf** file located in the **/etc/nginx/conf.d** directory as follows:
+
+   .. code-block:: console
+
+      $ sudo vi /etc/nginx/conf.d/nginx.conf
+	 
+   .. code-block:: console      
+
+       server {
+           listen       80;
+           listen       [::]:80;
+           server_name  _;
+           root         /usr/share/nginx/html;
+
+           # Load configuration files for the default server block.
+           include /etc/nginx/default.d/*.conf;
+
+           error_page 404 /404.html;
+           location = /404.html {
+           }
+
+           error_page 500 502 503 504 /50x.html;
+           location = /50x.html {
+           }
+       }
+	   
+Redirecting Studio Access from HTTP to HTTPS
+==================
+After configuring NGINX to use SSL, you must redirect Studio access from HTTP to HTTPS.
+
+According to your current configuration, NGINX responds with encrypted content for requests on port 443, but with **unencrypted** content for requests on **port 80**. This means that our site offers encryption, but does not enforce its usage. This may be fine for some use cases, but it is usually better to require encryption. This is especially important when confidential data like passwords may be transferred between the browser and the server.
+
+The default NGINX configuration file allows us to easily add directives to the default port 80 server block by adding files in the /etc/nginx/default.d directory.
+
+**To create a redirect from HTTP to HTTPS:**
+
+1. Create a new file called **ssl-redirect.conf** and open it for editing:
+
+   .. code-block:: console
+
+      $ sudo vi /etc/nginx/default.d/ssl-redirect.conf
+
+2. Copy and paste this line:
+
+   .. code-block:: console
+
+      $ return 301 https://$host$request_uri:8080/;
+	  
+Activating Your NGINX Configuration
+==============
+After redirecting from HTTP to HTTPs, you must restart NGINX to activate your new configuration.
+
+**To activate your NGINX configuration:**
+
+1. Verify that your files contain no syntax errors:
+
+   .. code-block:: console
+
+      $ sudo nginx -t
+   
+   The following output is generated if your files contain no syntax errors:
+
+   .. code-block:: console
+
+      nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
+      nginx: configuration file /etc/nginx/nginx.conf test is successful
+
+2. Restart NGINX to activate your configuration:
+
+   .. code-block:: console
+
+      $ sudo systemctl restart nginx
+
+Verifying that NGINX is Running
+==============
+After activating your NGINX configuration, you must verify that NGINX is running correctly.
+
+**To verify that NGINX is running correctly:**
+
+1. Check that the service is up and running:
+
+   .. code-block:: console
+
+      $ systemctl status nginx
+  
+   The following is an example of the correct output:
+
+   .. code-block:: console
+
+      Output● nginx.service - The nginx HTTP and reverse proxy server
+         Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
+         Active: active (running) since Fri 2017-01-06 17:27:50 UTC; 28s ago
+
+      . . .
+
+      Jan 06 17:27:50 centos-512mb-nyc3-01 systemd[1]: Started The nginx HTTP and reverse proxy server.
+ 
+2. Run the following command:
+
+   .. code-block:: console
+
+      $ sudo netstat -nltp |grep nginx
+ 
+   The following is an example of the correct output:
+
+   .. code-block:: console
+
+      [sqream@dorb-pc etc]$ sudo netstat -nltp |grep nginx
+      tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      15486/nginx: master 
+      tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      15486/nginx: master 
+      tcp6       0      0 :::80                   :::*                    LISTEN      15486/nginx: master 
+      tcp6       0      0 :::443                  :::*                    LISTEN      15486/nginx: master
\ No newline at end of file
diff --git a/installation_guides/installing_prometheus_exporters.rst b/installation_guides/installing_prometheus_exporters.rst
new file mode 100644
index 000000000..a1381fea3
--- /dev/null
+++ b/installation_guides/installing_prometheus_exporters.rst
@@ -0,0 +1,196 @@
+.. _installing_prometheus_exporters:
+
+*********************************************
+Installing Prometheus Exporter
+*********************************************
+
+The **Installing Prometheus Exporters** guide includes the following sections:
+
+.. contents::
+   :local:
+   :depth: 1
+
+Overview
+==============================
+The **Prometheus** exporter is an open-source systems monitoring and alerting toolkit. It is used for collecting metrics from an operating system and exporting them to a graphic user interface. 
+
+The Installing Prometheus Exporters guide describes how to installing the following exporters:
+
+* The **Node_exporter** - the basic exporter used for displaying server metrics, such as CPU and memory.
+
+* The **Nvidia_exporter** - shows Nvidia GPU metrics.
+
+* The **process_exporter** - shows data belonging to the server's running processes.
+
+For information about more exporters, see `Exporters and Integration `_
+
+Adding a User and Group
+=====================
+Adding a user and group determines who can run processes.
+
+You can add users with the following command:
+
+.. code-block:: console
+     
+   $ sudo groupadd --system prometheus
+	  
+You can add groups with the following command:
+
+.. code-block:: console
+     
+   $ sudo useradd -s /sbin/nologin --system -g prometheus prometheus
+
+Cloning the Prometheus GIT Project
+=====================
+After adding a user and group you must clone the Prometheus GIT project.
+
+You can clone the Prometheus GIT project with the following command:
+
+.. code-block:: console
+     
+   $ git clone http://gitlab.sq.l/IT/promethues.git prometheus
+	  
+.. note:: If you experience difficulties cloning the Prometheus GIT project or receive an error, contact your IT department.
+
+The following shows the result of cloning your Prometheus GIT project:
+
+.. code-block:: console
+     
+   $ prometheus/
+   $ ├── node_exporter
+   $ │   └── node_exporter
+   $ ├── nvidia_exporter
+   $ │   └── nvidia_exporter
+   $ ├── process_exporter
+   $ │   └── process-exporter_0.5.0_linux_amd64.rpm
+   $ ├── README.md
+   $ └── services
+   $     ├── node_exporter.service
+   $     └── nvidia_exporter.service	  
+	  
+Installing the Node Exporter and NVIDIA Exporter
+=====================
+After cloning the Prometheus GIT project you must install the **node_exporter** and **NVIDIA_exporter**.
+
+**To install the node_exporter and NVIDIA_exporter:**
+
+1. Navigate to the cloned folder:
+
+   .. code-block:: console
+     
+      $ cd prometheus
+   
+2. Copy **node_exporter** and **nvidia_exporter** to **/usr/bin/**.	  
+
+   .. code-block:: console
+     
+      $ sudo cp node_exporter/node_exporter /usr/bin/
+      $ sudo cp nvidia_exporter/nvidia_exporter /usr/bin/
+   	  
+3. Copy the **services** files to the services folder:	  
+
+   .. code-block:: console
+     
+      $ sudo cp services/node_exporter.service /etc/systemd/system/
+      $ sudo cp services/nvidia_exporter.service /etc/systemd/system/
+   	  
+4. Reload the services so that they can be run:	  
+
+   .. code-block:: console
+     
+      $ sudo systemctl daemon-reload  
+   	  
+5. Set the permissions and group for both service files:
+
+   .. code-block:: console
+     
+      $ sudo chown prometheus:prometheus /usr/bin/node_exporter
+      $ sudo chmod u+x /usr/bin/node_exporter
+      $ sudo chown prometheus:prometheus /usr/bin/nvidia_exporter
+      $ sudo chmod u+x /usr/bin/nvidia_exporter
+   
+6. Start both services:
+
+   .. code-block:: console
+     
+      $ sudo systemctl start node_exporter && sudo systemctl enable node_exporter
+   
+7. Set both services to start automatically when the server is booted up:
+
+   .. code-block:: console
+
+      $ sudo systemctl start nvidia_exporter && sudo systemctl enable nvidia_exporter
+   
+8. Verify that the server's status is **active (running)**:
+
+   .. code-block:: console
+     
+      $ sudo systemctl status node_exporter && sudo systemctl status nvidia_exporter
+   
+   The following is the correct output:
+
+   .. code-block:: console
+     
+      $ ● node_exporter.service - Node Exporter
+      $    Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
+      $    Active: active (running) since Wed 2019-12-11 12:28:31 IST; 1 months 5 days ago
+      $  Main PID: 28378 (node_exporter)
+      $    CGroup: /system.slice/node_exporter.service
+      $ 
+      $ ● nvidia_exporter.service - Nvidia Exporter
+      $    Loaded: loaded (/etc/systemd/system/nvidia_exporter.service; enabled; vendor preset: disabled)
+      $    Active: active (running) since Wed 2020-01-22 13:40:11 IST; 31min ago
+      $  Main PID: 1886 (nvidia_exporter)
+      $    CGroup: /system.slice/nvidia_exporter.service
+      $            └─1886 /usr/bin/nvidia_exporter
+   	  
+Installing the Process Exporter
+=====================
+After installing the **node_exporter** and **Nvidia_exporter** you must install the **process_exporter**.
+
+**To install the process_exporter:**
+
+1. Do one of the following:
+
+   * For **CentOS**, run ``sudo rpm -i process_exporter/process-exporter_0.5.0_linux_amd64.rpm``.
+   * For **Ubuntu**, run ``sudo dpkg -i process_exporter/process-exporter_0.6.0_linux_amd64.deb``.
+   
+2. Verify that the process_exporter is running:
+
+   .. code-block:: console
+     
+      $ sudo systemctl status process-exporter  
+	  
+3. Set the process_exporter to start automatically when the server is booted up:
+	  
+   .. code-block:: console
+     
+      $ sudo systemctl enable process-exporter
+	  
+Opening the Firewall Ports
+=====================
+After installing the **process_exporter** you must open the firewall ports for the following services:
+
+* **node_exporter** - port: 9100
+
+* **nvidia_exporter** - port: 9445
+
+* **process-exporter** - port: 9256
+
+.. note:: This procedure is only relevant if your firwall is running.
+
+**To open the firewall ports:**
+
+1. Run the following command:
+	  
+   .. code-block:: console
+     
+      $ sudo firewall-cmd --zone=public --add-port=/tcp --permanent
+	  
+2. Reload the firewall:
+	  
+   .. code-block:: console
+     
+      $ sudo firewall-cmd --reload
+	  
+3. Verify that the changes have taken effect.
\ No newline at end of file
diff --git a/installation_guides/installing_prometheus_using_binary_packages.rst b/installation_guides/installing_prometheus_using_binary_packages.rst
new file mode 100644
index 000000000..1512abfe7
--- /dev/null
+++ b/installation_guides/installing_prometheus_using_binary_packages.rst
@@ -0,0 +1,235 @@
+.. _installing_prometheus_using_binary_packages:
+
+.. _install_prometheus_binary_top:
+
+***********************
+Installing Prometheus Using Binary Packages
+***********************
+
+
+
+The **Installing Prometheus Using Binary Packages** guide includes the following sections:
+
+.. contents::
+   :local:
+   :depth: 1
+
+Overview
+^^^^^^^^^^^^^^^
+Prometheus is an application used for event monitoring and alerting.
+
+Installing Prometheus
+^^^^^^^^^^^^^^^
+You must install Prometheus before installing the Dashboard Data Collector.
+
+**To install Prometheus:**
+
+1. Verify the following:
+
+   1. That you have **sudo** access to your Linux server.
+   2. That your server has access to the internet (for downloading the Prometheus binary package).
+   3. That your firewall rules are opened for accessing Prometheus Port 9090.
+   
+2. Navigate to the Prometheus `Download `_ page and download the **prometheus-2.32.0-rc.1.linux-amd64.tar.gz** package.
+
+    ::
+
+3. Do the following:
+
+   1. Download the source using the ``curl`` command:
+
+      .. code-block:: console
+     
+         $ curl -LO url -LO https://github.com/prometheus/prometheus/releases/download/v2.22.0/prometheus-2.22.0.linux-amd64.tar.gz
+
+   2. Extract the file contents:
+
+      .. code-block:: console
+     
+         $ tar -xvf prometheus-2.22.0.linux-amd64.tar.gz
+
+   3. Rename the extracted folder **prometheus-files**:
+
+      .. code-block:: console
+     
+         $ mv prometheus-2.22.0.linux-amd64 prometheus-files
+
+4. Create a Prometheus user:
+
+   .. code-block:: console
+     
+      $ sudo useradd --no-create-home --shell /bin/false prometheus
+
+5. Create your required directories:
+
+   .. code-block:: console
+     
+      $ sudo mkdir /etc/prometheus
+      $ sudo mkdir /var/lib/prometheus
+	  
+6. Set the Prometheus user as the owner of your required directories:
+
+   .. code-block:: console
+     
+      $ sudo chown prometheus:prometheus /etc/prometheus
+      $ sudo chown prometheus:prometheus /var/lib/prometheus
+	  
+7. Copy the Prometheus and Promtool binary packages from the **prometheus-files** folder to **/usr/local/bin**:
+
+   .. code-block:: console
+     
+      $ sudo cp prometheus-files/prometheus /usr/local/bin/
+      $ sudo cp prometheus-files/promtool /usr/local/bin/
+
+8. Change the ownership to the prometheus user:
+
+   .. code-block:: console
+     
+      $ sudo chown prometheus:prometheus /usr/local/bin/prometheus
+      $ sudo chown prometheus:prometheus /usr/local/bin/promtool
+
+9. Move the **consoles** and **consoles_libraries** directories from **prometheus-files** folder to **/etc/prometheus** folder:
+
+   .. code-block:: console
+     
+      $ sudo cp -r prometheus-files/consoles /etc/prometheus
+	  $ sudo cp -r prometheus-files/console_libraries /etc/prometheus
+
+10. Change the ownership to the prometheus user:
+
+    .. code-block:: console
+     
+       $ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
+       $ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
+
+For more information on installing the Dashboard Data Collector, see `Installing the Dashboard Data Collector `_.
+
+Back to :ref:`Installing Prometheus Using Binary Packages`
+
+Configuring Your Prometheus Settings
+^^^^^^^^^^^^^^^
+After installing Prometheus you must configure your Prometheus settings. You must perform all Prometheus configurations in the **/etc/prometheus/prometheus.yml** file.
+
+**To configure your Prometheus settings:**
+
+1. Create your **prometheus.yml** file:
+
+   .. code-block:: console
+     
+      $ sudo vi /etc/prometheus/prometheus.yml
+	  
+2. Copy the contents below into your prometheus.yml file:
+
+   .. code-block:: console
+     
+      $ #node_exporter port : 9100
+      $ #nvidia_exporter port: 9445
+      $ #process-exporter port: 9256
+      $ 
+      $ global:
+      $   scrape_interval: 10s
+      $ 
+      $ scrape_configs:
+      $   - job_name: 'prometheus'
+      $     scrape_interval: 5s
+      $     static_configs:
+      $       - targets:
+      $         - :9090
+      $   - job_name: 'processes'
+      $     scrape_interval: 5s
+      $     static_configs:
+      $       - targets:
+      $         - :9256
+      $         - :9256
+      $   - job_name: 'nvidia'
+      $     scrape_interval: 5s
+      $     static_configs:
+      $       - targets:
+      $         - :9445
+      $         - :9445
+      $   - job_name: 'nodes'
+      $     scrape_interval: 5s
+      $     static_configs:
+      $       - targets:
+      $         - :9100
+      $         - :9100
+  
+3. Change the ownership of the file to the prometheus user:
+
+   .. code-block:: console
+     
+      $ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
+	  
+Back to :ref:`Installing Prometheus Using Binary Packages`
+
+Configuring Your Prometheus Service File	  
+^^^^^^^^^^^^^^^
+After configuring your Prometheus settings you must configure your Prometheus service file.
+
+**To configure your Prometheus service file**:
+
+1. Create your **prometheus.yml** file:
+
+   .. code-block:: console
+     
+      $ sudo vi /etc/systemd/system/prometheus.service
+	  
+2. Copy the contents below into your prometheus service file:
+
+   .. code-block:: console
+     
+      $ [Unit]
+      $ Description=Prometheus
+      $ Wants=network-online.target
+      $ After=network-online.target
+      $ 
+      $ [Service]
+      $ User=prometheus
+      $ Group=prometheus
+      $ Type=simple
+      $ ExecStart=/usr/local/bin/prometheus \
+      $     --config.file /etc/prometheus/prometheus.yml \
+      $     --storage.tsdb.path /var/lib/prometheus/ \
+      $     --web.console.templates=/etc/prometheus/consoles \
+      $     --web.console.libraries=/etc/prometheus/console_libraries
+      $ 
+      $ [Install]
+      $ WantedBy=multi-user.target
+
+3. Register the prometheus service by reloading the **systemd** service:
+
+   .. code-block:: console
+     
+      $ sudo systemctl daemon-reload
+	  
+4. Start the prometheus service:
+
+   .. code-block:: console
+     
+      $ sudo systemctl start prometheus
+
+5. Check the status of the prometheus service:
+
+   .. code-block:: console
+     
+      $ sudo systemctl status prometheus
+	  
+ If the status is ``active (running)``, you have configured your Prometheus service file correctly.
+
+Back to :ref:`Installing Prometheus Using Binary Packages`
+
+Accessing the Prometheus User Interface
+^^^^^^^^^^^^^^^
+After configuring your prometheus service file, you can access the Prometheus user interface.
+
+You can access the Prometheus user interface by running the following command:
+
+.. code-block:: console
+     
+   $ http://:9090/graph
+
+The Prometheus user interface is displayed.
+
+From the **Query** tab you can query metrics.
+
+Back to :ref:`Installing Prometheus Using Binary Packages`
\ No newline at end of file
diff --git a/installation_guides/installing_sqream_with_binary.rst b/installation_guides/installing_sqream_with_binary.rst
new file mode 100644
index 000000000..f4079b598
--- /dev/null
+++ b/installation_guides/installing_sqream_with_binary.rst
@@ -0,0 +1,279 @@
+.. _installing_sqream_with_binary:
+
+*********************************************
+Installing SQream Using Binary Packages
+*********************************************
+This procedure describes how to install SQream using Binary packages and must be done on all servers.
+
+**To install SQream using Binary packages:**
+
+1. Copy the SQream package to the **/home/sqream** directory for the current version:
+
+   .. code-block:: console
+   
+      $ tar -xf sqream-db-v<2020.2>.tar.gz
+
+2. Append the version number to the name of the SQream folder. The version number in the following example is **v2020.2**:
+
+   .. code-block:: console
+   
+      $ mv sqream sqream-db-v<2020.2>
+
+3. Move the new version of the SQream folder to the **/usr/local/** directory:
+
+   .. code-block:: console
+   
+      $ sudo mv sqream-db-v<2020.2> /usr/local/
+      
+4. Change the ownership of the folder to **sqream folder**:
+
+   .. code-block:: console
+   
+      $ sudo chown -R sqream:sqream  /usr/local/sqream-db-v<2020.2>
+
+5. Navigate to the **/usr/local/** directory and create a symbolic link to SQream:
+
+   .. code-block:: console
+   
+      $ cd /usr/local
+      $ sudo ln -s sqream-db-v<2020.2> sqream
+      
+6. Verify that the symbolic link that you created points to the folder that you created:
+
+   .. code-block:: console
+   
+      $ ls -l
+      
+7. Verify that the symbolic link that you created points to the folder that you created:
+
+   .. code-block:: console
+   
+      $ sqream -> sqream-db-v<2020.2>
+      
+8. Create the SQream configuration file destination folders and set their ownership to **sqream**:
+
+   .. code-block:: console
+   
+      $ sudo mkdir /etc/sqream
+      $ sudo chown -R sqream:sqream /etc/sqream
+      
+9. Create the SQream service log destination folders and set their ownership to **sqream**:
+
+   .. code-block:: console
+   
+      $ sudo mkdir /var/log/sqream
+      $ sudo chown -R sqream:sqream /var/log/sqream
+
+10. Navigate to the **/usr/local/** directory and copy the SQream configuration files from them:
+
+   .. code-block:: console
+   
+      $ cd /usr/local/sqream/etc/
+      $ cp * /etc/sqream
+      
+The configuration files are **service configuration files**, and the JSON files are **SQream configuration files**, for a total of four files. The number of SQream configuration files and JSON files must be identical.
+      
+**NOTICE** - Verify that the JSON files have been configured correctly and that all required flags have been set to the correct values.
+
+In each JSON file, the following parameters **must be updated**:
+
+* instanceId
+* machineIP
+* metadataServerIp
+* spoolMemoryGB
+* limitQueryMemoryGB
+* gpu
+* port
+* ssl_port
+
+Note the following:
+
+* The value of the **metadataServerIp** parameter must point to the IP that the metadata is running on.
+* The value of the **machineIP** parameter must point to the IP of your local machine.
+
+It would be same on server running metadataserver and different on other server nodes.
+
+11. **Optional** - To run additional SQream services, copy the required configuration files and create additional JSON files:
+
+   .. code-block:: console
+   
+      $ cp sqream2_config.json sqream3_config.json
+      $ vim sqream3_config.json
+      
+**NOTICE:** A unique **instanceID** must be used in each JSON file. IN the example above, the instanceID **sqream_2** is changed to **sqream_3**.
+
+12. **Optional** - If you created additional services in **Step 11**, verify that you have also created their additional configuration files:
+
+    .. code-block:: console
+   
+       $ cp sqream2-service.conf sqream3-service.conf
+       $ vim sqream3-service.conf
+      
+13. For each SQream service configuration file, do the following:
+
+    1. Change the **SERVICE_NAME=sqream2** value to **SERVICE_NAME=sqream3**.
+    
+    2. Change **LOGFILE=/var/log/sqream/sqream2.log** to **LOGFILE=/var/log/sqream/sqream3.log**.
+    
+**NOTE:** If you are running SQream on more than one server, you must configure the ``serverpicker`` and ``metadatserver`` services to start on only one of the servers. If **metadataserver** is running on the first server, the ``metadataServerIP`` value in the second server's /etc/sqream/sqream1_config.json file must point to the IP of the server on which the ``metadataserver`` service is running.
+    
+14. Set up **servicepicker**:
+
+    1. Do the following:
+
+       .. code-block:: console
+   
+          $ vim /etc/sqream/server_picker.conf
+    
+    2. Change the IP **127.0.0.1** to the IP of the server that the **metadataserver** service is running on.    
+    
+    3. Change the **CLUSTER** to the value of the cluster path.
+     
+15. Set up your service files:      
+      
+    .. code-block:: console
+   
+       $ cd /usr/local/sqream/service/
+       $ cp sqream2.service sqream3.service
+       $ vim sqream3.service      
+       
+16. Increment each **EnvironmentFile=/etc/sqream/sqream2-service.conf** configuration file for each SQream service file, as shown below:
+
+    .. code-block:: console
+     
+       $ EnvironmentFile=/etc/sqream/sqream<3>-service.conf
+       
+17. Copy and register your service files into systemd:       
+       
+    .. code-block:: console
+     
+       $ sudo cp metadataserver.service /usr/lib/systemd/system/
+       $ sudo cp serverpicker.service /usr/lib/systemd/system/
+       $ sudo cp sqream*.service /usr/lib/systemd/system/
+       
+18. Verify that your service files have been copied into systemd:
+
+    .. code-block:: console
+     
+       $ ls -l /usr/lib/systemd/system/sqream*
+       $ ls -l /usr/lib/systemd/system/metadataserver.service
+       $ ls -l /usr/lib/systemd/system/serverpicker.service
+       $ sudo systemctl daemon-reload       
+       
+19. Copy the license into the **/etc/license** directory:
+
+    .. code-block:: console
+     
+       $ cp license.enc /etc/sqream/   
+
+       
+If you have an HDFS environment, see :ref:`Configuring an HDFS Environment for the User sqream `.
+
+
+
+
+
+
+Upgrading SQream Version
+-------------------------
+Upgrading your SQream version requires stopping all running services while you manually upgrade SQream.
+
+**To upgrade your version of SQream:**
+
+1. Stop all actively running SQream services.
+
+**Notice-** All SQream services must remain stopped while the upgrade is in process. Ensuring that SQream services remain stopped depends on the tool being used.
+
+For an example of stopping actively running SQream services, see :ref:`Launching SQream with Monit `.
+
+
+      
+2. Verify that SQream has stopped listening on ports **500X**, **510X**, and **310X**:
+
+   .. code-block:: console
+
+      $ sudo netstat -nltp    #to make sure sqream stopped listening on 500X, 510X and 310X ports.
+
+3. Replace the old version ``sqream-db-v2020.2``, with the new version ``sqream-db-v2021.1``:
+
+   .. code-block:: console
+    
+      $ cd /home/sqream
+      $ mkdir tempfolder
+      $ mv sqream-db-v2021.1.tar.gz tempfolder/
+      $ tar -xf sqream-db-v2021.1.tar.gz
+      $ sudo mv sqream /usr/local/sqream-db-v2021.1
+      $ cd /usr/local
+      $ sudo chown -R sqream:sqream sqream-db-v2021.1
+   
+4. Remove the symbolic link:
+
+   .. code-block:: console
+   
+      $ sudo rm sqream
+   
+5. Create a new symbolic link named "sqream" pointing to the new version:
+
+   .. code-block:: console  
+
+      $ sudo ln -s sqream-db-v2021.1 sqream
+
+6. Verify that the symbolic SQream link points to the real folder:
+
+   .. code-block:: console  
+
+      $ ls -l
+	 
+   The following is an example of the correct output:
+
+   .. code-block:: console
+    
+      $ sqream -> sqream-db-v2021.1
+
+7. **Optional-** (for major versions) Upgrade your version of SQream storage cluster, as shown in the following example:
+
+   .. code-block:: console  
+
+      $ cat /etc/sqream/sqream1_config.json |grep cluster
+      $ ./upgrade_storage 
+	  
+   The following is an example of the correct output:
+	  
+   .. code-block:: console  
+
+	  get_leveldb_version path{}
+	  current storage version 23
+      upgrade_v24
+      upgrade_storage to 24
+	  upgrade_storage to 24 - Done
+	  upgrade_v25
+	  upgrade_storage to 25
+	  upgrade_storage to 25 - Done
+	  upgrade_v26
+	  upgrade_storage to 26
+	  upgrade_storage to 26 - Done
+	  validate_leveldb
+	  ...
+      upgrade_v37
+	  upgrade_storage to 37
+	  upgrade_storage to 37 - Done
+	  validate_leveldb
+      storage has been upgraded successfully to version 37
+ 
+8. Verify that the latest version has been installed:
+
+   .. code-block:: console
+    
+      $ ./sqream sql --username sqream --password sqream --host localhost --databasename master -c "SELECT SHOW_VERSION();"
+      
+   The following is an example of the correct output:
+ 
+   .. code-block:: console
+    
+      v2021.1
+      1 row
+      time: 0.050603s 
+ 
+For more information, see the `upgrade_storage `_ command line program.
+
+For more information about installing Studio on a stand-alone server, see `Installing Studio on a Stand-Alone Server `_.
\ No newline at end of file
diff --git a/installation_guides/installing_sqream_with_kubernetes.rst b/installation_guides/installing_sqream_with_kubernetes.rst
new file mode 100644
index 000000000..093f21ba3
--- /dev/null
+++ b/installation_guides/installing_sqream_with_kubernetes.rst
@@ -0,0 +1,1805 @@
+.. _installing_sqream_with_kubernetes:
+
+*********************************************
+Installing SQream with Kubernetes
+*********************************************
+**Kubernetes**, also known as **k8s**, is a portable open source platform that automates Linux container operations. Kubernetes supports outsourcing data centers to public cloud service providers or can be scaled for web hosting. SQream uses Kubernetes as an orchestration and recovery solution.
+
+The **Installing SQream with Kubernetes** guide describes the following:
+
+.. contents:: 
+   :local:
+   :depth: 1
+   
+.. _preparing_sqream_environment:
+   
+Preparing the SQream Environment to Launch SQream Using Kubernetes
+===============
+
+The **Preparing the SQream environment to Launch SQream Using Kubernetes** section describes the following:
+
+.. contents:: 
+   :local:
+   :depth: 1
+   
+Overview
+--------------
+   
+A minimum of three servers is required for preparing the SQream environment using Kubernetes.
+
+Kubernetes uses clusters, which are sets of nodes running containterized applications. A cluster consists of at least two GPU nodes and one additional server without GPU to act as the quorum manager.
+
+Each server must have the following IP addresses:
+
+* An IP address located in the management network.
+* An additional IP address from the same subnet to function as a floating IP.
+
+All servers must be mounted in the same shared storage folder.
+
+The following list shows the server host name format requirements:
+
+* A maximum of 253 characters.
+* Only lowercase alphanumeric characters, such as ``-`` or ``.``.
+* Starts and ends with alphanumeric characters.
+
+Go back to :ref:`Preparing the SQream Environment to Launch SQream Using Kubernetes`
+
+
+Operating System Requirements
+------------------------------
+The required operating system is a version of x86 CentOS/RHEL between 7.6 and 7.9. Regarding PPC64le, the required version is RHEL 7.6.
+
+Go back to :ref:`Preparing the SQream Environment to Launch SQream Using Kubernetes`
+
+
+Compute Server Specifications
+------------------------------
+Installing SQream with Kubernetes includes the following compute server specifications:
+
+* **CPU:** 4 cores
+* **RAM:** 16GB
+* **HD:** 500GB
+
+Go back to :ref:`Preparing the SQream Environment to Launch SQream Using Kubernetes`
+
+.. _set_up_your_hosts:
+
+Setting Up Your Hosts
+===============================
+SQream requires you to set up your hosts. Setting up your hosts requires the following:
+
+.. contents:: 
+   :local:
+   :depth: 1
+
+Configuring the Hosts File
+--------------------------------
+**To configure the /etc/hosts file:**
+
+1. Edit the **/etc/hosts** file:
+
+   .. code-block:: console
+
+      $ sudo vim /etc/hosts
+
+2. Call your local host:
+
+   .. code-block:: console
+
+      $ 127.0.0.1	localhost
+      $ 	
+ 
+
+Installing the Required Packages
+----------------------------------
+The first step in setting up your hosts is to install the required packages.
+
+**To install the required packages:**
+
+1. Run the following command based on your operating system:
+
+   * RHEL:
+    
+    .. code-block:: postgres
+   
+       $ sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm    
+ 
+   * CentOS:
+    
+    .. code-block:: postgres
+   
+       $ sudo yum install epel-release	   
+       $ sudo yum install pciutils openssl-devel python36 python36-pip kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc jq net-tools ntp
+
+2. Verify that that the required packages were successfully installed. The following is the correct output:
+    
+    .. code-block:: postgres
+   
+       ntpq --version
+       jq --version
+       python3 --version
+       pip3 --version
+       rpm -qa |grep kernel-devel-$(uname -r)
+       rpm -qa |grep kernel-headers-$(uname -r)
+       gcc --version
+
+3. Enable the **ntpd (Network Time Protocol daemon)** program on all servers:
+    
+    .. code-block:: postgres
+   
+       $ sudo systemctl start ntpd
+       $ sudo systemctl enable ntpd
+       $ sudo systemctl status ntpd
+       $ sudo ntpq -p
+	   
+Go back to :ref:`Setting Up Your Hosts`
+
+     
+Disabling the Linux UI
+----------------------------------
+After installing the required packages, you must disable the Linux UI if it has been installed.
+
+You can disable Linux by running the following command:
+
+   .. code-block:: postgres
+   
+      $ sudo systemctl set-default multi-user.target
+
+Go back to :ref:`Setting Up Your Hosts`
+
+
+Disabling SELinux
+----------------------------------
+After disabling the Linux UI you must disable SELinux.
+
+**To disable SELinux:**
+
+ 1.  Run the following command:
+ 
+    .. code-block:: postgres
+   
+       $ sed -i -e s/enforcing/disabled/g /etc/selinux/config
+       $ sudo reboot
+      
+ 2. Reboot the system as a root user:
+      
+    .. code-block:: postgres
+   
+       $ sudo reboot      
+
+Go back to :ref:`Setting Up Your Hosts`
+
+Disabling Your Firewall
+----------------------------------
+After disabling SELinux, you must disable your firewall by running the following commands:
+   
+      .. code-block:: postgres
+   
+         $ sudo systemctl stop firewalld
+         $ sudo systemctl disable firewalld
+
+Go back to :ref:`Setting Up Your Hosts`
+
+  
+Checking the CUDA Version
+----------------------------------
+After completing all of the steps above, you must check the CUDA version.
+
+**To check the CUDA version:**
+
+1. Check the CUDA version:
+
+   .. code-block:: postgres
+   
+      $ nvidia-smi
+      
+   The following is an example of the correct output:
+
+   .. code-block:: postgres
+   
+      $ +-----------------------------------------------------------------------------+
+      $ | NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
+      $ |-------------------------------+----------------------+----------------------+
+      $ | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
+      $ | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
+      $ |===============================+======================+======================|
+      $ |   0  GeForce GTX 105...  Off  | 00000000:01:00.0 Off |                  N/A |
+      $ | 32%   38C    P0    N/A /  75W |      0MiB /  4039MiB |      0%      Default |
+      $ +-------------------------------+----------------------+----------------------+
+      $                                                                                
+      $ +-----------------------------------------------------------------------------+
+      $ | Processes:                                                       GPU Memory |
+      $ |  GPU       PID   Type   Process name                             Usage      |
+      $ |=============================================================================|
+      $ |  No running processes found                                                 |
+      $ +-----------------------------------------------------------------------------+
+
+In the above output, the CUDA version is **10.1**.
+
+If the above output is not generated, CUDA has not been installed. To install CUDA, see `Installing the CUDA driver `_.
+
+
+Go back to :ref:`Setting Up Your Hosts`
+
+.. _install_kubernetes_cluster:
+
+Installing Your Kubernetes Cluster
+===================================
+After setting up your hosts, you must install your Kubernetes cluster. The Kubernetes and SQream software must be installed from the management host, and can be installed on any server in the cluster.
+
+Installing your Kubernetes cluster requires the following:
+
+.. contents:: 
+   :local:
+   :depth: 1
+
+Generating and Sharing SSH Keypairs Across All Existing Nodes
+------------------------------------
+You can generate and share SSH keypairs across all existing nodes. Sharing SSH keypairs across all nodes enables passwordless access from the management server to all nodes in the cluster. All nodes in the cluster require passwordless access.
+
+.. note::  You must generate and share an SSH keypair across all nodes even if you are installing the Kubernetes cluster on a single host.
+
+**To generate and share an SSH keypair:**
+
+1. Switch to root user access:
+
+  .. code-block:: postgres
+   
+     $ sudo su -
+
+2. Generate an RSA key pair:
+
+  .. code-block:: postgres
+   
+     $ ssh-keygen
+
+The following is an example of the correct output:
+
+  .. code-block:: postgres
+   
+     $ ssh-keygen
+     $ Generating public/private rsa key pair.
+     $ Enter file in which to save the key (/root/.ssh/id_rsa):
+     $ Created directory '/root/.ssh'.
+     $ Enter passphrase (empty for no passphrase):
+     $ Enter same passphrase again:
+     $ Your identification has been saved in /root/.ssh/id_rsa.
+     $ Your public key has been saved in /root/.ssh/id_rsa.pub.
+     $ The key fingerprint is:
+     $ SHA256:xxxxxxxxxxxxxxdsdsdffggtt66gfgfg root@localhost.localdomain
+     $ The key's randomart image is:
+     $ +---[RSA 2048]----+
+     $ |            =*.  |
+     $ |            .o   |
+     $ |            ..o o|
+     $ |     .     .oo +.|
+     $ |      = S =...o o|
+     $ |       B + *..o+.|
+     $ |      o * *..o .+|
+     $ |       o * oo.E.o|
+     $ |      . ..+..B.+o|
+     $ +----[SHA256]-----+
+
+The generated file is ``/root/.ssh/id_rsa.pub``.
+	 
+3. Copy the public key to all servers in the cluster, including the one that you are running on. 
+
+  .. code-block:: postgres
+   
+     $ ssh-copy-id -i ~/.ssh/id_rsa.pub root@remote-host
+   
+4. Replace the ``remote host`` with your host IP address.
+      
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+Installing and Deploying a Kubernetes Cluster Using Kubespray
+------------------------------------
+SQream uses the Kubespray software package to install and deploy Kubernetes clusters.
+
+**To install and deploy a Kubernetes cluster using Kubespray:**
+
+
+1. Clone Kubernetes:
+
+   1. Clone the **kubespray.git** repository:
+
+      .. code-block:: postgres
+   
+         $ git clone https://github.com/kubernetes-incubator/kubespray.git
+     
+   2. Nagivate to the **kubespray** directory:
+     
+       .. code-block:: postgres
+   
+          $ cd kubespray
+     
+   3. Install the **requirements.txt** configuration file:
+   
+      .. code-block:: postgres
+   
+         $ pip3 install -r requirements.txt		 
+
+2. Create your SQream inventory directory:
+
+   1. Run the following command:
+   
+      .. code-block:: postgres
+   
+         $ cp -rp inventory/sample inventory/sqream
+   
+   2. Replace the **** with the defined cluster node IP address(es).
+   
+      .. code-block:: postgres
+   
+         $ declare -a IPS=(, ) 
+   
+      For example, the following replaces ``192.168.0.93`` with ``192.168.0.92``:
+
+      .. code-block:: postgres
+   
+         $ declare -a IPS=(host-93,192.168.0.93 host-92,192.168.0.92)
+
+Note the following:
+ * Running a declare requires defining a pair (host name and cluster node IP address), as shown in the above example.
+ * You can define more than one pair.
+ 
+3. When the reboot is complete, switch back to the root user:
+   
+    .. code-block:: postgres
+   
+       $ sudo su -
+
+4. Navigate to **root/kubespray**:
+   
+    .. code-block:: postgres
+   
+       $ cd /root/kubespray
+
+5. Copy ``inventory/sample`` as ``inventory/sqream``:
+   
+    .. code-block:: postgres
+   
+       $ cp -rfp inventory/sample inventory/sqream
+
+6. Update the Ansible inventory file with the inventory builder:
+   
+    .. code-block:: postgres
+   
+       $ declare -a IPS=(, , ,)
+	   
+7. In the **kubespray hosts.yml** file, set the node IP's: 
+
+   .. code-block:: postgres
+   
+      $ CONFIG_FILE=inventory/sqream/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
+	  
+   If you do not set a specific hostname in declare, the server hostnames will change to ``node1``, ``node2``, etc. To maintain specific hostnames, run declare as in the following example:
+
+   .. code-block:: postgres
+   
+      $ declare -a IPS=(eks-rhl-1,192.168.5.81 eks-rhl-2,192.168.5.82 eks-rhl-3,192.168.5.83)
+
+   Note that the declare must contain pairs (hostname,ip).
+   
+::
+	  
+8. Verify that the following have been done:
+ 
+   * That the **hosts.yml** file is configured correctly.
+   * That all children are included with their relevant nodes.
+
+You can save your current server hostname by replacing  with your server hostname.
+
+9. Generate the content output of the **hosts.yml** file. Make sure to include the file's directory:
+
+   .. code-block:: postgres
+   
+      $ cat  inventory/sqream/hosts.yml
+	  
+The hostname can be lowercase and contain ``-`` or ``.`` only, and must be aligned with the server's hostname.
+
+The following is an example of the correct output. Each host and IP address that you provided in Step 2 should be displayed once:
+
+   .. code-block:: postgres
+   
+      $ all:
+      $   hosts:
+      $     node1:
+      $       ansible_host: 192.168.5.81
+      $       ip: 192.168.5.81
+      $       access_ip: 192.168.5.81
+      $     node2:
+      $       ansible_host: 192.168.5.82
+      $       ip: 192.168.5.82
+      $       access_ip: 192.168.5.82
+      $     node3:
+      $       ansible_host: 192.168.5.83
+      $       ip: 192.168.5.83
+      $       access_ip: 192.168.5.83
+      $   children:
+      $     kube-master:
+      $       hosts:
+      $         node1:
+      $         node2:
+      $         node3:
+      $     kube-node:
+      $       hosts:
+      $         node1:
+      $         node2:
+      $         node3:
+      $     etcd:
+      $       hosts:
+      $         node1:
+      $         node2:
+      $         node3:
+      $     k8s-cluster:
+      $       children:
+      $         kube-master:
+      $         kube-node:
+      $     calico-rr:
+      $       hosts: {}
+	  
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+     
+Adjusting Kubespray Deployment Values
+-------------------------------------    
+After downloading and configuring Kubespray, you can adjust your Kubespray deployment values. A script is used to modify how the Kubernetes cluster is deployed, and you must set the cluster name variable before running this script.
+
+.. note:: The script must be run from the **kubespray** folder.
+
+**To adjust Kubespray deployment values:**
+
+1. Add the following export to the local user’s **~/.bashrc** file by replacing the  with the user's Virtual IP address:
+
+   .. code-block:: postgres
+   
+      $ export VIP_IP=
+	  
+2. Logout, log back in, and verify the following:
+
+   .. code-block:: postgres
+   
+      $ echo $VIP_IP
+	  
+3. Make the following replacements to the **kubespray.settings.sh** file:
+
+   .. code-block:: postgres
+   
+      $ cat < kubespray_settings.sh
+      $ sed -i "/cluster_name: cluster.local/c   \cluster_name: cluster.local.$cluster_name" inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ sed -i "/dashboard_enabled/c   \dashboard_enabled\: "false"" inventory/sqream/group_vars/k8s-cluster/addons.yml
+      $ sed -i "/kube_version/c   \kube_version\: "v1.18.3"" inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ sed -i "/metrics_server_enabled/c   \metrics_server_enabled\: "true"" inventory/sample/group_vars/k8s-cluster/addons.yml
+      $ echo 'kube_apiserver_node_port_range: "3000-6000"' >> inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ echo 'kube_controller_node_monitor_grace_period: 20s' >> inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ echo 'kube_controller_node_monitor_period: 2s' >> inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ echo 'kube_controller_pod_eviction_timeout: 30s' >> inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ echo 'kubelet_status_update_frequency: 4s' >> inventory/sqream/group_vars/k8s-cluster/k8s-cluster.yml
+      $ echo 'ansible ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
+      $ EOF
+	  
+.. note:: In most cases, the Docker data resides on the system disk. Because Docker requires a high volume of data (images, containers, volumes, etc.), you can change the default Docker data location to prevent the system disk from running out of space.
+
+4. *Optional* - Change the default Docker data location:
+
+   .. code-block:: postgres
+   
+      $ sed -i "/docker_daemon_graph/c   \docker_daemon_graph\: """ inventory/sqream/group_vars/all/docker.yml
+ 	  
+5. Make the **kubespray_settings.sh** file executable for your user:
+
+   .. code-block:: postgres
+   
+      $ chmod u+x kubespray_settings.sh && ./kubespray_settings.sh
+	  
+6. Run the following script:
+
+   .. code-block:: postgres
+   
+      $ ./kubespray_settings.sh
+
+7. Run a playbook on the **inventory/sqream/hosts.yml cluster.yml** file:
+
+   .. code-block:: postgres
+   
+      $ ansible-playbook -i inventory/sqream/hosts.yml cluster.yml -v
+
+The Kubespray installation takes approximately 10 - 15 minutes.
+
+The following is an example of the correct output:
+
+   .. code-block:: postgres
+   
+      $ PLAY RECAP
+      $ *********************************************************************************************
+      $ node-1             : ok=680  changed=133  unreachable=0    failed=0
+      $ node-2             : ok=583  changed=113  unreachable=0    failed=0
+      $ node-3             : ok=586  changed=115  unreachable=0    failed=0
+      $ localhost          : ok=1    changed=0    unreachable=0    failed=0
+
+In the event that the output is incorrect, or a failure occurred during the installation, please contact a SQream customer support representative.
+
+Go back to :ref:`Installing Your Kubernetes Cluster`.   
+      
+Checking Your Kubernetes Status
+-------------------------------
+After adjusting your Kubespray deployment values, you must check your Kubernetes status.
+
+**To check your Kuberetes status:**
+
+1. Check the status of the node:
+
+   .. code-block:: postgres
+   
+      $ kubectl get nodes
+	  
+The following is an example of the correct output:
+
+   .. code-block:: postgres
+   
+      $ NAME        STATUS   ROLES                  AGE   VERSION
+      $ eks-rhl-1   Ready    control-plane,master   29m   v1.21.1
+      $ eks-rhl-2   Ready    control-plane,master   29m   v1.21.1
+      $ eks-rhl-3   Ready                     28m   v1.21.1
+
+2. Check the status of the pod:
+
+   .. code-block:: postgres
+   
+      $ kubectl get pods --all-namespaces 
+
+   The following is an example of the correct output:
+
+      .. code-block:: postgres
+   
+         $ NAMESPACE                NAME                                         READY   STATUS    RESTARTS   AGE
+         $ kube-system              calico-kube-controllers-68dc8bf4d5-n9pbp     1/1     Running   0          160m
+         $ kube-system              calico-node-26cn9                            1/1     Running   1          160m
+         $ kube-system              calico-node-kjsgw                            1/1     Running   1          160m
+         $ kube-system              calico-node-vqvc5                            1/1     Running   1          160m
+         $ kube-system              coredns-58687784f9-54xsp                     1/1     Running   0          160m
+         $ kube-system              coredns-58687784f9-g94xb                     1/1     Running   0          159m
+         $ kube-system              dns-autoscaler-79599df498-hlw8k              1/1     Running   0          159m
+         $ kube-system              kube-apiserver-k8s-host-1-134                1/1     Running   0          162m
+         $ kube-system              kube-apiserver-k8s-host-194                  1/1     Running   0          161m
+         $ kube-system              kube-apiserver-k8s-host-68                   1/1     Running   0          161m
+         $ kube-system              kube-controller-manager-k8s-host-1-134       1/1     Running   0          162m
+         $ kube-system              kube-controller-manager-k8s-host-194         1/1     Running   0          161m
+         $ kube-system              kube-controller-manager-k8s-host-68          1/1     Running   0          161m
+         $ kube-system              kube-proxy-5f42q                             1/1     Running   0          161m
+         $ kube-system              kube-proxy-bbwvk                             1/1     Running   0          161m
+         $ kube-system              kube-proxy-fgcfb                             1/1     Running   0          161m
+         $ kube-system              kube-scheduler-k8s-host-1-134                1/1     Running   0          161m
+         $ kube-system              kube-scheduler-k8s-host-194                  1/1     Running   0          161m
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+        
+Adding a SQream Label to Your Kubernetes Cluster Nodes
+-------------------------------------------------
+After checking your Kubernetes status, you must add a SQream label on your Kubernetes cluster nodes.
+
+**To add a SQream label on your Kubernetes cluster nodes:**
+
+1. Get the cluster node list:
+
+   .. code-block:: postgres
+   
+      $ kubectl get nodes
+	  
+   The following is an example of the correct output:
+   
+   .. code-block:: postgres
+   
+      $ NAME        STATUS   ROLES                  AGE   VERSION
+      $ eks-rhl-1   Ready    control-plane,master   29m   v1.21.1
+      $ eks-rhl-2   Ready    control-plane,master   29m   v1.21.1
+      $ eks-rhl-3   Ready                     28m   v1.21.1
+	  
+2. Set the node label, change the ``node-name`` to the node NAME(s) in the above example:
+
+   .. code-block:: postgres
+
+      $ kubectl label nodes  cluster=sqream
+   
+   The following is an example of the correct output:
+
+   .. code-block:: postgres
+   
+      $ [root@edk-rhl-1 kubespray]# kubectl label nodes eks-rhl-1 cluster=sqream
+      $ node/eks-rhl-1 labeled
+      $ [root@edk-rhl-1 kubespray]# kubectl label nodes eks-rhl-2 cluster=sqream
+      $ node/eks-rhl-2 labeled
+      $ [root@edk-rhl-1 kubespray]# kubectl label nodes eks-rhl-3 cluster=sqream
+      $ node/eks-rhl-3 labeled
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+   
+Copying Your Kubernetes Configuration API File to the Master Cluster Nodes
+-------------------------------------------------  
+After adding a SQream label on your Kubernetes cluster nodes, you must copy your Kubernetes configuration API file to your Master cluster nodes.
+
+When the Kubernetes cluster installation is complete, an API configuration file is automatically created in the **.kube** folder of the root user. This file enables the **kubectl** command access Kubernetes' internal API service. Following this step lets you run **kubectl** commands from any node in the cluster.
+
+
+.. warning:: You must perform this on the management server only!
+
+**To copy your Kubernetes configuration API file to your Master cluster nodes:**
+
+1. Create the **.kube** folder in the **local user** directory:
+
+   .. code-block:: postgres
+   
+      $ mkdir /home//.kube
+
+2. Copy the configuration file from the root user directory to the  directory:
+
+   .. code-block:: postgres
+   
+      $ sudo cp /root/.kube/config /home//.kube
+
+3. Change the file owner from **root user** to the :
+
+   .. code-block:: postgres
+   
+      $  sudo chown . /home//.kube/config
+
+4. Create the **.kube** folder in the other nodes located in the  directory:
+
+   .. code-block:: postgres
+   
+      $ ssh @ mkdir .kube
+
+5. Copy the configuration file from the management node to the other nodes:
+
+   .. code-block:: postgres
+   
+      $ scp /home//.kube/config @:/home//.kube/
+	  
+6. Under local user on each server you copied **.kube** to, run the following command:
+
+   .. code-block:: postgres
+   
+      $ sudo usermod -aG docker $USER
+
+This grants the local user the necessary permissions to run Docker commands.
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+Creating an env_file in Your Home Directory
+-------------------------------------------------
+After copying your Kubernetes configuration API file to your Master cluster nodes, you must create an **env_file** in your home directory, and must set the VIP address as a variable.
+
+.. warning:: You must perform this on the management server only!
+
+
+
+**To create an env_file for local users in the user's home directory:**
+
+1. Set a variable that includes the VIP IP address:
+
+   .. code-block:: postgres
+   
+      $ export VIP_IP=
+	  
+.. note:: If you use Kerberos, replace the ``KRB5_SERVER`` value with the IP address of your Kerberos server.
+   
+2. Do one of the following:
+
+   * For local users:
+
+     .. code-block:: postgres
+   
+        $ mkdir /home/$USER/.sqream
+   
+3. Make the following replacements to the **kubespray.settings.sh** file, verifying that the ``KRB5_SERVER`` parameter is set to your server IP:
+
+
+   .. code-block:: postgres  
+   
+        $ cat < /home/$USER/.sqream/env_file
+        SQREAM_K8S_VIP=$VIP_IP
+        SQREAM_ADMIN_UI_PORT=8080
+        SQREAM_DASHBOARD_DATA_COLLECTOR_PORT=8100
+        SQREAM_DATABASE_NAME=master
+        SQREAM_K8S_ADMIN_UI=sqream-admin-ui
+        SQREAM_K8S_DASHBOARD_DATA_COLLECTOR=dashboard-data-collector
+        SQREAM_K8S_METADATA=sqream-metadata
+        SQREAM_K8S_NAMESPACE=sqream
+        SQREAM_K8S_PICKER=sqream-picker
+        SQREAM_K8S_PROMETHEUS=prometheus
+        SQREAM_K8S_REGISTRY_PORT=6000
+        SQREAM_METADATA_PORT=3105
+        SQREAM_PICKER_PORT=3108
+        SQREAM_PROMETHEUS_PORT=9090
+        SQREAM_SPOOL_MEMORY_RATIO=0.25
+        SQREAM_WORKER_0_PORT=5000
+        KRB5CCNAME=FILE:/tmp/tgt
+        KRB5_SERVER=kdc.sq.com:1
+        KRB5_CONFIG_DIR=${        $ SQREAM_MOUNT_DIR}/krb5
+        KRB5_CONFIG_FILE=${KRB5_CONFIG_DIR}/krb5.conf
+        HADOOP_CONFIG_DIR=${        $ SQREAM_MOUNT_DIR}/hadoop
+        HADOOP_CORE_XML=${HADOOP_CONFIG_DIR}/core-site.xml
+        HADOOP_HDFS_XML=${HADOOP_CONFIG_DIR}/hdfs-site.xml
+        EOF  
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+		
+
+
+
+
+Creating a Base Kubernetes Namespace
+------------------------------------
+After creating an env_file in the user's home directory, you must create a base Kubernetes namespace.
+
+You can create a Kubernetes namespace by running the following command:
+
+.. code-block:: postgres
+   
+   $ kubectl create namespace sqream-init    
+   
+The following is an example of the correct output:
+
+.. code-block:: postgres
+   
+   $ namespace/sqream-init created
+   
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+   
+Pushing the **env_file** File to the Kubernetes Configmap
+--------------------------------------   
+After creating a base Kubernetes namespace, you must push the **env_file** to the Kubernetes configmap. You must push the **env_file** file to the Kubernetes **configmap** in the **sqream-init** namespace.
+
+This is done by running the following command:
+
+.. code-block:: postgres
+   
+   $ kubectl create configmap sqream-init -n sqream-init --from-env-file=/home/$USER/.sqream/env_file
+
+The following is an example of the correct output:
+
+.. code-block:: postgres
+   
+   $ configmap/sqream-init created
+
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+
+Installing the NVIDIA Docker2 Toolkit
+-------------------------------------
+After pushing the **env_file** file to the Kubernetes configmap, you must install the NVIDIA Docker2 Toolkit. The **NVIDIA Docker2 Toolkit** lets users build and run GPU-accelerated Docker containers, and must be run only on GPU servers. The NVIDIA Docker2 Toolkit includes a container runtime library and utilities that automatically configure containers to leverage NVIDIA GPUs.
+
+
+Installing the NVIDIA Docker2 Toolkit on an x86_64 Bit Processor on CentOS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**To install the NVIDIA Docker2 Toolkit on an x86_64 bit processor on CentOS:**
+
+1. Add the repository for your distribution:
+
+   .. code-block:: postgres
+   
+      $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
+      $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \
+      $ sudo tee /etc/yum.repos.d/nvidia-docker.repo
+
+2. Install the **nvidia-docker2** package and reload the Docker daemon configuration:
+   
+   .. code-block:: postgres
+   
+      $ sudo yum install nvidia-docker2
+      $ sudo pkill -SIGHUP dockerd
+
+3. Verify that the **nvidia-docker2** package has been installed correctly:
+
+   .. code-block:: postgres
+   
+      $ docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi
+
+   The following is an example of the correct output:
+
+   .. code-block:: postgres
+   
+      $ docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi
+      $ Unable to find image 'nvidia/cuda:10.1-base' locally
+      $ 10.1-base: Pulling from nvidia/cuda
+      $ d519e2592276: Pull complete 
+      $ d22d2dfcfa9c: Pull complete 
+      $ b3afe92c540b: Pull complete 
+      $ 13a10df09dc1: Pull complete 
+      $ 4f0bc36a7e1d: Pull complete 
+      $ cd710321007d: Pull complete 
+      $ Digest: sha256:635629544b2a2be3781246fdddc55cc1a7d8b352e2ef205ba6122b8404a52123
+      $ Status: Downloaded newer image for nvidia/cuda:10.1-base
+      $ Sun Feb 14 13:27:58 2021       
+      $ +-----------------------------------------------------------------------------+
+      $ | NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
+      $ |-------------------------------+----------------------+----------------------+
+      $ | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
+      $ | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
+      $ |===============================+======================+======================|
+      $ |   0  GeForce GTX 105...  Off  | 00000000:01:00.0 Off |                  N/A |
+      $ | 32%   37C    P0    N/A /  75W |      0MiB /  4039MiB |      0%      Default |
+      $ +-------------------------------+----------------------+----------------------+
+      $                                                                                
+      $ +-----------------------------------------------------------------------------+
+      $ | Processes:                                                       GPU Memory |
+      $ |  GPU       PID   Type   Process name                             Usage      |
+      $ |=============================================================================|
+      $ |  No running processes found                                                 |
+      $ +-----------------------------------------------------------------------------+
+
+For more information on installing the NVIDIA Docker2 Toolkit on an x86_64 Bit Processor on CentOS, see `NVIDIA Docker Installation - CentOS distributions `_
+     
+Installing the NVIDIA Docker2 Toolkit on an x86_64 Bit Processor on Ubuntu
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**To install the NVIDIA Docker2 Toolkit on an x86_64 bit processor on Ubuntu:**
+
+1. Add the repository for your distribution:
+
+   .. code-block:: postgres
+   
+      $ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
+      $ sudo apt-key add -     
+      $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)  
+      $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
+      $ sudo tee /etc/apt/sources.list.d/nvidia-docker.list  
+      $ sudo apt-get update
+     
+2. Install the **nvidia-docker2** package and reload the Docker daemon configuration:
+   
+   .. code-block:: postgres
+   
+      $ sudo apt-get install nvidia-docker2
+      $ sudo pkill -SIGHUP dockerd
+     
+3. Verify that the nvidia-docker2 package has been installed correctly:
+
+   .. code-block:: postgres
+   
+      $ docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi  
+     
+For more information on installing the NVIDIA Docker2 Toolkit on an x86_64 Bit Processor on Ubuntu, see `NVIDIA Docker Installation - Ubuntu distributions `_
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+    
+Modifying the Docker Daemon JSON File for GPU and Compute Nodes
+-------------------------------------
+After installing the NVIDIA Docker2 toolkit, you must modify the Docker daemon JSON file for GPU and Compute nodes.
+
+Modifying the Docker Daemon JSON File for GPU Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+**To modify the Docker daemon JSON file for GPU nodes:**     
+     
+1. Enable GPU and set HTTP access to the local Kubernetes Docker registry.
+
+.. note:: The Docker daemon JSON file must be modified on all GPU nodes.
+
+.. note:: Contact your IT department for a virtual IP.
+
+2. Replace the ``VIP address`` with your assigned VIP address.
+
+::
+
+3. Connect as a root user:
+
+   .. code-block:: postgres
+   
+      $  sudo -i
+     
+4. Set a variable that includes the VIP address:    
+     
+   .. code-block:: postgres
+   
+      $ export VIP_IP=
+
+5. Replace the  with the VIP address:      
+     
+    .. code-block:: postgres
+   
+      $ cat < /etc/docker/daemon.json
+      $ {
+      $    "insecure-registries": ["$VIP_IP:6000"],
+      $     "default-runtime": "nvidia",
+      $     "runtimes": {
+      $         "nvidia": {
+      $             "path": "nvidia-container-runtime",
+      $             "runtimeArgs": []
+      $         }
+      $     }
+      $ }
+      $ EOF   
+
+6. Apply the changes and restart Docker:
+
+   .. code-block:: postgres
+   
+      $ systemctl daemon-reload && systemctl restart docker
+      
+7. Exit the root user:
+ 
+  .. code-block:: postgres
+   
+     $ exit
+	 
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+      
+Modifying the Docker Daemon JSON File for Compute Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+You must follow this procedure only if you have a Compute node.
+
+**To modify the Docker daemon JSON file for Compute nodes:**
+
+1. Switch to a root user:
+
+   .. code-block:: postgres
+   
+      $  sudo -i
+
+2. Set a variable that includes a VIP address.
+
+.. note:: Contact your IT department for a virtual IP.
+
+3. Replace the ``VIP address`` with your assigned VIP address.
+
+   .. code-block:: postgres
+   
+      $ cat < /etc/docker/daemon.json
+      $ {
+      $    "insecure-registries": ["$VIP_IP:6000"]
+      $ }
+      $ EOF 
+
+4. Restart the services:
+
+   .. code-block:: postgres
+   
+      $ systemctl daemon-reload && systemctl restart docker
+
+5. Exit the root user:
+ 
+ 
+  .. code-block:: postgres
+   
+     $ exit
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+   
+Installing the Nvidia-device-plugin Daemonset
+----------------------------------------------
+After modifying the Docker daemon JSON file for GPU or Compute Nodes, you must installing the Nvidia-device-plugin daemonset. The Nvidia-device-plugin daemonset is only relevant to GPU nodes.
+
+**To install the Nvidia-device-plugin daemonset:**
+
+1. Set ``nvidia.com/gpu`` to ``true`` on all GPU nodes:
+
+.. code-block:: postgres
+   
+   $ kubectl label nodes  nvidia.com/gpu=true
+      
+2. Replace the ** with your GPU node name:
+   
+   For a complete list of GPU node names, run the ``kubectl get nodes`` command.
+
+   The following is an example of the correct output:
+   
+   .. code-block:: postgres
+   
+      $ [root@eks-rhl-1 ~]# kubectl label nodes eks-rhl-1 nvidia.com/gpu=true
+      $ node/eks-rhl-1 labeled
+      $ [root@eks-rhl-1 ~]# kubectl label nodes eks-rhl-2 nvidia.com/gpu=true
+      $ node/eks-rhl-2 labeled
+      $ [root@eks-rhl-1 ~]# kubectl label nodes eks-rhl-3 nvidia.com/gpu=true
+      $ node/eks-rhl-3 labeled  
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+   
+Creating an Nvidia Device Plugin
+----------------------------------------------   
+After installing the Nvidia-device-plugin daemonset, you must create an Nvidia-device-plugin. You can create an Nvidia-device-plugin by running the following command 
+
+.. code-block:: postgres
+   
+   $  kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta6/nvidia-device-plugin.yml
+   
+If needed, you can check the status of the Nvidia-device-plugin-daemonset pod status:
+
+.. code-block:: postgres
+   
+   $ kubectl get pods -n kube-system -o wide | grep nvidia-device-plugin
+
+The following is an example of the correct output:   
+
+.. code-block:: postgres
+   
+   $ NAME                                       READY   STATUS    RESTARTS   AGE
+   $ nvidia-device-plugin-daemonset-fxfct       1/1     Running   0          6h1m
+   $ nvidia-device-plugin-daemonset-jdvxs       1/1     Running   0          6h1m
+   $ nvidia-device-plugin-daemonset-xpmsv       1/1     Running   0          6h1m
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+Checking GPU Resources Allocatable to GPU Nodes
+-------------------------------------
+After creating an Nvidia Device Plugin, you must check the GPU resources alloctable to the GPU nodes. Each GPU node has records, such as ``nvidia.com/gpu:     <#>``. The ``#`` indicates the number of allocatable, or available, GPUs in each node.
+
+You can output a description of allocatable resources by running the following command:
+
+.. code-block:: postgres
+   
+   $ kubectl describe node | grep -i -A 7 -B 2 allocatable: 
+
+The following is an example of the correct output:
+
+.. code-block:: postgres
+   
+   $ Allocatable:
+   $  cpu:                3800m
+   $  ephemeral-storage:  94999346224
+   $  hugepages-1Gi:      0
+   $  hugepages-2Mi:      0
+   $  memory:             15605496Ki
+   $  nvidia.com/gpu:     1
+   $  pods:               110 
+
+Go back to :ref:`Installing Your Kubernetes Cluster`     
+
+Preparing the WatchDog Monitor
+------------------------------
+SQream's deployment includes installing two watchdog services. These services monitor Kuberenetes management and the server's storage network.
+
+You can enable the storage watchdogs by adding entries in the **/etc/hosts** file on each server:
+
+.. code-block:: postgres
+   
+   $ 
k8s-node1.storage + $
k8s-node2.storage + $
k8s-node3.storage + +The following is an example of the correct syntax: + +.. code-block:: postgres + + $ 10.0.0.1 k8s-node1.storage + $ 10.0.0.2 k8s-node2.storage + $ 10.0.0.3 k8s-node3.storage + +Go back to :ref:`Installing Your Kubernetes Cluster` + +.. _installing_sqream_software: + +Installing the SQream Software +================================= +Once you've prepared the SQream environment for launching it using Kubernetes, you can begin installing the SQream software. + +The **Installing the SQream Software** section describes the following: + +.. contents:: + :local: + :depth: 1 + + +.. _getting_sqream_package: + +Getting the SQream Package +-------------------------------- +The first step in installing the SQream software is getting the SQream package. Please contact the SQream Support team to get the **sqream_k8s-nnn-DBnnn-COnnn-SDnnn-.tar.gz** tarball file. + +This file includes the following values: + +* **sqream_k8s-** - the SQream installer version. +* **DB** - the SQreamDB version. +* **CO** - the SQream console version. +* **SD** - the SQream Acceleration Studio version. +* **arch** - the server architecture. + +You can extract the contents of the tarball by running the following command: + +.. code-block:: postgres + + $ tar -xvf sqream_k8s-1.0.15-DB2020.1.0.2-SD0.7.3-x86_64.tar.gz + $ cd sqream_k8s-1.0.15-DB2020.1.0.2-SD0.7.3-x86_64 + $ ls + +Extracting the contents of the tarball file generates a new folder with the same name as the tarball file. + +The following shows the output of the extracted file: + +.. code-block:: postgres + + drwxrwxr-x. 2 sqream sqream 22 Jan 27 11:39 license + lrwxrwxrwx. 1 sqream sqream 49 Jan 27 11:39 sqream -> .sqream/sqream-sql-v2020.3.1_stable.x86_64/sqream + -rwxrwxr-x. 1 sqream sqream 9465 Jan 27 11:39 sqream-install + -rwxrwxr-x. 1 sqream sqream 12444 Jan 27 11:39 sqream-start + +Go back to :ref:`Installing Your SQream Software` + +Setting Up and Configuring Hadoop +-------------------------------- +After getting the SQream package, you can set up and configure Hadoop by configuring the **keytab** and **krb5.conf** files. + +.. note:: You only need to configure the **keytab** and **krb5.conf** files if you use Hadoop with Kerberos authentication. + + +**To set up and configure Hadoop:** + +1. Contact IT for the **keytab** and **krb5.conf** files. + +:: + +2. Copy both files into the respective empty **.hadoop/** and **.krb5/** directories: + +.. code-block:: postgres + + $ cp hdfs.keytab krb5.conf .krb5/ + $ cp core-site.xml hdfs-site.xml .hadoop/ + +The SQream installer automatically copies the above files during the installation process. + +Go back to :ref:`Installing Your SQream Software` + +Starting a Local Docker Image Registry +-------------------------------- +After getting the SQream package, or (optionally) setting up and configuring Hadoop, you must start a local Docker image registry. Because Kubernetes is based on Docker, you must start the local Docker image registry on the host's shared folder. This allows all hosts to pull the SQream Docker images. + +**To start a local Docker image registry:** + +1. Create a Docker registry folder: + + .. code-block:: postgres + + $ mkdir /docker-registry/ + +2. Set the ``docker_path`` for the Docker registry folder: + + .. code-block:: postgres + + $ export docker_path= + +3. Apply the **docker-registry** service to the cluster: + + .. code-block:: postgres + + $ cat .k8s/admin/docker_registry.yaml | envsubst | kubectl create -f - + + The following is an example of the correct output: + + .. code-block:: postgres + + namespace/sqream-docker-registry created + configmap/sqream-docker-registry-config created + deployment.apps/sqream-docker-registry created + service/sqream-docker-registry created + +4. Check the pod status of the **docker-registry** service: + + .. code-block:: postgres + + $ kubectl get pods -n sqream-docker-registry + +The following is an example of the correct output: + + .. code-block:: postgres + + NAME READY STATUS RESTARTS AGE + sqream-docker-registry-655889fc57-hmg7h 1/1 Running 0 6h40m + +Go back to :ref:`Installing Your SQream Software` + +Installing the Kubernetes Dashboard +-------------------------------- +After starting a local Docker image registry, you must install the Kubernetes dashboard. The Kubernetes dashboard lets you see the Kubernetes cluster, nodes, services, and pod status. + +**To install the Kubernetes dashboard:** + +1. Apply the **k8s-dashboard** service to the cluster: + + .. code-block:: postgres + + $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml + + The following is an example of the correct output: + + .. code-block:: postgres + + namespace/kubernetes-dashboard created + serviceaccount/kubernetes-dashboard created + service/kubernetes-dashboard created + secret/kubernetes-dashboard-certs created + secret/kubernetes-dashboard-csrf created + secret/kubernetes-dashboard-key-holder created + configmap/kubernetes-dashboard-settings created + role.rbac.authorization.k8s.io/kubernetes-dashboard created + clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created + rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created + clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created + deployment.apps/kubernetes-dashboard created + service/dashboard-metrics-scraper created + deployment.apps/dashboard-metrics-scraper created + +2. Grant the user external access to the Kubernetes dashboard: + + .. code-block:: postgres + + $ cat .k8s/admin/kubernetes-dashboard-svc-metallb.yaml | envsubst | kubectl create -f - + + The following is an example of the correct output: + + .. code-block:: postgres + + service/kubernetes-dashboard-nodeport created + +3. Create the ``cluster-admin-sa.yaml`` file: + + .. code-block:: postgres + + $ kubectl create -f .k8s/admin/cluster-admin-sa.yaml + + The following is an example of the correct output: + + .. code-block:: postgres + + clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-sa-cluster-admin created + +4. Check the pod status of the **K8s-dashboard** service: + + .. code-block:: postgres + + $ kubectl get pods -n kubernetes-dashboard + + The following is an example of the correct output: + + .. code-block:: postgres + + NAME READY STATUS RESTARTS AGE + dashboard-metrics-scraper-6b4884c9d5-n8p57 1/1 Running 0 4m32s + kubernetes-dashboard-7b544877d5-qc8b4 1/1 Running 0 4m32s + +5. Obtain the **k8s-dashboard** access token: + + .. code-block:: postgres + + $ kubectl -n kube-system describe secrets cluster-admin-sa-token + + The following is an example of the correct output: + + .. code-block:: postgres + + Name: cluster-admin-sa-token-rbl9p + Namespace: kube-system + Labels: + Annotations: kubernetes.io/service-account.name: cluster-admin-sa + kubernetes.io/service-account.uid: 81866d6d-8ef3-4805-840d-58618235f68d + + Type: kubernetes.io/service-account-token + + Data + ==== + ca.crt: 1025 bytes + namespace: 11 bytes + token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjRMV09qVzFabjhId09oamQzZGFFNmZBeEFzOHp3SlJOZWdtVm5lVTdtSW8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVyLWFkbWluLXNhLXRva2VuLXJibDlwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItYWRtaW4tc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI4MTg2NmQ2ZC04ZWYzLTQ4MDUtODQwZC01ODYxODIzNWY2OGQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3Rlci1hZG1pbi1zYSJ9.mNhp8JMr5y3hQ44QrvRDCMueyjSHSrmqZcoV00ZC7iBzNUqh3n-fB99CvC_GR15ys43jnfsz0tdsTy7VtSc9hm5ENBI-tQ_mwT1Zc7zJrEtgFiA0o_eyfYZOARdhdyFEJg84bzkIxJFPKkBWb4iPWU1Xb7RibuMCjNTarZMZbqzKYfQEcMZWJ5UmfUqp-HahZZR4BNbjSWybs7t6RWdcQZt6sO_rRCDrOeEJlqKKjx4-5jFZB8Du_0kKmnw2YJmmSCEOXrpQCyXIiZJpX08HyDDYfFp8IGzm61arB8HDA9dN_xoWvuz4Cj8klUtTzL9effJJPjHJlZXcEqQc9hE3jw + +6. Navigate to ``https://:5999``. + +:: + +7. Select the **Token** radio button, paste the token from the previous command output, and click **Sign in**. + +The Kubernetes dashboard is displayed. + +Go back to :ref:`Installing Your SQream Software` + +Installing the SQream Prometheus Package +-------------------------------- +After installing the Kubernetes dashboard, you must install the SQream Prometheus package. To properly monitor the host and GPU statistics the **exporter service** must be installed on each Kubernetes cluster node. + +This section describes how to install the following: + +* **node_exporter** - collects host data, such as CPU memory usage. +* **nvidia_exporter** - collects GPU utilization data. + + +.. note:: The steps in this section must be done on **all** cluster nodes. + +To install the **sqream-prometheus** package, you must do the following: + +1. :ref:`Install the exporter service ` + +:: + +2. :ref:`Check the exporter service ` + +Go back to :ref:`Installing Your SQream Software` + + +.. _install_exporter_service: + +Installing the Exporter Service +~~~~~~~~~~~~~~~~~~~~~~~~ + +**To install the exporter service:** + +1. Create a user and group that will be used to run the exporter services: + + .. code-block:: console + + $ sudo groupadd --system prometheus && sudo useradd -s /sbin/nologin --system -g prometheus prometheus + +2. Extract the **sqream_exporters_prometheus.0.1.tar.gz** file: + + .. code-block:: console + + $ cd .prometheus + $ tar -xf sqream_exporters_prometheus.0.1.tar.gz + +3. Copy the exporter software files to the **/usr/bin** directory: + + .. code-block:: console + + $ cd sqream_exporters_prometheus.0.1 + $ sudo cp node_exporter/node_exporter /usr/bin/ + $ sudo cp nvidia_exporter/nvidia_exporter /usr/bin/ + +4. Copy the exporters service file to the **/etc/systemd/system/** directory: + + .. code-block:: console + + $ sudo cp services/node_exporter.service /etc/systemd/system/ + $ sudo cp services/nvidia_exporter.service /etc/systemd/system/ + +5. Set the permission and group of the service files: + + .. code-block:: console + + $ sudo chown prometheus:prometheus /usr/bin/node_exporter + $ sudo chmod u+x /usr/bin/node_exporter + $ sudo chown prometheus:prometheus /usr/bin/nvidia_exporter + $ sudo chmod u+x /usr/bin/nvidia_exporter + +6. Reload the services: + + .. code-block:: console + + $ sudo systemctl daemon-reload + +7. Start both services and set them to start when the server is booted up: + + * Node_exporter: + + .. code-block:: console + + $ sudo systemctl start node_exporter && sudo systemctl enable node_exporter + + * Nvidia_exporter: + + .. code-block:: console + + $ sudo systemctl start nvidia_exporter && sudo systemctl enable nvidia_exporter + + + +.. _check_exporter_status: + +Checking the Exporter Status +~~~~~~~~~~~~~~~~~~~~~~~~ +After installing the **exporter** service, you must check its status. + +You can check the exporter status by running the following command: + +.. code-block:: console + + $ sudo systemctl status node_exporter && sudo systemctl status nvidia_exporter + +Go back to :ref:`Installing Your SQream Software` + + +.. _running_sqream_install_service: + +Running the Sqream-install Service +================================ +The **Running the Sqream-install Service** section describes the following: + +.. contents:: + :local: + :depth: 1 + +Installing Your License +-------------------------------- +After install the SQream Prometheus package, you must install your license. + +**To install your license:** + +1. Copy your license package to the sqream **/license** folder. + +.. note:: You do not need to untar the license package after copying it to the **/license** folder because the installer script does it automatically. + +The following flags are **mandatory** during your first run: + +.. code-block:: console + + $ sudo ./sqream-install -i -k -m + +.. note:: If you cannot run the script with **sudo**, verify that you have the right permission (**rwx** for the user) on the relevant directories (config, log, volume, and data-in directories). + +Go back to :ref:`Running the SQream_install Service`. + +Changing Your Data Ingest Folder +-------------------------------- +After installing your license, you must change your data ingest folder. + +You can change your data ingest folder by running the following command: + +.. code-block:: console + + $ sudo ./sqream-install -d /media/nfs/sqream/data_in + +Go back to :ref:`Running the SQream_install Service`. + +Checking Your System Settings +-------------------------------- +After changing your data ingest folder, you must check your system settings. + +The following command shows you all the variables that your SQream system is running with: + +.. code-block:: console + + $ ./sqream-install -s + +After optionally checking your system settings, you can use the **sqream-start** application to control your Kubernetes cluster. + +Go back to :ref:`Running the SQream_install Service`. + +SQream Installation Command Reference +-------------------------------- +If needed, you can use the **sqream-install** flag reference for any needed flags by typing: + +.. code-block:: console + + $ ./sqream-install --help + +The following shows the **sqream--install** flag descriptions: + +.. list-table:: + :widths: 22 59 25 + :header-rows: 1 + + * - Flag + - Function + - Note + * - **-i** + - Loads all the software from the hidden **.docker** folder. + - Mandatory + * - **-k** + - Loads the license package from the **/license** directory. + - Mandatory + * - **-m** + - Sets the relative path for all SQream folders under the shared filesystem available from all nodes (sqreamdb, config, logs and data_in). No other flags are required if you use this flag (such as c, v, l or d). + - Mandatory + * - **-c** + - Sets the path where to write/read SQream configuration files from. The default is **/etc/sqream/**. + - Optional + * - **-v** + - Shows the location of the SQream cluster. ``v`` creates a cluster if none exist, and mounts it if does. + - Optional + * - **-l** + - Shows the location of the SQream system startup logs. The logs contain startup and Docker logs. The default is **/var/log/sqream/**. + - Optional + * - **-d** + - Shows the folder containing data that you want to import into or copy from SQream. + - Optional + * - **-n** + - Sets the Kubernetes namespace. The default is **sqream**. + - Optional + * - **-N** + - Deletes a specific Kubernetes namespace and sets the factory default namespace (sqream). + - Optional + * - **-f** + - Overwrite existing folders and all files located in mounted directories. + - Optional + * - **-r** + - Resets the system configuration. This flag is run without any other variables. + - Optional + * - **-s** + - Shows the system settings. + - Optional + * - **-e** + - Sets the Kubernetes cluster's virtual IP address. + - Optional + * - **-h** + - Help, shows all available flags. + - Optional + +Go back to :ref:`Running the SQream_install Service`. + +Controlling Your Kubernetes Cluster Using SQream Flags +-------------------------------- +You can control your Kubernetes cluster using SQream flags. + +The following command shows you the available Kubernetes cluster control options: + +.. code-block:: console + + $ ./sqream-start -h + +The following describes the **sqream-start** flags: + +.. list-table:: + :widths: 22 59 25 + :header-rows: 1 + + * - Flag + - Function + - Note + * - **-s** + - Starts the sqream services, starting metadata, server picker, and workers. The number of workers started is based on the number of available GPU’s. + - Mandatory + * - **-p** + - Sets specific ports to the workers services. You must enter the starting port for the sqream-start application to allocate it based on the number of workers. + - + * - **-j** + - Uses an external .json configuration file. The file must be located in the **configuration** directory. + - The workers must each be started individually. + * - **-m** + - Allocates worker spool memory. + - The workers must each be started individually. + * - **-a** + - Starts the SQream Administration dashboard and specifies the listening port. + - + * - **-d** + - Deletes all running SQream services. + - + * - **-h** + - Shows all available flags. + - Help + +Go back to :ref:`Running the SQream_install Service`. + +.. _using_sqream_start_commands: + +Using the sqream-start Commands +======================= +In addition to controlling your Kubernetes cluster using SQream flags, you can control it using **sqream-start** commands. + +The **Using the sqream-start Commands** section describes the following: + +.. contents:: + :local: + :depth: 1 + +Starting Your SQream Services +------------------------ +You can run the **sqream-start** command with the **-s** flag to start SQream services on all available GPU's: + +.. code-block:: console + + $ sudo ./sqream-start -s + +This command starts the SQream metadata, server picker, and sqream workers on all available GPU’s in the cluster. + +The following is an example of the correct output: + +.. code-block:: console + + ./sqream-start -s + Initializing network watchdogs on 3 hosts... + Network watchdogs are up and running + + Initializing 3 worker data collectors ... + Worker data collectors are up and running + + Starting Prometheus ... + Prometheus is available at 192.168.5.100:9090 + + Starting SQream master ... + SQream master is up and running + + Starting up 3 SQream workers ... + All SQream workers are up and running, SQream-DB is available at 192.168.5.100:3108 + All SQream workers are up and running, SQream-DB is available at 192.168.5.100:3108 + + +Go back to :ref:`Using the SQream-start Commands`. + +Starting Your SQream Services in Split Mode +------------------------ +Starting SQream services in split mode refers to running multiple SQream workers on a single GPU. You can do this by running the **sqream-start** command with the **-s** and **-z** flags. In addition, you can define the amount of hosts to run the multiple workers on. In the example below, the command defines to run the multiple workers on three hosts. + +**To start SQream services in split mode:** + +1. Run the following command: + +.. code-block:: console + + $ ./sqream-start -s -z 3 + +This command starts the SQream metadata, server picker, and sqream workers on a single GPU for three hosts: + +The following is an example of the correct output: + +.. code-block:: console + + Initializing network watchdogs on 3 hosts... + Network watchdogs are up and running + + Initializing 3 worker data collectors ... + Worker data collectors are up and running + + Starting Prometheus ... + Prometheus is available at 192.168.5.101:9090 + + Starting SQream master ... + SQream master is up and running + + Starting up 9 SQream workers over <#> available GPUs ... + All SQream workers are up and running, SQream-DB is available at 192.168.5.101:3108 + +2. Verify all pods are properly running in k8s cluster (**STATUS** column): + +.. code-block:: console + + kubectl -n sqream get pods + + NAME READY STATUS RESTARTS AGE + prometheus-bcf877867-kxhld 1/1 Running 0 106s + sqream-metadata-fbcbc989f-6zlkx 1/1 Running 0 103s + sqream-picker-64b8c57ff5-ndfr9 1/1 Running 2 102s + sqream-split-workers-0-1-2-6bdbfbbb86-ml7kn 1/1 Running 0 57s + sqream-split-workers-3-4-5-5cb49d49d7-596n4 1/1 Running 0 57s + sqream-split-workers-6-7-8-6d598f4b68-2n9z5 1/1 Running 0 56s + sqream-workers-start-xj75g 1/1 Running 0 58s + watchdog-network-management-6dnfh 1/1 Running 0 115s + watchdog-network-management-tfd46 1/1 Running 0 115s + watchdog-network-management-xct4d 1/1 Running 0 115s + watchdog-network-storage-lr6v4 1/1 Running 0 116s + watchdog-network-storage-s29h7 1/1 Running 0 116s + watchdog-network-storage-sx9mw 1/1 Running 0 116s + worker-data-collector-62rxs 0/1 Init:0/1 0 54s + worker-data-collector-n8jsv 0/1 Init:0/1 0 55s + worker-data-collector-zp8vf 0/1 Init:0/1 0 54s + +Go back to :ref:`Using the SQream-start Commands`. + +Starting the Sqream Studio UI +-------------------------------- +You can run the following command the to start the SQream Studio UI (Editor and Dashboard): + +.. code-block:: console + + $ ./sqream-start -a + +The following is an example of the correct output: + +.. code-block:: console + + $ ./sqream-start -a + Please enter USERNAME: + sqream + Please enter PASSWORD: + ****** + Please enter port value or press ENTER to keep 8080: + + Starting up SQream Admin UI... + SQream admin ui is available at 192.168.5.100:8080 + +Go back to :ref:`Using the SQream-start Commands`. + +Stopping the SQream Services +-------------------------------- +You can run the following command to stop all SQream services: + +.. code-block:: console + + $ ./sqream-start -d + +The following is an example of the correct output: + +.. code-block:: console + + $ ./sqream-start -d + $ Cleaning all SQream services in sqream namespace ... + $ All SQream service removed from sqream namespace + +Go back to :ref:`Using the SQream-start Commands`. + +Advanced sqream-start Commands +-------------------------------- +Controlling Your SQream Spool Size +~~~~~~~~~~~~~~~~~~~~~~~~ +If you do not specify the SQream spool size, the console automatically distributes the available RAM between all running workers. + +You can define a specific spool size by running the following command: + +.. code-block:: console + + $ ./sqream-start -s -m 4 + +Using a Custom .json File +~~~~~~~~~~~~~~~~~~~~~~~~ +You have the option of using your own .json file for your own custom configurations. Your .json file must be placed within the path mounted in the installation. SQream recommends placing your .json file in the **configuration** folder. + +The SQream console does not validate the integrity of external .json files. + +You can use the following command (using the ``j`` flag) to set the full path of your .json file to the configuration file: + +.. code-block:: console + + $ ./sqream-start -s -f .json + +This command starts one worker with an external configuration file. + +.. note:: The configuration file must be available in the shared configuration folder. + + +Checking the Status of the SQream Services +~~~~~~~~~~~~~~~~~~~~~~~~ +You can show all running SQream services by running the following command: + +.. code-block:: console + + $ kubectl get pods -n -o wide + +This command shows all running services in the cluster and which nodes they are running in. + +Go back to :ref:`Using the SQream-start Commands`. + +Upgrading Your SQream Version +================================ +The **Upgrading Your SQream Version** section describes the following: + +.. contents:: + :local: + :depth: 1 + +Before Upgrading Your System +---------------------------- +Before upgrading your system you must do the following: + +1. Contact SQream support for a new SQream package tarball file. + +:: + +2. Set a maintenance window. + + +.. note:: You must stop the system while upgrading it. + + +Upgrading Your System +---------------------------- +After completing the steps in **Before Upgrading Your System** above, you can upgrade your system. + +**To upgrade your system:** + +1. Extract the contents of the tarball file that you received from SQream support. Make sure to extract the contents to the same directory as in :ref:`getting_sqream_package` and for the same user: + + .. code-block:: console + + $ tar -xvf sqream_installer-2.0.5-DB2019.2.1-CO1.6.3-ED3.0.0-x86_64/ + $ cd sqream_installer-2.0.5-DB2019.2.1-CO1.6.3-ED3.0.0-x86_64/ + +2. Start the upgrade process run the following command: + + .. code-block:: console + + $ ./sqream-install -i + +The upgrade process checks if the SQream services are running and will prompt you to stop them. + +3. Do one of the following: + + * Stop the upgrade by writing ``No``. + * Continue the upgrade by writing ``Yes``. + +If you continue upgrading, all running SQream workers (master and editor) are stopped. When all services have been stopped, the new version is loaded. + + +.. note:: SQream periodically upgrades its metadata structure. If an upgrade version includes an upgraded metadata service, an approval request message is displayed. This approval is required to finish the upgrade process. Because SQream supports only specific metadata versions, all SQream services must be upgraded at the same time. + + +4. When SQream has successfully upgraded, load the SQream console and restart your services. + +For questions, contact SQream Support. diff --git a/installation_guides/installing_studio_on_stand_alone_server.rst b/installation_guides/installing_studio_on_stand_alone_server.rst new file mode 100644 index 000000000..ecec67c7f --- /dev/null +++ b/installation_guides/installing_studio_on_stand_alone_server.rst @@ -0,0 +1,610 @@ +.. _installing_studio_on_stand_alone_server: + +.. _install_studio_top: + +*********************** +Installing Studio on a Stand-Alone Server +*********************** +The **Installing Studio on a Stand-Alone Server** guide describes how to install SQream Studio on a stand-alone server. A stand-alone server is a server that does not run SQream based on binary files, Docker, or Kubernetes. + +The Installing Studio on a Stand-Alone Server guide includes the following sections: + +.. contents:: + :local: + :depth: 1 + +Installing NodeJS Version 12 on the Server +^^^^^^^^^^^^^^^ +Before installing Studio you must install NodeJS version 12 on the server. + +**To install NodeJS version 12 on the server:** + +1. Check if a version of NodeJS older than version *12.* has been installed on the target server. + + .. code-block:: console + + $ node -v + + The following is the output if a version of NodeJS has already been installed on the target server: + + .. code-block:: console + + bash: /usr/bin/node: No such file or directory + +2. If a version of NodeJS older than *12.* has been installed, remove it as follows: + + * On CentOS: + + .. code-block:: console + + $ sudo yum remove -y nodejs + + * On Ubuntu: + + .. code-block:: console + + $ sudo apt remove -y nodejs + +3. If you have not installed NodeJS version 12, run the following commands: + + * On CentOS: + + .. code-block:: console + + $ curl -sL https://rpm.nodesource.com/setup_12.x | sudo bash - + $ sudo yum clean all && sudo yum makecache fast + $ sudo yum install -y nodejs + + * On Ubuntu: + + .. code-block:: console + + $ curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - + $ sudo apt-get install -y nodejs + + The following output is displayed if your installation has completed successfully: + + .. code-block:: console + + Transaction Summary + ============================================================================================================================== + Install 1 Package + + Total download size: 22 M + Installed size: 67 M + Downloading packages: + warning: /var/cache/yum/x86_64/7/nodesource/packages/nodejs-12.22.1-1nodesource.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 34fa74dd: NOKEY + Public key for nodejs-12.22.1-1nodesource.x86_64.rpm is not installed + nodejs-12.22.1-1nodesource.x86_64.rpm | 22 MB 00:00:02 + Retrieving key from file:///etc/pki/rpm-gpg/NODESOURCE-GPG-SIGNING-KEY-EL + Importing GPG key 0x34FA74DD: + Userid : "NodeSource " + Fingerprint: 2e55 207a 95d9 944b 0cc9 3261 5ddb e8d4 34fa 74dd + Package : nodesource-release-el7-1.noarch (installed) + From : /etc/pki/rpm-gpg/NODESOURCE-GPG-SIGNING-KEY-EL + Running transaction check + Running transaction test + Transaction test succeeded + Running transaction + Warning: RPMDB altered outside of yum. + Installing : 2:nodejs-12.22.1-1nodesource.x86_64 1/1 + Verifying : 2:nodejs-12.22.1-1nodesource.x86_64 1/1 + + Installed: + nodejs.x86_64 2:12.22.1-1nodesource + + Complete! + +4. Confirm the Node version. + + .. code-block:: console + + $ node -v + + The following is an example of the correct output: + + .. code-block:: console + + v12.22.1 + +5. Install Prometheus using binary packages. + + For more information on installing Prometheus using binary packages, see :ref:`installing_prometheus_using_binary_packages`. + +Back to :ref:`Installing Studio on a Stand-Alone Server` + + + +Installing Studio +^^^^^^^^^^^^^^^ +After installing the Dashboard Data Collector, you can install Studio. + +**To install Studio:** + +1. Copy the SQream Studio package from SQream Artifactory into the target server. For access to the Sqream Studio package, contact SQream Support. + +:: + +2. Extract the package: + + .. code-block:: console + + $ tar -xvf sqream-acceleration-studio-.x86_64.tar.gz + +:: + +3. Navigate to the new package folder. + + .. code-block:: console + + $ cd sqream-admin + +.. _add_parameter: + +4. Build the configuration file to set up Sqream Studio. You can use IP address **127.0.0.1** on a single server. + + .. code-block:: console + + $ npm run setup -- -y --host= --port=3108 --data-collector-url=http://:8100/api/dashboard/data + + The above command creates the **sqream-admin-config.json** configuration file in the **sqream-admin** folder and shows the following output: + + .. code-block:: console + + Config generated successfully. Run `npm start` to start the app. + + For more information about the available set-up arguments, see :ref:`Set-Up Arguments`. + + :: + +5. To access Studio over a secure connection, in your configuration file do the following: + + #. Change your ``port`` value to **3109**. + + :: + + #. Change your ``ssl`` flag value to **true**. + + The following is an example of the correctly modified configuration file: + + .. code-block:: console + + { + "debugSqream": false, + "webHost": "localhost", + "webPort": 8080, + "webSslPort": 8443, + "logsDirectory": "", + "clusterType": "standalone", + "dataCollectorUrl": "", + "connections": [ + { + "host": "127.0.0.1", + "port":3109, + "isCluster": true, + "name": "default", + "service": "sqream", + "ssl":true, + "networkTimeout": 60000, + "connectionTimeout": 3000 + } + ] + } + +5. If you have installed Studio on a server where SQream is already installed, move the **sqream-admin-config.json** file to **/etc/sqream/**: + + .. code-block:: console + + $ mv sqream-admin-config.json /etc/sqream + +Back to :ref:`Installing Studio on a Stand-Alone Server` + +Starting Studio Manually +^^^^^^^^^^^^^^^ +You can start Studio manually by running the following command: + +.. code-block:: console + + $ cd /home/sqream/sqream-admin + $ NODE_ENV=production pm2 start ./server/build/main.js --name=sqream-studio -- start + +The following output is displayed: + +.. code-block:: console + + [PM2] Starting /home/sqream/sqream-admin/server/build/main.js in fork_mode (1 instance) + [PM2] Done. + ┌─────┬──────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐ + │ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │ + ├─────┼──────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤ + │ 0 │ sqream-studio │ default │ 0.1.0 │ fork │ 11540 │ 0s │ 0 │ online │ 0% │ 15.6mb │ sqream │ disabled │ + └─────┴──────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘ + +Starting Studio as a Service +^^^^^^^^^^^^^^^ +Sqream uses the **Process Manager (PM2)** to maintain Studio. + +**To start Studio as a service:** + +1. Run the following command: + + .. code-block:: console + + $ sudo npm install -g pm2 + +:: + +2. Verify that the PM2 has been installed successfully. + + .. code-block:: console + + $ pm2 list + + The following is the output: + + .. code-block:: console + + ┌─────┬──────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐ + │ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │ + ├─────┼──────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤ + │ 0 │ sqream-studio │ default │ 0.1.0 │ fork │ 11540 │ 2m │ 0 │ online │ 0% │ 31.5mb │ sqream │ disabled │ + └─────┴──────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘ + +:: + +2. Start the service with PM2: + + * If the **sqream-admin-config.json** file is located in **/etc/sqream/**, run the following command: + + .. code-block:: console + + $ cd /home/sqream/sqream-admin + $ NODE_ENV=production pm2 start ./server/build/main.js --name=sqream-studio -- start --config-location=/etc/sqream/sqream-admin-config.json + + * If the **sqream-admin-config.json** file is not located in **/etc/sqream/**, run the following command: + + .. code-block:: console + + $ cd /home/sqream/sqream-admin + $ NODE_ENV=production pm2 start ./server/build/main.js --name=sqream-studio -- start + +:: + +3. Verify that Studio is running. + + .. code-block:: console + + $ netstat -nltp + +4. Verify that SQream_studio is listening on port 8080, as shown below: + + .. code-block:: console + + (Not all processes could be identified, non-owned process info + will not be shown, you would have to be root to see it all.) + Active Internet connections (only servers) + Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name + tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - + tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN - + tcp6 0 0 :::8080 :::* LISTEN 11540/sqream-studio + tcp6 0 0 :::22 :::* LISTEN - + tcp6 0 0 ::1:25 :::* LISTEN - + + + +:: + +5. Verify the following: + + 1. That you can access Studio from your browser (``http://:8080``). + + :: + + 2. That you can log in to SQream. + +6. Save the configuration to run on boot. + + .. code-block:: console + + $ pm2 startup + + The following is an example of the output: + + .. code-block:: console + + $ sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u sqream --hp /home/sqream + +7. Copy and paste the output above and run it. + +:: + +8. Save the configuration. + + .. code-block:: console + + $ pm2 save + +Back to :ref:`Installing Studio on a Stand-Alone Server` + +Accessing Studio +^^^^^^^^^^^^^^^ +The Studio page is available on port 8080: ``http://:8080``. + +If port 8080 is blocked by the server firewall, you can unblock it by running the following command: + + .. code-block:: console + + $ firewall-cmd --zone=public --add-port=8080/tcp --permanent + $ firewall-cmd --reload + +Back to :ref:`Installing Studio on a Stand-Alone Server` + +Maintaining Studio with the Process Manager (PM2) +^^^^^^^^^^^^^^^ +Sqream uses the **Process Manager (PM2)** to maintain Studio. + +You can use PM2 to do one of the following: + +* To check the PM2 service status: ``pm2 list`` + + :: + +* To restart the PM2 service: ``pm2 reload sqream-studio`` + + :: + +* To see the PM2 service logs: ``pm2 logs sqream-studio`` + +Back to :ref:`Installing Studio on a Stand-Alone Server` + +Upgrading Studio +^^^^^^^^^^^^^^^ +To upgrade Studio you need to stop the version that you currently have. + +**To stop the current version of Studio:** + +1. List the process name: + + .. code-block:: console + + $ pm2 list + + The process name is displayed. + + .. code-block:: console + + + +:: + +2. Run the following command with the process name: + + .. code-block:: console + + $ pm2 stop + +:: + +3. If only one process is running, run the following command: + + .. code-block:: console + + $ pm2 stop all + +:: + +4. Change the name of the current **sqream-admin** folder to the old version. + + .. code-block:: console + + $ mv sqream-admin sqream-admin- + +:: + +5. Extract the new Studio version. + + .. code-block:: console + + $ tar -xf sqream-acceleration-studio-tar.gz + +:: + +6. Rebuild the configuration file. You can use IP address **127.0.0.1** on a single server. + + .. code-block:: console + + $ npm run setup -- -y --host= --port=3108 + + The above command creates the **sqream-admin-config.json** configuration file in the **sqream_admin** folder. + +:: + +7. Copy the **sqream-admin-config.json** configuration file to **/etc/sqream/** to overwrite the old configuration file. + +:: + +8. Start PM2. + + .. code-block:: console + + $ pm2 start all + +Back to :ref:`Installing Studio on a Stand-Alone Server` + +.. _install_studio_docker_container: + +Installing Studio in a Docker Container +^^^^^^^^^^^^^^^^^^^^^^^ +This guide explains how to install SQream Studio in a Docker container and includes the following sections: + +.. contents:: + :local: + :depth: 1 + +Installing Studio +-------------- +If you have already installed Docker, you can install Studio in a Docker container. + +**To install Studio:** + +1. Copy the downloaded image onto the target server. + +:: + +2. Load the Docker image. + + .. code-block:: console + + $ docker load -i + +:: + +3. If the downloaded image is called **sqream-acceleration-studio-5.1.3.x86_64.docker18.0.3.tar,** run the following command: + + .. code-block:: console + + $ docker load -i sqream-acceleration-studio-5.1.3.x86_64.docker18.0.3.tar + +:: + +4. Start the Docker container. + + .. code-block:: console + + $ docker run -d --restart=unless-stopped -p :8080 -e runtime=docker -e SQREAM_K8S_PICKER= -e SQREAM_PICKER_PORT= -e SQREAM_DATABASE_NAME= -e SQREAM_ADMIN_UI_PORT=8080 --name=sqream-admin-ui + + The following is an example of the command above: + + .. code-block:: console + + $ docker run -d --name sqream-studio -p 8080:8080 -e runtime=docker -e SQREAM_K8S_PICKER=192.168.0.183 -e SQREAM_PICKER_PORT=3108 -e SQREAM_DATABASE_NAME=master -e SQREAM_ADMIN_UI_PORT=8080 sqream-acceleration-studio:5.1.3 + +Back to :ref:`Installing Studio in a Docker Container` + +Accessing Studio +----------------- +You can access Studio from Port 8080: ``http://:8080``. + +If you want to use Studio over a secure connection (https), you must use the parameter values shown in the following table: + +.. list-table:: + :widths: 10 25 65 + :header-rows: 1 + + * - Parameter + - Default Value + - Description + * - ``--web-ssl-port`` + - 8443 + - + * - ``--web-ssl-key-path`` + - None + - The path of SSL key PEM file for enabling https. Leave empty to disable. + * - ``--web-ssl-cert-path`` + - None + - The path of SSL certificate PEM file for enabling https. Leave empty to disable. + +You can configure the above parameters using the following syntax: + +.. code-block:: console + + $ npm run setup -- -y --host=127.0.0.1 --port=3108 + +.. _using_docker_container_commands: + +Back to :ref:`Installing Studio in a Docker Container` + +Using Docker Container Commands +--------------- +When installing Studio in Docker, you can run the following commands: + +* View Docker container logs: + + .. code-block:: console + + $ docker logs -f sqream-admin-ui + +* Restart the Docker container: + + .. code-block:: console + + $ docker restart sqream-admin-ui + +* Kill the Docker container: + + .. code-block:: console + + $ docker rm -f sqream-admin-ui + +Back to :ref:`Installing Studio in a Docker Container` + +Setting Up Argument Configurations +---------------- +When creating the **sqream-admin-config.json** configuration file, you can add ``-y`` to create the configuration file in non-interactive mode. Configuration files created in non-interactive mode use all the parameter defaults not provided in the command. + +The following table shows the available arguments: + +.. list-table:: + :widths: 10 25 65 + :header-rows: 1 + + * - Parameter + - Default Value + - Description + * - ``--web--host`` + - 8443 + - + * - ``--web-port`` + - 8080 + - + * - ``--web-ssl-port`` + - 8443 + - + * - ``--web-ssl-key-path`` + - None + - The path of the SSL Key PEM file for enabling https. Leave empty to disable. + * - ``--web-ssl-cert-path`` + - None + - The path of the SSL Certificate PEM file for enabling https. Leave empty to disable. + * - ``--debug-sqream (flag)`` + - false + - + * - ``--host`` + - 127.0.0.1 + - + * - ``--port`` + - 3108 + - + * - ``is-cluster (flag)`` + - true + - + * - ``--service`` + - sqream + - + * - ``--ssl (flag)`` + - false + - Enables the SQream SSL connection. + * - ``--name`` + - default + - + * - ``--data-collector-url`` + - localhost:8100/api/dashboard/data + - Enables the Dashboard. Leaving this blank disables the Dashboard. Using a mock URL uses mock data. + * - ``--cluster-type`` + - standalone (``standalone`` or ``k8s``) + - + * - ``--config-location`` + - ./sqream-admin-config.json + - + * - ``--network-timeout`` + - 60000 (60 seconds) + - + * - ``--access-key`` + - None + - If defined, UI access is blocked unless ``?ui-access=`` is included in the URL. + +Back to :ref:`Installing Studio in a Docker Container` + + :: + +Back to :ref:`Installing Studio on a Stand-Alone Server` \ No newline at end of file diff --git a/installation_guides/launching_sqream_with_monit.rst b/installation_guides/launching_sqream_with_monit.rst new file mode 100644 index 000000000..b8dc4f1a6 --- /dev/null +++ b/installation_guides/launching_sqream_with_monit.rst @@ -0,0 +1,290 @@ +.. _launching_sqream_with_monit: + +********************************************* +Launching SQream with Monit +********************************************* +This procedure describes how to launch SQream using Monit. + +Launching SQream +==================================== + +After doing the following, you can launch SQream according to the instructions on this page. + + + +1. :ref:`Installing Monit ` +2. :ref:`Installing SQream with Binary ` + + + + +The following is an example of a working monitrc file configured to monitor the ***metadataserver** and **serverpicker** commands, and **four sqreamd services**. The **monitrc** configuration file is located in the **conf/monitrc** directory. + +Note that the **monitrc** in the following example is configured for eight ``sqreamd`` services, but that only the first four are enabled: + +.. code-block:: console + + $ set daemon 5 # check services at 30 seconds intervals + $ set logfile syslog + $ + $ set httpd port 2812 and + $ use address localhost # only accept connection from localhost + $ allow localhost # allow localhost to connect to the server and + $ allow admin:monit # require user 'admin' with password 'monit' + $ + $ ##set mailserver smtp.gmail.com port 587 + $ ## using tlsv12 + $ #METADATASERVER-START + $ check process metadataserver with pidfile /var/run/metadataserver.pid + $ start program = "/usr/bin/systemctl start metadataserver" + $ stop program = "/usr/bin/systemctl stop metadataserver" + $ #METADATASERVER-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: metadataserver $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ #SERVERPICKER-START + $ check process serverpicker with pidfile /var/run/serverpicker.pid + $ start program = "/usr/bin/systemctl start serverpicker" + $ stop program = "/usr/bin/systemctl stop serverpicker" + $ #SERVERPICKER-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: serverpicker $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # + $ # + $ #SQREAM1-START + $ check process sqream1 with pidfile /var/run/sqream1.pid + $ start program = "/usr/bin/systemctl start sqream1" + $ stop program = "/usr/bin/systemctl stop sqream1" + $ #SQREAM1-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream1 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ #SQREAM2-START + $ check process sqream2 with pidfile /var/run/sqream2.pid + $ start program = "/usr/bin/systemctl start sqream2" + $ #SQREAM2-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream1 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ #SQREAM3-START + $ check process sqream3 with pidfile /var/run/sqream3.pid + $ start program = "/usr/bin/systemctl start sqream3" + $ stop program = "/usr/bin/systemctl stop sqream3" + $ #SQREAM3-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream2 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ #SQREAM4-START + $ check process sqream4 with pidfile /var/run/sqream4.pid + $ start program = "/usr/bin/systemctl start sqream4" + $ stop program = "/usr/bin/systemctl stop sqream4" + $ #SQREAM4-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream2 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ # + $ #SQREAM5-START + $ #check process sqream5 with pidfile /var/run/sqream5.pid + $ #start program = "/usr/bin/systemctl start sqream5" + $ #stop program = "/usr/bin/systemctl stop sqream5" + $ #SQREAM5-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream2 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ # + $ #SQREAM6-START + $ #check process sqream6 with pidfile /var/run/sqream6.pid + $ #start program = "/usr/bin/systemctl start sqream6" + $ #stop program = "/usr/bin/systemctl stop sqream6" + $ #SQREAM6-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream2 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ # + $ #SQREAM7-START + $ #check process sqream7 with pidfile /var/run/sqream7.pid + $ #start program = "/usr/bin/systemctl start sqream7" + $ #stop program = "/usr/bin/systemctl stop sqream7" + $ #SQREAM7-END + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream2 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + $ # + $ #SQREAM8-START + $ #check process sqream8 with pidfile /var/run/sqream8.pid + $ #start program = "/usr/bin/systemctl start sqream8" + $ #stop program = "/usr/bin/systemctl stop sqream8" + $ #SQREAM8-END + $ # alert user@domain.com on {nonexist, timeout} + $ # with mail-format { + $ # from: Monit@$HOST + $ # subject: sqream2 $EVENT - $ACTION + $ # message: This is an automate mail, sent from monit. + $ # } + +Monit Usage Examples +==================================== + +This section shows examples of two methods for stopping the **sqream3** service use Monit's command syntax: + + + +* :ref:`Stopping Monit and SQream separately ` +* :ref:`Stopping SQream using a Monit command ` + +.. _stopping_monit_and_sqream_separately: + +Stopping Monit and SQream Separately +------------------------------------- + +You can stop the Monit service and SQream separately as follows: + +.. code-block:: console + + $ sudo systemctl stop monit + $ sudo systemctl stop sqream3 + +You can restart Monit as follows: + +.. code-block:: console + + $ sudo systemctl start monit + +Restarting Monit automatically restarts the SQream services. + +.. _stopping_sqream_using_a_monit_command: + +Stopping SQream Using a Monit Command +------------------------------------- + +You can stop SQream using a Monit command as follows: + +.. code-block:: console + + $ sudo monit stop sqream3 + +This command stops SQream only (and not Monit). + +You can restart SQream as follows: + +.. code-block:: console + + $ sudo monit start sqream3 + +Monit Command Line Options +------------------------------------- +The **Monit Command Line Options** section describes some of the most commonly used Monit command options. + +You can show the command line options by running: + +.. code-block:: console + + $ monit --help + +.. code-block:: console + + $ start all - Start all services + $ start - Only start the named service + $ stop all - Stop all services + $ stop - Stop the named service + $ restart all - Stop and start all services + $ restart - Only restart the named service + $ monitor all - Enable monitoring of all services + $ monitor - Only enable monitoring of the named service + $ unmonitor all - Disable monitoring of all services + $ unmonitor - Only disable monitoring of the named service + $ reload - Reinitialize monit + $ status [name] - Print full status information for service(s) + $ summary [name] - Print short status information for service(s) + $ report [up|down|..] - Report state of services. See manual for options + $ quit - Kill the monit daemon process + $ validate - Check all services and start if not running + $ procmatch - Test process matching pattern + +Using Monit While Upgrading Your Version of SQream +================================================== + +While upgrading your version of SQream, you can use Monit to avoid conflicts (such as service start). This is done by pausing or stopping all running services while you manually upgrade SQream. When you finish successfully upgrading SQream, you can use Monit to restart all SQream services + +**To use Monit while upgrading your version of SQream:** + +1. Stop all actively running SQream services: + + .. code-block:: console + + $ sudo monit stop all + +2. Verify that SQream has stopped listening on ports **500X**, **510X**, and **310X**: + + .. code-block:: console + + $ sudo netstat -nltp #to make sure sqream stopped listening on 500X, 510X and 310X ports. + + The example below shows the old version ``sqream-db-v2020.2`` being replaced with the new version ``sqream-db-v2025.200``. + + .. code-block:: console + + $ cd /home/sqream + $ mkdir tempfolder + $ mv sqream-db-v2025.200.tar.gz tempfolder/ + $ tar -xf sqream-db-v2025.200.tar.gz + $ sudo mv sqream /usr/local/sqream-db-v2025.200 + $ cd /usr/local + $ sudo chown -R sqream:sqream sqream-db-v2025.200 + $ sudo rm sqream #This only should remove symlink + $ sudo ln -s sqream-db-v2025.200 sqream #this will create new symlink named "sqream" pointing to new version + $ ls -l + + The symbolic SQream link should point to the real folder: + + .. code-block:: console + + $ sqream -> sqream-db-v2025.200 + +4. Restart the SQream services: + + .. code-block:: console + + $ sudo monit start all + +5. Verify that the latest version has been installed: + + .. code-block:: console + + $ SELECT SHOW_VERSION(); + + The correct version is output. + + :: + +6. Restart the UI: + + .. code-block:: console + + $ pm2 start all diff --git a/installation_guides/recommended_pre-installation_configurations.rst b/installation_guides/recommended_pre-installation_configurations.rst new file mode 100644 index 000000000..cf6328a34 --- /dev/null +++ b/installation_guides/recommended_pre-installation_configurations.rst @@ -0,0 +1,1289 @@ +.. _recommended_pre-installation_configurations: + +********************************************* +Recommended Pre-Installation Configuration +********************************************* +Before :ref:`installing SQream DB`, SQream recommends you to tune your system for better performance and stability. + +This page provides recommendations for production deployments of SQream and describes the following: + +.. contents:: + :local: + :depth: 1 + +Recommended BIOS Settings +========================== +The first step when setting your pre-installation configurations is to use the recommended BIOS settings. + +The BIOS settings may have a variety of names, or may not exist on your system. Each system vendor has a different set of settings and variables. It is safe to skip any and all of the configuration steps, but this may impact performance. + +If any doubt arises, consult the documentation for your server or your hardware vendor for the correct way to apply the settings. + +.. list-table:: + :widths: 25 25 50 + :header-rows: 1 + + * - Item + - Setting + - Rationale + * - **Management console access** + - **Connected** + - Connection to OOB required to preserve continuous network uptime. + * - **All drives** + - **Connected and displayed on RAID interface** + - Prerequisite for cluster or OS installation. + * - **RAID volumes.** + - **Configured according to project guidelines. Must be rebooted to take effect.** + - Clustered to increase logical volume and provide redundancy. + * - **Fan speed Thermal Configuration.** + - Dell fan speed: **High Maximum**. Specified minimum setting: **60**. HPe thermal configuration: **Increased cooling**. + - NVIDIA Tesla GPUs are passively cooled and require high airflow to operate at full performance. + * - **Power regulator or iDRAC power unit policy** + - HPe: **HP static high performance** mode enabled. Dell: **iDRAC power unit policy** (power cap policy) disabled. + - Other power profiles (such as "balanced") throttle the CPU and diminishes performance. Throttling may also cause GPU failure. + * - **System Profile**, **Power Profile**, or **Performance Profile** + - **High Performance** + - The Performance profile provides potentially increased performance by maximizing processor frequency, and the disabling certain power saving features such as C-states. Use this setting for environments that are not sensitive to power consumption. + * - **Power Cap Policy** or **Dynamic power capping** + - **Disabled** + - Other power profiles (like "balanced") throttle the CPU and may diminish performance or cause GPU failure. This setting may appear together with the above (Power profile or Power regulator). This setting allows disabling system ROM power calibration during the boot process. Power regulator settings are named differently in BIOS and iLO/iDRAC. + * - **Intel Turbo Boost** + - **Enabled** + - Intel Turbo Boost enables overclocking the processor to boost CPU-bound operation performance. Overclocking may risk computational jitter due to changes in the processor's turbo frequency. This causes brief pauses in processor operation, introducing uncertainty into application processing time. Turbo operation is a function of power consumption, processor temperature, and the number of active cores. + * - **Logical Processor** + - **HPe**: Enable **Hyperthreading** **Dell**: Enable **Logical Processor** + - Hyperthreading doubles the amount of logical processors, which may improve performance by ~5-10% for CPU-bound operations. + * - **Intel Virtualization Technology** (VT-d) + - **Disable** + - VT-d is optimal for running VMs. However, when running Linux natively, disabling VT-d boosts performance by up to 10%. + * - **Logical Processor** + - **HPe**: Enable **Hyperthreading** **Dell**: Enable **Logical Processor** + - Hyperthreading doubles the amount of logical processors, which may improve performance by ~5-10% for CPU-bound operations. + * - **Intel Virtualization Technology** (VT-d) + - **Disable** + - VT-d is optimal for running VMs. However, when running Linux natively, disabling VT-d boosts performance by up to 10%. + * - **Processor C-States** (Minimum processor idle power core state) + - **Disable** + - Processor C-States reduce server power when the system is in an idle state. This causes slower cold-starts when the system transitions from an idle to a load state, and may reduce query performance by up to 15%. + * - **HPe**: **Energy/Performance bias** + - **Maximum performance** + - Configures processor sub-systems for high-performance and low-latency. Other power profiles (like "balanced") throttle the CPU and may diminish performance. Use this setting for environments that are not sensitive to power consumption. + * - **HPe**: **DIMM voltage** + - **Optimized for Performance** + - Setting a higher voltage for DIMMs may increase performance. + * - **Memory Operating Mode** + - **Optimizer Mode**, **Disable Node Interleaving**, **Auto Memory Operating Voltage** + - Memory Operating Mode is tuned for performance in **Optimizer** mode. Other modes may improve reliability, but reduce performance. **Node Interleaving** should be disabled because enabling it interleaves the memory between memory nodes, which harms NUMA-aware applications such as SQream DB. + * - **HPe**: **Memory power savings mode** + - **Maximum performance** + - This setting configures several memory parameters to optimize the performance of memory sub-systems. The default setting is **Balanced**. + * - **HPe ACPI SLIT** + - **Enabled** + - ACPI SLIT sets the relative access times between processors and memory and I/O sub-systems. ACPI SLIT enables operating systems to use this data to improve performance by more efficiently allocating resources and workloads. + * - **QPI Snoop** + - **Cluster on Die** or **Home Snoop** + - QPI (QuickPath Interconnect) Snoop lets you configure different Snoop modes that impact the QPI interconnect. Changing this setting may improve the performance of certain workloads. The default setting of **Home Snoop** provides high memory bandwidth in an average NUMA environment. **Cluster on Die** may provide increased memory bandwidth in highly optimized NUMA workloads. **Early Snoop** may decrease memory latency, but may result in lower overall bandwidth compared to other modes. + +Installing the Operating System +=================================================== +Once the BIOS settings have been set, you must install the operating system. Either the CentOS (versions 7.6-7.9) or RHEL (versions 7.6-7.9) must be installed before installing the SQream database, by either the customer or a SQream representative. + +**To install the operating system:** + +#. Select a language (English recommended). +#. From **Software Selection**, select **Minimal**. +#. Select the **Development Tools** group checkbox. +#. Continue the installation. +#. Set up the necessary drives and users as per the installation process. + + Using Debugging Tools is recommended for future problem-solving if necessary. + +Selecting the **Development Tools** group installs the following tools: + + * autoconf + * automake + * binutils + * bison + * flex + * gcc + * gcc-c++ + * gettext + * libtool + * make + * patch + * pkgconfig + * redhat-rpm-config + * rpm-build + * rpm-sign + +The root user is created and the OS shell is booted up. + +Configuring the Operating System +=================================================== +Once you've installted your operation system, you can configure it. When configuring the operating system, several basic settings related to creating a new server are required. Configuring these as part of your basic set-up increases your server's security and usability. + +Logging In to the Server +-------------------------------- +You can log in to the server using the server's IP address and password for the **root** user. The server's IP address and **root** user were created while installing the operating system above. + +Automatically Creating a SQream User +------------------------------------ + +**To automatically create a SQream user:** + +#. If a SQream user was created during installation, verify that the same ID is used on every server: + + .. code-block:: console + + $ sudo id sqream + + The ID **1000** is used on each server in the following example: + + .. code-block:: console + + $ uid=1000(sqream) gid=1000(sqream) groups=1000(sqream) + +2. If the ID's are different, delete the SQream user and SQream group from both servers: + + .. code-block:: console + + $ sudo userdel sqream + +3. Recreate it using the same ID: + + .. code-block:: console + + $ sudo rm /var/spool/mail/sqream + +Manually Creating a SQream User +-------------------------------- + +**To manually create a SQream user:** + +SQream enables you to manually create users. This section shows you how to manually create a user with the UID **1111**. You cannot manually create during the operating system installation procedure. + +1. Add a user with an identical UID on all cluster nodes: + + .. code-block:: console + + $ useradd -u 1111 sqream + +2. Add the user **sqream** to the **wheel** group. + + .. code-block:: console + + $ sudo usermod -aG wheel sqream + + You can remove the SQream user from the **wheel** group when the installation and configuration are complete: + + .. code-block:: console + + $ passwd sqream + +3. Log out and log back in as **sqream**. + + .. note:: If you deleted the **sqream** user and recreated it with different ID, to avoid permission errors, you must change its ownership to /home/sqream. + +4. Change the **sqream** user's ownership to /home/sqream: + + .. code-block:: console + + $ sudo chown -R sqream:sqream /home/sqream + +Setting Up A Locale +-------------------------------- + +SQream enables you to set up a locale. In this example, the locale used is your own location. + +**To set up a locale:** + +1. Set the language of the locale: + + .. code-block:: console + + $ sudo localectl set-locale LANG=en_US.UTF-8 + +2. Set the time stamp (time and date) of the locale: + + .. code-block:: console + + $ sudo timedatectl set-timezone Asia/Jerusalem + +If needed, you can run the **timedatectl list-timezones** command to see your current time-zone. + + +Installing the Required Packages +-------------------------------- +You can install the required packages by running the following command: + +.. code-block:: console + + $ sudo yum install ntp pciutils monit zlib-devel openssl-devel kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc net-tools wget jq + + +Installing the Recommended Tools +-------------------------------- +You can install the recommended tools by running the following command: + +.. code-block:: console + + $ sudo yum install bash-completion.noarch vim-enhanced vim-common net-tools iotop htop psmisc screen xfsprogs wget yum-utils deltarpm dos2unix + + +Installing Python 3.6.7 +-------------------------------- +1. Download the Python 3.6.7 source code tarball file from the following URL into the **/home/sqream** directory: + + .. code-block:: console + + $ wget https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tar.xz + +2. Extract the Python 3.6.7 source code into your current directory: + + .. code-block:: console + + $ tar -xf Python-3.6.7.tar.xz + +3. Navigate to the Python 3.6.7 directory: + + .. code-block:: console + + $ cd Python-3.6.7 + +4. Run the **./configure** script: + + .. code-block:: console + + $ ./configure + +5. Build the software: + + .. code-block:: console + + $ make -j30 + +6. Install the software: + + .. code-block:: console + + $ sudo make install + +7. Verify that Python 3.6.7 has been installed: + + .. code-block:: console + + $ python3 + +Installing NodeJS on CentOS +-------------------------------- +**To install the node.js on CentOS:** + +1. Download the `setup_12.x file `__ as a root user logged in shell: + + .. code-block:: console + + $ curl -sL https://rpm.nodesource.com/setup_12.x | sudo bash - + +2. Clear the YUM cache and update the local metadata: + + .. code-block:: console + + $ sudo yum clean all && sudo yum makecache fast + +3. Install the **node.js** file: + + .. code-block:: console + + $ sudo yum install -y nodejs + +4. Install npm and make it available for all users: + + .. code-block:: console + + $ sudo npm install pm2 -g + +Installing NodeJS on Ubuntu +-------------------------------- +**To install the node.js file on Ubuntu:** + +1. Download the `setup_12.x file `__ as a root user logged in shell: + + .. code-block:: console + + $ curl -sL https://rpm.nodesource.com/setup_12.x | sudo bash - + +2. Install the node.js file: + + .. code-block:: console + + $ sudo apt-get install -y nodejs + +3. Install npm and make it available for all users: + + .. code-block:: console + + $ sudo npm install pm2 -g + +Installing NodeJS Offline +------------------------------------------- +**To install NodeJS Offline** + +1. Download the NodeJS source code tarball file from the following URL into the **/home/sqream** directory: + + .. code-block:: console + + $ wget https://nodejs.org/dist/v12.13.0/node-v12.13.0-linux-x64.tar.xz + +2. Move the node-v12.13.0-linux-x64 file to the */usr/local* directory. + + .. code-block:: console + + $ sudo mv node-v12.13.0-linux-x64 /usr/local + +3. Navigate to the */usr/bin/* directory: + + .. code-block:: console + + $ cd /usr/bin + +4. Create a symbolic link to the */local/node-v12.13.0-linux-x64/bin/node node* directory: + + .. code-block:: console + + $ sudo ln -s ../local/node-v12.13.0-linux-x64/bin/node node + +5. Create a symbolic link to the */local/node-v12.13.0-linux-x64/bin/npm npm* directory: + + .. code-block:: console + + $ sudo ln -s ../local/node-v12.13.0-linux-x64/bin/npm npm + +6. Create a symbolic link to the */local/node-v12.13.0-linux-x64/bin/npx npx* directory: + + .. code-block:: console + + $ sudo ln -s ../local/node-v12.13.0-linux-x64/bin/npx npx + +7. Verify that the node versions for the above are correct: + + .. code-block:: console + + $ node --version + +Installing the pm2 Service Offline +------------------------------------------- +**To install the pm2 Service Offline** + +1. On a machine with internet access, install the following: + + * nodejs + * npm + * pm2 + +2. Extract the pm2 module to the correct directory: + + .. code-block:: console + + $ cd /usr/local/node-v12.13.0-linux-x64/lib/node_modules + $ tar -czvf pm2_x86.tar.gz pm2 + +3. Copy the **pm2_x86.tar.gz** file to a server without access to the internet and extract it. + + :: + +4. Move the **pm2** folder to the */usr/local/node-v12.13.0-linux-x64/lib/node_modules* directory: + + .. code-block:: console + + $ sudo mv pm2 /usr/local/node-v12.13.0-linux-x64/lib/node_modules + +5. Navigate back to the */usr/bin* directory: + + .. code-block:: console + + $ cd /usr/bin again + +6. Create a symbolink to the **pm2** service: + + .. code-block:: console + + $ sudo ln -s /usr/local/node-v12.22.3-linux-x64/lib/node_modules/pm2/bin/pm2 pm2 + +7. Verify that installation was successful: + + .. code-block:: console + + $ pm2 list + + .. note:: This must be done as a **sqream** user, and not as a **sudo** user. + +8. Verify that the node version is correct: + + .. code-block:: console + + $ node -v + +Configuring the Network Time Protocol +------------------------------------------- +This section describes how to configure your **Network Time Protocol (NTP)**. + +If you don't have internet access, see `Configure NTP Client to Synchronize with NTP Server `__. + +**To configure your NTP:** + +1. Install the NTP file. + + .. code-block:: console + + $ sudo yum install ntp + +2. Enable the **ntpd** program. + + .. code-block:: console + + $ sudo systemctl enable ntpd + +3. Start the **ntdp** program. + + .. code-block:: console + + $ sudo systemctl start ntpd + +4. Print a list of peers known to the server and a summary of their states. + + .. code-block:: console + + $ sudo ntpq -p + +Configuring the Network Time Protocol Server +-------------------------------------------- +If your organization has an NTP server, you can configure it. + +**To configure your NTP server:** + +1. Output your NTP server address and append ``/etc/ntpd.conf`` to the outuput. + + .. code-block:: console + + $ echo -e "\nserver \n" | sudo tee -a /etc/ntp.conf + +2. Restart the service. + + .. code-block:: console + + $ sudo systemctl restart ntpd + +3. Check that synchronization is enabled: + + .. code-block:: console + + $ sudo timedatectl + + Checking that synchronization is enabled generates the following output: + + .. code-block:: console + + $ Local time: Sat 2019-10-12 17:26:13 EDT + Universal time: Sat 2019-10-12 21:26:13 UTC + RTC time: Sat 2019-10-12 21:26:13 + Time zone: America/New_York (EDT, -0400) + NTP enabled: yes + NTP synchronized: yes + RTC in local TZ: no + DST active: yes + Last DST change: DST began at + Sun 2019-03-10 01:59:59 EST + Sun 2019-03-10 03:00:00 EDT + Next DST change: DST ends (the clock jumps one hour backwards) at + Sun 2019-11-03 01:59:59 EDT + Sun 2019-11-03 01:00:00 EST + +Configuring the Server to Boot Without the UI +--------------------------------------------- +You can configure your server to boot without a UI in cases when it is not required (recommended) by running the following command: + +.. code-block:: console + + $ sudo systemctl set-default multi-user.target + +Running this command activates the **NO-UI** server mode. + +Configuring the Security Limits +-------------------------------- +The security limits refers to the number of open files, processes, etc. + +You can configure the security limits by running the **echo -e** command as a root user logged in shell: + +.. code-block:: console + + $ sudo bash + +.. code-block:: console + + $ echo -e "sqream soft nproc 1000000\nsqream hard nproc 1000000\nsqream soft nofile 1000000\nsqream hard nofile 1000000\nsqream soft core unlimited\nsqream hard core unlimited" >> /etc/security/limits.conf + +Configuring the Kernel Parameters +--------------------------------- +**To configure the kernel parameters:** + +1. Insert a new line after each kernel parameter: + + .. code-block:: console + + $ echo -e "vm.dirty_background_ratio = 5 \n vm.dirty_ratio = 10 \n vm.swappiness = 10 \n vm.vfs_cache_pressure = 200 \n vm.zone_reclaim_mode = 0 \n" >> /etc/sysctl.conf + + .. note:: In the past, the **vm.zone_reclaim_mode** parameter was set to **7.** In the latest Sqream version, the vm.zone_reclaim_mode parameter must be set to **0**. If it is not set to **0**, when a numa node runs out of memory, the system will get stuck and will be unable to pull memory from other numa nodes. + +2. Check the maximum value of the **fs.file**. + + .. code-block:: console + + $ sysctl -n fs.file-max + +3. If the maximum value of the **fs.file** is smaller than **2097152**, run the following command: + + .. code-block:: console + + $ echo "fs.file-max=2097152" >> /etc/sysctl.conf + + **IP4 forward** must be enabled for Docker and K8s installation only. + +4. Run the following command: + + .. code-block:: console + + $ sudo echo “net.ipv4.ip_forward = 1” >> /etc/sysctl.conf + +5. Reboot your system: + + .. code-block:: console + + $ sudo reboot + +Configuring the Firewall +-------------------------------- +The example in this section shows the open ports for four sqreamd sessions. If more than four are required, open the required ports as needed. Port 8080 in the example below is a new UI port. + +**To configure the firewall:** + +1. Start the service and enable FirewallID on boot: + + .. code-block:: console + + $ systemctl start firewalld + +2. Add the following ports to the permanent firewall: + + .. code-block:: console + + $ firewall-cmd --zone=public --permanent --add-port=8080/tcp + $ firewall-cmd --zone=public --permanent --add-port=3105/tcp + $ firewall-cmd --zone=public --permanent --add-port=3108/tcp + $ firewall-cmd --zone=public --permanent --add-port=5000-5003/tcp + $ firewall-cmd --zone=public --permanent --add-port=5100-5103/tcp + $ firewall-cmd --permanent --list-all + +3. Reload the firewall: + + .. code-block:: console + + $ firewall-cmd --reload + +4. Enable FirewallID on boot: + + .. code-block:: console + + $ systemctl enable firewalld + + If you do not need the firewall, you can disable it: + + .. code-block:: console + + $ sudo systemctl disable firewalld + +Disabling selinux +-------------------------------- +**To disable selinux:** + +1. Show the status of **selinux**: + + .. code-block:: console + + $ sudo sestatus + +2. If the output is not **disabled**, edit the **/etc/selinux/config** file: + + .. code-block:: console + + $ sudo vim /etc/selinux/config + +3. Change **SELINUX=enforcing** to **SELINUX=disabled**. + + The above changes will only take effect after rebooting the server. + + You can disable selinux immediately after rebooting the server by running the following command: + + .. code-block:: console + + $ sudo setenforce 0 + +Configuring the /etc/hosts File +-------------------------------- +**To configure the /etc/hosts file:** + +1. Edit the **/etc/hosts** file: + + .. code-block:: console + + $ sudo vim /etc/hosts + +2. Call your local host: + + .. code-block:: console + + $ 127.0.0.1 localhost + $ + $ + +Configuring the DNS +-------------------------------- +**To configure the DNS:** + +1. Run the **ifconfig** commasnd to check your NIC name. In the following example, **eth0** is the NIC name: + + .. code-block:: console + + $ sudo vim /etc/sysconfig/network-scripts/ifcfg-eth0 + +2. Replace the DNS lines from the example above with your own DNS addresses : + + .. code-block:: console + + $ DNS1="4.4.4.4" + $ DNS2="8.8.8.8" + +Installing the Nvidia CUDA Driver +=================================================== +After configuring your operating system, you must install the Nvidia CUDA driver. + + .. warning:: If your UI runs on the server, the server must be stopped before installing the CUDA drivers. + +CUDA Driver Prerequisites +-------------------------------- +1. Verify that the NVIDIA card has been installed and is detected by the system: + + .. code-block:: console + + $ lspci | grep -i nvidia + +2. Check which version of gcc has been installed: + + .. code-block:: console + + $ gcc --version + +3. If gcc has not been installed, install it for one of the following operating systems: + + * On RHEL/CentOS: + + .. code-block:: console + + $ sudo yum install -y gcc + + * On Ubuntu: + + .. code-block:: console + + $ sudo apt-get install gcc + +Updating the Kernel Headers +-------------------------------- +**To update the kernel headers:** + +1. Update the kernel headers on one of the following operating systems: + + * On RHEL/CentOS: + + .. code-block:: console + + $ sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) + + * On Ubuntu: + + .. code-block:: console + + $ sudo apt-get install linux-headers-$(uname -r) + +2. Install **wget** one of the following operating systems: + + * On RHEL/CentOS: + + .. code-block:: console + + $ sudo yum install wget + + * On Ubuntu: + + .. code-block:: console + + $ sudo apt-get install wget + +Disabling Nouveau +-------------------------------- +You can disable Nouveau, which is the default driver. + +**To disable Nouveau:** + +1. Check if the Nouveau driver has been loaded: + + .. code-block:: console + + $ lsmod | grep nouveau + + If the Nouveau driver has been loaded, the command above generates output. + +2. Blacklist the Nouveau drivers to disable them: + + .. code-block:: console + + $ cat <`__ for the additional set-up requirements. + +To Tune Up NVIDIA Performance when Driver Installed from the Runfile +~~~~~~~~~~~~~~~~~~~~ +**To tune up NVIDIA performance when the driver was installed from the runfile:** + +1. Change the permissions on the **rc.local** file to **executable**: + + .. code-block:: console + + $ sudo chmod +x /etc/rc.local + +2. Edit the **/etc/yum.repos.d/cuda-10-1-local.repo** file: + + .. code-block:: console + + $ sudo vim /etc/rc.local + +3. Add the following lines: + + * **For V100/A100**: + + .. code-block:: console + + $ nvidia-persistenced + + * **For IBM (mandatory)**: + + .. code-block:: console + + $ sudo systemctl start nvidia-persistenced + $ sudo systemctl enable nvidia-persistenced + + * **For K80**: + + .. code-block:: console + + $ nvidia-persistenced + $ nvidia-smi -pm 1 + $ nvidia-smi -acp 0 + $ nvidia-smi --auto-boost-permission=0 + $ nvidia-smi --auto-boost-default=0 + +4. Reboot the server and run the **NVIDIA System Management Interface (NVIDIA SMI)**: + + .. code-block:: console + + $ nvidia-smi + +.. note:: Setting up the NVIDIA POWER9 CUDA driver includes additional set-up requirements. The NVIDIA POWER9 CUDA driver will not function properly if the additional set-up requirements are not followed. See `POWER9 Setup `__ for the additional set-up requirements. + +Disabling Automatic Bug Reporting Tools +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +**To disable automatic bug reporting tools:** + +1. Run the following **abort** commands: + + .. code-block:: console + + $ for i in abrt-ccpp.service abrtd.service abrt-oops.service abrt-pstoreoops.service abrt-vmcore.service abrt-xorg.service ; do sudo systemctl disable $i; sudo systemctl stop $i; done + +The server is ready for the SQream software installation. + +2. Run the following checks: + + a. Check the OS release: + + .. code-block:: console + + $ cat /etc/os-release + + b. Verify that a SQream user exists and has the same ID on all cluster member services: + + .. code-block:: console + + $ id sqream + + c. Verify that the storage is mounted: + + .. code-block:: console + + $ mount + + d. Verify that the driver has been installed correctly: + + .. code-block:: console + + $ nvidia-smi + + e. Check the maximum value of the **fs.file**: + + .. code-block:: console + + $ sysctl -n fs.file-max + + f. Run the following command as a SQream user: + + .. code-block:: console + + $ ulimit -c -u -n + + The following shows the desired output: + + .. code-block:: console + + $ core file size (blocks, -c) unlimited + $ max user processes (-u) 1000000 + $ open files (-n) 1000000 + +Enabling Core Dumps +=================================================== +After installing the Nvidia CUDA driver, you can enable your core dumps. While SQream recommends enabling your core dumps, it is optional. + +The **Enabling Core Dumps** section describes the following: + +.. contents:: + :local: + :depth: 1 + +Checking the abrtd Status +--------------------------------------------------- +**To check the abrtd status:** + +1. Check if **abrtd** is running: + + .. code-block:: console + + $ sudo ps -ef |grep abrt + +2. If **abrtd** is running, stop it: + + .. code-block:: console + + $ sudo service abrtd stop + $ sudo chkconfig abrt-ccpp off + $ sudo chkconfig abrt-oops off + $ sudo chkconfig abrt-vmcore off + $ sudo chkconfig abrt-xorg off + $ sudo chkconfig abrtd off + +Setting the Limits +--------------------------------------------------- +**To set the limits:** + +1. Set the limits: + + .. code-block:: console + + $ ulimit -c + +2. If the output is **0**, add the following lines to the **limits.conf** file (/etc/security): + + .. code-block:: console + + $ * soft core unlimited + $ * hard core unlimited + +3. Log out and log in to apply the limit changes. + +Creating the Core Dumps Directory +--------------------------------------------------- +**To set the core dumps directory:** + +1. Make the **/tmp/core_dumps** directory: + + .. code-block:: console + + $ mkdir /tmp/core_dumps + +2. Set the ownership of the **/tmp/core_dumps** directory: + + .. code-block:: console + + $ sudo chown sqream.sqream /tmp/core_dumps + +3. Grant read, write, and execute permissions to all users: + + .. code-block:: console + + $ sudo chmod -R 777 /tmp/core_dumps + +.. warning:: Because the core dump file may be the size of total RAM on the server, verify that you have sufficient disk space. In the example above, the core dump is configured to the */tmp/core_dumps* directory. You must replace path according to your own environment and disk space. + +Setting the Output Directory of the /etc/sysctl.conf File +----------------------------------------------------------------- +**To set the output directory of the /etc/sysctl.conf file:** + +1. Edit the **/etc/sysctl.conf** file: + + .. code-block:: console + + $ sudo vim /etc/sysctl.conf + +2. Add the following to the bottom of the file: + + .. code-block:: console + + $ kernel.core_uses_pid = 1 + $ kernel.core_pattern = //core-%e-%s-%u-%g-%p-%t + $ fs.suid_dumpable = 2 + +3. To apply the changes without rebooting the server, run the following: + + .. code-block:: console + + $ sudo sysctl -p + +4. Check that the core output directory points to the following: + + .. code-block:: console + + $ sudo cat /proc/sys/kernel/core_pattern + + The following shows the correct generated output: + + .. code-block:: console + + $ /tmp/core_dumps/core-%e-%s-%u-%g-%p-%t + +5. Verify that the core dumping works: + + .. code-block:: console + + $ select abort_server(); + +Verifying that the Core Dumps Work +--------------------------------------------------- +You can verify that the core dumps work only after installing and running SQream. This causes the server to crash and a new core.xxx file to be included in the folder that is written in **/etc/sysctl.conf** + +**To verify that the core dumps work:** + +1. Stop and restart all SQream services. + + :: + +2. Connect to SQream with ClientCmd and run the following command: + + .. code-block:: console + + $ select abort_server(); + +Troubleshooting Core Dumping +--------------------------------------------------- +This section describes the troubleshooting procedure to be followed if all parameters have been configured correctly, but the cores have not been created. + +**To troubleshoot core dumping:** + +1. Reboot the server. + + :: + +2. Verify that you have folder permissions: + + .. code-block:: console + + $ sudo chmod -R 777 /tmp/core_dumps + +3. Verify that the limits have been set correctly: + + .. code-block:: console + + $ ulimit -c + + If all parameters have been configured correctly, the correct output is: + + .. code-block:: console + + $ core file size (blocks, -c) unlimited + $ open files (-n) 1000000 + +4. If all parameters have been configured correctly, but running **ulimit -c** outputs **0**, run the following: + + .. code-block:: console + + $ sudo vim /etc/profile + +5. Search for line and tag it with the **hash** symbol: + + .. code-block:: console + + $ ulimit -S -c 0 > /dev/null 2>&1 + +6. Log out and log in. + + :: + +7. Run the ulimit -c command: + + .. code-block:: console + + $ ulimit -c command + +8. If the line is not found in **/etc/profile** directory, do the following: + + a. Run the following command: + + .. code-block:: console + + $ sudo vim /etc/init.d/functions + + b. Search for the following: + + .. code-block:: console + + $ ulimit -S -c ${DAEMON_COREFILE_LIMIT:-0} >/dev/null 2>&1 + + c. If the line is found, tag it with the **hash** symbol and reboot the server. diff --git a/installation_guides/running_sqream_in_a_docker_container.rst b/installation_guides/running_sqream_in_a_docker_container.rst new file mode 100644 index 000000000..e85eac925 --- /dev/null +++ b/installation_guides/running_sqream_in_a_docker_container.rst @@ -0,0 +1,1470 @@ +.. _running_sqream_in_a_docker_container: + +*********************** +Installing and Running SQream in a Docker Container +*********************** +The **Installing and Running SQream in a Docker Container** page describes how to prepare your machine's environment for installing and running SQream in a Docker container. + +This page describes the following: + +.. contents:: + :local: + :depth: 1 + +Setting Up a Host +==================================== + +Operating System Requirements +------------------------------------ +SQream was tested and verified on the following versions of Linux: + + * x86 CentOS/RHEL 7.6 - 7.9 + * IBM RHEL 7.6 + +SQream recommends installing a clean OS on the host to avoid any installation issues. + +.. warning:: Docker-based installation supports only single host deployment and cannot be used on a multi-node cluster. Installing Docker on a single host you will not be able to scale it to a multi-node cluster. + + +Creating a Local User +---------------- +To run SQream in a Docker container you must create a local user. + +**To create a local user:** + +1. Add a local user: + + .. code-block:: + + $ useradd -m -U + +2. Set the local user's password: + + .. code-block:: + + $ passwd + +3. Add the local user to the ``wheel`` group: + + .. code-block:: + + $ usermod -aG wheel + + You can remove the local user from the ``wheel`` group when you have completed the installation. + +4. Log out and log back in as the local user. + +Setting a Local Language +---------------- +After creating a local user you must set a local language. + +**To set a local language:** + +1. Set the local language: + + .. code-block:: + + $ sudo localectl set-locale LANG=en_US.UTF-8 + +2. Set the time stamp (time and date) of the locale: + + .. code-block:: + + $ sudo timedatectl set-timezone Asia/Jerusalem + +You can run the ``timedatectl list-timezones`` command to see your timezone. + +Adding the EPEL Repository +---------------- +After setting a local language you must add the EPEL repository. + +**To add the EPEL repository:** + +1. As a root user, upgrade the **epel-release-latest-7.noarch.rpm** repository: + + 1. RedHat (RHEL 7): + + .. code-block:: + + $ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + + 2. CentOS 7 + + .. code-block:: + + $ sudo yum install epel-release + +Installing the Required NTP Packages +---------------- +After adding the EPEL repository, you must install the required NTP packages. + +You can install the required NTP packages by running the following command: + +.. code-block:: + + $ sudo yum install ntp pciutils python36 kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc + +Installing the Recommended Tools +---------------- +After installin gthe required NTP packages you must install the recommended tools. + +SQream recommends installing the following recommended tools: + +.. code-block:: + + $ sudo yum install bash-completion.noarch vim-enhanced.x86_64 vim-common.x86_64 net-tools iotop htop psmisc screen xfsprogs wget yum-utils deltarpm dos2unix + +Updating to the Current Version of the Operating System +---------------- +After installing the recommended tools you must update to the current version of the operating system. + +SQream recommends updating to the current version of the operating system. This is not recommended if the nvidia driver has **not been installed.** + + + +Configuring the NTP Package +---------------- +After updating to the current version of the operating system you must configure the NTP package. + +**To configure the NTP package:** + +1. Add your local servers to the NTP configuration. + + :: + +2. Configure the **ntpd** service to begin running when your machine is started: + + .. code-block:: + + $ sudo systemctl enable ntpd + $ sudo systemctl start ntpd + $ sudo ntpq -p + +Configuring the Performance Profile +---------------- +After configuring the NTP package you must configure the performance profile. + +**To configure the performance profile:** + +1. *Optional* - Switch the active profile: + + .. code-block:: + + $ sudo tuned-adm profile throughput-performance + +2. Change the multi-user's default run level: + + .. code-block:: + + $ sudo systemctl set-default multi-user.target + +Configuring Your Security Limits +---------------- +After configuring the performance profile you must configure your security limits. Configuring your security limits refers to configuring the number of open files, processes, etc. + +**To configure your security limits:** + +1. Run the **bash** shell as a super-user: + + .. code-block:: + + $ sudo bash + +2. Run the following command: + + .. code-block:: + + $ echo -e "sqream soft nproc 500000\nsqream hard nproc 500000\nsqream soft nofile 500000\nsqream hard nofile 500000\nsqream soft core unlimited\nsqream hard core unlimited" >> /etc/security/limits.conf + +3. Run the following command: + + .. code-block:: + + $ echo -e "vm.dirty_background_ratio = 5 \n vm.dirty_ratio = 10 \n vm.swappiness = 10 \n vm.zone_reclaim_mode = 0 \n vm.vfs_cache_pressure = 200 \n" >> /etc/sysctl.conf + +Disabling Automatic Bug-Reporting Tools +---------------- +After configuring your security limits you must disable the following automatic bug-reporting tools: + +* ccpp.service +* oops.service +* pstoreoops.service +* vmcore.service +* xorg.service + +You can abort the above but-reporting tools by running the following command: + +.. code-block:: + + $ for i in abrt-ccpp.service abrtd.service abrt-oops.service abrt-pstoreoops.service abrt-vmcore.service abrt-xorg.service ; do sudo systemctl disable $i; sudo systemctl stop $i; done + +Installing the Nvidia CUDA Driver +------------------------------------- + +1. Verify that the Tesla NVIDIA card has been installed and is detected by the system: + + .. code-block:: + + $ lspci | grep -i nvidia + + The correct output is a list of Nvidia graphic cards. If you do not receive this output, verify that an NVIDIA GPU card has been installed. + +#. Verify that the open-source upstream Nvidia driver is running: + + .. code-block:: + + $ lsmod | grep nouveau + + No output should be generated. + +#. If you receive any output, do the following: + + 1. Disable the open-source upstream Nvidia driver: + + .. code-block:: + + $ sudo bash + $ echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nouveau.conf + $ echo "options nouveau modeset=0" >> /etc/modprobe.d/blacklist-nouveau.conf + $ dracut --force + $ modprobe --showconfig | grep nouveau + + 2. Reboot the server and verify that the Nouveau model has not been loaded: + + .. code-block:: + + $ lsmod | grep nouveau + +#. Check if the Nvidia CUDA driver has already been installed: + + .. code-block:: + + $ nvidia-smi + + The following is an example of the correct output: + + .. code-block:: + + nvidia-smi + Wed Oct 30 14:05:42 2019 + +-----------------------------------------------------------------------------+ + | NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 | + |-------------------------------+----------------------+----------------------+ + | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | + | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | + |===============================+======================+======================| + | 0 Tesla V100-SXM2... On | 00000004:04:00.0 Off | 0 | + | N/A 32C P0 37W / 300W | 0MiB / 16130MiB | 0% Default | + +-------------------------------+----------------------+----------------------+ + | 1 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 | + | N/A 33C P0 37W / 300W | 0MiB / 16130MiB | 0% Default | + +-------------------------------+----------------------+----------------------+ + + +-----------------------------------------------------------------------------+ + | Processes: GPU Memory | + | GPU PID Type Process name Usage | + |=============================================================================| + | No running processes found | + +-----------------------------------------------------------------------------+ + +#. Verify that the installed CUDA version shown in the output above is ``10.1``. + + :: + + +#. Do one of the following: + + :: + + 1. If CUDA version 10.1 has already been installed, skip to Docktime Runtime (Community Edition). + :: + + 2. If CUDA version 10.1 has not been installed yet, continue with Step 7 below. + +#. Do one of the following: + + * Install :ref:`CUDA Driver version 10.1 for x86_64 `. + + :: + + * Install :ref:`CUDA driver version 10.1 for IBM Power9 `. + +.. _CUDA_10.1_x8664: + +Installing the CUDA Driver Version 10.1 for x86_64 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**To install the CUDA driver version 10.1 for x86_64:** + +1. Make the following target platform selections: + + :: + + * **Operating system**: Linux + * **Architecture**: x86_64 + * **Distribution**: CentOS + * **Version**: 7 + * **Installer type**: the relevant installer type + +For installer type, SQream recommends selecting **runfile (local)**. The available selections shows only the supported platforms. + +2. Download the base installer for Linux CentOS 7 x86_64: + + .. code-block:: + + wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.243-418.87.00-1.0-1.x86_64.rpm + + +3. Install the base installer for Linux CentOS 7 x86_64 by running the following commands: + + .. code-block:: + + $ sudo yum localinstall cuda-repo-rhel7-10-1-local-10.1.243-418.87.00-1.0-1.x86_64.rpm + $ sudo yum clean all + $ sudo yum install nvidia-driver-latest-dkms + +.. warning:: Verify that the output indicates that driver **418.87** will be installed. + +4. Follow the command line prompts. + + + :: + + +5. Enable the Nvidia service to start at boot and start it: + + .. code-block:: + + $ sudo systemctl enable nvidia-persistenced.service && sudo systemctl start nvidia-persistenced.service + +7. Reboot the server. + + :: +8. Verify that the Nvidia driver has been installed and shows all available GPU's: + + .. code-block:: + + $ nvidia-smi + + The following is the correct output: + + .. code-block:: + + nvidia-smi + Wed Oct 30 14:05:42 2019 + +-----------------------------------------------------------------------------+ + | NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 | + |-------------------------------+----------------------+----------------------+ + | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | + | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | + |===============================+======================+======================| + | 0 Tesla V100-SXM2... On | 00000004:04:00.0 Off | 0 | + | N/A 32C P0 37W / 300W | 0MiB / 16130MiB | 0% Default | + +-------------------------------+----------------------+----------------------+ + | 1 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 | + | N/A 33C P0 37W / 300W | 0MiB / 16130MiB | 0% Default | + +-------------------------------+----------------------+----------------------+ + + +-----------------------------------------------------------------------------+ + | Processes: GPU Memory | + | GPU PID Type Process name Usage | + |=============================================================================| + | No running processes found | + +-----------------------------------------------------------------------------+ + +.. _CUDA_10.1_IBMPower9: + +Installing the CUDA Driver Version 10.1 for IBM Power9 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**To install the CUDA driver version 10.1 for IBM Power9:** + +1. Download the base installer for Linux CentOS 7 PPC64le: + + .. code-block:: + + wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.243-418.87.00-1.0-1.ppc64le.rpm + + +#. Install the base installer for Linux CentOS 7 x86_64 by running the following commands: + + .. code-block:: + + $ sudo rpm -i cuda-repo-rhel7-10-1-local-10.1.243-418.87.00-1.0-1.ppc64le.rpm + $ sudo yum clean all + $ sudo yum install nvidia-driver-latest-dkms + +.. warning:: Verify that the output indicates that driver **418.87** will be installed. + + + +3. Copy the file to the **/etc/udev/rules.d** directory. + + :: + +4. If you are using RHEL 7 version (7.6 or later), comment out, remove, or change the hot-pluggable memory rule located in file copied to the **/etc/udev/rules.d** directory by running the following command: + + .. code-block:: + + $ sudo cp /lib/udev/rules.d/40-redhat.rules /etc/udev/rules.d + $ sudo sed -i 's/SUBSYSTEM!="memory",.*GOTO="memory_hotplug_end"/SUBSYSTEM=="*", GOTO="memory_hotplug_end"/' /etc/udev/rules.d/40-redhat.rules + +#. Enable the **nvidia-persisted.service** file: + + .. code-block:: + + $ sudo systemctl enable nvidia-persistenced.service + +#. Reboot your system to initialize the above modifications. + + :: + +#. Verify that the Nvidia driver and the **nvidia-persistenced.service** files are running: + + .. code-block:: + + $ nvidia smi + + The following is the correct output: + + .. code-block:: + + nvidia-smi + Wed Oct 30 14:05:42 2019 + +-----------------------------------------------------------------------------+ + | NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 | + |-------------------------------+----------------------+----------------------+ + | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | + | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | + |===============================+======================+======================| + | 0 Tesla V100-SXM2... On | 00000004:04:00.0 Off | 0 | + | N/A 32C P0 37W / 300W | 0MiB / 16130MiB | 0% Default | + +-------------------------------+----------------------+----------------------+ + | 1 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 | + | N/A 33C P0 37W / 300W | 0MiB / 16130MiB | 0% Default | + +-------------------------------+----------------------+----------------------+ + + +-----------------------------------------------------------------------------+ + | Processes: GPU Memory | + | GPU PID Type Process name Usage | + |=============================================================================| + | No running processes found | + +-----------------------------------------------------------------------------+ + +#. Verify that the **nvidia-persistenced** service is running: + + .. code-block:: + + $ systemctl status nvidia-persistenced + + The following is the correct output: + + .. code-block:: + + root@gpudb ~]systemctl status nvidia-persistenced + nvidia-persistenced.service - NVIDIA Persistence Daemon + Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: disabled) + Active: active (running) since Tue 2019-10-15 21:43:19 KST; 11min ago + Process: 8257 ExecStart=/usr/bin/nvidia-persistenced --verbose (code=exited, status=0/SUCCESS) + Main PID: 8265 (nvidia-persiste) + Tasks: 1 + Memory: 21.0M + CGroup: /system.slice/nvidia-persistenced.service + └─8265 /usr/bin/nvidia-persistenced --verbose + +Installing the Docker Engine (Community Edition) +======================= +After installing the Nvidia CUDA driver you must install the Docker engine. + +This section describes how to install the Docker engine using the following processors: + +* :ref:`Using x86_64 processor on CentOS ` +* :ref:`Using x86_64 processor on Ubuntu ` +* :ref:`Using IBM Power9 (PPC64le) processor ` + + +.. _dockerx8664centos: + +Installing the Docker Engine Using an x86_64 Processor on CentOS +--------------------------------- +The x86_64 processor supports installing the **Docker Community Edition (CE)** versions 18.03 and higher. + +For more information on installing the Docker Engine CE on an x86_64 processor, see `Install Docker Engine on CentOS `_ + + + +.. _dockerx8664ubuntu: + +Installing the Docker Engine Using an x86_64 Processor on Ubuntu +----------------------------------------------------- + + +The x86_64 processor supports installing the **Docker Community Edition (CE)** versions 18.03 and higher. + +For more information on installing the Docker Engine CE on an x86_64 processor, see `Install Docker Engine on Ubuntu `_ + +.. _docker_ibmpower9: + +Installing the Docker Engine on an IBM Power9 Processor +---------------------------------------- +The x86_64 processor only supports installing the **Docker Community Edition (CE)** version 18.03. + +**To install the Docker Engine on an IBM Power9 processor:** + +You can install the Docker Engine on an IBM Power9 processor by running the following command: + +.. code-block:: + + wget http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/container-selinux-2.9-4.el7.noarch.rpm + wget http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/docker-ce-18.03.1.ce-1.el7.centos.ppc64le.rpm + yum install -y container-selinux-2.9-4.el7.noarch.rpm docker-ce-18.03.1.ce-1.el7.centos.ppc64le.rpm + +For more information on installing the Docker Engine CE on an IBM Power9 processor, see `Install Docker Engine on Ubuntu `_. + +Docker Post-Installation +================================= +After installing the Docker engine you must configure Docker on your local machine. + +**To configure Docker on your local machine:** + +1. Enable Docker to start on boot: + + .. code-block:: + + $ sudo systemctl enable docker && sudo systemctl start docker + +2. Enable managing Docker as a non-root user: + + .. code-block:: + + $ sudo usermod -aG docker $USER + +3. Log out and log back in via SSH. This causes Docker to re-evaluate your group membership. + + :: + +4. Verify that you can run the following Docker command as a non-root user (without ``sudo``): + + .. code-block:: + + $ docker run hello-world + +If you can run the above Docker command as a non-root user, the following occur: + +* Docker downloads a test image and runs it in a container. +* When the container runs, it prints an informational message and exits. + +For more information on installing the Docker Post-Installation, see `Docker Post-Installation `_. + +Installing the Nvidia Docker2 ToolKit +========================================== +After configuring Docker on your local machine you must install the Nvidia Docker2 ToolKit. The NVIDIA Docker2 Toolkit lets you build and run GPU-accelerated Docker containers. The Toolkit includes a container runtime library and related utilities for automatically configuring containers to leverage NVIDIA GPU's. + +This section describes the following: + +* :ref:`Installing the NVIDIA Docker2 Toolkit on an x86_64 processor ` +* :ref:`Installing the NVIDIA Docker2 Toolkit on a PPC64le processor ` + +.. _install_nvidia_docker2_toolkit_x8664_processor: + +Installing the NVIDIA Docker2 Toolkit on an x86_64 Processor +---------------------------------------- + +This section describes the following: + +* :ref:`Installing the NVIDIA Docker2 Toolkit on a CentOS operating system ` + +* :ref:`Installing the NVIDIA Docker2 Toolkit on an Ubuntu operating system ` + +.. _install_nvidia_docker2_toolkit_centos: + +Installing the NVIDIA Docker2 Toolkit on a CentOS Operating System +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**To install the NVIDIA Docker2 Toolkit on a CentOS operating system:** + +1. Install the repository for your distribution: + + .. code-block:: + + distribution=$(. /etc/os-release;echo $ID$VERSION_ID) + curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \ + sudo tee /etc/yum.repos.d/nvidia-docker.repo + +2. Install the ``nvidia-docker2`` package and reload the Docker daemon configuration: + + .. code-block:: + + $ sudo yum install nvidia-docker2 + $ sudo pkill -SIGHUP dockerd + +3. Do one of the following: + + * If you received an error when installing the ``nvidia-docker2`` package, skip to :ref:`Step 4 `. + * If you successfully installed the ``nvidia-docker2`` package, skip to :ref:`Step 5 `. + +.. _step_4_centos: + +4. Do the following: + + 1. Run the ``sudo vi /etc/yum.repos.d/nvidia-docker.repo`` command if the following error is displayed when installing the ``nvidia-docker2`` package: + + + .. code-block:: + + https://nvidia.github.io/nvidia-docker/centos7/ppc64le/repodata/repomd.xml: + [Errno -1] repomd.xml signature could not be verified for nvidia-docker + + 2. Change ``repo_gpgcheck=1`` to ``repo_gpgcheck=0``. + +.. _step_5_centos: + +5. Verify that the NVIDIA-Docker run has been installed correctly: + + .. code-block:: + + $ docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi + +For more information on installing the NVIDIA Docker2 Toolkit on a CentOS operating system, see :ref:`Installing the NVIDIA Docker2 Toolkit on a CentOS operating system ` + + +.. _install_nvidia_docker2_toolkit_ubuntu: + +Installing the NVIDIA Docker2 Toolkit on an Ubuntu Operating System +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**To install the NVIDIA Docker2 Toolkit on an Ubuntu operating system:** + +1. Install the repository for your distribution: + + .. code-block:: + + curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - + distribution=$(. /etc/os-release;echo $ID$VERSION_ID) + curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list + sudo apt-get update + +2. Install the ``nvidia-docker2`` package and reload the Docker daemon configuration: + + .. code-block:: + + $ sudo apt-get install nvidia-docker2 + $ sudo pkill -SIGHUP dockerd + +3. Do one of the following: + + * If you received an error when installing the ``nvidia-docker2`` package, skip to :ref:`Step 4 `. + * If you successfully installed the ``nvidia-docker2`` package, skip to :ref:`Step 5 `. + + .. _step_4_ubuntu: + +4. Do the following: + + 1. Run the ``sudo vi /etc/yum.repos.d/nvidia-docker.repo`` command if the following error is displayed when installing the ``nvidia-docker2`` package: + + .. code-block:: + + https://nvidia.github.io/nvidia-docker/centos7/ppc64le/repodata/repomd.xml: + [Errno -1] repomd.xml signature could not be verified for nvidia-docker + + 2. Change ``repo_gpgcheck=1`` to ``repo_gpgcheck=0``. + +.. _step_5_ubuntu: + +5. Verify that the NVIDIA-Docker run has been installed correctly: + + .. code-block:: + + $ docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi + +For more information on installing the NVIDIA Docker2 Toolkit on a CentOS operating system, see :ref:`Installing the NVIDIA Docker2 Toolkit on an Ubuntu operating system ` + +.. _install_nvidia_docker2_toolkit_ppc64le_processor: + +Installing the NVIDIA Docker2 Toolkit on a PPC64le Processor +-------------------------------------- + +This section describes how to install the NVIDIA Docker2 Toolkit on an IBM RHEL operating system: + +**To install the NVIDIA Docker2 Toolkit on an IBM RHEL operating system:** + +1. Import the repository and install the ``libnvidia-container`` and the ``nvidia-container-runtime`` containers. + + .. code-block:: + + $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) + $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \ + sudo tee /etc/yum.repos.d/nvidia-docker.repo + $ sudo yum install -y libnvidia-container* + +2. Do one of the following: + + * If you received an error when installing the containers, skip to :ref:`Step 3 `. + * If you successfully installed the containers, skip to :ref:`Step 4 `. + +.. _step_3_installing_nvidia_docker2_toolkit_ppc64le_processor: + +3. Do the following: + + 1. Run the ``sudo vi /etc/yum.repos.d/nvidia-docker.repo`` command if the following error is displayed when installing the containers: + + .. code-block:: + + https://nvidia.github.io/nvidia-docker/centos7/ppc64le/repodata/repomd.xml: + [Errno -1] repomd.xml signature could not be verified for nvidia-docker + + 2. Change ``repo_gpgcheck=1`` to ``repo_gpgcheck=0``. + + :: + + 3. Install the ``libnvidia-container`` container. + + .. code-block:: + + $ sudo yum install -y libnvidia-container* + + .. _step_4_installing_nvidia_docker2_toolkit_ppc64le_processor: + +4. Install the ``nvidia-container-runtime`` container: + + .. code-block:: + + $ sudo yum install -y nvidia-container-runtime* + +5. Add ``nvidia runtime`` to the Docker daemon: + + .. code-block:: + + $ sudo mkdir -p /etc/systemd/system/docker.service.d/ + $ sudo vi /etc/systemd/system/docker.service.d/override.conf + + $ [Service] + $ ExecStart= + $ ExecStart=/usr/bin/dockerd + +6. Restart Docker: + + .. code-block:: + + $ sudo systemctl daemon-reload + $ sudo systemctl restart docker + +7. Verify that the NVIDIA-Docker run has been installed correctly: + + .. code-block:: + + $ docker run --runtime=nvidia --rm nvidia/cuda-ppc64le nvidia-smi + +.. _accessing_hadoop_kubernetes_configuration_files: + +Accessing the Hadoop and Kubernetes Configuration Files +-------------------------------------- +The information this section is optional and is only relevant for Hadoop users. If you require Hadoop and Kubernetes (Krb5) connectivity, contact your IT department for access to the following configuration files: + +* Hadoop configuration files: + + * core-site.xml + * hdfs-site.xml + + :: + +* Kubernetes files: + + * Configuration file - krb.conf + * Kubernetes Hadoop client certificate - hdfs.keytab + +Once you have the above files, you must copy them into the correct folders in your working directory. + +For more information about the correct directory to copy the above files into, see the :ref:`Installing the SQream Software ` section below. + +For related information, see the following sections: + +* :ref:`Configuring the Hadoop and Kubernetes Configuration Files ` +* :ref:`Setting the Hadoop and Kubernetes Configuration Parameters ` + +.. _installing_sqream_software: + +Installing the SQream Software +============================== + +Preparing Your Local Environment +------------------------- +After installing the Nvidia Docker2 toolKit you must prepare your local environment. + +.. note:: You must install the SQream software under a *sqream* and not a *root* user. + +The Linux user preparing the local environment must have **read/write** access to the following directories for the SQream software to correctly read and write the required resources: + +* **Log directory** - default: /var/log/sqream/ +* **Configuration directory** - default: /etc/sqream/ +* **Cluster directory** - the location where SQream writes its DB system, such as */mnt/sqreamdb* +* **Ingest directory** - the location where the required data is loaded, such as */mnt/data_source/* + +.. _download_sqream_software: + +Deploying the SQream Software +------------------------- +After preparing your local environment you must deploy the SQream software. Deploying the SQream software requires you to access and extract the required files and to place them in the correct directory. + +**To deploy the SQream software:** + +1. Contact the SQream Support team for access to the **sqream_installer-nnn-DBnnn-COnnn-EDnnn-.tar.gz** file. + +The **sqream_installer-nnn-DBnnn-COnnn-EDnnn-.tar.gz** file includes the following parameter values: + +* **sqream_installer-nnn** - sqream installer version +* **DBnnn** - SQreamDB version +* **COnnn** - SQream console version +* **EDnnn** - SQream editor version +* **arch** - server arch (applicable to X86.64 and ppc64le) + +2. Extract the tarball file: + + .. code-block:: + + $ tar -xvf sqream_installer-1.1.5-DB2019.2.1-CO1.5.4-ED3.0.0-x86_64.tar.gz + + When the tarball file has been extracted, a new folder will be created. The new folder is automatically given the name of the tarball file: + + .. code-block:: + + drwxrwxr-x 9 sqream sqream 4096 Aug 11 11:51 sqream_istaller-1.1.5-DB2019.2.1-CO1.5.4-ED3.0.0-x86_64/ + -rw-rw-r-- 1 sqream sqream 3130398797 Aug 11 11:20 sqream_installer-1.1.5-DB2019.2.1-CO1.5.4-ED3.0.0-x86_64.tar.gz + +3. Change the directory to the new folder that you created in the previous step. + +:: + +4. Verify that the folder you just created contains all of the required files. + + .. code-block:: + + $ ls -la + + The following is an example of the files included in the new folder: + + .. code-block:: + + drwxrwxr-x. 10 sqream sqream 198 Jun 3 17:57 . + drwx------. 25 sqream sqream 4096 Jun 7 18:11 .. + drwxrwxr-x. 2 sqream sqream 226 Jun 7 18:09 .docker + drwxrwxr-x. 2 sqream sqream 64 Jun 3 12:55 .hadoop + drwxrwxr-x. 2 sqream sqream 4096 May 31 14:18 .install + drwxrwxr-x. 2 sqream sqream 39 Jun 3 12:53 .krb5 + drwxrwxr-x. 2 sqream sqream 22 May 31 14:18 license + drwxrwxr-x. 2 sqream sqream 82 May 31 14:18 .sqream + -rwxrwxr-x. 1 sqream sqream 1712 May 31 14:18 sqream-console + -rwxrwxr-x. 1 sqream sqream 4608 May 31 14:18 sqream-install + +For information relevant to Hadoop users, see the following sections: + +* :ref:`Accessing the Hadoop and Kubernetes Configuration Files `. +* :ref:`Configuring the Hadoop and Kubernetes Configuration Files `. +* :ref:`Setting the Hadoop and Kubernetes Configuration Parameters `. + +.. _configure_hadoop_kubernetes_configuration_files: + +Configuring the Hadoop and Kubernetes Configuration Files +----------------------------- +The information in this section is optional and is only relevant for Hadoop users. If you require Hadoop and Kubernetes (Krb5) connectivity, you must copy the Hadoop and Kubernetes files into the correct folders in your working directory as shown below: + +* .hadoop/core-site.xml +* .hadoop/hdfs-site.xml +* .krb5/krb5.conf +* .krb5/hdfs.keytab + +For related information, see the following sections: + +* :ref:`Accessing the Hadoop and Kubernetes Configuration Files `. +* :ref:`Setting the Hadoop and Kubernetes Configuration Parameters `. + +Configuring the SQream Software +------------------------------- +After deploying the SQream software, and optionally configuring the Hadoop and Kubernetes configuration files, you must configure the SQream software. + +Configuring the SQream software requires you to do the following: + +* Configure your local environment +* Understand the ``sqream-install`` flags +* Install your SQream license +* Validate your SQream icense +* Change your data ingest folder + +Configuring Your Local Environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Once you've downloaded the SQream software, you can begin configuring your local environment. The following commands must be run (as **sudo**) from the same directory that you located your packages. + +For example, you may have saved your packages in **/home/sqream/sqream-console-package/**. + +The following table shows the flags that you can use to configure your local directory: + +.. list-table:: + :widths: 10 50 40 + :header-rows: 1 + + * - Flag + - Function + - Note + * - **-i** + - Loads all software from the hidden folder **.docker**. + - Mandatory + * - **-k** + - Loads all license packages from the **/license** directory. + - Mandatory + * - **-f** + - Overwrites existing folders. **Note** Using ``-f`` overwrites **all files** located in mounted directories. + - Mandatory + * - **-c** + - Defines the origin path for writing/reading SQream configuration files. The default location is ``/etc/sqream/``. + - If you are installing the Docker version on a server that already works with SQream, do not use the default path. + * - **-v** + - The SQream cluster location. If a cluster does not exist yet, ``-v`` creates one. If a cluster already exists, ``-v`` mounts it. + - Mandatory + * - **-l** + - SQream system startup logs location, including startup logs and docker logs. The default location is ``/var/log/sqream/``. + - + * - **-d** + - The directory containing customer data to be imported and/or copied to SQream. + - + * - **-s** + - Shows system settings. + - + * - **-r** + - Resets the system configuration. This value is run without any other variables. + - Mandatory + * - **-h** + - Help. Shows the available flags. + - Mandatory + * - **-K** + - Runs license validation + - + * - **-e** + - Used for inserting your RKrb5 server DNS name. For more information on setting your Kerberos configuration parameters, see :ref:`Setting the Hadoop and Kubernetes Configuration Parameters `. + - + * - **-p** + - Used for inserting your Kerberos user name. For more information on setting your Kerberos configuration parameters, see :ref:`Setting the Hadoop and Kubernetes Configuration Parameters `. + - + + +Installing Your License +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Once you've configured your local environment, you must install your license by copying it into the SQream installation package folder located in the **./license** folder: + +.. code-block:: + + $ sudo ./sqream-install -k + +You do not need to extract this folder after uploading into the **./license**. + + +Validating Your License +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +You can copy your license package into the SQream console folder located in the **/license** folder by running the following command: + +.. code-block:: + + $ sudo ./sqream-install -K + +The following mandatory flags must be used in the first run: + +.. code-block:: + + $ sudo ./sqream-install -i -k -v + +The following is an example of the correct command syntax: + +.. code-block:: + + $ sudo ./sqream-install -i -k -c /etc/sqream -v /home/sqream/sqreamdb -l /var/log/sqream -d /home/sqream/data_ingest + +.. _setting_hadoop_kubernetes_connectivity_parameters: + +Setting the Hadoop and Kubernetes Connectivity Parameters +------------------------------- +The information in this section is optional, and is only relevant for Hadoop users. If you require Hadoop and Kubernetes (Krb5) connectivity, you must set their connectivity parameters. + +The following is the correct syntax when setting the Hadoop and Kubernetes connectivity parameters: + +.. code-block:: + + $ sudo ./sqream-install -p -e : + +The following is an example of setting the Hadoop and Kubernetes connectivity parameters: + +.. code-block:: + + $ sudo ./sqream-install -p -e kdc.sq.com:<192.168.1.111> + +For related information, see the following sections: + +* :ref:`Accessing the Hadoop and Kubernetes Configuration Files `. +* :ref:`Configuring the Hadoop and Kubernetes Configuration Files `. + +Modifying Your Data Ingest Folder +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Once you've validated your license, you can modify your data ingest folder after the first run by running the following command: + +.. code-block:: + + $ sudo ./sqream-install -d /home/sqream/data_in + +Configuring Your Network for Docker +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Once you've modified your data ingest folder (if needed), you must validate that the server network and Docker network that you are setting up do not overlap. + +**To configure your network for Docker:** + +1. To verify that your server network and Docker network do not overlap, run the following command: + +.. code-block:: + + $ ifconfig | grep 172. + +2. Do one of the following: + + * If running the above command output no results, continue the installation process. + * If running the above command output results, run the following command: + + .. code-block:: + + $ ifconfig | grep 192.168. + + +Checking and Verifying Your System Settings +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Once you've configured your network for Docker, you can check and verify your system settings. + +Running the following command shows you all the variables used by your SQream system: + +.. code-block:: + + $ ./sqream-install -s + +The following is an example of the correct output: + +.. code-block:: + + SQREAM_CONSOLE_TAG=1.5.4 + SQREAM_TAG=2019.2.1 + SQREAM_EDITOR_TAG=3.0.0 + license_worker_0=f0:cc: + license_worker_1=26:91: + license_worker_2=20:26: + license_worker_3=00:36: + SQREAM_VOLUME=/media/sqreamdb + SQREAM_DATA_INGEST=/media/sqreamdb/data_in + SQREAM_CONFIG_DIR=/etc/sqream/ + LICENSE_VALID=true + SQREAM_LOG_DIR=/var/log/sqream/ + SQREAM_USER=sqream + SQREAM_HOME=/home/sqream + SQREAM_ENV_PATH=/home/sqream/.sqream/env_file + PROCESSOR=x86_64 + METADATA_PORT=3105 + PICKER_PORT=3108 + NUM_OF_GPUS=2 + CUDA_VERSION=10.1 + NVIDIA_SMI_PATH=/usr/bin/nvidia-smi + DOCKER_PATH=/usr/bin/docker + NVIDIA_DRIVER=418 + SQREAM_MODE=single_host + +Using the SQream Console +========================= +After configuring the SQream software and veriying your system settings you can begin using the SQream console. + +SQream Console - Basic Commands +--------------------------------- +The SQream console offers the following basic commands: + +* :ref:`Starting your SQream console ` +* :ref:`Starting Metadata and Picker ` +* :ref:`Starting the running services ` +* :ref:`Listing the running services ` +* :ref:`Stopping the running services ` +* :ref:`Using the SQream editor ` +* :ref:`Using the SQream Client ` + +.. _starting_sqream_console: + +Starting Your SQream Console +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can start your SQream console by running the following command: + +.. code-block:: + + $ ./sqream-console + +.. _starting_metadata_and_picker: + +Starting the SQream Master +~~~~~~~~~~~~~~~~~ + +**To listen to metadata and picker:** + +1. Start the metadata server (default port 3105) and picker (default port 3108) by running the following command: + + .. code-block:: + + $ sqream master --start + + The following is the correct output: + + .. code-block:: + + sqream-console> sqream master --start + starting master server in single_host mode ... + sqream_single_host_master is up and listening on ports: 3105,3108 + + +2. *Optional* - Change the metadata and server picker ports by adding ``-p `` and ``-m ``: + + .. code-block:: + + $ sqream-console>sqream master --start -p 4105 -m 43108 + $ starting master server in single_host mode ... + $ sqream_single_host_master is up and listening on ports: 4105,4108 + + + +.. _starting_running_services: + +Starting SQream Workers +~~~~~~~~~~~~~~~~~ + + +When starting SQream workers, setting the ```` value sets how many workers to start. Leaving the ```` value unspecified runs all of the available resources. + + +.. code-block:: + + $ sqream worker --start + + The following is an example of expected output when setting the ```` value to ``2``: + + .. code-block:: + + sqream-console>sqream worker --start 2 + started sqream_single_host_worker_0 on port 5000, allocated gpu: 0 + started sqream_single_host_worker_1 on port 5001, allocated gpu: 1 + + +.. _listing_running_services: + +Listing the Running Services +~~~~~~~~~~~~~~~~~ + +You can list running SQream services to look for container names and ID's by running the following command: + +.. code-block:: + + $ sqream master --list + +The following is an example of the expected output: + +.. code-block:: + + sqream-console>sqream master --list + container name: sqream_single_host_worker_0, container id: c919e8fb78c8 + container name: sqream_single_host_master, container id: ea7eef80e038-- + + +.. _stopping_running_services: + +Stopping the Running Services +~~~~~~~~~~~~~~~~~ + +You can stop running services either for a single SQream worker, or all SQream services for both master and worker. + +The following is the command for stopping a running service for a single SQream worker: + +.. code-block:: + + $ sqream worker --stop + +The following is an example of expected output when stopping a running service for a single SQream worker: + +.. code-block:: + + sqream worker stop + stopped container sqream_single_host_worker_0, id: 892a8f1a58c5 + + +You can stop all running SQream services (both master and worker) by running the following command: + +.. code-block:: + + $ sqream-console>sqream master --stop --all + +The following is an example of expected output when stopping all running services: + +.. code-block:: + + sqream-console>sqream master --stop --all + stopped container sqream_single_host_worker_0, id: 892a8f1a58c5 + stopped container sqream_single_host_master, id: 55cb7e38eb22 + + +.. _using_sqream_editor: + +Using SQream Studio +~~~~~~~~~~~~~~~~~ +SQream Studio is an SQL statement editor. + +**To start SQream Studio:** + +1. Run the following command: + + .. code-block:: + + $ sqream studio --start + +The following is an example of the expected output: + + .. code-block:: + + SQream Acceleration Studio is available at http://192.168.1.62:8080 + +2. Click the ``http://192.168.1.62:8080`` link shown in the CLI. + + +**To stop SQream Studio:** + +You can stop your SQream Studio by running the following command: + +.. code-block:: + + $ sqream studio --stop + +The following is an example of the expected output: + +.. code-block:: + + sqream_admin stopped + + +.. _using_sqream_client: + +Using the SQream Client +~~~~~~~~~~~~~~~~~ +You can use the embedded SQream Client on the following nodes: + +* Master node +* Worker node + + +When using the SQream Client on the Master node, the following default settings are used: + +* **Default port**: 3108. You can change the default port using the ``-p`` variable. +* **Default database**: master. You can change the default database using the ``-d`` variable. + +The following is an example: + +.. code-block:: + + $ sqream client --master -u sqream -w sqream + + +When using the SQream Client on a Worker node (or nodes), you should use the ``-p`` variable for Worker ports. The default database is ``master``, but you can use the ``-d`` variable to change databases. + +The following is an example: + +.. code-block:: + + $ sqream client --worker -p 5000 -u sqream -w sqream + + +Moving from Docker Installation to Standard On-Premises Installation +----------------------------------------------- + +Because Docker creates all files and directories on the host at the **root** level, you must grant ownership of the SQream storage folder to the working directory user. + +SQream Console - Advanced Commands +----------------------------- + +The SQream console offers the following advanced commands: + + +* :ref:`Controlling the spool size ` +* :ref:`Splitting a GPU ` +* :ref:`Splitting a GPU and setting the spool size ` +* :ref:`Using a custom configuration file ` +* :ref:`Clustering your Docker environment ` + + + + +.. _controlling_spool_size: + +Controlling the Spool Size +~~~~~~~~~~~~~~~~~~ + +From the console you can define a spool size value. + +The following example shows the spool size being set to ``50``: + +.. code-block:: + + $ sqream-console>sqream worker --start 2 -m 50 + + +If you don't define the SQream spool size, the SQream console automatically distributes the available RAM between all running workers. + +.. _splitting_gpu: + +Splitting a GPU +~~~~~~~~~~~~~~~~~~ + +You can start more than one sqreamd on a single GPU by splitting it. + + +The following example shows the GPU being split into **two** sqreamd's on the GPU in **slot 0**: + +.. code-block:: + + $ sqream-console>sqream worker --start 2 -g 0 + +.. _splitting_gpu_setting_spool_size: + +Splitting GPU and Setting the Spool Size +~~~~~~~~~~~~~~~~~~ + +You can simultaneously split a GPU and set the spool size by appending the ``-m`` flag: + +.. code-block:: + + $ sqream-console>sqream worker --start 2 -g 0 -m 50 + +.. note:: The console does not validate whether the user-defined spool size is available. Before setting the spool size, verify that the requested resources are available. + +.. _using_custom_configuration_file: + +Using a Custom Configuration File +~~~~~~~~~~~~~~~~~~ + +SQream lets you use your own external custom configuration json files. You must place these json files in the path mounted in the installation. SQream recommends placing the json file in the Configuration folder. + +The SQream console does not validate the integrity of your external configuration files. + +When using your custom configuration file, you can use the ``-j`` flag to define the full path to the Configuration file, as in the example below: + +.. code-block:: + + $ sqream-console>sqream worker --start 1 -j /etc/sqream/configfile.json + +.. note:: To start more than one sqream daemon, you must provide files for each daemon, as in the example below: + +.. code-block:: + + $ sqream worker --start 2 -j /etc/sqream/configfile.json /etc/sqream/configfile2.json + +.. note:: To split a specific GPU, you must also list the GPU flag, as in the example below: + +.. code-block:: + + $ sqream worker --start 2 -g 0 -j /etc/sqream/configfile.json /etc/sqream/configfile2.json + +.. _clustering_docker_environment: + +Clustering Your Docker Environment +~~~~~~~~~~~~~~~~~~ + +SQream lets you connect to a remote Master node to start Docker in Distributed mode. If you have already connected to a Slave node server in Distributed mode, the **sqream Master** and **Client** commands are only available on the Master node. + +.. code-block:: + + $ --master-host + $ sqream-console>sqream worker --start 1 --master-host 192.168.0.1020 + +Checking the Status of SQream Services +--------------------------- +SQream lets you check the status of SQream services from the following locations: + +* :ref:`From the Sqream console ` +* :ref:`From outside the Sqream console ` + +.. _inside_sqream_console: + +Checking the Status of SQream Services from the SQream Console +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +From the SQream console, you can check the status of SQream services by running the following command: + +.. code-block:: + + $ sqream-console>sqream master --list + +The following is an example of the expected output: + +.. code-block:: + + $ sqream-console>sqream master --list + $ checking 3 sqream services: + $ sqream_single_host_worker_1 up, listens on port: 5001 allocated gpu: 1 + $ sqream_single_host_worker_0 up, listens on port: 5000 allocated gpu: 1 + $ sqream_single_host_master up listens on ports: 3105,3108 + +.. _outside_sqream_console: + +Checking the Status of SQream Services from Outside the SQream Console +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +From outside the Sqream Console, you can check the status of SQream services by running the following commands: + +.. code-block:: + + $ sqream-status + $ NAMES STATUS PORTS + $ sqream_single_host_worker_1 Up 3 minutes 0.0.0.0:5001->5001/tcp + $ sqream_single_host_worker_0 Up 3 minutes 0.0.0.0:5000->5000/tcp + $ sqream_single_host_master Up 3 minutes 0.0.0.0:3105->3105/tcp, 0.0.0.0:3108->3108/tcp + $ sqream_editor_3.0.0 Up 3 hours (healthy) 0.0.0.0:3000->3000/tcp + +Upgrading Your SQream System +---------------------------- +This section describes how to upgrade your SQream system. + +**To upgrade your SQream system:** + +1. Contact the SQream Support team for access to the new SQream package tarball file. + + :: + +2. Set a maintenance window to enable stopping the system while upgrading it. + + :: + +3. Extract the following tarball file received from the SQream Support team, under it with the same user and in the same folder that you used while :ref:`Downloading the SQream Software <_download_sqream_software>`. + + + .. code-block:: + + $ tar -xvf sqream_installer-2.0.5-DB2019.2.1-CO1.6.3-ED3.0.0-x86_64/ + +4. Navigate to the new folder created as a result of extracting the tarball file: + + .. code-block:: + + $ cd sqream_installer-2.0.5-DB2019.2.1-CO1.6.3-ED3.0.0-x86_64/ + +5. Initiate the upgrade process: + + .. code-block:: + + $ ./sqream-install -i + + Initiating the upgrade process checks if any SQream services are running. If any services are running, you will be prompted to stop them. + +6. Do one of the following: + + * Select **Yes** to stop all running SQream workers (Master and Editor) and continue the upgrade process. + * Select **No** to stop the upgrade process. + + SQream periodically upgrades the metadata structure. If an upgrade version includes a change to the metadata structure, you will be prompted with an approval request message. Your approval is required to finish the upgrade process. + + Because SQream supports only certain metadata versions, all SQream services must be upgraded at the same time. + +7. When the upgrade is complete, load the SQream console and restart your services. + + For assistance, contact SQream Support. diff --git a/installation_guides/sqream_studio_installation.rst b/installation_guides/sqream_studio_installation.rst new file mode 100644 index 000000000..53c89772d --- /dev/null +++ b/installation_guides/sqream_studio_installation.rst @@ -0,0 +1,16 @@ +.. _sqream_studio_installation: + +************************* +Installing SQream Studio +************************* +The **Installing SQream Studio** page incudes the following installation guides: + +.. toctree:: + :maxdepth: 1 + :glob: + + installing_prometheus_exporters + installing_prometheus_using_binary_packages + installing_dashboard_data_collector + installing_studio_on_stand_alone_server + installing_nginx_proxy_over_secure_connection \ No newline at end of file diff --git a/loading_and_unloading_data/index.rst b/loading_and_unloading_data/index.rst new file mode 100644 index 000000000..e90606719 --- /dev/null +++ b/loading_and_unloading_data/index.rst @@ -0,0 +1,36 @@ +.. _loading_and_unloading_data: + +*********************** +Loading and Unloading Data +*********************** +The **Loading Data** section describes concepts and operations related to importing data into your SQream database: + +* `Overview of loading data `_ - Describes best practices and considerations for loading data into SQream from a variety of sources and locations. + +* `Alternatives to loading data (foreign tables) `_ - Useful for running queries directly on external data without importing into your SQream database. + +* `Supported data types `_ - Overview of supported data types, including descriptions, examples, and relevant aliases. + +* `Ingesting data from external sources `_ - List of data ingestion sources that SQream supports. + +* `Inserting data from external tables `_ - Inserts one or more rows into a table. + +* `Ingesting data from third party client platforms `_ - Gives you direct access to a variety of drivers, connectors, tools, vizualisers, and utilities.. + +* `Using the COPY FROM statement `_ - Used for loading data from files located on a filesystem into SQream tables. + +* `Importing data using Studio `_ - SQream's web-based client providing users with all functionality available from the command line in an intuitive and easy-to-use format. + +* `Loading data using Amazon S3 `_ - Used for loading data from Amazon S3. + +* Troubleshooting - Describes troubleshooting solutions related to importing data from the following: + + * `SAS Viya `_ + + * `Tableau `_ + +The **Unloading Data** section describes concepts and operations related to exporting data from your SQream database: + +* `Overview of unloading data `_ - Describes best practices and considerations for unloading data from SQream to a variety of sources and locations. + +* `The COPY TO statement `_ - Used for unloading data from a SQream database table or query to a file on a filesystem. \ No newline at end of file diff --git a/operational_guides/access_control.rst b/operational_guides/access_control.rst new file mode 100644 index 000000000..c8ffe3097 --- /dev/null +++ b/operational_guides/access_control.rst @@ -0,0 +1,17 @@ +.. _access_control: + +************** +Access Control +************** +The **Access Control** page describes the following: + +.. toctree:: + :maxdepth: 1 + :titlesonly: + + access_control_overview + access_control_managing_roles + access_control_permissions + access_control_departmental_example + + diff --git a/operational_guides/access_control_departmental_example.rst b/operational_guides/access_control_departmental_example.rst new file mode 100644 index 000000000..0a6b55e54 --- /dev/null +++ b/operational_guides/access_control_departmental_example.rst @@ -0,0 +1,185 @@ +.. _access_control_departmental_example: + +************** +Departmental Example +************** + +You work in a company with several departments. + +The example below shows you how to manage permissions in a database shared by multiple departments, where each department has different roles for the tables by schema. It walks you through how to set the permissions up for existing objects and how to set up default permissions rules to cover newly created objects. + +The concept is that you set up roles for each new schema with the correct permissions, then the existing users can use these roles. + +A superuser must do new setup for each new schema which is a limitation, but superuser permissions are not needed at any other time, and neither are explicit grant statements or object ownership changes. + +In the example, the database is called ``my_database``, and the new or existing schema being set up to be managed in this way is called ``my_schema``. + +Our departmental example has four user group roles and seven users roles + +There will be a group for this schema for each of the following: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Group + - Activities + + * - database designers + - create, alter and drop tables + + * - updaters + - insert and delete data + + * - readers + - read data + + * - security officers + - add and remove users from these groups + +Setting up the department permissions +------------------------------------------ + +As a superuser, you connect to the system and run the following: + +.. code-block:: postgres + + -- create the groups + + CREATE ROLE my_schema_security_officers; + CREATE ROLE my_schema_database_designers; + CREATE ROLE my_schema_updaters; + CREATE ROLE my_schema_readers; + + -- grant permissions for each role + -- we grant permissions for existing objects here too, + -- so you don't have to start with an empty schema + + -- security officers + + GRANT connect ON DATABASE my_database TO my_schema_security_officers; + GRANT usage ON SCHEMA my_schema TO my_schema_security_officers; + + GRANT my_schema_database_designers TO my_schema_security_officers WITH ADMIN OPTION; + GRANT my_schema_updaters TO my_schema_security_officers WITH ADMIN OPTION; + GRANT my_schema_readers TO my_schema_security_officers WITH ADMIN OPTION; + + -- database designers + + GRANT connect ON DATABASE my_database TO my_schema_database_designers; + GRANT usage ON SCHEMA my_schema TO my_schema_database_designers; + + GRANT create,ddl ON SCHEMA my_schema TO my_schema_database_designers; + + -- updaters + + GRANT connect ON DATABASE my_database TO my_schema_updaters; + GRANT usage ON SCHEMA my_schema TO my_schema_updaters; + + GRANT SELECT,INSERT,DELETE ON ALL TABLES IN SCHEMA my_schema TO my_schema_updaters; + + -- readers + + GRANT connect ON DATABASE my_database TO my_schema_readers; + GRANT usage ON SCHEMA my_schema TO my_schema_readers; + + GRANT SELECT ON ALL TABLES IN SCHEMA my_schema TO my_schema_readers; + GRANT EXECUTE ON ALL FUNCTIONS TO my_schema_readers; + + + -- create the default permissions for new objects + + ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema + FOR TABLES GRANT SELECT,INSERT,DELETE TO my_schema_updaters; + + -- For every table created by my_schema_database_designers, give access to my_schema_readers: + + ALTER DEFAULT PERMISSIONS FOR my_schema_database_designers IN my_schema + FOR TABLES GRANT SELECT TO my_schema_readers; + +.. note:: + * This process needs to be repeated by a user with ``SUPERUSER`` permissions each time a new schema is brought into this permissions management approach. + + * + By default, any new object created will not be accessible by our new ``my_schema_readers`` group. + Running a ``GRANT SELECT ...`` only affects objects that already exist in the schema or database. + + If you're getting a ``Missing the following permissions: SELECT on table 'database.public.tablename'`` error, make sure that + you've altered the default permissions with the ``ALTER DEFAULT PERMISSIONS`` statement. + +Creating new users in the departments +----------------------------------------- + +After the group roles have been created, you can now create user roles for each of your users. + +.. code-block:: postgres + + -- create the new database designer users + + CREATE ROLE ecodd; + GRANT LOGIN TO ecodd; + GRANT PASSWORD 'ecodds_secret_password' TO ecodd; + GRANT CONNECT ON DATABASE my_database TO ecodd; + GRANT my_schema_database_designers TO ecodd; + + CREATE ROLE ebachmann; + GRANT LOGIN TO ebachmann; + GRANT PASSWORD 'another_secret_password' TO ebachmann; + GRANT CONNECT ON DATABASE my_database TO ebachmann; + GRANT my_database_designers TO ebachmann; + + -- If a user already exists, we can assign that user directly to the group + + GRANT my_schema_updaters TO rhendricks; + + -- Create users in the readers group + + CREATE ROLE jbarker; + GRANT LOGIN TO jbarker; + GRANT PASSWORD 'action_jack' TO jbarker; + GRANT CONNECT ON DATABASE my_database TO jbarker; + GRANT my_schema_readers TO jbarker; + + CREATE ROLE lbream; + GRANT LOGIN TO lbream; + GRANT PASSWORD 'artichoke123' TO lbream; + GRANT CONNECT ON DATABASE my_database TO lbream; + GRANT my_schema_readers TO lbream; + + CREATE ROLE pgregory; + GRANT LOGIN TO pgregory; + GRANT PASSWORD 'c1ca6a' TO pgregory; + GRANT CONNECT ON DATABASE my_database TO pgregory; + GRANT my_schema_readers TO pgregory; + + -- Create users in the security officers group + + CREATE ROLE hoover; + GRANT LOGIN TO hoover; + GRANT PASSWORD 'mintchip' TO hoover; + GRANT CONNECT ON DATABASE my_database TO hoover; + GRANT my_schema_security_officers TO hoover; + + +.. todo: + create some example users + show that they have the right permission + try out the with admin option. we can't really do a security officer because + only superusers can create users and logins. see what can be done + need 1-2 users in each group, for at least 2 schemas/departments + this example will be very big just to show what this setup can do ... + example: a security officer for a department which will only have + read only access to a schema can only get that with admin option + access granted to them + +After this setup: + +* Database designers will be able to run any ddl on objects in the schema and create new objects, including ones created by other database designers +* Updaters will be able to insert and delete to existing and new tables +* Readers will be able to read from existing and new tables + +All this will happen without having to run any more ``GRANT`` statements. + +Any security officer will be able to add and remove users from these +groups. Creating and dropping login users themselves must be done by a +superuser. \ No newline at end of file diff --git a/operational_guides/access_control_managing_roles.rst b/operational_guides/access_control_managing_roles.rst new file mode 100644 index 000000000..373b9a212 --- /dev/null +++ b/operational_guides/access_control_managing_roles.rst @@ -0,0 +1,124 @@ +.. _access_control_managing_roles: + +************** +Managing Roles +************** +Roles are used for both users and groups, and are global across all databases in the SQream cluster. For a ``ROLE`` to be used as a user, it requires a password and log-in and connect permissionss to the relevant databases. + +The Managing Roles section describes the following role-related operations: + +.. contents:: + :local: + :depth: 1 + +Creating New Roles (Users) +------------------------------ +A user role logging in to the database requires ``LOGIN`` permissions and as a password. + +The following is the syntax for creating a new role: + +.. code-block:: postgres + + CREATE ROLE ; + GRANT LOGIN to ; + GRANT PASSWORD <'new_password'> to ; + GRANT CONNECT ON DATABASE to ; + +The following is an example of creating a new role: + +.. code-block:: postgres + + CREATE ROLE new_role_name ; + GRANT LOGIN TO new_role_name; + GRANT PASSWORD 'my_password' to new_role_name; + GRANT CONNECT ON DATABASE master to new_role_name; + +A database role may have a number of permissions that define what tasks it can perform, which are assigned using the :ref:`grant` command. + +Dropping a User +------------------------------ +The following is the syntax for dropping a user: + +.. code-block:: postgres + + DROP ROLE ; + +The following is an example of dropping a user: + +.. code-block:: postgres + + DROP ROLE admin_role ; + +Altering a User Name +------------------------------ +The following is the syntax for altering a user name: + +.. code-block:: postgres + + ALTER ROLE RENAME TO ; + +The following is an example of altering a user name: + +.. code-block:: postgres + + ALTER ROLE admin_role RENAME TO copy_role ; + +Changing a User Password +------------------------------ +You can change a user role's password by granting the user a new password. + +The following is an example of changing a user password: + +.. code-block:: postgres + + GRANT PASSWORD <'new_password'> TO rhendricks; + +.. note:: Granting a new password overrides any previous password. Changing the password while the role has an active running statement does not affect that statement, but will affect subsequent statements. + +Altering Public Role Permissions +------------------------------ + +There is a public role which always exists. Each role is granted to the ``PUBLIC`` role (i.e. is a member of the public group), and this cannot be revoked. You can alter the permissions granted to the public role. + +The ``PUBLIC`` role has ``USAGE`` and ``CREATE`` permissions on ``PUBLIC`` schema by default, therefore, new users can create, :ref:`insert`, :ref:`delete`, and :ref:`select` from objects in the ``PUBLIC`` schema. + + +Altering Role Membership (Groups) +------------------------------ + +Many database administrators find it useful to group user roles together. By grouping users, permissions can be granted to, or revoked from a group with one command. In SQream DB, this is done by creating a group role, granting permissions to it, and then assigning users to that group role. + +To use a role purely as a group, omit granting it ``LOGIN`` and ``PASSWORD`` permissions. + +The ``CONNECT`` permission can be given directly to user roles, and/or to the groups they are part of. + +.. code-block:: postgres + + CREATE ROLE my_group; + +Once the group role exists, you can add user roles (members) using the ``GRANT`` command. For example: + +.. code-block:: postgres + + -- Add my_user to this group + GRANT my_group TO my_user; + + +To manage object permissions like databases and tables, you would then grant permissions to the group-level role (see :ref:`the permissions table` below. + +All member roles then inherit the permissions from the group. For example: + +.. code-block:: postgres + + -- Grant all group users connect permissions + GRANT CONNECT ON DATABASE a_database TO my_group; + + -- Grant all permissions on tables in public schema + GRANT ALL ON all tables IN schema public TO my_group; + +Removing users and permissions can be done with the ``REVOKE`` command: + +.. code-block:: postgres + + -- remove my_other_user from this group + REVOKE my_group FROM my_other_user; \ No newline at end of file diff --git a/operational_guides/access_control_overview.rst b/operational_guides/access_control_overview.rst new file mode 100644 index 000000000..080797fec --- /dev/null +++ b/operational_guides/access_control_overview.rst @@ -0,0 +1,20 @@ +.. _access_control_overview: + +************** +Overview +************** +Access control refers to SQream's authentication and authorization operations, managed using a **Role-Based Access Control (RBAC)** system, such as ANSI SQL or other SQL products. SQream's default permissions system is similar to Postgres, but is more powerful. SQream's method lets administrators prepare the system to automatically provide objects with their required permissions. + +SQream users can log in from any worker, which verify their roles and permissions from the metadata server. Each statement issues commands as the role that you're currently logged into. Roles are defined at the cluster level, and are valid for all databases in the cluster. To bootstrap SQream, new installations require one ``SUPERUSER`` role, typically named ``sqream``. You can only create new roles by connecting as this role. + +Access control refers to the following basic concepts: + + * **Role** - A role can be a user, a group, or both. Roles can own database objects (such as tables) and can assign permissions on those objects to other roles. Roles can be members of other roles, meaning a user role can inherit permissions from its parent role. + + :: + + * **Authentication** - Verifies the identity of the role. User roles have usernames (or **role names**) and passwords. + + :: + + * **Authorization** - Checks that a role has permissions to perform a particular operation, such as the :ref:`grant` command. \ No newline at end of file diff --git a/operational_guides/access_control_permissions.rst b/operational_guides/access_control_permissions.rst new file mode 100644 index 000000000..5afc23009 --- /dev/null +++ b/operational_guides/access_control_permissions.rst @@ -0,0 +1,218 @@ +.. _access_control_permissions: + +************** +Permissions +************** + +The following table displays the access control permissions: + ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Permission** | **Description** | ++====================+=========================================================================================================================+ +| **Object/Layer: All Databases** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``LOGIN`` | use role to log into the system (the role also needs connect permission on the database it is connecting to) | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``PASSWORD`` | the password used for logging into the system | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``SUPERUSER`` | no permission restrictions on any activity | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Database** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``SUPERUSER`` | no permission restrictions on any activity within that database (this does not include modifying roles or permissions) | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CONNECT`` | connect to the database | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CREATE`` | create schemas in the database | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CREATE FUNCTION``| create and drop functions | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Schema** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``USAGE`` | allows additional permissions within the schema | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``CREATE`` | create tables in the schema | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Table** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``SELECT`` | :ref:`select` from the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``INSERT`` | :ref:`insert` into the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``UPDATE`` | UPDATE the value of certain columns in existing rows without creating a table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``DELETE`` | :ref:`delete` and :ref:`truncate` on the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``DDL`` | drop and alter on the table | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``ALL`` | all the table permissions | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| **Object/Layer: Function** | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``EXECUTE`` | use the function | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``DDL`` | drop and alter on the function | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ +| ``ALL`` | all function permissions | ++--------------------+-------------------------------------------------------------------------------------------------------------------------+ + + + + +GRANT +----- + +:ref:`grant` gives permissions to a role. + +.. code-block:: postgres + + -- Grant permissions at the instance/ storage cluster level: + GRANT + + { SUPERUSER + | LOGIN + | PASSWORD '' + } + TO [, ...] + + -- Grant permissions at the database level: + GRANT {{CREATE | CONNECT| DDL | SUPERUSER | CREATE FUNCTION} [, ...] | ALL [PERMISSIONS]} + + ON DATABASE [, ...] + TO [, ...] + + -- Grant permissions at the schema level: + GRANT {{ CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [ + PERMISSIONS ]} + ON SCHEMA [, ...] + TO [, ...] + + -- Grant permissions at the object level: + GRANT {{SELECT | INSERT | DELETE | DDL } [, ...] | ALL [PERMISSIONS]} + ON { TABLE [, ...] | ALL TABLES IN SCHEMA [, ...]} + TO [, ...] + + -- Grant execute function permission: + GRANT {ALL | EXECUTE | DDL} ON FUNCTION function_name + TO role; + + -- Allows role2 to use permissions granted to role1 + GRANT [, ...] + TO + + -- Also allows the role2 to grant role1 to other roles: + GRANT [, ...] + TO + WITH ADMIN OPTION + +``GRANT`` examples: + +.. code-block:: postgres + + GRANT LOGIN,superuser TO admin; + + GRANT CREATE FUNCTION ON database master TO admin; + + GRANT SELECT ON TABLE admin.table1 TO userA; + + GRANT EXECUTE ON FUNCTION my_function TO userA; + + GRANT ALL ON FUNCTION my_function TO userA; + + GRANT DDL ON admin.main_table TO userB; + + GRANT ALL ON all tables IN schema public TO userB; + + GRANT admin TO userC; + + GRANT superuser ON schema demo TO userA + + GRANT admin_role TO userB; + +REVOKE +------ + +:ref:`revoke` removes permissions from a role. + +.. code-block:: postgres + + -- Revoke permissions at the instance/ storage cluster level: + REVOKE + { SUPERUSER + | LOGIN + | PASSWORD + } + FROM [, ...] + + -- Revoke permissions at the database level: + REVOKE {{CREATE | CONNECT | DDL | SUPERUSER | CREATE FUNCTION}[, ...] |ALL [PERMISSIONS]} + ON DATABASE [, ...] + FROM [, ...] + + -- Revoke permissions at the schema level: + REVOKE { { CREATE | DDL | USAGE | SUPERUSER } [, ...] | ALL [PERMISSIONS]} + ON SCHEMA [, ...] + FROM [, ...] + + -- Revoke permissions at the object level: + REVOKE { { SELECT | INSERT | DELETE | DDL } [, ...] | ALL } + ON { [ TABLE ] [, ...] | ALL TABLES IN SCHEMA + + [, ...] } + FROM [, ...] + + -- Removes access to permissions in role1 by role 2 + REVOKE [, ...] FROM [, ...] WITH ADMIN OPTION + + -- Removes permissions to grant role1 to additional roles from role2 + REVOKE [, ...] FROM [, ...] WITH ADMIN OPTION + + +Examples: + +.. code-block:: postgres + + REVOKE superuser on schema demo from userA; + + REVOKE delete on admin.table1 from userB; + + REVOKE login from role_test; + + REVOKE CREATE FUNCTION FROM admin; + +Default permissions +------------------- + +The default permissions system (See :ref:`alter_default_permissions`) +can be used to automatically grant permissions to newly +created objects (See the departmental example below for one way it can be used). + +A default permissions rule looks for a schema being created, or a +table (possibly by schema), and is table to grant any permission to +that object to any role. This happens when the create table or create +schema statement is run. + + +.. code-block:: postgres + + + ALTER DEFAULT PERMISSIONS FOR target_role_name + [IN schema_name, ...] + FOR { TABLES | SCHEMAS } + { grant_clause | DROP grant_clause} + TO ROLE { role_name | public }; + + grant_clause ::= + GRANT + { CREATE FUNCTION + | SUPERUSER + | CONNECT + | CREATE + | USAGE + | SELECT + | INSERT + | DELETE + | DDL + | EXECUTE + | ALL + } \ No newline at end of file diff --git a/guides/operations/creating_or_cloning_a_storage_cluster.rst b/operational_guides/creating_or_cloning_a_storage_cluster.rst similarity index 99% rename from guides/operations/creating_or_cloning_a_storage_cluster.rst rename to operational_guides/creating_or_cloning_a_storage_cluster.rst index 0ff7fe294..0406bda77 100644 --- a/guides/operations/creating_or_cloning_a_storage_cluster.rst +++ b/operational_guides/creating_or_cloning_a_storage_cluster.rst @@ -1,9 +1,8 @@ .. _creating_or_cloning_a_storage_cluster: **************************************** -Creating or cloning a storage cluster +Creating or Cloning Storage Clusters **************************************** - When SQream DB is installed, it comes with a default storage cluster. This guide will help if you need a fresh storage cluster or a separate copy of an existing storage cluster. Creating a new storage cluster diff --git a/guides/features/delete.rst b/operational_guides/delete_guide.rst similarity index 99% rename from guides/features/delete.rst rename to operational_guides/delete_guide.rst index b72a9617b..24ab5a218 100644 --- a/guides/features/delete.rst +++ b/operational_guides/delete_guide.rst @@ -1,7 +1,7 @@ .. _delete_guide: *********************** -Deleting data +Deleting Data *********************** SQream DB supports deleting data, but it's important to understand how this works and how to maintain deleted data. diff --git a/operational_guides/exporting_data.rst b/operational_guides/exporting_data.rst new file mode 100644 index 000000000..62d32f1c8 --- /dev/null +++ b/operational_guides/exporting_data.rst @@ -0,0 +1,15 @@ +.. _exporting_data: + +*********************** +Exporting Data +*********************** +You can export data from SQream, which you may want to do for the following reasons: + + +* To use data in external tables. See `Working with External Data `_. +* To share data with other clients or consumers with different systems. +* To copy data into another SQream cluster. + +SQream provides the following methods for exporting data: + +* Copying data from a SQream database table or query to another file - See `COPY TO `_. \ No newline at end of file diff --git a/guides/features/external_tables.rst b/operational_guides/foreign_tables.rst similarity index 63% rename from guides/features/external_tables.rst rename to operational_guides/foreign_tables.rst index b3b566083..74c476331 100644 --- a/guides/features/external_tables.rst +++ b/operational_guides/foreign_tables.rst @@ -1,128 +1,79 @@ -.. _external_tables: +.. _foreign_tables: -************************** +*********************** Foreign Tables -************************** - +*********************** Foreign tables can be used to run queries directly on data without inserting it into SQream DB first. +SQream DB supports read only external tables, so you can query from external tables, but you cannot insert to them, or run deletes or updates on them. -SQream DB supports read only foreign tables, so you can query from foreign tables, but you cannot insert to them, or run deletes or updates on them. - -Running queries directly on foreign (external) data is most effectively used for things like one off querying. If you will be repeatedly querying data, the performance will usually be better if you insert the data into SQream DB first. - -Although foreign tables can be used without inserting data into SQream DB, one of their main use cases is to help with the insertion process. An insert select statement on an foreign table can be used to insert data into SQream using the full power of the query engine to perform ETL. +Running queries directly on external data is most effectively used for things like one off querying. If you will be repeatedly querying data, the performance will usually be better if you insert the data into SQream DB first. +Although external tables can be used without inserting data into SQream DB, one of their main use cases is to help with the insertion process. An insert select statement on an external table can be used to insert data into SQream using the full power of the query engine to perform ETL. .. contents:: In this topic: :local: -What kind of data is supported? +Supported Data Formats ===================================== +SQream DB supports external tables over: -SQream DB uses foreign data wrappers (FDW) to abstract external sources. - -SQream DB supports these FDWs: - -* text files (e.g. CSV, PSV, TSV) via the ``csv_fdw`` -* ORC via the ``orc_fdw`` -* Parquet via the ``parquet_fdw`` +* Text files (e.g. CSV, PSV, TSV) +* ORC +* Parquet -What kind of data staging is supported? +Supported Data Staging ============================================ - -SQream DB can stage data from: +SQream can stage data from: * a local filesystem (e.g. ``/mnt/storage/....``) * :ref:`s3` buckets (e.g. ``s3://pp-secret-bucket/users/*.parquet``) * :ref:`hdfs` (e.g. ``hdfs://hadoop-nn.piedpiper.com/rhendricks/*.csv``) -Using foreign tables - a practical example +Using External Tables ============================================== +Use an external table to stage data before loading from CSV, Parquet or ORC files. -Use an foreign table to stage data before loading from CSV, Parquet or ORC files. - -Planning for data staging +Planning for Data Staging -------------------------------- - For the following examples, we will want to interact with a CSV file. Here's a peek at the table contents: - + .. csv-table:: nba.csv :file: nba-t10.csv :widths: auto - :header-rows: 1 + :header-rows: 1 The file is stored on :ref:`s3`, at ``s3://sqream-demo-data/nba_players.csv``. - We will make note of the file structure, to create a matching ``CREATE_EXTERNAL_TABLE`` statement. -Creating the foreign table ------------------------------------- - -Based on the source file structure, we we :ref:`create a foreign table` with the appropriate structure, and point it to the file. - -The file format in this case is CSV, with a DOS newline (``\r\n``). - -Here's how the table would be created from S3 and HDFS: - -From S3 -^^^^^^^^^^ - - -.. code-block:: postgres - - CREATE FOREIGN TABLE nba - ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float - ) - WRAPPER csv_fdw -- Text file - OPTIONS - ( LOCATION = 's3://sqream-demo-data/nba_players.csv', - RECORD_DELIMITER = '\r\n'; -- DOS delimited file - ) - ; - -.. _hdfs_external_table_demo: - -From HDFS -^^^^^^^^^^^^^^ - +Creating an External Table +----------------------------- +Based on the source file structure, we we :ref:`create an external table` with the appropriate structure, and point it to the file. .. code-block:: postgres - CREATE FOREIGN TABLE nba + CREATE EXTERNAL TABLE nba ( - Name varchar(40), - Team varchar(40), + Name varchar, + Team varchar, Number tinyint, - Position varchar(2), + Position varchar, Age tinyint, - Height varchar(4), + Height varchar, Weight real, - College varchar(40), + College varchar, Salary float ) - WRAPPER csv_fdw -- Text file - OPTIONS - ( - LOCATION = 'hdfs://hadoop-nn.piedpiper.com:8020/demo-data/nba_players.csv', - RECORD_DELIMITER = '\r\n'; -- DOS delimited file - ) - ; + USING FORMAT CSV -- Text file + WITH PATH 's3://sqream-demo-data/nba_players.csv' + RECORD DELIMITER '\r\n'; -- DOS delimited file +The file format in this case is CSV, and it is stored as an :ref:`s3` object (if the path is on :ref:`hdfs`, change the URI accordingly). +We also took note that the record delimiter was a DOS newline (``\r\n``). - -Querying foreign tables +Querying External Tables ------------------------------ -Let's peek at the data from the foreign table: +Let's peek at the data from the external table: .. code-block:: psql @@ -140,11 +91,9 @@ Let's peek at the data from the foreign table: Terry Rozier | Boston Celtics | 12 | PG | 22 | 6-2 | 190 | Louisville | 1824360 Marcus Smart | Boston Celtics | 36 | PG | 22 | 6-4 | 220 | Oklahoma State | 3431040 -Modifying data from staging +Modifying Data from Staging ------------------------------- - One of the main reasons for staging data is to examine the contents and modify them before loading them. - Assume we are unhappy with weight being in pounds, because we want to use kilograms instead. We can apply the transformation as part of a query: .. code-block:: psql @@ -152,6 +101,7 @@ Assume we are unhappy with weight being in pounds, because we want to use kilogr t=> SELECT name, team, number, position, age, height, (weight / 2.205) as weight, college, salary . FROM nba . ORDER BY weight; + name | team | number | position | age | height | weight | college | salary -------------------------+------------------------+--------+----------+-----+--------+----------+-----------------------+--------- Nikola Pekovic | Minnesota Timberwolves | 14 | C | 30 | 6-11 | 139.229 | | 12100000 @@ -165,14 +115,14 @@ Assume we are unhappy with weight being in pounds, because we want to use kilogr Cristiano Felicio | Chicago Bulls | 6 | PF | 23 | 6-10 | 124.7166 | | 525093 [...] -Now, if we're happy with the results, we can convert the staged foreign table to a standard table +Now, if we're happy with the results, we can convert the staged external table to a standard table -Converting a foreign table to a standard database table +Converting an External Table to a Standard Database Table --------------------------------------------------------------- -:ref:`create_table_as` can be used to materialize a foreign table into a regular table. +:ref:`create_table_as` can be used to materialize an external table into a regular table. -.. tip:: If you intend to use the table multiple times, convert the foreign table to a standard table. +.. tip:: If you intend to use the table multiple times, convert the external table to a standard table. .. code-block:: psql @@ -182,6 +132,7 @@ Converting a foreign table to a standard database table . ORDER BY weight; executed t=> SELECT * FROM real_nba LIMIT 5; + name | team | number | position | age | height | weight | college | salary -----------------+------------------------+--------+----------+-----+--------+----------+-------------+--------- Nikola Pekovic | Minnesota Timberwolves | 14 | C | 30 | 6-11 | 139.229 | | 12100000 @@ -190,15 +141,12 @@ Converting a foreign table to a standard database table Jusuf Nurkic | Denver Nuggets | 23 | C | 21 | 7-0 | 126.9841 | | 1842000 Andre Drummond | Detroit Pistons | 0 | C | 22 | 6-11 | 126.5306 | Connecticut | 3272091 - -Error handling and limitations +Error Handling and Limitations ================================== - -* Error handling in foreign tables is limited. Any error that occurs during source data parsing will result in the statement aborting. +* Error handling in external tables is limited. Any error that occurs during source data parsing will result in the statement aborting. * - Foreign tables are logical and do not contain any data, their structure is not verified or enforced until a query uses the table. - + External tables are logical and do not contain any data, their structure is not verified or enforced until a query uses the table. For example, a CSV with the wrong delimiter may cause a query to fail, even though the table has been created successfully: .. code-block:: psql @@ -206,5 +154,4 @@ Error handling and limitations t=> SELECT * FROM nba; master=> select * from nba; Record delimiter mismatch during CSV parsing. User defined line delimiter \n does not match the first delimiter \r\n found in s3://sqream-demo-data/nba.csv - -* Since the data for a foreign table is not stored in SQream DB, it can be changed or removed at any time by an external process. As a result, the same query can return different results each time it runs against a foreign table. Similarly, a query might fail if the external data is moved, removed, or has changed structure. +* Since the data for an external table is not stored in SQream DB, it can be changed or removed at any time by an external process. As a result, the same query can return different results each time it runs against an external table. Similarly, a query might fail if the external data is moved, removed, or has changed structure. \ No newline at end of file diff --git a/operational_guides/index.rst b/operational_guides/index.rst new file mode 100644 index 000000000..e72ff27c8 --- /dev/null +++ b/operational_guides/index.rst @@ -0,0 +1,25 @@ +.. _operational_guides: + +********************************** +Operational Guides +********************************** +The **Operational Guides** section describes processes that SQream users can manage to affect the way their system operates, such as creating storage clusters and monitoring query performance. + +This section summarizes the following operational guides: + +.. toctree:: + :maxdepth: 1 + :glob: + :titlesonly: + + access_control + creating_or_cloning_a_storage_cluster + foreign_tables + delete_guide + exporting_data + logging + monitoring_query_performance + security + saved_queries + seeing_system_objects_as_ddl + optimization_best_practices diff --git a/guides/operations/logging.rst b/operational_guides/logging.rst similarity index 93% rename from guides/operations/logging.rst rename to operational_guides/logging.rst index 2a4f5c317..64ac5c8d8 100644 --- a/guides/operations/logging.rst +++ b/operational_guides/logging.rst @@ -4,7 +4,7 @@ Logging *********************** -Locating the log files +Locating the Log Files ========================== The :ref:`storage cluster` contains a ``logs`` directory. Each worker produces a log file in its own directory, which can be identified by the worker's hostname and port. @@ -23,7 +23,7 @@ The worker logs contain information messages, warnings, and errors pertaining to * Statement execution success / failure * Statement execution statistics -Log structure and contents +Log Structure and Contents --------------------------------- The log is a CSV, with several fields. @@ -68,7 +68,7 @@ The log is a CSV, with several fields. .. _information_level: -.. list-table:: Information level +.. list-table:: Information Level :widths: auto :header-rows: 1 @@ -87,7 +87,7 @@ The log is a CSV, with several fields. .. _message_type: -.. list-table:: Message type +.. list-table:: Message Type :widths: auto :header-rows: 1 @@ -158,7 +158,7 @@ The log is a CSV, with several fields. * - ``1000`` - ``SYSTEM`` - Worker startup message - - ``"Server Start Time - 2019-12-30 21:18:31, SQream ver{v2020.1}"`` + - ``"Server Start Time - 2019-12-30 21:18:31, SQream ver{v2020.2}"`` * - ``1002`` - ``SYSTEM`` - ``Metadata`` @@ -191,7 +191,7 @@ The log is a CSV, with several fields. - Worker shutdown - ``"Server shutdown"`` -Log naming +Log-Naming --------------------------- Log file name syntax @@ -210,13 +210,13 @@ For example, ``/home/rhendricks/sqream_storage/192.168.1.91_5000``. See the :ref:`log_rotation` below for information about controlling this setting. -Logging control and maintenance +Log Control and Maintenance ====================================== -Change log verbosity +Changing Log Verbosity -------------------------- -A few configuration settings alter the verbosity of the logs +A few configuration settings alter the verbosity of the logs: .. list-table:: Log verbosity configuration :widths: auto @@ -239,10 +239,10 @@ A few configuration settings alter the verbosity of the logs .. _log_rotation: -Change log rotation +Changing Log Rotation ----------------------- -A few configuration settings alter the log rotation policy +A few configuration settings alter the log rotation policy: .. list-table:: Log rotation configuration :widths: auto @@ -267,10 +267,10 @@ A few configuration settings alter the log rotation policy .. _collecting_logs2: -Collect logs from your cluster +Collecting Logs from Your Cluster ==================================== -Collecting logs from your cluster can be as simple as creating an archive from the ``logs`` subdirectory: ``tar -czvf logs.tgz *.log`` +Collecting logs from your cluster can be as simple as creating an archive from the ``logs`` subdirectory: ``tar -czvf logs.tgz *.log``. However, SQream DB comes bundled with a data collection utility and an SQL utility intended for collecting logs and additional information that can help SQream support drill down into possible issues. @@ -289,10 +289,10 @@ SQL Syntax log | db | db_and_log -Command line utility +Command Line Utility -------------------------- -If SQream DB can't be accessed for any reason, a command line tool can also be used to collect the same information: +If you cannot access SQream DB for any reason, you can also use a command line toolto collect the same information: .. code-block:: console @@ -342,10 +342,10 @@ Using the command line utility: $ ./bin/report_collection /home/rhendricks/sqream_storage /home/rhendricks db_and_log -Troubleshooting with logs +Troubleshooting with Logs =============================== -Loading logs with foreign tables +Loading Logs with Foreign Tables --------------------------------------- Assuming logs are stored at ``/home/rhendricks/sqream_storage/logs/``, a database administrator can access the logs using the :ref:`external_tables` concept through SQream DB. @@ -374,11 +374,17 @@ Assuming logs are stored at ``/home/rhendricks/sqream_storage/logs/``, a databas OPTIONS ( LOCATION = '/home/rhendricks/sqream_storage/logs/**/sqream*.log', - DELIMITER = '|' + DELIMITER = '|', + CONTINUE_ON_ERROR = true ) ; + +For more information, see `Loading Logs with Foreign Tables `_. + + + -Count message types +Counting Message Types ------------------------------ .. code-block:: psql @@ -404,7 +410,7 @@ Count message types 1004 | 19 1010 | 5 -Find fatal errors +Finding Fatal Errors ---------------------- .. code-block:: psql @@ -416,7 +422,7 @@ Find fatal errors Mismatch in storage version, upgrade is needed,Storage version: 25, Server version is: 26 Internal Runtime Error,open cluster metadata database:IO error: lock /home/rhendricks/sqream_storage/LOCK: Resource temporarily unavailable -Count error events within a certain timeframe +Countng Error Events Within a Certain Timeframe --------------------------------------------------- .. code-block:: psql @@ -435,7 +441,7 @@ Count error events within a certain timeframe .. _tracing_errors: -Tracing errors to find offending statements +Tracing Errors to Find Offending Statements ------------------------------------------------- If we know an error occured, but don't know which statement caused it, we can find it using the connection ID and statement ID. @@ -464,6 +470,3 @@ Use the ``connection_id`` and ``statement_id`` to narrow down the results. .. how logs are read with csvkit, find a better working solution - - - diff --git a/guides/operations/monitoring_query_performance.rst b/operational_guides/monitoring_query_performance.rst similarity index 97% rename from guides/operations/monitoring_query_performance.rst rename to operational_guides/monitoring_query_performance.rst index 53815b78b..a542f61e6 100644 --- a/guides/operations/monitoring_query_performance.rst +++ b/operational_guides/monitoring_query_performance.rst @@ -1,34 +1,25 @@ .. _monitoring_query_performance: ********************************* -Monitoring query performance +Monitoring Query Performance ********************************* - -When analyzing options for query tuning, the first step is to analyze the query plan and execution. - +When analyzing options for query tuning, the first step is to analyze the query plan and execution. The query plan and execution details explains how SQream DB processes a query and where time is spent. - This document details how to analyze query performance with execution plans. - This guide focuses specifically on identifying bottlenecks and possible optimization techniques to improve query performance. - Performance tuning options for each query are different. You should adapt the recommendations and tips for your own workloads. - See also our :ref:`sql_best_practices` guide for more information about data loading considerations and other best practices. .. contents:: In this section: :local: -Setting up the system for monitoring +Setting Up the System for Monitoring ================================================= - By default, SQream DB logs execution details for every statement that runs for more than 60 seconds. If you want to see the execution details for a currently running statement, see :ref:`using_show_node_info` below. - -Adjusting the logging frequency +Adjusting the Logging Frequency --------------------------------------- - To adjust the frequency of logging for statements, you may want to reduce the interval from 60 seconds down to, say, 5 or 10 seconds. Modify the configuration files and set the ``nodeInfoLoggingSec`` parameter as you see fit: @@ -46,18 +37,14 @@ say, 5 or 10 seconds. Modify the configuration files and set the ``nodeInfoLoggi "server":{ } } - After restarting the SQream DB cluster, the execution plan details will be logged to the :ref:`standard SQream DB logs directory`, as a message of type ``200``. - You can see these messages with a text viewer or with queries on the log :ref:`external_tables`. -Reading execution plans with a foreign table +Reading Execution Plans with a Foreign Table ----------------------------------------------------- - First, create a foreign table for the logs .. code-block:: postgres - CREATE FOREIGN TABLE logs ( start_marker VARCHAR(4), @@ -83,12 +70,10 @@ First, create a foreign table for the logs DELIMITER = '|' ) ; - Once you've defined the foreign table, you can run queries to observe the previously logged execution plans. This is recommended over looking at the raw logs. .. code-block:: psql - t=> SELECT message . FROM logs . WHERE message_type_id = 200 @@ -117,17 +102,13 @@ This is recommended over looking at the raw logs. .. _using_show_node_info: -The ``SHOW_NODE_INFO`` command +Using the ``SHOW_NODE_INFO`` Command ===================================== - The :ref:`show_node_info` command returns a snapshot of the current query plan, similar to ``EXPLAIN ANALYZE`` from other databases. - The :ref:`show_node_info` result, just like the periodically-logged execution plans described above, are an at-the-moment view of the compiler's execution plan and runtime statistics for the specified statement. - To inspect a currently running statement, execute the ``show_node_info`` utility function in a SQL client like :ref:`sqream sql`, the :ref:`SQream Studio Editor`, or any other :ref:`third party SQL terminal`. - In this example, we inspect a statement with statement ID of 176. The command looks like this: .. code-block:: psql @@ -147,17 +128,13 @@ In this example, we inspect a statement with statement ID of 176. The command lo 176 | 10 | CpuDecompress | 457 | 1 | 457 | 2019-12-25 23:53:13 | 9 | | | | 0 176 | 11 | ReadTable | 457 | 1 | 457 | 2019-12-25 23:53:13 | 10 | 4MB | | public.nba | 0.0004 -Understanding the query execution plan output +Understanding the Query Execution Plan Output ================================================== - Both :ref:`show_node_info` and the logged execution plans represents the query plan as a graph hierarchy, with data separated into different columns. - Each row represents a single logical database operation, which is also called a **node** or **chunk producer**. A node reports several metrics during query execution, such as how much data it has read and written, how many chunks and rows, and how much time has elapsed. - Consider the example show_node_info presented above. The source node with ID #11 (``ReadTable``), has a parent node ID #10 (``CpuDecompress``). If we were to draw this out in a graph, it'd look like this: - .. figure:: /_static/images/show_node_info_graph.png :height: 70em :align: center @@ -175,8 +152,6 @@ The last node, also called the sink, has a parent node ID of -1, meaning it has rankdir=tb; ranksep=0.95; node[shape=box3d, width=3.0, height=0.6, fontname="Consolas", fillcolor=SteelBlue2, style=filled]; - - PushToNetworkQueue [shape=house, fillcolor=SeaGreen1, style=filled]; ReadTable->CpuDecompress; @@ -195,7 +170,6 @@ The last node, also called the sink, has a parent node ID of -1, meaning it has DeferredGather->ReorderInput4; ReorderInput4->Rechunk2; Rechunk2->PushToNetworkQueue; - Rechunk2[label="Rechunk"]; ReorderInput4[label="ReorderInput"]; ReorderInput3[label="ReorderInput"]; @@ -204,24 +178,19 @@ The last node, also called the sink, has a parent node ID of -1, meaning it has GpuTransform3[label="GpuTransform"]; ReadTable [shape=house, style=filled, fillcolor=SeaGreen4]; - } - - When using :ref:`show_node_info`, a tabular representation of the currently running statement execution is presented. - See the examples below to understand how the query execution plan is instrumental in identifying bottlenecks and optimizing long-running statements. -Information presented in the execution plan +Information Presented in the Execution Plan ---------------------------------------------------- .. include:: /reference/sql/sql_statements/monitoring_commands/show_node_info.rst :start-line: 47 :end-line: 78 -Commonly seen nodes +Commonly Seen Nodes ---------------------- - .. list-table:: Node types :widths: auto :header-rows: 1 @@ -325,34 +294,26 @@ Commonly seen nodes .. tip:: The full list of nodes appears in the :ref:`Node types table`, as part of the :ref:`show_node_info` reference. - Examples ================== - In general, looking at the top three longest running nodes (as is detailed in the ``timeSum`` column) can indicate the biggest bottlenecks. - In the following examples you will learn how to identify and solve some common issues. .. contents:: In this section: :local: -1. Spooling to disk +1. Spooling to Disk ----------------------- - When there is not enough RAM to process a statement, SQream DB will spill over data to the ``temp`` folder in the storage disk. While this ensures that a statement can always finish processing, it can slow down the processing significantly. - It's worth identifying these statements, to figure out if the cluster is configured correctly, as well as potentially reduce the statement size. - You can identify a statement that spools to disk by looking at the ``write`` column in the execution details. A node that spools will have a value, shown in megabytes in the ``write`` column. - Common nodes that write spools include ``Join`` or ``LoopJoin``. -Identifying the offending nodes +Identifying the Offending Nodes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - #. Run a query. @@ -376,7 +337,6 @@ Identifying the offending nodes WHERE o_orderdate BETWEEN '1995-01-01' AND '1996-12-31') AS all_nations GROUP BY o_year ORDER BY o_year; - #. Observe the execution information by using the foreign table, or use ``show_node_info`` @@ -429,34 +389,27 @@ Identifying the offending nodes : 150,LoopJoin ,182369485,10,18236948,2020-09-04 18:31:47,149,12860MB,12860MB,inner,23.62 [...] : 199,ReadTable ,20000000,1,20000000,2020-09-04 18:30:33,198,0MB,,public.part,0.83 - Because of the relatively low amount of RAM in the machine and because the data set is rather large at around 10TB, SQream DB needs to spool. The total spool used by this query is around 20GB (1915MB + 2191MB + 3064MB + 12860MB). -Common solutions for reducing spool +Common Solutions for Reducing Spool ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * Increase the amount of spool memory available for the workers, as a proportion of the maximum statement memory. When the amount of spool memory is increased, SQream DB may not need to write to disk. This setting is called ``spoolMemoryGB``. Refer to the :ref:`configuration` guide. - * Reduce the amount of **workers** per host, and increase the amount of spool available to the (now reduced amount of) active workers. This may reduce the amount of concurrent statements, but will improve performance for heavy statements. - -2. Queries with large result sets +2. Queries with Large Result Sets ------------------------------------ - When queries have large result sets, you may see a node called ``DeferredGather``. - This gathering occurs when the result set is assembled, in preparation for sending it to the client. -Identifying the offending nodes +Identifying the Offending Nodes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - #. Run a query. @@ -483,7 +436,6 @@ Identifying the offending nodes WHERE r_name = 'AMERICA' AND o_orderdate BETWEEN '1995-01-01' AND '1996-12-31' AND high_selectivity(p_type = 'ECONOMY BURNISHED NICKEL'); - #. Observe the execution information by using the foreign table, or use ``show_node_info`` @@ -508,12 +460,11 @@ Identifying the offending nodes 494 | 194 | DeferredGather | 133241 | 20 | 6662 | 2020-09-04 19:07:03 | 193 | | | | 0.41 [...] 494 | 221 | ReadTable | 20000000 | 20 | 1000000 | 2020-09-04 19:07:01 | 220 | 20MB | | public.part | 0.1 - + When you see ``DeferredGather`` operations taking more than a few seconds, that's a sign that you're selecting too much data. In this case, the DeferredGather with node ID 166 took over 21 seconds. #. Modify the statement to see the difference - Altering the select clause to be more restrictive will reduce the deferred gather time back to a few milliseconds. .. code-block:: postgres @@ -523,38 +474,29 @@ Identifying the offending nodes n2.n_name as nation FROM ... -Common solutions for reducing gather time +Common Solutions for Reducing Gather Time ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * Reduce the effect of the preparation time. Avoid selecting unnecessary columns (``SELECT * FROM...``), or reduce the result set size by using more filters. - .. `` - -3. Inefficient filtering +3. Inefficient Filtering -------------------------------- - When running statements, SQream DB tries to avoid reading data that is not needed for the statement by :ref:`skipping chunks`. - If statements do not include efficient filtering, SQream DB will read a lot of data off disk. In some cases, you need the data and there's nothing to do about it. However, if most of it gets pruned further down the line, it may be efficient to skip reading the data altogether by using the :ref:`metadata`. -Identifying the situation +Identifying the Situation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - We consider the filtering to be inefficient when the ``Filter`` node shows that the number of rows processed is less than a third of the rows passed into it by the ``ReadTable`` node. - For example: - #. Run a query. In this example, we execute a modified query from the TPC-H benchmark. Our ``lineitem`` table contains 600,037,902 rows. - .. code-block:: postgres SELECT o_year, @@ -576,7 +518,6 @@ For example: AND high_selectivity(p_type = 'ECONOMY BURNISHED NICKEL')) AS all_nations GROUP BY o_year ORDER BY o_year; - #. Observe the execution information by using the foreign table, or use ``show_node_info`` @@ -614,7 +555,6 @@ For example: 559 | 214 | Rechunk | 20000000 | 20 | 1000000 | 2020-09-07 11:11:57 | 213 | | | | 0 559 | 215 | CpuDecompress | 20000000 | 20 | 1000000 | 2020-09-07 11:11:57 | 214 | | | | 0 559 | 216 | ReadTable | 20000000 | 20 | 1000000 | 2020-09-07 11:11:57 | 215 | 20MB | | public.part | 0 - * The ``Filter`` on line 9 has processed 12,007,447 rows, but the output of ``ReadTable`` on ``public.lineitem`` @@ -627,7 +567,6 @@ For example: of the data, but the entire table was read. However, this table is small enough that we can ignore it. #. Modify the statement to see the difference - Altering the statement to have a ``WHERE`` condition on the clustered ``l_orderkey`` column of the ``lineitem`` table will help SQream DB skip reading the data. .. code-block:: postgres @@ -671,37 +610,27 @@ For example: 586 | 197 | CpuDecompress | 494927872 | 8 | 61865984 | 2020-09-07 13:20:44 | 196 | | | | 0 586 | 198 | ReadTable | 494927872 | 8 | 61865984 | 2020-09-07 13:20:44 | 197 | 6595MB | | public.lineitem | 0.09 [...] - - In this example, the filter processed 494,621,593 rows, while the output of ``ReadTable`` on ``public.lineitem`` was 494,927,872 rows. This means that it has filtered out all but 0.01% (:math:`1 - \dfrac{494621593}{494927872} = 0.01\%`) of the data that was read. The metadata skipping has performed very well, and has pre-filtered the data for us by pruning unnecessary chunks. -Common solutions for improving filtering +Common Solutions for Improving Filtering ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * Use :ref:`clustering keys and naturally ordered data` in your filters. - * Avoid full table scans when possible - -4. Joins with ``varchar`` keys +4. Joins with ``varchar`` Keys ----------------------------------- - Joins on long text keys, such as ``varchar(100)`` do not perform as well as numeric data types or very short text keys. - -Identifying the situation +Identifying the Situation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - When a join is inefficient, you may note that a query spends a lot of time on the ``Join`` node. - For example, consider these two table structures: .. code-block:: postgres - CREATE TABLE t_a ( amt FLOAT NOT NULL, @@ -711,14 +640,12 @@ For example, consider these two table structures: flag VARCHAR(10) NOT NULL, fk VARCHAR(50) NOT NULL ); - CREATE TABLE t_b ( id VARCHAR(50) NOT NULL prob FLOAT NOT NULL, j INT NOT NULL, ); - #. Run a query. @@ -731,7 +658,6 @@ For example, consider these two table structures: FROM t_a JOIN t_b ON (t_a.fk = t_b.id) GROUP BY t_a.country_code - #. Observe the execution information by using the foreign table, or use ``show_node_info`` @@ -760,12 +686,9 @@ For example, consider these two table structures: 5 | 41 | CpuDecompress | 10000000 | 2 | 5000000 | 2020-09-08 18:26:09 | 40 | | | | 0 5 | 42 | ReadTable | 10000000 | 2 | 5000000 | 2020-09-08 18:26:09 | 41 | 14MB | | public.t_a | 0 - -Improving query performance +Improving Query Performance ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * In general, try to avoid ``VARCHAR`` as a join key. As a rule of thumb, ``BIGINT`` works best as a join key. - * Convert text values on-the-fly before running the query. For example, the :ref:`crc64` function takes a text input and returns a ``BIGINT`` hash. @@ -779,7 +702,6 @@ Improving query performance FROM t_a JOIN t_b ON (crc64_join(t_a.fk) = crc64_join(t_b.id)) GROUP BY t_a.country_code - The execution below has been shortened, but note the highlighted rows for ``Join``. The ``Join`` node went from taking nearly 70 seconds, to just 6.67 seconds for joining 1.5 billion records. @@ -804,23 +726,16 @@ Improving query performance * You can map some text values to numeric types by using a dimension table. Then, reconcile the values when you need them by joining the dimension table. - 5. Sorting on big ``VARCHAR`` fields --------------------------------------- - In general, SQream DB automatically inserts a ``Sort`` node which arranges the data prior to reductions and aggregations. - When running a ``GROUP BY`` on large ``VARCHAR`` fields, you may see nodes for ``Sort`` and ``Reduce`` taking a long time. - -Identifying the situation +Identifying the Situation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - When running a statement, inspect it with :ref:`show_node_info`. If you see ``Sort`` and ``Reduce`` among your top five longest running nodes, there is a potential issue. - For example: - #. Run a query to test it out. @@ -885,14 +800,11 @@ For example: max --- 3 - With a maximum string length of just 3 characters, our ``VARCHAR(100)`` is way oversized. - #. We can recreate the table with a more restrictive ``VARCHAR(3)``, and can examine the difference in performance: .. code-block:: psql - t=> CREATE TABLE t_efficient . AS SELECT i, . amt, @@ -918,36 +830,28 @@ For example: This time, the entire query took just 4.75 seconds, or just about 91% faster. -Improving sort performance on text keys +Improving Sort Performance on Text Keys ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - When using VARCHAR, ensure that the maximum length defined in the table structure is as small as necessary. For example, if you're storing phone numbers, don't define the field as ``VARCHAR(255)``, as that affects sort performance. You can run a query to get the maximum column length (e.g. ``MAX(LEN(a_column))``), and potentially modify the table structure. - .. _high_selectivity_data_opt: -6. High selectivity data +6. High Selectivity Data -------------------------- - Selectivity is the ratio of cardinality to the number of records of a chunk. We define selectivity as :math:`\frac{\text{Distinct values}}{\text{Total number of records in a chunk}}` - SQream DB has a hint called ``HIGH_SELECTIVITY``, which is a function you can wrap a condition in. - The hint signals to SQream DB that the result of the condition will be very sparse, and that it should attempt to rechunk the results into fewer, fuller chunks. - .. note:: SQream DB doesn't do this automatically because it adds a significant overhead on naturally ordered and well-clustered data, which is the more common scenario. -Identifying the situation +Identifying the Situation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - This is easily identifiable - when the amount of average of rows in a chunk is small, following a ``Filter`` operation. - Consider this execution plan: .. code-block:: psql @@ -959,34 +863,25 @@ Consider this execution plan: 30 | 38 | Filter | 18160 | 74 | 245 | 2020-09-10 12:17:09 | 37 | | | | 0.012 [...] 30 | 44 | ReadTable | 77000000 | 74 | 1040540 | 2020-09-10 12:17:09 | 43 | 277MB | | public.dim | 0.058 - - The table was read entirely - 77 million rows into 74 chunks. - The filter node reduced the output to just 18,160 relevant rows, but they're distributed across the original 74 chunks. All of these rows could fit in one single chunk, instead of spanning 74 rather sparse chunks. -Improving performance with high selectivity hints +Improving Performance with High Selectivity Hints ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * Use when there's a ``WHERE`` condition on an :ref:`unclustered column`, and when you expect the filter to cut out more than 60% of the result set. - * Use when the data is uniformly distributed or random - 7. Performance of unsorted data in joins ------------------------------------------ - When data is not well-clustered or naturally ordered, a join operation can take a long time. -Identifying the situation +Identifying the Situation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - When running a statement, inspect it with :ref:`show_node_info`. If you see ``Join`` and ``DeferredGather`` among your top five longest running nodes, there is a potential issue. - In this case, we're also interested in the number of chunks produced by these nodes. Consider this execution plan: @@ -1009,23 +904,18 @@ Consider this execution plan: 30 | 38 | Filter | 18160 | 74 | 245 | 2020-09-10 12:17:09 | 37 | | | | 0.012 [...] 30 | 44 | ReadTable | 77000000 | 74 | 1040540 | 2020-09-10 12:17:09 | 43 | 277MB | | public.dim | 0.058 - * ``Join`` is the node that matches rows from both table relations. * ``DeferredGather`` gathers the required column chunks to decompress - Pay special attention to the volume of data removed by the ``Filter`` node. The table was read entirely - 77 million rows into 74 chunks. - The filter node reduced the output to just 18,160 relevant rows, but they're distributed across the original 74 chunks. All of these rows could fit in one single chunk, instead of spanning 74 rather sparse chunks. -Improving join performance when data is sparse +Improving Join Performance when Data is Sparse ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - You can tell SQream DB to reduce the amount of chunks involved, if you know that the filter is going to be quite agressive by using the :ref:`HIGH_SELECTIVITY` hint described :ref:`above`. This forces the compiler to rechunk the data into fewer chunks. - To tell SQream DB to rechunk the data, wrap a condition (or several) in the ``HIGH_SELECTIVITY`` hint: .. code-block:: postgres @@ -1047,26 +937,20 @@ To tell SQream DB to rechunk the data, wrap a condition (or several) in the ``HI AND EnterpriseID=1150 AND MSISDN='9724871140341'; - -8. Manual join reordering +8. Manual Join Reordering -------------------------------- - When joining multiple tables, you may wish to change the join order to join the smallest tables first. Identifying the situation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - When joining more than two tables, the ``Join`` nodes will be the most time-consuming nodes. -Changing the join order +Changing the Join Order ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Always prefer to join the smallest tables first. - .. note:: We consider small tables to be tables that only retain a small amount of rows after conditions are applied. This bears no direct relation to the amount of total rows in the table. - Changing the join order can reduce the query runtime significantly. In the examples below, we reduce the time from 27.3 seconds to just 6.4 seconds. @@ -1077,7 +961,6 @@ from 27.3 seconds to just 6.4 seconds. SELECT SUM(l_extendedprice / 100.0*(1 - l_discount / 100.0)) AS revenue, c_nationkey FROM lineitem --6B Rows, ~183GB - JOIN orders --1.5B Rows, ~55GB ON l_orderkey = o_orderkey JOIN customer --150M Rows, ~12GB @@ -1097,7 +980,6 @@ from 27.3 seconds to just 6.4 seconds. SELECT SUM(l_extendedprice / 100.0*(1 - l_discount / 100.0)) AS revenue, c_nationkey FROM orders --1.5B Rows, ~55GB - JOIN customer --150M Rows, ~12GB ON c_custkey = o_custkey JOIN lineitem --6B Rows, ~183GB @@ -1110,9 +992,6 @@ from 27.3 seconds to just 6.4 seconds. AND l_shipdate <= dateadd(DAY,122,'1994-01-01') GROUP BY c_nationkey - - -Further reading +Further Reading ================== - See our :ref:`sql_best_practices` guide for more information about query optimization and data loading considerations. \ No newline at end of file diff --git a/guides/operations/optimization_best_practices.rst b/operational_guides/optimization_best_practices.rst similarity index 99% rename from guides/operations/optimization_best_practices.rst rename to operational_guides/optimization_best_practices.rst index e1abecd21..fe36269c6 100644 --- a/guides/operations/optimization_best_practices.rst +++ b/operational_guides/optimization_best_practices.rst @@ -1,7 +1,7 @@ .. _sql_best_practices: ********************************** -Optimization and best practices +Optimization and Best Practices ********************************** This topic explains some best practices of working with SQream DB. @@ -78,7 +78,6 @@ While it won't make a big difference in storage, large strings allocate a lot of If a column's string length never exceeds 50 characters, specify ``VARCHAR(50)`` rather than an arbitrarily large number. - Sorting ============== diff --git a/guides/features/saved_queries.rst b/operational_guides/saved_queries.rst similarity index 98% rename from guides/features/saved_queries.rst rename to operational_guides/saved_queries.rst index 23f47647b..d554b4dc8 100644 --- a/guides/features/saved_queries.rst +++ b/operational_guides/saved_queries.rst @@ -1,10 +1,10 @@ .. _saved_queries: *********************** -Saved queries +Saved Queries *********************** -Saved queries can be used to reuse a query plan for a query to eliminate compilation times for repeated queries. They also provide a way to implement 'parameterized views'. +Saved queries can be used to reuse a query plan for a query to eliminate compilation times for repeated queries. They also provide a way to implement 'parameterized views'. How saved queries work ========================== diff --git a/guides/operations/security.rst b/operational_guides/security.rst similarity index 94% rename from guides/operations/security.rst rename to operational_guides/security.rst index 7f50a3f81..598c40196 100644 --- a/guides/operations/security.rst +++ b/operational_guides/security.rst @@ -47,8 +47,6 @@ After creating a second ``SUPERUSER`` role, remove or change the default credent No database user should ever use the default ``SUPERUSER`` role in a production environment. -If you don't change the user role itself, change the password of the default ``SUPERUSER``. See the :ref:`change password` section of our :ref:`access control` guide. - Create distinct user roles -------------------------------- @@ -80,11 +78,6 @@ Follow these recommendations to strengthen passwords: * Mix letters and numbers * Include non-alphanumeric characters (except ``"`` and ``'``) - - - - - Use TLS/SSL when possible ---------------------------- diff --git a/guides/features/viewing_system_objects_as_ddl.rst b/operational_guides/seeing_system_objects_as_ddl.rst similarity index 99% rename from guides/features/viewing_system_objects_as_ddl.rst rename to operational_guides/seeing_system_objects_as_ddl.rst index ce1fc59b3..fc8ff3b83 100644 --- a/guides/features/viewing_system_objects_as_ddl.rst +++ b/operational_guides/seeing_system_objects_as_ddl.rst @@ -1,4 +1,4 @@ -.. _seeing_system_objects_as_sql: +.. _seeing_system_objects_as_ddl: ******************************** Seeing system objects as DDL diff --git a/reference/catalog_reference.rst b/reference/catalog_reference.rst index 8cfa8e832..8fc0593b8 100644 --- a/reference/catalog_reference.rst +++ b/reference/catalog_reference.rst @@ -1,606 +1,16 @@ .. _catalog_reference: ************************************* -Catalog reference +Catalog Reference Guide ************************************* +The **Catalog Reference Guide** describes the following: -SQream DB contains a schema called ``sqream_catalog`` that contains information about your database's objects - tables, columns, views, permissions, and more. +.. toctree:: + :maxdepth: 1 + :glob: -Some additional catalog tables are used primarily for internal introspection, which could change across SQream DB versions. - - -.. contents:: In this topic: - :local: - -Types of data exposed by ``sqream_catalog`` -============================================== - -.. list-table:: Database objects - :widths: auto - :header-rows: 1 - - * - Object - - Table - * - Clustering keys - - ``clustering_keys`` - * - Columns - - ``columns``, ``external_table_columns`` - * - Databases - - ``databases`` - * - Permissions - - ``table_permissions``, ``database_permissions``, ``schema_permissions``, ``permission_types``, ``udf_permissions`` - * - Roles - - ``roles``, ``roles_memeberships`` - * - Schemas - - ``schemas`` - * - Sequences - - ``identity_key`` - * - Tables - - ``tables``, ``external_tables`` - * - Views - - ``views`` - * - UDFs - - ``user_defined_functions`` - -The catalog contains a few more tables which contain storage details for internal use - -.. list-table:: Storage objects - :widths: auto - :header-rows: 1 - - * - Object - - Table - * - Extents - - ``extents`` - * - Chunks - - ``chunks`` - * - Delete predicates - - ``delete_predicates`` - -Tables in the catalog -======================== - -clustering_keys ------------------------ - -Explicit clustering keys for tables. - -When more than one clustering key is defined, each key is listed in a separate row. - - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the table - * - ``table_id`` - - ID of the table containing the column - * - ``schema_name`` - - Name of the schema containing the table - * - ``table_name`` - - Name of the table containing the column - * - ``clustering_key`` - - Name of the column that is a clustering key for this table - -columns --------- - -Column objects for standard tables - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the table - * - ``schema_name`` - - Name of the schema containing the table - * - ``table_id`` - - ID of the table containing the column - * - ``table_name`` - - Name of the table containing the column - * - ``column_id`` - - Ordinal of the column in the table (begins at 0) - * - ``column_name`` - - Name of the column - * - ``type_name`` - - :ref:`Data type ` of the column - * - ``column_size`` - - The maximum length in bytes. - * - ``has_default`` - - ``NULL`` if the column has no default value. ``1`` if the default is a fixed value, or ``2`` if the default is an :ref:`identity` - * - ``default_value`` - - :ref:`Default value` for the column - * - ``compression_strategy`` - - User-overridden compression strategy - * - ``created`` - - Timestamp when the column was created - * - ``altered`` - - Timestamp when the column was last altered - - -.. _external_tables_table: - -external_tables ----------------- - -``external_tables`` identifies external tables in the database. - -For ``TABLES`` see :ref:`tables ` - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the table - * - ``table_id`` - - Database-unique ID for the table - * - ``schema_name`` - - Name of the schema containing the table - * - ``table_name`` - - Name of the table - * - ``format`` - - - Identifies the foreign data wrapper used. - - ``0`` for csv_fdw, ``1`` for parquet_fdw, ``2`` for orc_fdw. - - * - ``created`` - - Identifies the clause used to create the table - -external_table_columns ------------------------- - -Column objects for external tables - -databases ------------ - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_Id`` - - Unique ID of the database - * - ``database_name`` - - Name of the database - * - ``default_disk_chunk_size`` - - Internal use - * - ``default_process_chunk_size`` - - Internal use - * - ``rechunk_size`` - - Internal use - * - ``storage_subchunk_size`` - - Internal use - * - ``compression_chunk_size_threshold`` - - Internal use - -database_permissions ----------------------- - -``database_permissions`` identifies all permissions granted to databases. - -There is one row for each combination of role (grantee) and permission granted to a database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database the permission applies to - * - ``role_id`` - - ID of the role granted permissions (grantee) - * - ``permission_type`` - - Identifies the permission type - - -identity_key --------------- - - -permission_types ------------------- - -``permission_types`` Identifies the permission names that exist in the database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``permission_type_id`` - - ID of the permission type - * - ``name`` - - Name of the permission type - -roles ------- - -``roles`` identifies the roles in the database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``role_id`` - - Database-unique ID of the role - * - ``name`` - - Name of the role - * - ``superuser`` - - Identifies if this role is a superuser. ``1`` for superuser or ``0`` otherwise. - * - ``login`` - - Identifies if this role can be used to log in to SQream DB. ``1`` for yes or ``0`` otherwise. - * - ``has_password`` - - Identifies if this role has a password. ``1`` for yes or ``0`` otherwise. - * - ``can_create_function`` - - Identifies if this role can create UDFs. ``1`` for yes, ``0`` otherwise. - -roles_memberships -------------------- - -``roles_memberships`` identifies the role memberships in the database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``role_id`` - - Role ID - * - ``member_role_id`` - - ID of the parent role from which this role will inherit - * - ``inherit`` - - Identifies if permissions are inherited. ``1`` for yes or ``0`` otherwise. - -savedqueries ----------------- - -``savedqueries`` identifies the :ref:`saved_queries` in the database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``name`` - - Saved query name - * - ``num_parameters`` - - Number of parameters to be replaced at run-time - -schemas ----------- - -``schemas`` identifies all the database's schemas. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``schema_id`` - - Unique ID of the schema - * - ``schema_name`` - - Name of the schema - * - ``schema_owner`` - - Name of the role who owns this schema - * - ``rechunker_ignore`` - - Internal use - - -schema_permissions --------------------- - -``schema_permissions`` identifies all permissions granted to schemas. - -There is one row for each combination of role (grantee) and permission granted to a schema. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the schema - * - ``schema_id`` - - ID of the schema the permission applies to - * - ``role_id`` - - ID of the role granted permissions (grantee) - * - ``permission_type`` - - Identifies the permission type - - -.. _tables_table: - -tables ----------- - -``tables`` identifies proper SQream tables in the database. - -For ``EXTERNAL TABLES`` see :ref:`external_tables ` - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the table - * - ``table_id`` - - Database-unique ID for the table - * - ``schema_name`` - - Name of the schema containing the table - * - ``table_name`` - - Name of the table - * - ``row_count_valid`` - - Identifies if the ``row_count`` can be used - * - ``row_count`` - - Number of rows in the table - * - ``rechunker_ignore`` - - Internal use - - -table_permissions ------------------- - -``table_permissions`` identifies all permissions granted to tables. - -There is one row for each combination of role (grantee) and permission granted to a table. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the table - * - ``table_id`` - - ID of the table the permission applies to - * - ``role_id`` - - ID of the role granted permissions (grantee) - * - ``permission_type`` - - Identifies the permission type - - -udf_permissions ------------------- - -user_defined_functions -------------------------- - -``user_defined_functions`` identifies UDFs in the database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the database containing the view - * - ``function_id`` - - Database-unique ID for the UDF - * - ``function_name`` - - Name of the UDF - -views -------- - -``views`` identifies views in the database. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``view_id`` - - Database-unique ID for the view - * - ``view_schema`` - - Name of the schema containing the view - * - ``view_name`` - - Name of the view - * - ``view_data`` - - Internal use - * - ``view_query_text`` - - Identifies the ``AS`` clause used to create the view - - -Additional tables -====================== - -There are additional tables in the catalog that can be used for performance monitoring and inspection. - -The definition for these tables is provided below could change across SQream DB versions. - -extents ----------- - -``extents`` identifies storage extents. - -Each storage extents can contain several chunks. - -.. note:: This is an internal table designed for low-level performance troubleshooting. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the databse containing the extent - * - ``table_id`` - - ID of the table containing the extent - * - ``column_id`` - - ID of the column containing the extent - * - ``extent_id`` - - ID for the extent - * - ``size`` - - Extent size in megabytes - * - ``path`` - - Full path to the extent on the file system - -chunk_columns -------------------- - -``chunk_columns`` lists chunk information by column. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the databse containing the extent - * - ``table_id`` - - ID of the table containing the extent - * - ``column_id`` - - ID of the column containing the extent - * - ``chunk_id`` - - ID for the chunk - * - ``extent_id`` - - ID for the extent - * - ``compressed_size`` - - Actual chunk size in bytes - * - ``uncompressed_size`` - - Uncompressed chunk size in bytes - * - ``compression_type`` - - Actual compression scheme for this chunk - * - ``long_min`` - - Minimum numeric value in this chunk (if exists) - * - ``long_max`` - - Maximum numeric value in this chunk (if exists) - * - ``string_min`` - - Minimum text value in this chunk (if exists) - * - ``string_max`` - - Maximum text value in this chunk (if exists) - * - ``offset_in_file`` - - Internal use - -.. note:: This is an internal table designed for low-level performance troubleshooting. - -chunks -------- - -``chunks`` identifies storage chunks. - -.. note:: This is an internal table designed for low-level performance troubleshooting. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the databse containing the chunk - * - ``table_id`` - - ID of the table containing the chunk - * - ``column_id`` - - ID of the column containing the chunk - * - ``rows_num`` - - Amount of rows contained in the chunk - * - ``deletion_status`` - - When data is deleted from the table, it is first deleted logically. This value identifies how much data is deleted from the chunk. ``0`` for no data, ``1`` for some data, ``2`` to specify the entire chunk is deleted. - -delete_predicates -------------------- - -``delete_predicates`` identifies the existing delete predicates that have not been cleaned up. - -Each :ref:`DELETE ` command may result in several entries in this table. - -.. note:: This is an internal table designed for low-level performance troubleshooting. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Column - - Description - * - ``database_name`` - - Name of the databse containing the predicate - * - ``table_id`` - - ID of the table containing the predicate - * - ``max_chunk_id`` - - Internal use. Placeholder marker for the highest ``chunk_id`` logged during the DELETE operation. - * - ``delete_predicate`` - - Identifies the DELETE predicate - - -Examples -=========== - -List all tables in the database ----------------------------------- - -.. code-block:: psql - - master=> SELECT * FROM sqream_catalog.tables; - database_name | table_id | schema_name | table_name | row_count_valid | row_count | rechunker_ignore - --------------+----------+-------------+----------------+-----------------+-----------+----------------- - master | 1 | public | nba | true | 457 | 0 - master | 12 | public | cool_dates | true | 5 | 0 - master | 13 | public | cool_numbers | true | 9 | 0 - master | 27 | public | jabberwocky | true | 8 | 0 - -List all schemas in the database ------------------------------------- - -.. code-block:: psql - - master=> SELECT * FROM sqream_catalog.schemas; - schema_id | schema_name | schema_owner | rechunker_ignore - ----------+---------------+--------------+----------------- - 0 | public | sqream | false - 1 | secret_schema | mjordan | false - - -List columns and their types for a specific table ---------------------------------------------------- - -.. code-block:: postgres - - SELECT column_name, type_name - FROM sqream_catalog.columns - WHERE table_name='cool_animals'; - -List delete predicates ------------------------- - -.. code-block:: postgres - - SELECT t.table_name, d.* FROM - sqream_catalog.delete_predicates AS d - INNER JOIN sqream_catalog.tables AS t - ON d.table_id=t.table_id; - - -List :ref:`saved_queries` ------------------------------ - -.. code-block:: postgres - - SELECT * FROM sqream_catalog.savedqueries; + catalog_reference_overview + catalog_reference_schema_information + catalog_reference_catalog_tables + catalog_reference_additonal_tables + catalog_reference_examples \ No newline at end of file diff --git a/reference/catalog_reference_additonal_tables.rst b/reference/catalog_reference_additonal_tables.rst new file mode 100644 index 000000000..7d34429d3 --- /dev/null +++ b/reference/catalog_reference_additonal_tables.rst @@ -0,0 +1,120 @@ +.. _catalog_reference_additonal_tables: + +************************************* +Additional Tables +************************************* +The Reference Catalog includes additional tables that can be used for performance monitoring and inspection. The definition for these tables described on this page may change across SQream versions. + +.. contents:: + :local: + :depth: 1 + +Extents +---------- +The ``extents`` storage object identifies storage extents, and each storage extents can contain several chunks. + +.. note:: This is an internal table designed for low-level performance troubleshooting. + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the databse containing the extent. + * - ``table_id`` + - Shows the ID of the table containing the extent. + * - ``column_id`` + - Shows the ID of the column containing the extent. + * - ``extent_id`` + - Shows the ID for the extent. + * - ``size`` + - Shows the extent size in megabytes. + * - ``path`` + - Shows the full path to the extent on the file system. + +Chunk Columns +------------------- +The ``chunk_columns`` storage object lists chunk information by column. + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the databse containing the extent. + * - ``table_id`` + - Shows the ID of the table containing the extent. + * - ``column_id`` + - Shows the ID of the column containing the extent. + * - ``chunk_id`` + - Shows the chunk ID. + * - ``extent_id`` + - Shows the extent ID. + * - ``compressed_size`` + - Shows the compressed chunk size in bytes. + * - ``uncompressed_size`` + - Shows the uncompressed chunk size in bytes. + * - ``compression_type`` + - Shows the chunk's actual compression scheme. + * - ``long_min`` + - Shows the minimum numeric value in the chunk (if one exists). + * - ``long_max`` + - Shows the maximum numeric value in the chunk (if one exists). + * - ``string_min`` + - Shows the minimum text value in the chunk (if one exists). + * - ``string_max`` + - Shows the maximum text value in the chunk (if one exists). + * - ``offset_in_file`` + - Reserved for internal use. + +.. note:: This is an internal table designed for low-level performance troubleshooting. + +Chunks +------- +The ``chunks`` storage object identifies storage chunks. + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the databse containing the chunk. + * - ``table_id`` + - Shows the ID of the table containing the chunk. + * - ``column_id`` + - Shows the ID of the column containing the chunk. + * - ``rows_num`` + - Shows the amount of rows in the chunk. + * - ``deletion_status`` + - Determines what data to logically delete from the table first, and identifies how much data to delete from the chunk. The value ``0`` is ued for no data, ``1`` for some data, and ``2`` to delete the entire chunk. + +.. note:: This is an internal table designed for low-level performance troubleshooting. + +Delete Predicates +------------------- +The ``delete_predicates`` storage object identifies the existing delete predicates that have not been cleaned up. + +Each :ref:`DELETE ` command may result in several entries in this table. + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the databse containing the predicate. + * - ``table_id`` + - Shows the ID of the table containing the predicate. + * - ``max_chunk_id`` + - Reserved for internal use, this is a placeholder marker for the highest ``chunk_id`` logged during the ``DELETE`` operation. + * - ``delete_predicate`` + - Identifies the DELETE predicate. + +.. note:: This is an internal table designed for low-level performance troubleshooting. \ No newline at end of file diff --git a/reference/catalog_reference_catalog_tables.rst b/reference/catalog_reference_catalog_tables.rst new file mode 100644 index 000000000..5a5b77fec --- /dev/null +++ b/reference/catalog_reference_catalog_tables.rst @@ -0,0 +1,447 @@ +.. _catalog_reference_catalog_tables: + +************************************* +Catalog Tables +************************************* +The ``sqream_catalog`` includes the following tables: + +.. contents:: + :local: + :depth: 1 + +.. _clustering_keys: + +Clustering Keys +---------------- +The ``clustering_keys`` data object is used for explicit clustering keys for tables. If you define more than one clustering key, each key is listed in a separate row, and is described in the following table: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the table. + * - ``table_id`` + - Shows the ID of the table containing the column. + * - ``schema_name`` + - Shows the name of the schema containing the table. + * - ``table_name`` + - Shows the name of the table containing the column. + * - ``clustering_key`` + - Shows the name of the column used as a clustering key for this table. + +.. _columns: + +Columns +---------------- +The **Columns** database object shows the following tables: + +.. contents:: + :local: + :depth: 1 + +Columns +*********** +The ``column`` data object is used with standard tables and is described in the following table: + +.. list-table:: + :widths: 20 150 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the table. + * - ``schema_name`` + - Shows the name of the schema containing the table. + * - ``table_id`` + - Shows the ID of the table containing the column. + * - ``table_name`` + - Shows the name of the table containing the column. + * - ``column_id`` + - Shows the ordinal number of the column in the table (begins at **0**). + * - ``column_name`` + - Shows the column's name. + * - ``type_name`` + - Shows the column's data type. For more information see :ref:`Supported Data Types `. + * - ``column_size`` + - Shows the maximum length in bytes. + * - ``has_default`` + - Shows ``NULL`` if the column has no default value, ``1`` if the default is a fixed value, or ``2`` if the default is an identity. For more information, see :ref:`identity`. + * - ``default_value`` + - Shows the column's default value. For more information, see :ref:`Default Value Constraints`. + * - ``compression_strategy`` + - Shows the compression strategy that a user has overridden. + * - ``created`` + - Shows the timestamp displaying when the column was created. + * - ``altered`` + - Shows the timestamp displaying when the column was last altered. + +External Table Columns +*********** +The ``external_table_columns`` is used for viewing data from foreign tables. + +For more information on foreign tables, see :ref:`CREATE FOREIGN TABLE`. + +.. _databases: + +Databases +---------------- +The ``databases`` data object is used for displaying database information, and is described in the following table: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_Id`` + - Shows the database's unique ID. + * - ``database_name`` + - Shows the database's name. + * - ``default_disk_chunk_size`` + - Reserved for internal use. + * - ``default_process_chunk_size`` + - Reserved for internal use. + * - ``rechunk_size`` + - Reserved for internal use. + * - ``storage_subchunk_size`` + - Reserved for internal use. + * - ``compression_chunk_size_threshold`` + - Reserved for internal use. + +.. _permissions: + +Permissions +---------------- +The ``permissions`` data object is used for displaying permissions information, such as roles (also known as **grantees**), and is described in the following tables: + +.. contents:: + :local: + :depth: 1 + +Permission Types +*********** +The ``permission_types`` object identifies the permission names existing in the database. + +The following table describes the ``permission_types`` data object: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``permission_type_id`` + - Shows the permission type's ID. + * - ``name`` + - Shows the name of the permission type. + +Default Permissions +*********** +The commands included in the **Default Permissions** section describe how to check the following default permissions: + +.. contents:: + :local: + :depth: 1 + +Default Table Permissions +~~~~~~~~~~~~~~~~ +The ``sqream_catalog.table_default_permissions`` command shows the columns described below: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the database that the default permission rule applies to. + * - ``schema_id`` + - Shows the schema that the rule applies to, or ``NULL`` if the ``ALTER`` statement does not specify a schema. + * - ``modifier_role_id`` + - Shows the role to apply the rule to. + * - ``getter_role_id`` + - Shows the role that the permission is granted to. + * - ``permission_type`` + - Shows the type of permission granted. + +Default Schema Permissions +~~~~~~~~~~~~~~~~ +The ``sqream_catalog.schema_default_permissions`` command shows the columns described below: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the database that the default permission rule applies to. + * - ``modifier_role_id`` + - Shows the role to apply the rule to. + * - ``getter_role_id`` + - Shows the role that the permission is granted to. + * - ``permission_type`` + - Shows the type of permission granted. + +For an example of using the ``sqream_catalog.table_default_permissions`` command, see `Granting Default Table Permissions `_. + +Table Permissions +*********** +The ``table_permissions`` data object identifies all permissions granted to tables. Each role-permission combination displays one row. + +The following table describes the ``table_permissions`` data object: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the table. + * - ``table_id`` + - Shows the ID of the table the permission applies to. + * - ``role_id`` + - Shows the ID of the role granted permissions. + * - ``permission_type`` + - Identifies the permission type. + +Database Permissions +*********** +The ``database_permissions`` data object identifies all permissions granted to databases. Each role-permission combination displays one row. + +The following table describes the ``database_permissions`` data object: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database the permission applies to + * - ``role_id`` + - Shows the ID of the role granted permissions. + * - ``permission_type`` + - Identifies the permission type. + +Schema Permissions +*********** +The ``schema_permissions`` data object identifies all permissions granted to schemas. Each role-permission combination displays one row. + +The following table describes the ``schema_permissions`` data object: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the schema. + * - ``schema_id`` + - Shows the ID of the schema the permission applies to. + * - ``role_id`` + - Shows the ID of the role granted permissions. + * - ``permission_type`` + - Identifies the permission type. + +.. _queries: + +Queries +---------------- +The ``savedqueries`` data object identifies the saved_queries in the database, as shown in the following table: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``name`` + - Shows the saved query name. + * - ``num_parameters`` + - Shows the number of parameters to be replaced at run-time. + +For more information, see :ref:`saved_queries`. + +.. _roles: + +Roles +---------------- +The ``roles`` data object is used for displaying role information, and is described in the following tables: + +.. contents:: + :local: + :depth: 1 + +Roles +*********** +The ``roles`` data object identifies the roles in the database, as shown in the following table: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``role_id`` + - Shows the role's database-unique ID. + * - ``name`` + - Shows the role's name. + * - ``superuser`` + - Identifies whether the role is a superuser (``1`` - superuser, ``0`` - regular user). + * - ``login`` + - Identifies whether the role can be used to log in to SQream (``1`` - yes, ``0`` - no). + * - ``has_password`` + - Identifies whether the role has a password (``1`` - yes, ``0`` - no). + * - ``can_create_function`` + - Identifies whether role can create UDFs (``1`` - yes, ``0`` - no). + +Role Memberships +*********** +The ``roles_memberships`` data object identifies the role memberships in the database, as shown below: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``role_id`` + - Shows the role ID. + * - ``member_role_id`` + - Shows the ID of the parent role that this role inherits from. + * - ``inherit`` + - Identifies whether permissions are inherited (``1`` - yes, ``0`` - no). + +.. _schemas: + +Schemas +---------------- +The ``schemas`` data object identifies all the database's schemas, as shown below: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``schema_id`` + - Shows the schema's unique ID. + * - ``schema_name`` + - Shows the schema's name. + * - ``schema_owner`` + - Shows the name of the role that owns the schema. + * - ``rechunker_ignore`` + - Reserved for internal use. + +.. _sequences: + +Sequences +---------------- +The ``sequences`` data object is used for displaying identity key information. + +.. _tables: + +Tables +---------------- +The ``tables`` data object is used for displaying table information, and is described in the following tables: + +.. contents:: + :local: + :depth: 1 + +Tables +*********** +The ``tables`` data object identifies proper (**Comment** - *What does "proper" mean?*) SQream tables in the database, as shown in the following table: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the table. + * - ``table_id`` + - Shows the table's database-unique ID. + * - ``schema_name`` + - Shows the name of the schema containing the table. + * - ``table_name`` + - Shows the name of the table. + * - ``row_count_valid`` + - Identifies whether the ``row_count`` can be used. + * - ``row_count`` + - Shows the number of rows in the table. + * - ``rechunker_ignore`` + - Relevant for internal use. + +Foreign Tables +*********** +The ``external_tables`` data object identifies foreign tables in the database, as shown below: + +.. list-table:: + :widths: 20 200 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the table. + * - ``table_id`` + - Shows the table's database-unique ID. + * - ``schema_name`` + - Shows the name of the schema containing the table. + * - ``table_name`` + - Shows the name of the table. + * - ``format`` + - Identifies the foreign data wrapper used. ``0`` for ``csv_fdw``, ``1`` for ``parquet_fdw``, ``2`` for ``orc_fdw``. + * - ``created`` + - Identifies the clause used to create the table. + +.. _views: + +Views +---------------- +The ``views`` data object is used for displaying views in the database, as shown below: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``view_id`` + - Shows the view's database-unique ID. + * - ``view_schema`` + - Shows the name of the schema containing the view. + * - ``view_name`` + - Shows the name of the view. + * - ``view_data`` + - Reserved for internal use. + * - ``view_query_text`` + - Identifies the ``AS`` clause used to create the view. + +.. _udfs: + +User Defined Functions +---------------- +The ``udf`` data object is used for displaying UDFs in the database, as shown below: + +.. list-table:: + :widths: 20 180 + :header-rows: 1 + + * - Column + - Description + * - ``database_name`` + - Shows the name of the database containing the view. + * - ``function_id`` + - Shows the UDF's database-unique ID. + * - ``function_name`` + - Shows the name of the UDF. \ No newline at end of file diff --git a/reference/catalog_reference_examples.rst b/reference/catalog_reference_examples.rst new file mode 100644 index 000000000..4531dfefc --- /dev/null +++ b/reference/catalog_reference_examples.rst @@ -0,0 +1,64 @@ +.. _catalog_reference_examples: + +************************************* +Examples +************************************* +The **Examples** page includes the following examples: + +.. contents:: + :local: + :depth: 1 + +Listing All Tables in a Database +---------------------------------- + +.. code-block:: psql + + master=> SELECT * FROM sqream_catalog.tables; + database_name | table_id | schema_name | table_name | row_count_valid | row_count | rechunker_ignore + --------------+----------+-------------+----------------+-----------------+-----------+----------------- + master | 1 | public | nba | true | 457 | 0 + master | 12 | public | cool_dates | true | 5 | 0 + master | 13 | public | cool_numbers | true | 9 | 0 + master | 27 | public | jabberwocky | true | 8 | 0 + +Listing All Schemas in a Database +------------------------------------ + +.. code-block:: psql + + master=> SELECT * FROM sqream_catalog.schemas; + schema_id | schema_name | schema_owner | rechunker_ignore + ----------+---------------+--------------+----------------- + 0 | public | sqream | false + 1 | secret_schema | mjordan | false + + +Listing Columns and Their Types for a Specific Table +--------------------------------------------------- + +.. code-block:: postgres + + SELECT column_name, type_name + FROM sqream_catalog.columns + WHERE table_name='cool_animals'; + +Listing Delete Predicates +------------------------ + +.. code-block:: postgres + + SELECT t.table_name, d.* FROM + sqream_catalog.delete_predicates AS d + INNER JOIN sqream_catalog.tables AS t + ON d.table_id=t.table_id; + + +Listing Saved Queries +----------------------------- + +.. code-block:: postgres + + SELECT * FROM sqream_catalog.savedqueries; + +For more information, see :ref:`saved_queries`. \ No newline at end of file diff --git a/reference/catalog_reference_overview.rst b/reference/catalog_reference_overview.rst new file mode 100644 index 000000000..b74663509 --- /dev/null +++ b/reference/catalog_reference_overview.rst @@ -0,0 +1,11 @@ +.. _catalog_reference_overview: + +************************************* +Overview +************************************* +The SQream database uses a schema called ``sqream_catalog`` that contains information about your database's objects, such tables, columns, views, and permissions. Some additional catalog tables are used primarily for internal analysis and which may be different across SQream versions. + +* :ref:`catalog_reference_schema_information` +* :ref:`catalog_reference_catalog_tables` +* :ref:`catalog_reference_additonal_tables` +* :ref:`catalog_reference_examples` \ No newline at end of file diff --git a/reference/catalog_reference_schema_information.rst b/reference/catalog_reference_schema_information.rst new file mode 100644 index 000000000..6cd43ab6a --- /dev/null +++ b/reference/catalog_reference_schema_information.rst @@ -0,0 +1,62 @@ +.. _catalog_reference_schema_information: + +************************************* +What Information Does the Schema Contain? +************************************* +The schema includes tables designated and relevant for both external and internal use: + +.. contents:: + :local: + :depth: 1 + +External Tables +----------------- +The following table shows the data objects contained in the ``sqream_catalog`` schema designated for external use: + +.. list-table:: Database Objects + :widths: 20 180 + :header-rows: 1 + + * - Database Object + - Table + * - :ref:`Clustering Keys` + - ``clustering_keys`` + * - :ref:`Columns` + - ``columns``, ``external_table_columns`` + * - :ref:`Databases` + - ``databases`` + * - :ref:`Permissions` + - ``table_permissions``, ``database_permissions``, ``schema_permissions``, ``permission_types``, ``udf_permissions``, ``sqream_catalog.table_default_permissions`` + * - :ref:`Queries` + - ``saved_queries`` + * - :ref:`Roles` + - ``roles``, ``roles_memeberships`` + * - :ref:`Schemas` + - ``schemas`` + * - :ref:`Sequences` + - ``identity_key`` + * - :ref:`Tables` + - ``tables``, ``external_tables`` + * - :ref:`Views` + - ``views`` + * - :ref:`User Defined Functions` + - ``user_defined_functions`` + +Internal Tables +----------------- +The following table shows the data objects contained in the ``sqream_catalog`` schema designated for internal use: + +.. list-table:: Storage Objects + :widths: 20 180 + :header-rows: 1 + + * - Database Object + - Table + * - Extents + - Shows ``extents``. + * - Chunk columns + - Shows ``chunks_columns``. + * - Chunks + - Shows ``chunks``. + * - Delete predicates + - Shows ``delete_predicates``. For more information, see :ref:`Deleting Data`. \ No newline at end of file diff --git a/reference/cli/sqream_sql.rst b/reference/cli/sqream_sql.rst index 079f1543d..87bb374c0 100644 --- a/reference/cli/sqream_sql.rst +++ b/reference/cli/sqream_sql.rst @@ -1,7 +1,7 @@ .. _sqream_sql_cli_reference: ********************************* -sqream sql CLI reference +Sqream SQL CLI Reference ********************************* SQream DB comes with a built-in client for executing SQL statements either interactively or from the command-line. @@ -13,15 +13,11 @@ This page serves as a reference for the options and parameters. Learn more about Installing sqream sql ========================= - If you have a SQream DB installation on your server, ``sqream sql`` can be found in the ``bin`` directory of your SQream DB installation, under the name ``sqream``. -.. note:: If you installed SQream DB via Docker, the command is named ``sqream-client sql``, and can be found in the same location as the console. - - .. versionchanged:: 2020.1 As of version 2020.1, ``ClientCmd`` has been renamed to ``sqream sql``. - + To run ``sqream sql`` on any other Linux host: @@ -40,7 +36,7 @@ To run ``sqream sql`` on any other Linux host: master=> _ -Troubleshooting sqream sql installation +Troubleshooting SQream SQl Installation ------------------------------------------- Upon running sqream sql for the first time, you may get an error ``error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory``. @@ -69,12 +65,13 @@ Solving this error requires installing the ncruses or libtinfo libraries, depend ``$ sudo ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so.5`` -Using sqream sql + +Using SQream SQL ================= By default, sqream sql runs in interactive mode. You can issue commands or SQL statements. -Running commands interactively (SQL shell) +Running Commands Interactively in an SQL Shell -------------------------------------------- When starting sqream sql, after entering your password, you are presented with the SQL shell. @@ -138,7 +135,7 @@ The prompt for a multi-line statement will change from ``=>`` to ``.``, to alert time: 0.009320s -Executing batch scripts (``-f``) +Executing Batch Scripts --------------------------------- To run an SQL script, use the ``-f `` argument. @@ -151,7 +148,7 @@ For example, .. tip:: Output can be saved to a file by using redirection (``>``). -Executing commands immediately (``-c``) +Executing Commands Immediately ------------------------------------------- To run a statement from the console, use the ``-c `` argument. @@ -175,7 +172,7 @@ For example, Examples =========== -Start a regular interactive shell +Starting a Regular Interactive Shell ----------------------------------- Connect to local server 127.0.0.1 on port 5000, to the default built-in database, `master`: @@ -202,7 +199,7 @@ Connect to local server 127.0.0.1 via the built-in load balancer on port 3108, t master=>_ -Execute statements in the interactive shell +Executing Statements in the Interactive Shell ----------------------------------------------- Note that all SQL commands end with a semicolon. @@ -243,7 +240,7 @@ Creating a new database and switching over to it without reconnecting: 2 rows time: 0.029299s -Execute SQL statements from the command line +Executing SQL Statements from the Command Line ---------------------------------------------- .. code-block:: console @@ -255,7 +252,7 @@ Execute SQL statements from the command line .. _controlling_output: -Controlling the output of the client +Controlling the Output of the Client ---------------------------------------- Two parameters control the dispay of results from the client: @@ -263,7 +260,7 @@ Two parameters control the dispay of results from the client: * ``--results-only`` - removes row counts and timing information * ``--delimiter`` - changes the record delimiter -Export SQL query result to CSV +Exportin an SQL Query Result to CSV ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using the ``--results-only`` flag removes the row counts and timing. @@ -277,7 +274,7 @@ Using the ``--results-only`` flag removes the row counts and timing. 3,chicken ,0 4,bull ,1 -Change a CSV to a TSV +Changing a CSV to a TSV ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``--delimiter`` parameter accepts any printable character. @@ -294,7 +291,7 @@ The ``--delimiter`` parameter accepts any printable character. 4 bull 1 -Execute a series of statements from a file +Executing a Series of Statements from a File -------------------------------------------- Assuming a file containing SQL statements (separated by semicolons): @@ -303,7 +300,7 @@ Assuming a file containing SQL statements (separated by semicolons): $ cat some_queries.sql CREATE TABLE calm_farm_animals - ( id INT IDENTITY(0, 1), name VARCHAR(30) + ( id INT IDENTITY(0, 1), name TEXT ); INSERT INTO calm_farm_animals (name) @@ -317,7 +314,7 @@ Assuming a file containing SQL statements (separated by semicolons): executed time: 0.090697s -Connect using environment variables +Connecting Using Environment Variables ------------------------------------- You can save connection parameters as environment variables: @@ -328,7 +325,7 @@ You can save connection parameters as environment variables: $ export SQREAM_DATABASE=farm; $ sqream sql --port=3105 --clustered --username=$SQREAM_USER -d $SQREAM_DATABASE -Connecting to a specific queue +Connecting to a Specific Queue ----------------------------------- When using the :ref:`dynamic workload manager` - connect to ``etl`` queue instead of using the default ``sqream`` queue @@ -344,10 +341,10 @@ When using the :ref:`dynamic workload manager` - connect to `` master=>_ -Operations and flag reference +Operations and Flag Reference =============================== -Command line arguments +Command Line Arguments ----------------------- **SQream SQL** supports the following command line arguments: @@ -400,7 +397,7 @@ Command line arguments .. _supported_record_delimiters: -Supported record delimiters +Supported Record Delimiters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The supported record delimiters are printable ASCII values (32-126). @@ -455,7 +452,7 @@ Basic Commands -Moving about the command line +Navigating the Command Line --------------------------------- .. list-table:: diff --git a/reference/cli/sqream_storage.rst b/reference/cli/sqream_storage.rst index 72f931b96..f47b04a87 100644 --- a/reference/cli/sqream_storage.rst +++ b/reference/cli/sqream_storage.rst @@ -3,20 +3,17 @@ ************************* SqreamStorage ************************* +You can use the **SqreamStorage** program to create a new :ref:`storage cluster`. -``SqreamStorage`` allows the creation of a new :ref:`storage cluster`. - -This page serves as a reference for the options and parameters. +The **SqreamStorage** page serves as a reference for the options and parameters. Running SqreamStorage ======================= +The **SqreamStorage** program is located in the **bin** directory of your SQream installation.. -``SqreamStorage`` can be found in the ``bin`` directory of your SQream DB installation.. - -Command line arguments +Command Line Arguments ========================== - -``SqreamStorage`` supports the following command line arguments: +The **SqreamStorage** program supports the following command line arguments: .. list-table:: :widths: auto @@ -32,20 +29,13 @@ Command line arguments - ``-r`` - Specifies the cluster path. The path must **not** already exist. - -Examples +Example ============= - -Create a new storage cluster ----------------------------------- - -Create a new cluster at ``/home/rhendricks/raviga_database``: +The **Examples** section describes how to create a new storage cluster at ``/home/rhendricks/raviga_database``: .. code-block:: console $ SqreamStorage --create-cluster --cluster-root /home/rhendricks/raviga_database Setting cluster version to: 26 -This can also be written shorthand as ``SqreamStorage -C -r /home/rhendricks/raviga_database``. - -This message confirms the creation of the cluster successfully. +Alternatively, you can write this in shorthand as ``SqreamStorage -C -r /home/rhendricks/raviga_database``. A message is displayed confirming that your cluster has been created. diff --git a/reference/configuration.rst b/reference/configuration.rst deleted file mode 100644 index bf487496e..000000000 --- a/reference/configuration.rst +++ /dev/null @@ -1,5 +0,0 @@ -.. _configuration_reference: - -************************* -Configuration -************************* diff --git a/reference/index.rst b/reference/index.rst index 9360f33c5..c21c756ed 100644 --- a/reference/index.rst +++ b/reference/index.rst @@ -1,19 +1,18 @@ .. _reference: ************************* -Reference +Reference Guides ************************* This section provides reference for using SQream DB's interfaces and SQL features. .. toctree:: - :maxdepth: 5 + :maxdepth: 1 :caption: In this section: :glob: - sql_data_types sql/index catalog_reference cli/index sql_feature_support - xxconfiguration + python_api_reference_guide diff --git a/reference/python_api_reference_guide.rst b/reference/python_api_reference_guide.rst new file mode 100644 index 000000000..bbcea1300 --- /dev/null +++ b/reference/python_api_reference_guide.rst @@ -0,0 +1,125 @@ +.. _python_api_reference_guide: + +************************* +Python API Reference Guide +************************* +The **Python API Reference Guide** describes the following: + +.. contents:: + :local: + :depth: 1 + +Overview +=================== +The SQream Python connector allows Python programs to connect to SQream, and conforms to Python DB-API specifications `PEP-249 `_. The main Python connector module is pysqream, which contains the ``connection`` class. + +The following is the syntax for the ``connect`` class: + +.. code-block:: postgres + + $ connect(host, port, database, username, password, clustered = False, use_ssl = False, service='sqream', reconnect_attempts=3, reconnect_interval=10) + +The ``Connection`` class creates a new ``Connection`` object and connects to SQream. + +The following table describes the ``connect`` object parameters: + +.. list-table:: + :widths: 15 75 + :header-rows: 1 + + * - Parameter + - Description + * - host + - Sets the SQream hostname or IP. + * - port + - Sets the SQream port. + * - database + - Sets the database name. + * - username + - Sets the username for establishing a connection. + * - password + - Sets the password used with your ``username``. + * - clustered + - Connects using either the load balancer or direct to worker (Default: false - direct to worker). + * - use_ssl + - Uses an SSL connection (default: false). + * - service + - (Optional) Sets the service queue (default: 'sqream'). + * - reconnect_attempts + - Sets the number of failed reconnection attempts to attempt before closing the connection. + * - reconnect_interval + - Sets the time in seconds between reconnection attempts. + +.. _connection_object_parameters: + +Connection Object Parameters +=================== +The following table describes the ``Connection`` class parameters: + +.. list-table:: + :widths: 15 117 + :header-rows: 1 + + * - Parameter + - Description + * - arraysize + - Specifies the number of rows to fetch at a time using ``fetchmany()``. Default - ``1``. + * - rowcount + - Not used. Return ``-1``. + * - description + - Displays read-only attributes containing the following result set metadata: + + * ``name`` - Displays the column name. + * ``type_code`` - Displays the internal type code. + * ``display_size`` - Not used, identical to ``internal_size`` below. + * ``internal_size`` - Displays the Data size (bytes). + * ``precision`` - Not used. Displays the precision of numeric data. + * ``scale`` - Not used. Displays the scale for numeric data. + * ``null_ok`` - Specifices whether ``NULL`` values are allowed for the column. + * - execute + - Executes statements (self, query, params=None). Parameters are not supported. + + * ``self`` - :ref:`Connection Object Parameters`. + * ``query`` - Displays a statment or query text. + * ``params`` - Not used. + * - executemany + - Prepares a statement and executes it against all parameter sequences found in ``rows_or_cols`` (self, query, rows_or_cols=None, data_as='rows', amount=None). + + * ``self`` - :ref:`Connection Object Parameters`. + * ``query`` - INSERT statement. + * ``rows_or_cols`` - Data buffer to insert. This should be a sequence of lists or tuples.. + * ``data_as`` - (Optional) Read data as rows or columns. + * ``amount`` - (Optional) count of rows to insert. + * - close + - Closes a statement and connection (self). Closed statements must be reopened by creating a new cursor. + + * ``self`` - :ref:`Connection Object Parameters`. + * - cursor + - Creates a new :ref:`Connection Object Parameters` cursor (self). Recommended - create a new cursor for every statement. + + * ``self`` - :ref:`Connection Object Parameters`. + * - fetchall + - Fetches all remaining records from the result set (self, data_as='rows'). Returns an empty sequence if no more rows are available. + + * ``self`` - :ref:`Connection Object Parameters`. + * ``data_as`` - (Optional) Reads data as rows or columns. + * - fetchone + - Fetches one record from the result set. Returns an empty sequence if no more rows are available. + + * ``self`` - :ref:`Connection Object Parameters`. + * ``size`` - Sets the number of records to fetch. If not set, fetches ``Connection.arraysize`` records (default = 1). + * ``data_as`` - (Optional) Reads data as rows or columns. + * - __iter__() + - Makes the cursor iterable. + +API Level Object Parameters +=================== +The API Level object parameter is a string constant stating the supported API level. The Python connector supports ``apilevel`` 2.0. + +Thread Safety Object Parameters +=================== +The **Thread Safety** object parameter displays the thread safety level the interface supports. The Python connector currently supports ``threadsafety`` level 1, which states that threads can share the module, but not connections. + +Parameter Style Object Parameters +=================== +The **paramstyle** object parameter sets the placeholder marker and is set to to ``qmark``, which is a **question mark** (``?``). \ No newline at end of file diff --git a/reference/sql/sql_functions/aggregate_functions/count.rst b/reference/sql/sql_functions/aggregate_functions/count.rst index 5a056d9fe..803e529c0 100644 --- a/reference/sql/sql_functions/aggregate_functions/count.rst +++ b/reference/sql/sql_functions/aggregate_functions/count.rst @@ -4,18 +4,20 @@ COUNT ************************** -Returns the count of numeric values, or only the distinct values. +The ``COUNT`` function returns the count of numeric values, or only the distinct values. Syntax ========== - +The following is the correct syntax for using the ``COUNT`` function as an **aggregate**: .. code-block:: postgres - -- As an aggregate COUNT( { [ DISTINCT ] expr | * } ) --> BIGINT - -- As a window function +The following is the correct syntax for using the ``COUNT`` function as a **window function**: + +.. code-block:: postgres + COUNT ( { [ DISTINCT ] expr | * } ) OVER ( [ PARTITION BY value_expression [, ...] ] [ ORDER BY value_expression [ ASC | DESC ] [ NULLS { FIRST | LAST } ] [, ...] ] @@ -24,6 +26,7 @@ Syntax Arguments ============ +The following table describes the ``COUNT`` arguments: .. list-table:: :widths: auto @@ -40,22 +43,25 @@ Arguments Returns ============ - -* Count returns ``BIGINT``. +* The ``COUNT`` function returns ``BIGINT``. Notes ======= - -* ``NULL`` values are *not* ignored by ``COUNT`` +The following notes apply to the ``COUNT`` function: * When all rows contain ``NULL`` values, the function returns ``NULL``. +* ``COUNT(*)`` returns the number of items in a group, including duplicates and ``NULL`` values. + +* ``COUNT(ALL expression)`` evaluates expressions for each row in a group, returning the number of non-null values. + +* ``COUNT(DISTINCT expression)`` evaluates expressions for each row in a group, returning the number of unique, non-null values. + Examples =========== - -For these examples, assume a table named ``nba``, with the following structure: +The examples in this section are based on a table named ``nba``, structured as follows: .. code-block:: postgres @@ -72,16 +78,22 @@ For these examples, assume a table named ``nba``, with the following structure: "Salary" float ); - -Here's a peek at the table contents (:download:`Download nba.csv `): +The following table is a preview of the source :download:`nba.csv ` table shown below: .. csv-table:: nba.csv :file: nba-t10.csv :widths: auto :header-rows: 1 + +This section includes the following examples: -Count rows in a table +.. contents:: + :local: + :depth: 1 + +Counting Rows in a Table --------------------------- +This example shows how to count rows in a table: .. code-block:: psql @@ -90,10 +102,11 @@ Count rows in a table ----- 457 -Count distinct values in a table +Counting Distinct Values in a Table ---------------------------------- +This example shows how to count distinct values in a table: -These two forms are equivalent: +The following structures generate the same result: .. code-block:: psql @@ -102,14 +115,17 @@ These two forms are equivalent: ----- 22 +.. code-block:: psql + t=> SELECT COUNT(*) FROM (SELECT "Age" FROM nba GROUP BY 1); count ----- 22 -Combine COUNT with other aggregates +Combining COUNT with Other Aggregates ------------------------------------- +This example shows how to combine the ``COUNT`` function with other aggregates: .. code-block:: psql @@ -138,4 +154,3 @@ Combine COUNT with other aggregates 38 | 1840041 | 4 39 | 2517872 | 2 40 | 4666916 | 3 - diff --git a/reference/sql/sql_functions/aggregate_functions/index.rst b/reference/sql/sql_functions/aggregate_functions/index.rst index 5f38ac1c5..5d3dfc125 100644 --- a/reference/sql/sql_functions/aggregate_functions/index.rst +++ b/reference/sql/sql_functions/aggregate_functions/index.rst @@ -1,31 +1,32 @@ .. _aggregate_functions: ******************** -Aggregate functions +Aggregate Functions ******************** -Aggregate functions performs calculation on a set of values, and return a single value. +Overview +=========== +Aggregate functions perform calculations based on a set of values and return a single value. Most aggregate functions ignore null values. Aggregate functions are often used with the ``GROUP BY`` clause of the :ref:`select` statement. -Most aggregate functions ignore null values. +Available Aggregate Functions +=============== +The following list shows the available aggregate functions: -Aggregate functions are often used with the ``GROUP BY`` clause of the :ref:`select` statement. - - - -.. toctree:: - :maxdepth: 1 - :caption: Aggregate functions - :glob: +.. hlist:: + :columns: 2 - avg - corr - count - covar_pop - covar_samp - max - min - stddev_pop - stddev_samp - sum - var_pop - var_samp \ No newline at end of file + * :ref:`AVG` + * :ref:`CORR` + * :ref:`COUNT` + * :ref:`COVAR_POP` + * :ref:`COVAR_SAMP` + * :ref:`MAX` + * :ref:`MIN` + * :ref:`MODE` + * :ref:`PERCENTILE_CONT` + * :ref:`PERCENTILE_DISC` + * :ref:`STDDEV_POP` + * :ref:`STDDEV_SAMP` + * :ref:`SUM` + * :ref:`VAR_POP` + * :ref:`VAR_SAMP` \ No newline at end of file diff --git a/reference/sql/sql_functions/aggregate_functions/mode.rst b/reference/sql/sql_functions/aggregate_functions/mode.rst new file mode 100644 index 000000000..a6d46e114 --- /dev/null +++ b/reference/sql/sql_functions/aggregate_functions/mode.rst @@ -0,0 +1,65 @@ +.. _mode: + +************************** +MODE +************************** +The **MODE** function returns the most common value in the selected column. If there are no repeating values, or if there is the same frequency of multiple values, this function returns the top value based on the ``ORDER BY`` clause. + +The **MODE** function is commonly used with the following functions: + +* `PERCENTILE_CONT `_ function +* `PERCENTILE_DISC `_ function + +Syntax +======== +The following is the correct syntax for the ``MODE`` function: + +.. code-block:: postgres + + MODE() WITHIN GROUP (ORDER BY column) + +Example +======== +The example in this section is based on the ``players`` table below: + +.. list-table:: + :widths: 33 33 33 + :header-rows: 1 + ++-----------------+----------+-----------+ +| **Player_Name** | **Team** | **Score** | ++-----------------+----------+-----------+ +| T_Tock | Blue | 13 | ++-----------------+----------+-----------+ +| N_Stein | Blue | 20 | ++-----------------+----------+-----------+ +| F_Dirk | Blue | 20 | ++-----------------+----------+-----------+ +| Y_Hyung | Blue | 10 | ++-----------------+----------+-----------+ +| A_Rodrick | Blue | 13 | ++-----------------+----------+-----------+ +| R_Evans | Red | 55 | ++-----------------+----------+-----------+ +| C_Johnston | Red | 20 | ++-----------------+----------+-----------+ +| K_Stoll | Red | 25 | ++-----------------+----------+-----------+ +| J_Loftus | Red | 22 | ++-----------------+----------+-----------+ +| L_Ellis | Red | 7 | ++-----------------+----------+-----------+ +| G_Elroy | Red | 23 | ++-----------------+----------+-----------+ + +The following is an example of the ``MODE`` function: + +.. code-block:: postgres + + SELECT + MODE() WITHIN GROUP (ORDER BY score) as Mode + FROM players; + + Mode| + ----+ + 20 diff --git a/reference/sql/sql_functions/aggregate_functions/percentile_cont.rst b/reference/sql/sql_functions/aggregate_functions/percentile_cont.rst new file mode 100644 index 000000000..115101a1d --- /dev/null +++ b/reference/sql/sql_functions/aggregate_functions/percentile_cont.rst @@ -0,0 +1,64 @@ +.. _percentile_cont: + +************************** +PERCENTILE_CONT +************************** +The **PERCENTILE_CONT** function returns the value located in the selected percentile of a specified column. The required input is a number between **0** and **1** that represents the wanted percentile and the column containing the wanted percentile. If the selected percentile does not have an exact value, the PERCENTILE_CONT function extrapolates a value estimated from the existing values. + +The **PERCENTILE_CONT** function returns an extrapolated value closest to the values in the requested percentile. If you want to see the closest value in your data to the requested percentile, use `PERCENTILE DISC `_. + +Syntax +======== +The following is the correct syntax for the ``PERCENTILE_CONT`` function: + +.. code-block:: postgres + + PERCENTILE_CONT(double) WITHIN GROUP (ORDER BY ) + +Example +======== +The example in this section is based on the ``players`` table below: + +.. list-table:: + :widths: 33 33 33 + :header-rows: 1 + ++-----------------+----------+-----------+ +| **Player_Name** | **Team** | **Score** | ++-----------------+----------+-----------+ +| T_Tock | Blue | 12 | ++-----------------+----------+-----------+ +| N_Stein | Blue | 23 | ++-----------------+----------+-----------+ +| F_Dirk | Blue | 45 | ++-----------------+----------+-----------+ +| Y_Hyung | Blue | 19 | ++-----------------+----------+-----------+ +| A_Rodrick | Blue | 37 | ++-----------------+----------+-----------+ +| R_Evans | Red | 52 | ++-----------------+----------+-----------+ +| C_Johnston | Red | 43 | ++-----------------+----------+-----------+ +| K_Stoll | Red | 27 | ++-----------------+----------+-----------+ +| J_Loftus | Red | 8 | ++-----------------+----------+-----------+ +| L_Ellis | Red | 13 | ++-----------------+----------+-----------+ +| G_Elroy | Red | 31 | ++-----------------+----------+-----------+ + +The following is an example of the ``PERCENTILE_CONT`` function: + +.. code-block:: postgres + + SELECT + PERCENTILE_CONT(0.18) WITHIN GROUP (ORDER BY score) + FROM players; + + percentile_cont| + ---------------+ + 12.8 + +.. note:: The output value is an extrapolation of the halfway point between the values. diff --git a/reference/sql/sql_functions/aggregate_functions/percentile_disc.rst b/reference/sql/sql_functions/aggregate_functions/percentile_disc.rst new file mode 100644 index 000000000..e5c1fffc3 --- /dev/null +++ b/reference/sql/sql_functions/aggregate_functions/percentile_disc.rst @@ -0,0 +1,62 @@ +.. _percentile_disc: + +************************** +PERCENTILE_DISC +************************** +The **PERCENTILE_DISC** function returns the value located in the selected percentile of a specified column. The required input is a number between **0** and **1** that represents the wanted percentile and the column that you want to rank. If the selected percentile does not have an exact value, this function returns the closest value beneath the requested percentile. + +The **PERCENTILE_DISC** function returns an exact value based on the closest values in your data to the requested percentile. If you want to see an extrapolated value in your data closest to the requested percentile, use `PERCENTILE CONT `_. + +Syntax +======== +The following is the correct syntax for the ``PERCENTILE_DISC`` function: + +.. code-block:: postgres + + PERCENTILE_DISC(double) WITHIN GROUP (ORDER BY ) + +Example +======== +The example in this section is based on the ``players`` table below: + +.. list-table:: + :widths: 33 33 33 + :header-rows: 1 + ++-----------------+----------+-----------+ +| **Player_Name** | **Team** | **Score** | ++-----------------+----------+-----------+ +| T_Tock | Blue | 12 | ++-----------------+----------+-----------+ +| N_Stein | Blue | 23 | ++-----------------+----------+-----------+ +| F_Dirk | Blue | 45 | ++-----------------+----------+-----------+ +| Y_Hyung | Blue | 19 | ++-----------------+----------+-----------+ +| A_Rodrick | Blue | 37 | ++-----------------+----------+-----------+ +| R_Evans | Red | 52 | ++-----------------+----------+-----------+ +| C_Johnston | Red | 43 | ++-----------------+----------+-----------+ +| K_Stoll | Red | 27 | ++-----------------+----------+-----------+ +| J_Loftus | Red | 8 | ++-----------------+----------+-----------+ +| L_Ellis | Red | 13 | ++-----------------+----------+-----------+ +| G_Elroy | Red | 31 | ++-----------------+----------+-----------+ + +The following is an example of the ``PERCENTILE_DISC`` function: + +.. code-block:: postgres + + SELECT + PERCENTILE_DISC(0.15) WITHIN GROUP (ORDER BY score) + FROM players; + + percentile_disc| + ---------------+ + 12 diff --git a/reference/sql/sql_functions/index.rst b/reference/sql/sql_functions/index.rst index 5d02b27a6..a66760b7f 100644 --- a/reference/sql/sql_functions/index.rst +++ b/reference/sql/sql_functions/index.rst @@ -1,25 +1,26 @@ .. _sql_functions: **************** -SQL functions +SQL Functions **************** -SQream DB supports functions from ANSI SQL, as well as others for compatibility. +SQream supports functions from ANSI SQL, as well as others for compatibility. -Summary of functions +Summary of Functions ======================= .. contents:: :local: -Scalar functions +Built-In Scalar Functions ------------------- +For more information about built-in scalar functions, see :ref:`scalar_functions`. -See more about :ref:`scalar_functions` - -Bitwise operations +Bitwise Operations ^^^^^^^^^^^^^^^^^^^^^^^ +The following table shows the **bitwise operations** functions: + .. list-table:: :widths: auto :header-rows: 1 @@ -41,6 +42,7 @@ Bitwise operations Conditionals ^^^^^^^^^^^^^^ +The following table shows the **conditionals** functions: .. list-table:: :widths: auto @@ -65,6 +67,7 @@ Conditionals Conversion ^^^^^^^^^^^^ +The following table shows the **conversion** functions: .. list-table:: :widths: auto @@ -79,8 +82,9 @@ Conversion * - :ref:`to_unixts` - Converts a ``DATE`` or ``DATETIME`` to a UNIX Timestamp -Date and time +Date and Time ^^^^^^^^^^^^^^^^ +The following table shows the **date and time** functions: .. list-table:: :widths: auto @@ -113,10 +117,9 @@ Date and time Numeric ^^^^^^^^^^^ +The following table shows the **arithmetic operators**: -See more about :ref:`arithmetic_operators` - -.. list-table:: Arithmetic operators +.. list-table:: Arithmetic Operators :widths: auto :header-rows: 1 @@ -145,7 +148,11 @@ See more about :ref:`arithmetic_operators` - ``a % b`` - Modulu of ``a`` by ``b``. See also :ref:`mod` -.. list-table:: Functions +For more information about arithmetic operators, see :ref:`arithmetic_operators`. + +The following table shows the **arithmetic operator** functions: + +.. list-table:: Arithemtic Operator Functions :widths: auto :header-rows: 1 @@ -202,6 +209,7 @@ See more about :ref:`arithmetic_operators` Strings ^^^^^^^^^^^ +The following table shows the **string** functions: .. list-table:: :widths: auto @@ -215,6 +223,8 @@ Strings - Calculates the position where a string starts inside another string * - :ref:`concat` - Concatenates two strings + * - :ref:`decode` + - Decodes or extracts binary data from a textual input string * - :ref:`isprefixof` - Matches if a string is the prefix of another string * - :ref:`left` @@ -235,6 +245,8 @@ Strings - Calculates the number of matches of a regular expression match in an argument * - :ref:`regexp_instr` - Returns the start position of a regular expression match in an argument + * - :ref:`regexp_replace` + - Replaces and returns the text column substrings of a regular expression match in an argument * - :ref:`regexp_substr` - Returns a substring of an argument that matches a regular expression * - :ref:`repeat` @@ -256,11 +268,14 @@ Strings * - :ref:`upper` - Converts an argument to an upper-case equivalent - -Aggregate functions +User-Defined Scalar Functions --------------------- +For more information about user-defined scalar functions, see :ref:`scalar_sql_udf`. -See more about :ref:`aggregate_functions` + +Aggregate Functions +--------------------- +The following table shows the **aggregate** functions: .. list-table:: :widths: auto @@ -306,10 +321,11 @@ See more about :ref:`aggregate_functions` - ``varp`` - Calculates population variance of values -Window functions -------------------- +For more information about aggregate functions, see :ref:`aggregate_functions`. -See more about :ref:`window_functions` +Window Functions +------------------- +The following table shows the **window** functions: .. list-table:: :widths: auto @@ -325,41 +341,30 @@ See more about :ref:`window_functions` - Calculates the maximum value * - :ref:`min` - Calculates the minimum value - * - :ref:`rank` - - Calculates the rank of a row - * - :ref:`row_number` - - Calculates the row number * - :ref:`sum` - - Calculates the sum of all of the values - -System functions ------------------- - -System functions allow you to execute actions in the system, such as aborting a query or get information about system processes. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Function - - Description - * - :ref:`explain` - - Returns a static query plan for a statement - * - :ref:`show_connections` - - Returns a list of jobs and statements on the current worker - * - :ref:`show_locks` - - Returns any existing locks in the database - * - :ref:`show_node_info` - - Returns a query plan for an actively running statement with timing information - * - :ref:`show_server_status` - - Shows running statements across the cluster - * - :ref:`show_version` - - Returns the version of SQream DB - * - :ref:`stop_statement` - - Stops a query (or statement) if it is currently running - -Workload management functions + - Calculates the sum of all of the values + * - :ref:`rank` + - Calculates the rank of a row + * - :ref:`first_value` + - Returns the value in the first row of a window + * - :ref:`last_value` + - Returns the value in the last row of a window + * - :ref:`nth_value` + - Returns the value in a specified ``(n)`` row of a window + * - :ref:`dense_rank` + - Returns the rank of the current row with no gaps + * - :ref:`percent_rank` + - Returns the relative rank of the current row + * - :ref:`cume_dist` + - Returns the cumulative distribution of rows + * - :ref:`ntile` + - Returns an integer ranging between ``1`` and the argument value, dividing the partitions as equally as possible + +For more information about window functions, see :ref:`window_functions`. + +Workload Management Functions --------------------------------- +The following table shows the **workload management** functions: .. list-table:: :widths: auto @@ -382,6 +387,6 @@ Workload management functions :glob: scalar_functions/index + user_defined_functions/index aggregate_functions/index - window_functions/* - system_functions/* + window_functions/index \ No newline at end of file diff --git a/reference/sql/sql_functions/scalar_functions/index.rst b/reference/sql/sql_functions/scalar_functions/index.rst index b7befbb28..bd05f9bbe 100644 --- a/reference/sql/sql_functions/scalar_functions/index.rst +++ b/reference/sql/sql_functions/scalar_functions/index.rst @@ -1,20 +1,85 @@ .. _scalar_functions: **************** -Scalar functions +Built-In Scalar Functions **************** +The **Built-In Scalar Functions** page describes functions that return one value per call: -Scalar functions return one value per call. - - -.. toctree:: - :maxdepth: 1 - :caption: Scalar functions - :glob: - - bitwise/* - conditionals/* - conversion/* - date_and_time/* - numeric/* - string/* \ No newline at end of file +.. hlist:: + :columns: 5 + + * `AND `_ + * `NOT `_ + * `OR `_ + * `SHIFT_LEFT `_ + * `SHIFT_RIGHT `_ + * `XOR `_ + * :ref:`between` + * :ref:`case` + * :ref:`coalesce` + * :ref:`in` + * :ref:`is_ascii` + * :ref:`is_null` + * :ref:`isnull` + * :ref:`from_unixts` + * :ref:`to_hex` + * :ref:`to_unixts` + * :ref:`curdate` + * :ref:`current_date` + * :ref:`current_timestamp` + * :ref:`dateadd` + * :ref:`datediff` + * :ref:`datepart` + * :ref:`eomonth` + * :ref:`extract` + * :ref:`getdate` + * :ref:`sysdate` + * :ref:`trunc` + * :ref:`abs` + * :ref:`acos` + * :ref:`asin` + * :ref:`atan` + * :ref:`atn2` + * :ref:`ceiling` + * :ref:`cos` + * :ref:`cot` + * :ref:`crc64` + * :ref:`degrees` + * :ref:`exp` + * :ref:`floor` + * :ref:`log` + * :ref:`log10` + * :ref:`mod` + * :ref:`pi` + * :ref:`power` + * :ref:`radians` + * :ref:`round` + * :ref:`sin` + * :ref:`sqrt` + * :ref:`square` + * :ref:`tan` + * :ref:`trunc` + * :ref:`char_length` + * :ref:`charindex` + * :ref:`concat` + * :ref:`isprefixof` + * :ref:`left` + * :ref:`len` + * :ref:`like` + * :ref:`lower` + * :ref:`ltrim` + * :ref:`octet_length` + * :ref:`patindex` + * :ref:`regexp_count` + * :ref:`regexp_instr` + * :ref:`regexp_replace` + * :ref:`regexp_substr` + * :ref:`repeat` + * :ref:`replace` + * :ref:`reverse` + * :ref:`right` + * :ref:`rlike` + * :ref:`rtrim` + * :ref:`substring` + * :ref:`trim` + * :ref:`upper` \ No newline at end of file diff --git a/reference/sql/sql_functions/scalar_functions/string/concat.rst b/reference/sql/sql_functions/scalar_functions/string/concat.rst index 0409216cd..9c18902be 100644 --- a/reference/sql/sql_functions/scalar_functions/string/concat.rst +++ b/reference/sql/sql_functions/scalar_functions/string/concat.rst @@ -48,15 +48,15 @@ For these examples, assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float + "Name" varchar(40), + "Team" varchar(40), + "Number" tinyint, + "Position" varchar(2), + "Age" tinyint, + "Height" varchar(4), + "Weight" real, + "College" varchar(40), + "Salary" float ); diff --git a/reference/sql/sql_functions/scalar_functions/string/isprefixof.rst b/reference/sql/sql_functions/scalar_functions/string/isprefixof.rst index 8d3531059..4a978b1ff 100644 --- a/reference/sql/sql_functions/scalar_functions/string/isprefixof.rst +++ b/reference/sql/sql_functions/scalar_functions/string/isprefixof.rst @@ -69,4 +69,4 @@ Filtering using ``ISPREFIXOF`` ----------------------------------- And the mome raths outgrabe. -.. tip:: Use :ref:`trim` to avoid leading and trailing whitespace issues \ No newline at end of file +.. tip:: Use :ref:`trim` to avoid leading and trailing whitespace issues diff --git a/reference/sql/sql_functions/scalar_functions/string/left.rst b/reference/sql/sql_functions/scalar_functions/string/left.rst index 041957819..77c99b0f1 100644 --- a/reference/sql/sql_functions/scalar_functions/string/left.rst +++ b/reference/sql/sql_functions/scalar_functions/string/left.rst @@ -72,4 +72,4 @@ Using ``LEFT`` "Beware th | , my son! The | at catch! Beware the | and shun - The | rsnatch!" \ No newline at end of file + The | rsnatch!" diff --git a/reference/sql/sql_functions/scalar_functions/string/like.rst b/reference/sql/sql_functions/scalar_functions/string/like.rst index 4640ba9be..9a1b257be 100644 --- a/reference/sql/sql_functions/scalar_functions/string/like.rst +++ b/reference/sql/sql_functions/scalar_functions/string/like.rst @@ -83,15 +83,15 @@ For these examples, assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float + "Name" varchar(40), + "Team" varchar(40), + "Number" tinyint, + "Position" varchar(2), + "Age" tinyint, + "Height" varchar(4), + "Weight" real, + "College" varchar(40), + "Salary" float ); diff --git a/reference/sql/sql_functions/scalar_functions/string/nba-t10.csv b/reference/sql/sql_functions/scalar_functions/string/nba-t10.csv index 024530355..c3d41e729 100644 --- a/reference/sql/sql_functions/scalar_functions/string/nba-t10.csv +++ b/reference/sql/sql_functions/scalar_functions/string/nba-t10.csv @@ -1,10 +1,9 @@ -Name,Team,Number,Position,Age,Height,Weight,College,Salary -Avery Bradley,Boston Celtics,0.0,PG,25.0,6-2,180.0,Texas,7730337.0 -Jae Crowder,Boston Celtics,99.0,SF,25.0,6-6,235.0,Marquette,6796117.0 -John Holland,Boston Celtics,30.0,SG,27.0,6-5,205.0,Boston University, -R.J. Hunter,Boston Celtics,28.0,SG,22.0,6-5,185.0,Georgia State,1148640.0 -Jonas Jerebko,Boston Celtics,8.0,PF,29.0,6-10,231.0,,5000000.0 -Amir Johnson,Boston Celtics,90.0,PF,29.0,6-9,240.0,,12000000.0 -Jordan Mickey,Boston Celtics,55.0,PF,21.0,6-8,235.0,LSU,1170960.0 -Kelly Olynyk,Boston Celtics,41.0,C,25.0,7-0,238.0,Gonzaga,2165160.0 -Terry Rozier,Boston Celtics,12.0,PG,22.0,6-2,190.0,Louisville,1824360.0 +Avery Bradley,Boston Celtics,0,PG,25,2-Jun,180,Texas,7730337 +Jae Crowder,Boston Celtics,99,SF,25,6-Jun,235,Marquette,6796117 +John Holland,Boston Celtics,30,SG,27,5-Jun,205,Boston University, +R.J. Hunter,Boston Celtics,28,SG,22,5-Jun,185,Georgia State,1148640 +Jonas Jerebko,Boston Celtics,8,PF,29,10-Jun,231,,5000000 +Amir Johnson,Boston Celtics,90,PF,29,9-Jun,240,,12000000 +Jordan Mickey,Boston Celtics,55,PF,21,8-Jun,235,LSU,1170960 +Kelly Olynyk,Boston Celtics,41,C,25,Jul-00,238,Gonzaga,2165160 +Terry Rozier,Boston Celtics,12,PG,22,2-Jun,190,Louisville,1824360 diff --git a/reference/sql/sql_functions/scalar_functions/string/octet_length.rst b/reference/sql/sql_functions/scalar_functions/string/octet_length.rst index 61ac2fa85..8bb1e3daf 100644 --- a/reference/sql/sql_functions/scalar_functions/string/octet_length.rst +++ b/reference/sql/sql_functions/scalar_functions/string/octet_length.rst @@ -16,7 +16,7 @@ Calculates the number of bytes in a string. Syntax ========== - +The following is the correct syntax for the ``OCTET_LENGTH`` function: .. code-block:: postgres @@ -24,6 +24,7 @@ Syntax Arguments ============ +The following table describes the ``OCTET_LENGTH`` arguments: .. list-table:: :widths: auto @@ -36,33 +37,29 @@ Arguments Returns ============ - -Returns an integer containing the number of bytes in the string. +The ``OCTET_LENGTH`` function returns an integer containing the number of bytes in the string. Notes ======= +The following notes are applicable to the ``OCTET_LENGTH`` function: * To get the length in characters, see :ref:`char_length` * If the value is NULL, the result is NULL. -Examples +Length in Characters and Bytes of Strings =========== - -For these examples, consider the following table and contents: +The **Length in characters and bytes of strings** example is based on the following table and contents: .. code-block:: postgres - CREATE TABLE alphabets(line NVARCHAR(50)); + CREATE TABLE alphabets(line TEXT(50)); INSERT INTO alphabets VALUES ('abcdefghijklmnopqrstuvwxyz'), ('กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรฤลฦวศษสหฬอฮฯ') , ('אבגדהוזחטיכלמנסעפצקרשת'); -Length in characters and bytes of strings --------------------------------------------------- - -ASCII characters take up 1 byte per character, while Thai takes up 3 bytes and Hebrew takes up 2 bytes. +ASCII characters take up 1 byte per character, while Thai takes up 3 bytes and Hebrew takes up 2 bytes, as shown below: .. code-block:: psql @@ -72,6 +69,3 @@ ASCII characters take up 1 byte per character, while Thai takes up 3 bytes and H 26 | 26 | 26 47 | 47 | 141 22 | 22 | 44 - - - diff --git a/reference/sql/sql_functions/scalar_functions/string/patindex.rst b/reference/sql/sql_functions/scalar_functions/string/patindex.rst index fddaa1afb..7e2ce8f79 100644 --- a/reference/sql/sql_functions/scalar_functions/string/patindex.rst +++ b/reference/sql/sql_functions/scalar_functions/string/patindex.rst @@ -84,7 +84,9 @@ For these examples, consider the following table and contents: .. code-block:: postgres - CREATE TABLE jabberwocky(line VARCHAR(50)); + + CREATE TABLE jabberwocky(line TEXT(50)); + INSERT INTO jabberwocky VALUES ('''Twas brillig, and the slithy toves '), (' Did gyre and gimble in the wabe: ') diff --git a/reference/sql/sql_functions/scalar_functions/string/replace.rst b/reference/sql/sql_functions/scalar_functions/string/replace.rst index 5552be269..5d489e0d3 100644 --- a/reference/sql/sql_functions/scalar_functions/string/replace.rst +++ b/reference/sql/sql_functions/scalar_functions/string/replace.rst @@ -8,7 +8,6 @@ Replaces all occurrences of a specified string value with another string value. .. warning:: With ``VARCHAR``, a substring can only be replaced with another substring of equal **byte length**. See :ref:`octet_length`. - Syntax ========== @@ -40,7 +39,7 @@ Returns the same type as the argument supplied. Notes ======= -* In ``VARCHAR`` strings, the ``source_expr`` and ``replacement_expr`` must be the same **byte length**. See :ref:`octet_length`. +* In ``TEXT`` strings, the ``source_expr`` and ``replacement_expr`` must be the same **byte length**. See :ref:`octet_length`. * If the value is NULL, the result is NULL. diff --git a/reference/sql/sql_functions/scalar_functions/string/right.rst b/reference/sql/sql_functions/scalar_functions/string/right.rst index 8473b5da4..158de7da0 100644 --- a/reference/sql/sql_functions/scalar_functions/string/right.rst +++ b/reference/sql/sql_functions/scalar_functions/string/right.rst @@ -72,4 +72,4 @@ Using ``RIGHT`` "Beware th | , my son! The | at catch! Beware the | and shun - The | rsnatch!" \ No newline at end of file + The | rsnatch!" diff --git a/reference/sql/sql_functions/scalar_functions/string/rlike.rst b/reference/sql/sql_functions/scalar_functions/string/rlike.rst index 324a6e525..15dab212a 100644 --- a/reference/sql/sql_functions/scalar_functions/string/rlike.rst +++ b/reference/sql/sql_functions/scalar_functions/string/rlike.rst @@ -99,15 +99,15 @@ For these examples, assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float + "Name" varchar(40), + "Team" varchar(40), + "Number" tinyint, + "Position" varchar(2), + "Age" tinyint, + "Height" varchar(4), + "Weight" real, + "College" varchar(40), + "Salary" float ); diff --git a/reference/sql/sql_functions/scalar_functions/string/substring.rst b/reference/sql/sql_functions/scalar_functions/string/substring.rst index b07d951fb..3bfd98b73 100644 --- a/reference/sql/sql_functions/scalar_functions/string/substring.rst +++ b/reference/sql/sql_functions/scalar_functions/string/substring.rst @@ -54,15 +54,15 @@ For these examples, assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float + "Name" varchar(40), + "Team" varchar(40), + "Number" tinyint, + "Position" varchar(2), + "Age" tinyint, + "Height" varchar(4), + "Weight" real, + "College" varchar(40), + "Salary" float ); diff --git a/reference/sql/sql_functions/user_defined_functions/index.rst b/reference/sql/sql_functions/user_defined_functions/index.rst new file mode 100644 index 000000000..028e35a07 --- /dev/null +++ b/reference/sql/sql_functions/user_defined_functions/index.rst @@ -0,0 +1,12 @@ +.. _user_defined_functions_index: + +******************** +User-Defined Functions +******************** + +The following user-defined functions are functions that can be defined and configured by users. + +The **User-Defined Functions** page describes the following: + +* `Python user-defined functions `_ +* `Scalar SQL user-defined functions `_ \ No newline at end of file diff --git a/reference/sql/sql_functions/user_defined_functions/scalar_sql_udf.rst b/reference/sql/sql_functions/user_defined_functions/scalar_sql_udf.rst new file mode 100644 index 000000000..de399c3ff --- /dev/null +++ b/reference/sql/sql_functions/user_defined_functions/scalar_sql_udf.rst @@ -0,0 +1,101 @@ +.. _scalar_sql_udf: + +Scalar SQL UDF +----------------------- +A **scalar SQL UDF** is a user-defined function that returns a single value, such as the sum of a group of values. Scalar UDFs are different than table-value functions, which return a result set in the form of a table. + +This page describes the correct syntax when building simple scalar UDFs and provides three examples. + +Syntax +~~~~~~~~~~~~ + +The following example shows the correct syntax for simple scalar SQL UDF's returning the type name: + +.. code-block:: console + + $ create_function_statement ::= + $ CREATE [ OR REPLACE ] FUNCTION function_name (argument_list) + $ RETURNS return_type + $ AS $$ + $ { function_body } + $ $$ LANGUAGE SQL + $ ; + $ + $ function_name ::= identifier + $ argument_list :: = { value_name type_name [, ...] } + $ value_name ::= identifier + $ return_type ::= type_name + $ function_body ::= A valid SQL statement + +Examples +~~~~~~~~~~ +Example 1 – Support for Different Syntax +############ +Scalar SQL UDF supports standard functionality even when different syntax is used. + +In the example below, the syntax ``dateadd`` is used instead of ``add_months``, although the function of each is identical. In addition, the operation works correctly even though the order of the expressions in ``add_months`` (``dt``, ``datetime``, and ``n int``) is different than ``MONTH``, ``n``, and ``dt`` in ``dateadd``. + +.. code-block:: console + + $ CREATE or replace FUNCTION add_months(dt datetime,n int) + $ RETURNS datetime + $ AS $$ + $ SELECT dateadd(MONTH ,n,dt) + $ $$ LANGUAGE SQL; + +Example 2 – Manipulating Strings +############ +The Scalar SQL UDF can be used to manipulate strings. + +The following example shows the correct syntax for converting a TEXT date to the DATE type: + +.. code-block:: console + + $ CREATE or replace FUNCTION STR_TO_DATE(f text) + $ RETURNS date + $ AS $$ + $ select (substring(f,1,4)||'-'||substring(f,5,2)||'-'||substring(f,7,2))::date + $ $$ LANGUAGE SQL; + +Example 3 – Manually Building Functionality +############ +You can use the Scalar SQL UDF to manually build functionality for otherwise unsupported operations. + +.. code-block:: console + + $ CREATE OR REPLACE function "least_sq" (a float, b float) -- Replace the LEAST(from hql) function + $ returns float as + $ $$select case + $ when a <= b then a + $ when b < a then b + $ when a is null then b + $ when b is null then a + $ else null + $ end; + $ $$ + $ language sql; + +Usage Notes +~~~~~~~~~~~~~~ +The following usage notes apply when using simple scalar SQL UDF's: + +* During this stage, the SQL embedded in the function body must be of the type ``SELECT expr;``. Creating a UDF with invalid SQL, or with valid SQL of any other type, results in an error. +* As with Python UDFs, the argument list can be left empty. +* SQL UDFs can reference other UDF's, including Python UDF's. + +**NOTICE:** A function cannot (directly or indirectly) reference itself (such as by referencing another function that references it). + +Because SQL UDF's are one type of supported UDFs, the following Python UDF characteristics apply: + +* UDF permission rules - see `Access Control `_. + +* The ``get_function_ddl`` utility function works on these functions - see `Getting the DDL for a Function `_. + +* SQL UDF's should appear in the catalog with Python UDF's - see `Finding Existing UDFs in the Catalog `_. + +Restrictions +~~~~~~~~~~~~~~~~ +The following restrictions apply to simple scalar SQL UDFs: + +* Simple scalar SQL UDF's cannot currently reference other UDFs. +* Like Python UDF's, Sqream does not support overloading. diff --git a/reference/sql/sql_functions/user_defined_functions/simple_scalar_sql_udf.rst b/reference/sql/sql_functions/user_defined_functions/simple_scalar_sql_udf.rst new file mode 100644 index 000000000..1a78e2e8c --- /dev/null +++ b/reference/sql/sql_functions/user_defined_functions/simple_scalar_sql_udf.rst @@ -0,0 +1,51 @@ +.. _simple_scalar_sql_udf: + +************************** +Simple Scalar SQL UDF's +************************** + +Syntax +~~~~~~~~~~~~ +The following example shows the correct syntax for simple scalar SQL UDF's: + + + .. code-block:: console + + $ create_function_statement ::= + $ CREATE [ OR REPLACE ] FUNCTION function_name (argument_list) + $ RETURNS return_type + $ AS $$ + $ { function_body } + $ $$ LANGUAGE SQL + $ ; + $ + $ function_name ::= identifier + $ argument_list :: = { value_name type_name [, ...] } + $ value_name ::= identifier + $ return_type ::= type_name + $ function_body ::= A valid SQL statement + +Usage Notes +~~~~~~~~~~~~~~ +The following usage notes apply when using simple scalar SQL UDF's: + +* During this stage, the SQL embedded in the function body must be of the type ``SELECT expr;``. Creating a UDF with invalid SQL, or with valid SQL of any other type, results in an error. +* As with Python UDFs, the argument list can be left empty. +* SQL UDFs can reference other UDF's, including Python UDF's. + +**NOTICE:** A function cannot (directly or indirectly) reference itself (such as by referencing another function that references it). + +Because SQL UDF's are one type of supported UDFs, the following Python UDF characteristics apply: + +* UDF permission rules - see `Access Control `_. + +* The ``get_function_ddl`` utility function works on these functions - see `Getting the DDL for a Function `_. + +* SQL UDF's should appear in the catalog with Python UDF's - see `Finding Existing UDFs in the Catalog `_. + +Restrictions +~~~~~~~~~~~~~~~~~~~~~ +The following restrictions apply to simple scalar SQL UDF's: + +* Simple scalar SQL UDF's cannot currently reference other UDF's. +* Like Python UDF's, Sqream does not support overloading. diff --git a/reference/sql/sql_functions/window_functions/cume_dist.rst b/reference/sql/sql_functions/window_functions/cume_dist.rst new file mode 100644 index 000000000..1af80bb78 --- /dev/null +++ b/reference/sql/sql_functions/window_functions/cume_dist.rst @@ -0,0 +1,23 @@ +.. _cume_dist: + +************************** +CUME_DIST +************************** + +The **CUME_DIST** function returns the cumulative distribution for each of your partitioned rows between the range of ``1/n`` (with ``n`` being the total number of rows), and ``1``. For example, in a table with ten rows, the cumulative distance for the first row is ``0.1`` and the second is ``0.2``, etc. The cumulative distance of the final row is always ``1``. + +Syntax +------- +The following shows the correct syntax for the CUME_DIST function. + +.. code-block:: postgres + + CUME_DIST ( ) OVER (...) + +Arguments +--------- +None + +Returns +--------- +Returns the cumulative distribution for each of your partitioned rows. diff --git a/reference/sql/sql_functions/window_functions/dense_rank.rst b/reference/sql/sql_functions/window_functions/dense_rank.rst new file mode 100644 index 000000000..d9aabf5dc --- /dev/null +++ b/reference/sql/sql_functions/window_functions/dense_rank.rst @@ -0,0 +1,23 @@ +.. _dense_rank: + +************************** +DENSE_RANK +************************** + +The **DENSE_RANK** function returns data ranking information and is similar to the **RANK** function. Its main difference is related to the results of identical line values. While the RANK function proceeds with the next row after providing the same value for identical consecutive rows (for example, 1,1,3), the DENSE_RANK function proceeds with the next number in the sequence (1,1,2). + +Syntax +------- +The following shows the correct syntax for the DENSE_RANK function. + +.. code-block:: postgres + + DENSE_RANK ( ) OVER (...) + +Arguments +--------- +None + +Returns +--------- +Returns data ranking information. diff --git a/reference/sql/sql_functions/window_functions/first_value.rst b/reference/sql/sql_functions/window_functions/first_value.rst new file mode 100644 index 000000000..07708872d --- /dev/null +++ b/reference/sql/sql_functions/window_functions/first_value.rst @@ -0,0 +1,24 @@ +.. _first_value: + +************************** +FIRST_VALUE +************************** +The **FIRST_VALUE** function returns the value located in the selected column of the first row of a segment. If the table is not segmented, the FIRST_VALUE function returns the value from the first row of the whole table. + +This function returns the same type of variable that you input for your requested value. For example, requesting the value for the first employee in a list using an ``int`` type output returns an ``int`` type ID column. If you use a ``varchar`` type, the function returns a ``varchar`` type name column. + +Syntax +------- +The following shows the correct syntax for the FIRST_VALUE function. + +.. code-block:: postgres + + FIRST_VALUE(Selected_Column) OVER (...) + +Arguments +--------- +None + +Returns +--------- +Returns the value located in the selected column of the first row of a segment. diff --git a/reference/sql/sql_functions/window_functions/index.rst b/reference/sql/sql_functions/window_functions/index.rst index fffba9a0a..e949e0e3c 100644 --- a/reference/sql/sql_functions/window_functions/index.rst +++ b/reference/sql/sql_functions/window_functions/index.rst @@ -1,23 +1,24 @@ -.. _window_functions_functions: +.. _window_functions: ******************** -Window functions +Window Functions ******************** -Window functions are functions applied over a subset (known as a window) of the rows returned by a :ref:`select` query. +Window functions are functions applied over a subset (known as a window) of the rows returned by a :ref:`select` query and describes the following: -Read more about :ref:`window_functions` in the :ref:`sql_syntax` section. - -.. toctree:: - :maxdepth: 1 - :glob: - :hidden: +.. hlist:: + :columns: 1 - lag - lead - row_number - rank + * :ref:`lag` + * :ref:`lead` + * :ref:`row_number` + * :ref:`rank` + * :ref:`first_value` + * :ref:`last_value` + * :ref:`nth_value` + * :ref:`dense_rank` + * :ref:`percent_rank` + * :ref:`cume_dist` + * :ref:`ntile` -.. include:: /reference/sql/sql_syntax/window_functions.rst - :start-line: 9 - :end-line: 100 +For more information, see :ref:`window_functions` in the :ref:`sql_syntax` section. \ No newline at end of file diff --git a/reference/sql/sql_functions/window_functions/last_value.rst b/reference/sql/sql_functions/window_functions/last_value.rst new file mode 100644 index 000000000..3cadaa9d1 --- /dev/null +++ b/reference/sql/sql_functions/window_functions/last_value.rst @@ -0,0 +1,24 @@ +.. _last_value: + +************************** +LAST_VALUE +************************** +The **LAST_VALUE** function returns the value located in the selected column of the last row of a segment. If the table is not segmented, the LAST_VALUE function returns the value from the last row of the whole table. + +This function returns the same type of variable that you input for your requested value. For example, requesting the value for the last employee in a list using an ``int`` type output returns an ``int`` type ID column. If you use a ``varchar`` type, the function returns a ``varchar`` type name column. + +Syntax +------- +The following shows the correct syntax for the LAST_VALUE function. + +.. code-block:: postgres + + LAST_VALUE(Selected_Column) OVER (...) + +Arguments +--------- +None + +Returns +--------- +Returns the value located in the selected column of the last row of a segment. diff --git a/reference/sql/sql_functions/window_functions/nba-t10.csv b/reference/sql/sql_functions/window_functions/nba-t10.csv index 024530355..c3d41e729 100644 --- a/reference/sql/sql_functions/window_functions/nba-t10.csv +++ b/reference/sql/sql_functions/window_functions/nba-t10.csv @@ -1,10 +1,9 @@ -Name,Team,Number,Position,Age,Height,Weight,College,Salary -Avery Bradley,Boston Celtics,0.0,PG,25.0,6-2,180.0,Texas,7730337.0 -Jae Crowder,Boston Celtics,99.0,SF,25.0,6-6,235.0,Marquette,6796117.0 -John Holland,Boston Celtics,30.0,SG,27.0,6-5,205.0,Boston University, -R.J. Hunter,Boston Celtics,28.0,SG,22.0,6-5,185.0,Georgia State,1148640.0 -Jonas Jerebko,Boston Celtics,8.0,PF,29.0,6-10,231.0,,5000000.0 -Amir Johnson,Boston Celtics,90.0,PF,29.0,6-9,240.0,,12000000.0 -Jordan Mickey,Boston Celtics,55.0,PF,21.0,6-8,235.0,LSU,1170960.0 -Kelly Olynyk,Boston Celtics,41.0,C,25.0,7-0,238.0,Gonzaga,2165160.0 -Terry Rozier,Boston Celtics,12.0,PG,22.0,6-2,190.0,Louisville,1824360.0 +Avery Bradley,Boston Celtics,0,PG,25,2-Jun,180,Texas,7730337 +Jae Crowder,Boston Celtics,99,SF,25,6-Jun,235,Marquette,6796117 +John Holland,Boston Celtics,30,SG,27,5-Jun,205,Boston University, +R.J. Hunter,Boston Celtics,28,SG,22,5-Jun,185,Georgia State,1148640 +Jonas Jerebko,Boston Celtics,8,PF,29,10-Jun,231,,5000000 +Amir Johnson,Boston Celtics,90,PF,29,9-Jun,240,,12000000 +Jordan Mickey,Boston Celtics,55,PF,21,8-Jun,235,LSU,1170960 +Kelly Olynyk,Boston Celtics,41,C,25,Jul-00,238,Gonzaga,2165160 +Terry Rozier,Boston Celtics,12,PG,22,2-Jun,190,Louisville,1824360 diff --git a/reference/sql/sql_functions/window_functions/nth_value.rst b/reference/sql/sql_functions/window_functions/nth_value.rst new file mode 100644 index 000000000..a2c1dd9a6 --- /dev/null +++ b/reference/sql/sql_functions/window_functions/nth_value.rst @@ -0,0 +1,74 @@ +.. _nth_value: + +************************** +NTH_VALUE +************************** + +The **NTH_VALUE** function returns the value located in the selected column of a specified row of a segment. While the NTH_VALUE function is identical to the FIRST_VALUE and LAST_VALUE functions, it requires you to add a literal and whole ``int`` type ``n`` value representing the row number containing the value that you want. If you select an ``n`` value larger than the number of rows in the table, the NTH_VALUE function returns ``NULL``. + +Syntax +------- +The following shows the correct syntax for the NTH_VALUE function. + +.. code-block:: postgres + + NTH_VALUE(Selected_Column, n) OVER (...) + +Examples +--------- +The following example shows the syntax for a table named ``superstore`` used for tracking item sales in thousands: + +.. code-block:: postgres + + CREATE TABLE superstore + ( + "Section" varchar(40), + "Product_Name" varchar(40), + "Sales_In_K" int, + ); + +The following example shows the output of the syntax above: + +.. list-table:: + :widths: 24 26 50 + :header-rows: 1 + + * - Section + - Product_Name + - Sales_In_K + * - bed + - pillow_case + - 17 + * - bath + - rubber_duck + - 22 + * - beyond + - curtain + - 15 + * - bed + - side_table + - 8 + * - bath + - shampoo + - 9 + * - bed + - blanket + - 7 + * - bath + - bath_bomb + - 13 + * - bath + - conditioner + - 17 + * - beyond + - lamp + - 7 + * - bath + - soap + - 13 + * - beyond + - rug + - 12 + * - bed + - pillow + - 17 diff --git a/reference/sql/sql_functions/window_functions/ntile.rst b/reference/sql/sql_functions/window_functions/ntile.rst new file mode 100644 index 000000000..d40b197bf --- /dev/null +++ b/reference/sql/sql_functions/window_functions/ntile.rst @@ -0,0 +1,22 @@ +.. _ntile: + +************************** +NTILE +************************** + +The **NTILE** function returns an integer ranging between ``1`` and the argument value, and divides a table's rows into ``n`` ranked buckets according to the following: + +* The selected column. +* The selected positive, literal ``n``. + +Selecting an ``n`` that exceeds the number of rows in a table outputs a result identical to the RANK function. When this happens, each ranked bucket is populated with a single value and any bucket numbered higher than the number of rows not even being created. + +Additionally, if the number of rows in a bucket is not exactly divisible by ``n``, the function gives precedence to values at the top. For example, if ``n`` is ``6`` for 63 values, the top three buckets each have 11 values, and the bottom three each have 10. + +Syntax +------- +The following shows the correct syntax for the NTILE function. + +.. code-block:: postgres + + NTILE(n) OVER (...) diff --git a/reference/sql/sql_functions/window_functions/percent_rank.rst b/reference/sql/sql_functions/window_functions/percent_rank.rst new file mode 100644 index 000000000..9f1120a46 --- /dev/null +++ b/reference/sql/sql_functions/window_functions/percent_rank.rst @@ -0,0 +1,28 @@ +.. _percent_rank: + +************************** +PERCENT_RANK +************************** + +The **PERCENT_RANK** function returns relative data ranking information and is similar to the **RANK** function. + +It differs from the RANK function in the following ways: + +* The output type is ``double`` instead of ``bigint``. +* Shows ranking as a *relative* rank number (for example, a number between and including ``0`` and ``1``), and not an *absolute* rank number (for example, ``1``, ``1``, ``3``). + +Syntax +------- +The following shows the correct syntax for the PERCENT_RANK function. + +.. code-block:: postgres + + PERCENT_RANK ( ) OVER (...) + +Arguments +--------- +None + +Returns +--------- +Returns relative data ranking information in the form of a number between (and including) ``0`` and ``1``. diff --git a/reference/sql/sql_statements/access_control_commands/alter_default_permissions.rst b/reference/sql/sql_statements/access_control_commands/alter_default_permissions.rst index bef17f22f..a49e53f88 100644 --- a/reference/sql/sql_statements/access_control_commands/alter_default_permissions.rst +++ b/reference/sql/sql_statements/access_control_commands/alter_default_permissions.rst @@ -3,22 +3,27 @@ ***************************** ALTER DEFAULT PERMISSIONS ***************************** +The **ALTER DEFAULT PERMISSIONS** page describes the following: -``ALTER DEFAULT PERMISSIONS`` allows granting automatic permissions to future objects. +.. contents:: + :local: + :depth: 1 -By default, if one user creates a table, another user will not have ``SELECT`` permissions on it. -By modifying the target role's default permissions, a database administrator can ensure that -all objects created by that role will be accessible to others. +Overview +============= +The ``ALTER DEFAULT PERMISSIONS`` command lets you grant automatic permissions to future objects. + +By default, users do not have ``SELECT`` permissions on tables created by other users. Database administrators can grant access to other users by modifying the target role default permissions. -Learn more about the permission system in the :ref:`access control guide`. +For more information about access control, see :ref:`Access Control`. Permissions ============= - -To alter default permissions, the current role must have the ``SUPERUSER`` permission. +The ``SUPERUSER`` permission is required to alter default permissions. Syntax ========== +The following is the syntax for altering default permissions: .. code-block:: postgres @@ -38,6 +43,7 @@ Syntax | USAGE | SELECT | INSERT + | UPDATE | DELETE | DDL | EXECUTE @@ -55,31 +61,70 @@ Syntax :start-line: 127 :end-line: 180 - Examples ============ +This section includes the following examples: -Automatic permissions for newly created schemas +.. contents:: + :local: + :depth: 1 + +Granting Default Table Permissions ------------------------------------------------- +This example is based on the roles **r1** and **r2**, created as follows: -When role ``demo`` creates a new schema, roles u1,u2 will get USAGE and CREATE permissions in the new schema: +.. code-block:: postgres + + create role r1; + create role r2; + alter default permissions for r1 for tables grant select to r2; + +Once created, you can build and run the following query based on the above: .. code-block:: postgres - ALTER DEFAULT PERMISSIONS FOR demo FOR SCHEMAS GRANT USAGE, CREATE TO u1,u2; + select + tdp.database_name as "database_name", + ss.schema_name as "schema_name", + rs1.name as "table_creator", + rs2.name as "grant_to", + pts.name as "permission_type" + from sqream_catalog.table_default_permissions tdp + inner join sqream_catalog.roles rs1 on tdp.modifier_role_id = rs1.role_id + inner join sqream_catalog.roles rs2 on tdp.getter_role_id = rs2.role_id + left join sqream_catalog.schemas ss on tdp.schema_id = ss.schema_id + inner join sqream_catalog.permission_types pts on pts.permission_type_id=tdp.permission_type + ; + +The following is an example of the output generated from the above queries: ++-----------------------+----------------------+-------------------+--------------+------------------------------+ +| **database_name** | **schema_name** | **table_creator** | **grant_to** | **permission_type** | ++-----------------------+----------------------+-------------------+--------------+------------------------------+ +| master | NULL | public | public | select | ++-----------------------+----------------------+-------------------+--------------+------------------------------+ -Automatic permissions for newly created tables in a schema ----------------------------------------------------------------- +For more information about default permissions, see `Default Permissions `_. + +Granting Automatic Permissions for Newly Created Schemas +------------------------------------------------- +When the role ``demo`` creates a new schema, roles **u1,u2** are granted ``USAGE`` and ``CREATE`` permissions in the new schema, as shown below: -When role ``demo`` creates a new table in schema ``s1``, roles u1,u2 wil be granted with SELECT on it: +.. code-block:: postgres + + ALTER DEFAULT PERMISSIONS FOR demo FOR SCHEMAS GRANT USAGE, CREATE TO u1,u2; + +Granting Automatic Permissions for Newly Created Tables in a Schema +---------------------------------------------------------------- +When the role ``demo`` creates a new table in schema ``s1``, roles **u1,u2** are granted ``SELECT`` permissions, as shown below: .. code-block:: postgres ALTER DEFAULT PERMISSIONS FOR demo IN s1 FOR TABLES GRANT SELECT TO u1,u2; -Revoke (``DROP GRANT``) permissions for newly created tables +Revoking Permissions from Newly Created Tables --------------------------------------------------------------- +Revoking permissions refers to using the ``DROP GRANT`` command, as shown below: .. code-block:: postgres diff --git a/reference/sql/sql_statements/access_control_commands/grant.rst b/reference/sql/sql_statements/access_control_commands/grant.rst index dfb48c212..d86522900 100644 --- a/reference/sql/sql_statements/access_control_commands/grant.rst +++ b/reference/sql/sql_statements/access_control_commands/grant.rst @@ -15,13 +15,9 @@ Learn more about the permission system in the :ref:`access control guide`, and use a more flexible foreign data wrapper concept. See :ref:`create_foreign_table` instead. - - Upgrading to a new version of SQream DB converts existing tables automatically. When creating a new external tables, use the new foreign table syntax. - - -``CREATE TABLE`` creates a new external table in an existing database. - -See more in the :ref:`External tables guide`. - -.. tip:: - - * Data in an external table can change if the sources change, and frequent access to remote files may harm performance. - - * To create a regular table, see :ref:`CREATE TABLE ` - -Permissions -============= - -The role must have the ``CREATE`` permission at the database level. - -Syntax -========== - -.. code-block:: postgres - - create_table_statement ::= - CREATE [ OR REPLACE ] EXTERNAL TABLE [schema_name].table_name ( - { column_def [, ...] } - ) - USING FORMAT format_def - WITH { external_table_option [ ...] } - ; - - schema_name ::= identifier - - table_name ::= identifier - - format_def ::= { PARQUET | ORC | CSV } - - external_table_option ::= { - PATH '{ path_spec }' - | FIELD DELIMITER '{ field_delimiter }' - | RECORD DELIMITER '{ record_delimiter }' - | AWS_ID '{ AWS ID }' - | AWS_SECRET '{ AWS SECRET }' - } - - path_spec ::= { local filepath | S3 URI | HDFS URI } - - field_delimiter ::= delimiter_character - - record_delimiter ::= delimiter_character - - column_def ::= { column_name type_name [ default ] [ column_constraint ] } - - column_name ::= identifier - - column_constraint ::= - { NOT NULL | NULL } - - default ::= - - DEFAULT default_value - | IDENTITY [ ( start_with [ , increment_by ] ) ] - -.. _cet_parameters: - -Parameters -============ - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Parameter - - Description - * - ``OR REPLACE`` - - Create a new table, and overwrite any existing table by the same name. Does not return an error if the table already exists. ``CREATE OR REPLACE`` does not check the table contents or structure, only the table name. - * - ``schema_name`` - - The name of the schema in which to create the table. - * - ``table_name`` - - The name of the table to create, which must be unique inside the schema. - * - ``column_def`` - - A comma separated list of column definitions. A minimal column definition includes a name identifier and a datatype. Other column constraints and default values can be added optionally. - * - ``USING FORMAT ...`` - - Specifies the format of the source files, such as ``PARQUET``, ``ORC``, or ``CSV``. - * - ``WITH PATH ...`` - - Specifies a path or URI of the source files, such as ``/path/to/*.parquet``. - * - ``FIELD DELIMITER`` - - Specifies the field delimiter for CSV files. Defaults to ``,``. - * - ``RECORD DELIMITER`` - - Specifies the record delimiter for CSV files. Defaults to a newline, ``\n`` - * - ``AWS_ID``, ``AWS_SECRET`` - - Credentials for authenticated S3 access - - -Examples -=========== - -A simple table from Tab-delimited file (TSV) ----------------------------------------------- - -.. code-block:: postgres - - CREATE OR REPLACE EXTERNAL TABLE cool_animals - (id INT NOT NULL, name VARCHAR(30) NOT NULL, weight FLOAT NOT NULL) - USING FORMAT csv - WITH PATH '/home/rhendricks/cool_animals.csv' - FIELD DELIMITER '\t'; - - -A table from a directory of Parquet files on HDFS ------------------------------------------------------ - -.. code-block:: postgres - - CREATE EXTERNAL TABLE users - (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL) - USING FORMAT Parquet - WITH PATH 'hdfs://hadoop-nn.piedpiper.com/rhendricks/users/*.parquet'; - -A table from a bucket of files on S3 --------------------------------------- - -.. code-block:: postgres - - CREATE EXTERNAL TABLE users - (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL) - USING FORMAT Parquet - WITH PATH 's3://pp-secret-bucket/users/*.parquet' - AWS_ID 'our_aws_id' - AWS_SECRET 'our_aws_secret'; - - -Changing an external table to a regular table ------------------------------------------------- - -Materializes an external table into a regular table. - -.. tip: Using an external table allows you to perform ETL-like operations in SQream DB by applying SQL functions and operations to raw files - -.. code-block:: postgres - - CREATE TABLE real_table - AS SELECT * FROM external_table; - diff --git a/reference/sql/sql_statements/ddl_commands/create_foreign_table.rst b/reference/sql/sql_statements/ddl_commands/create_foreign_table.rst index d50e13380..cc87c7b8f 100644 --- a/reference/sql/sql_statements/ddl_commands/create_foreign_table.rst +++ b/reference/sql/sql_statements/ddl_commands/create_foreign_table.rst @@ -113,7 +113,7 @@ A simple table from Tab-delimited file (TSV) .. code-block:: postgres CREATE OR REPLACE FOREIGN TABLE cool_animals - (id INT NOT NULL, name VARCHAR(30) NOT NULL, weight FLOAT NOT NULL) + (id INT NOT NULL, name TEXT(30) NOT NULL, weight FLOAT NOT NULL) WRAPPER csv_fdw OPTIONS ( LOCATION = '/home/rhendricks/cool_animals.csv', @@ -128,7 +128,7 @@ A table from a directory of Parquet files on HDFS .. code-block:: postgres CREATE FOREIGN TABLE users - (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL) + (id INT NOT NULL, name TEXT(30) NOT NULL, email TEXT(50) NOT NULL) WRAPPER parquet_fdw OPTIONS ( @@ -141,7 +141,7 @@ A table from a bucket of ORC files on S3 .. code-block:: postgres CREATE FOREIGN TABLE users - (id INT NOT NULL, name VARCHAR(30) NOT NULL, email VARCHAR(50) NOT NULL) + (id INT NOT NULL, name TEXT(30) NOT NULL, email TEXT(50) NOT NULL) WRAPPER orc_fdw OPTIONS ( diff --git a/reference/sql/sql_statements/ddl_commands/create_table.rst b/reference/sql/sql_statements/ddl_commands/create_table.rst index 83fbe568d..c5f80c294 100644 --- a/reference/sql/sql_statements/ddl_commands/create_table.rst +++ b/reference/sql/sql_statements/ddl_commands/create_table.rst @@ -4,19 +4,17 @@ CREATE TABLE ***************** -``CREATE TABLE`` creates a new table in an existing database. +The ``CREATE TABLE`` statement is used to create a new table in an existing database. .. tip:: * To create a table based on the result of a select query, see :ref:`CREATE TABLE AS `. * To create a table based on files like Parquet and ORC, see :ref:`CREATE FOREIGN TABLE ` -Permissions -============= -The role must have the ``CREATE`` permission at the schema level. Syntax ========== +The following is the correct syntax for creating a table: .. code-block:: postgres @@ -44,6 +42,7 @@ Syntax Parameters ============ +The following parameters can be used when creating a table: .. list-table:: :widths: auto @@ -52,7 +51,7 @@ Parameters * - Parameter - Description * - ``OR REPLACE`` - - Create a new table, and overwrite any existing table by the same name. Does not return an error if the table already exists. ``CREATE OR REPLACE`` does not check the table contents or structure, only the table name. + - Creates a new tables and overwrites any existing table by the same name. Does not return an error if the table already exists. ``CREATE OR REPLACE`` does not check the table contents or structure, only the table name. * - ``schema_name`` - The name of the schema in which to create the table. * - ``table_name`` @@ -63,21 +62,26 @@ Parameters - A commma separated list of clustering column keys. - See :ref:`data_clustering` for more information. - + See :ref:`flexible_data_clustering` for more information. + * - ``LIKE`` + - Duplicates the column structure of an existing table. + + .. _default_values: -Default values +Default Value Constraints =============== -The ``DEFAULT`` value constraint specifies a value to use if a value isn't defined in an :ref:`insert` or :ref:`copy_from` statement. +The ``DEFAULT`` value constraint specifies a value to use if one is not defined in an :ref:`insert` or :ref:`copy_from` statement. -The value may be either a literal or `GETDATE()`, which is s evaluated at the time the row is created. +The value may either be a literal, **GETDATE()**, or Null, which is evaluated at the time the row is created. .. note:: The ``DEFAULT`` constraint only applies if the column does not have a value specified in the :ref:`insert` or :ref:`copy_from` statement. You can still insert a ``NULL`` into an nullable column by explicitly inserting ``NULL``. For example, ``INSERT INTO cool_animals VALUES (1, 'Gnu', NULL)``. Syntax --------- +The following is the correct syntax for using the **DEFAULT** value constraints: + .. code-block:: postgres @@ -99,15 +103,16 @@ Syntax .. _identity: -Identity (sequence) +Identity ----------------------- +The ``Identity`` (or sequence) columns can be used for generating key values. Some databases call this ``AUTOINCREMENT``. -Identity columns can be used for generating key values. Some databases call this ``AUTOINCREMENT``. - -The identity property on a column guarantees that each new row inserted is generated based on the current seed & increment. +The **identity** property on a column guarantees that each new row inserted is generated based on the current seed & increment. .. warning:: The identity property on a column does not guarantee uniqueness. The identity value can be bypassed by specifying it in an :ref:`insert` command. + +The following table describes the identity parameters: .. list-table:: :widths: auto @@ -122,9 +127,15 @@ The identity property on a column guarantees that each new row inserted is gener Examples =========== +This section includes the following examples: -A simple table +.. contents:: + :local: + :depth: 1 + +Creating a Standard Table ----------------- +The following is an example of the syntax used to create a standard table: .. code-block:: postgres @@ -135,8 +146,10 @@ A simple table is_agressive BOOL ); -A table with default values for some columns +Creating a Table with Default Value Constraints for Some Columns --------------------------------------------------- +The following is an example of the syntax used to create a table with default value constraints for some columns: + .. code-block:: postgres @@ -149,15 +162,17 @@ A table with default values for some columns .. note:: The nullable/non-nullable constraint appears at the end, after the default option -A table with an identity (autoincrement) column +Creating a Table with an Identity Column --------------------------------------------------- +The following is an example of the syntax used to create a table with an identity (auto-increment) column: + .. code-block:: postgres CREATE TABLE users ( id BIGINT IDENTITY(0,1) NOT NULL , -- Start with 0, increment by 1 - name VARCHAR(30) NOT NULL, - country VARCHAR(30) DEFAULT 'Unknown' NOT NULL + name varchar(30) NOT NULL, + country varchar(30) DEFAULT 'Unknown' NOT NULL ); .. note:: @@ -165,31 +180,117 @@ A table with an identity (autoincrement) column * Identity does not enforce the uniqueness of values. The identity value can be bypassed by specifying it in an :ref:`insert` command. -Creating a table from a SELECT query +Creating a Table from a SELECT Query ----------------------------------------- - -Use a :ref:`CREATE TABLE AS ` statement to create a new table from the results of a SELECT query. +The following is an example of the syntax used to create a table from a SELECT query: .. code-block:: postgres CREATE TABLE users_uk AS SELECT * FROM users WHERE country = 'United Kingdom'; + +For more information on creating a new table from the results of a SELECT query, see :ref:`CREATE TABLE AS `. -Creating a table with a clustering key +Creating a Table with a Clustering Key ---------------------------------------------- - When data in a table is stored in a sorted order, the sorted columns are considered clustered. Good clustering can have a significant positive impact on performance. In the following example, we expect the ``start_date`` column to be naturally clustered, as new users sign up and get a newer start date. When the clustering key is set, if the incoming data isn’t naturally clustered, it will be clustered by SQream DB during insert or bulk load. -See :ref:`data_clustering` for more information. +The following is an example of the syntax used to create a table with a clustering key: .. code-block:: postgres CREATE TABLE users ( - name VARCHAR(30) NOT NULL, + name varchar(30) NOT NULL, start_date datetime not null, - country VARCHAR(30) DEFAULT 'Unknown' NOT NULL + country varchar(30) DEFAULT 'Unknown' NOT NULL ) CLUSTER BY start_date; + +For more information on data clustering, see :ref:`data_clustering`. + +Duplicating the Column Structure of an Existing Table +----------------- + +Syntax +************ +The following is the correct syntax for duplicating the column structure of an existing table: + +.. code-block:: postgres + + CREATE [OR REPLACE] TABLE table_name + { + (column_name column_type [{NULL | NOT NULL}] [,...]) + | LIKE source_table_name + } + [CLUSTER BY ...] + ; + +Examples +************** +This section includes the following examples of duplicating the column structure of an existing table using the ``LIKE`` clause: + +.. contents:: + :local: + :depth: 3 + +Creating a Table Using an Explicit Column List +~~~~~~~~~~~~ +The following is an example of creating a table using an explict column list: + +.. code-block:: postgres + + CREATE TABLE t1(x int default 0 not null, y text(10) null); + +Creating a Second Table Based on the Structure of Another Table +~~~~~~~~~~~~ +Either of the following examples can be used to create a second table based on the structure of another table. + +**Example 1** + +.. code-block:: postgres + + CREATE TABLE t2 LIKE t1; + +**Example 2** +.. code-block:: postgres + + CREATE TABLE t2(x int default 0 not null, y text(10) null); + +The generated output of both of the statements above is identical. + +Creating a Table based on Foreign Tables and Views +~~~~~~~~~~~~ +The following is example of creating a table based on foreign tables and views: + + +.. code-block:: postgres + + CREATE VIEW v as SELECT x+1,y,y || 'abc' from t1; + CREATE TABLE t3 LIKE v; + +When duplicating the column structure of an existing table, the target table of the ``LIKE`` clause can be a regular or an external table, or a view. + +The following table describes the properties that must be copied from the target table: + ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| **Property** | **Native Table** | **External Table** | **View** | ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| Column names | Must be copied | Must be copied | Must be copied | ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| Column types | Must be copied | Must be copied | Must be copied | ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| ``NULL``/``NOT NULL`` | Must be copied | Must be copied | Must be copied | ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| ``text`` length constraints | Must be copied | Must be copied | Does not exist in source object | ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| Compression specification | Must be copied | Does not exist in source object | Does not exist in source object | ++-----------------------------+------------------+---------------------------------+---------------------------------+ +| Default/identity | Must be copied | Does not exist in source object | Does not exist in source object | ++-----------------------------+------------------+---------------------------------+---------------------------------+ + +Permissions +============= +The role must have the ``CREATE`` permission at the schema level. \ No newline at end of file diff --git a/reference/sql/sql_statements/ddl_commands/create_table_as.rst b/reference/sql/sql_statements/ddl_commands/create_table_as.rst index 19aaf8128..a7f9dd4d4 100644 --- a/reference/sql/sql_statements/ddl_commands/create_table_as.rst +++ b/reference/sql/sql_statements/ddl_commands/create_table_as.rst @@ -4,14 +4,13 @@ CREATE TABLE AS ***************** -``CREATE TABLE AS`` creates a new table from the result of a select query. +The ``CREATE TABLE AS`` commands creates a new table from the result of a select query. -Permissions -============= -The role must have the ``CREATE`` permission at the schema level, as well as ``SELECT`` permissions for any tables referenced by the statement. Syntax ========== +The following is the correct syntax for creating a table from the result of a select query: + .. CREATE [ OR REPLACE ] TABLE [schema_name].table_name ( .. { column_def [, ...] } @@ -32,6 +31,7 @@ Syntax Parameters ============ +The following parameters can be used when creating a table from the result of a select query: .. list-table:: :widths: auto @@ -51,15 +51,27 @@ Parameters .. * - ``column_def`` .. - A comma separated list of column definitions. A minimal column definition includes a name identifier and a datatype. Other column constraints and default values can be added optionally. +Permissions +============= +The role must have the ``CREATE`` permission at the schema level, as well as ``SELECT`` permissions for any tables referenced by the statement. + + Examples =========== +This section includes the following examples: -Create a copy of an :ref:`foreign table ` or view +.. contents:: + :local: + :depth: 1 + +Creating a Copy of a Foreign Table or View --------------------------------------------------------------------------- .. code-block:: postgres CREATE TABLE users AS SELECT * FROM users_source; + +For more information, see :ref:`CREATE FOREIGN TABLE `. Filtering ------------ @@ -68,14 +80,14 @@ Filtering CREATE TABLE users_uk AS SELECT * FROM users WHERE country = 'United Kingdom'; -Adding columns +Adding Columns ----------------------- .. code-block:: postgres CREATE TABLE users_uk_new AS SELECT GETDATE() as "Date",*,false as is_new FROM users_uk; -Creating a table from values +Creating a Table From Values ----------------------------------------- .. code-block:: postgres diff --git a/reference/sql/sql_statements/ddl_commands/rename_column.rst b/reference/sql/sql_statements/ddl_commands/rename_column.rst index f91933f71..1022ce0f2 100644 --- a/reference/sql/sql_statements/ddl_commands/rename_column.rst +++ b/reference/sql/sql_statements/ddl_commands/rename_column.rst @@ -3,16 +3,11 @@ ********************** RENAME COLUMN ********************** - -``RENAME COLUMN`` can be used to rename columns in a table. - -Permissions -============= - -The role must have the ``DDL`` permission at the database or table level. +The ``RENAME COLUMN`` command can be used to rename columns in a table. Syntax ========== +The following is the correct syntax for the ``RENAME_COLUMN`` command: .. code-block:: postgres @@ -30,6 +25,7 @@ Syntax Parameters ============ +The following table describes the `RENAME_COLUMN`` parameters: .. list-table:: :widths: auto @@ -48,18 +44,29 @@ Parameters Examples =========== +The **Examples** section includes the following examples: -Renaming a column +.. contents:: + :local: + :depth: 1 + +Renaming a Column ----------------------------------------- +The following is an example of renaming a column: .. code-block:: postgres -- Remove the 'weight' column ALTER TABLE users RENAME COLUMN weight TO mass; -Renaming a quoted name +Renaming a Quoted Name -------------------------- +The following is an example of renaming a quoted name: .. code-block:: postgres - ALTER TABLE users RENAME COLUMN "mass" TO "Mass (Kilograms); \ No newline at end of file + ALTER TABLE users RENAME COLUMN "mass" TO "Mass (Kilograms); + +Permissions +============= +The role must have the ``DDL`` permission at the database or table level. \ No newline at end of file diff --git a/reference/sql/sql_statements/dml_commands/copy_from.rst b/reference/sql/sql_statements/dml_commands/copy_from.rst index 960a3f89c..05d5083aa 100644 --- a/reference/sql/sql_statements/dml_commands/copy_from.rst +++ b/reference/sql/sql_statements/dml_commands/copy_from.rst @@ -4,11 +4,7 @@ COPY FROM ********************** -``COPY ... FROM`` is a statement that allows loading data from files on the filesystem and importing them into SQream tables. - -This is the recommended way for bulk loading CSV files into SQream DB. - -In general, ``COPY`` moves data between filesystem files and SQream DB tables. +``COPY ... FROM`` is a statement that allows loading data from files on the filesystem and importing them into SQream tables. This is the recommended way for bulk loading CSV files into SQream DB. In general, ``COPY`` moves data between filesystem files and SQream DB tables. .. note:: * Learn how to migrate from CSV files in the :ref:`csv` guide @@ -41,6 +37,8 @@ Syntax LOCATION = { filename | S3 URI | HDFS URI } + | QUOTE = {'C' | E'\ooo') + | OFFSET = { offset } | LIMIT = { limit } @@ -103,6 +101,10 @@ Elements - None - - Table to copy data into + * - ``QUOTE`` + - " + - + - Specifies an alternative quote character. The quote character must be a single, 1-byte printable ASCII character, and the equivalent octal syntax of the copy command can be used. The quote character cannot be contained in the field delimiter, the record delimiter, or the null marker. ``QUOTE`` can be used with ``csv_fdw`` in **COPY FROM** and foreign tables. * - ``name_fdw`` - - ``csv_fdw``, ``orc_fdw``, or ``parquet_fdw`` @@ -177,7 +179,7 @@ Elements .. _copy_date_parsers: -Supported date formats +Supported Date Formats ========================= .. list-table:: Supported date parsers @@ -251,19 +253,63 @@ Supported date formats .. _field_delimiters: -Supported field delimiters +Supported Field Delimiters ===================================================== Field delimiters can be one or more characters. -Multi-character delimiters +Customizing Quotations Using Alternative Characters +---------------------------- + +Syntax Example 1 - Customizing Quotations Using Alternative Characters +************ + +The following is the correct syntax for customizing quotations using alternative characters: + +.. code-block:: postgres + + copy t from wrapper csv_fdw options (location = '/tmp/source_file.csv', quote='@'); + copy t to wrapper csv_fdw options (location = '/tmp/destination_file.csv', quote='@'); + +Usage Example 1 - Customizing Quotations Using Alternative Characters +************ + +The following is an example of line taken from a CSV when customizing quotations using a character: + +.. code-block:: postgres + + Pepsi-"Cola",@Coca-"Cola"@,Sprite,Fanta + + +Syntax Example 2 - Customizing Quotations Using ASCII Character Codes +************ + +The following is the correct syntax for customizing quotations using ASCII character codes: + +.. code-block:: postgres + + copy t from wrapper csv_fdw options (location = '/tmp/source_file.csv', quote=E'\064'); + copy t to wrapper csv_fdw options (location = '/tmp/destination_file.csv', quote=E'\064'); + +Usage Example 2 - Customizing Quotations Using ASCII Character Codes +************ + +The following is an example of line taken from a CSV when customizing quotations using an ASCII character code: + +.. code-block:: postgres + + Pepsi-"Cola",@Coca-"Cola"@,Sprite,Fanta + + + +Multi-Character Delimiters ---------------------------------- SQream DB supports multi-character field delimiters, sometimes found in non-standard files. A multi-character delimiter can be specified. For example, ``DELIMITER '%%'``, ``DELIMITER '{~}'``, etc. -Printable characters +Printable Characters ----------------------- Any printable ASCII character (or characters) can be used as a delimiter without special syntax. The default CSV field delimiter is a comma (``,``). @@ -272,7 +318,7 @@ A printable character is any ASCII character in the range 32 - 126. :ref:`Literal quoting rules` apply with delimiters. For example, to use ``'`` as a field delimiter, use ``DELIMITER ''''`` -Non-printable characters +Non-Printable Characters ---------------------------- A non-printable character (1 - 31, 127) can be used in its octal form. @@ -283,7 +329,47 @@ For example, ASCII character ``15``, known as "shift in", can be specified using .. _capturing_rejected_rows: -Capturing rejected rows +Unsupported Field Delimiters +========================== +The following ASCII field delimiters (octal range 001 - 176) are not supported: + ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| **Character** | **Decimal** | **Symbol** | **Character** | **Decimal** | **Symbol** | **Character** | **Decimal** | **Symbol** | ++===============+=============+============+===============+=============+============+===============+=============+============+ +| - | 45 | 55 | b | 98 | 142 | q | 113 | 161 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| . | 46 | 56 | c | 99 | 143 | r | 114 | 162 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| : | 58 | 72 | d | 100 | 144 | s | 115 | 163 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| \ | 92 | 134 | e | 101 | 145 | t | 116 | 164 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 0 | 48 | 60 | f | 102 | 146 | u | 117 | 165 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 1 | 49 | 61 | g | 103 | 147 | v | 118 | 166 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 2 | 50 | 62 | h | 104 | 150 | w | 119 | 167 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 3 | 51 | 63 | i | 105 | 151 | x | 120 | 170 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 4 | 52 | 64 | j | 106 | 152 | y | 121 | 171 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 5 | 53 | 65 | k | 107 | 153 | z | 122 | 172 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 6 | 54 | 66 | l | 108 | 154 | N | 78 | 116 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 7 | 55 | 67 | m | 109 | 155 | 10 | 49 | 12 | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ +| 8 | 56 | 70 | n | 110 | 156 | 13 | 49 | 13 | ++---------------+-------------+------------+---------------+-------------+------------+ | | | +| 9 | 57 | 71 | o | 111 | 157 | | | | ++---------------+-------------+------------+---------------+-------------+------------+ | | | +| a | 97 | 141 | p | 112 | 160 | | | | ++---------------+-------------+------------+---------------+-------------+------------+---------------+-------------+------------+ + + + +Capturing Rejected Rows ========================== Prior to the column process and storage, the ``COPY`` command parses the data. @@ -298,12 +384,12 @@ Whenever the data can’t be parsed because it is improperly formatted or doesn #. When ``ERROR_LOG`` is set and ``ERROR_VERBOSITY`` is set to ``0``, rejected rows are saved to the file path specified, but errors are not logged. This is useful for replaying the file later. -CSV support +CSV Support ================ By default, SQream DB's CSV parser can handle `RFC 4180 standard CSVs `_ , but can also be modified to support non-standard CSVs (with multi-character delimiters, unquoted fields, etc). -All CSV files shoudl be prepared according to these recommendations: +All CSV files should be prepared according to these recommendations: * Files are UTF-8 or ASCII encoded @@ -324,7 +410,7 @@ All CSV files shoudl be prepared according to these recommendations: Other modes of escaping are not supported (e.g. ``1,"What are \"birds\"?"`` is not a valid way of escaping CSV values). -Null markers +Marking Null Markers --------------- ``NULL`` values can be marked in two ways in the CSV: @@ -337,7 +423,7 @@ Null markers Examples =========== -Loading a standard CSV file +Loading a Standard CSV File ------------------------------ .. code-block:: postgres @@ -345,7 +431,7 @@ Loading a standard CSV file COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.csv'); -Skipping faulty rows +Skipping Faulty Rows ------------------------------ .. code-block:: postgres @@ -353,7 +439,7 @@ Skipping faulty rows COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.csv', continue_on_error = true); -Skipping at most 100 faulty rows +Skipping a Maximum of 100 Faulty Rows ----------------------------------- .. code-block:: postgres @@ -361,14 +447,14 @@ Skipping at most 100 faulty rows COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.csv', continue_on_error = true, error_count = 100); -Loading a PSV (pipe separated value) file +Loading a Pipe Separated Value (PSV) File ------------------------------------------- .. code-block:: postgres COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.psv', delimiter = '|'); -Loading a TSV (tab separated value) file +Loading a Tab Separated Value (TSV) File ------------------------------------------- .. code-block:: postgres @@ -376,7 +462,7 @@ Loading a TSV (tab separated value) file COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.tsv', delimiter = '\t'); -Loading a ORC file +Loading an ORC File ------------------------------------------- .. code-block:: postgres @@ -384,7 +470,7 @@ Loading a ORC file COPY table_name FROM WRAPPER orc_fdw OPTIONS (location = '/tmp/file.orc'); -Loading a Parquet file +Loading a Parquet File ------------------------------------------- .. code-block:: postgres @@ -392,7 +478,7 @@ Loading a Parquet file COPY table_name FROM WRAPPER parquet_fdw OPTIONS (location = '/tmp/file.parquet'); -Loading a text file with non-printable delimiter +Loading a Text File with Non-Printable Delimiters ----------------------------------------------------- In the file below, the separator is ``DC1``, which is represented by ASCII 17 decimal or 021 octal. @@ -401,7 +487,7 @@ In the file below, the separator is ``DC1``, which is represented by ASCII 17 de COPY table_name FROM WRAPPER psv_fdw OPTIONS (location = '/tmp/file.txt', delimiter = E'\021'); -Loading a text file with multi-character delimiters +Loading a Text File with Multi-Character Delimiters ----------------------------------------------------- In the file below, the separator is ``^|``. @@ -417,7 +503,7 @@ In the file below, the separator is ``'|``. The quote character has to be repeat COPY table_name FROM WRAPPER psv_fdw OPTIONS (location = '/tmp/file.txt', delimiter = ''''|'); -Loading files with a header row +Loading Files with a Header Row ----------------------------------- Use ``OFFSET`` to skip rows. @@ -428,14 +514,14 @@ Use ``OFFSET`` to skip rows. COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.psv', delimiter = '|', offset = 2); -Loading files formatted for Windows (``\r\n``) +Loading Files Formatted for Windows (``\r\n``) --------------------------------------------------- .. code-block:: postgres COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/file.psv', delimiter = '\r\n'); -Loading a file from a public S3 bucket +Loading a File from a Public S3 Bucket ------------------------------------------ .. note:: The bucket must be publicly available and objects can be listed @@ -444,14 +530,14 @@ Loading a file from a public S3 bucket COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = 's3://sqream-demo-data/file.csv', delimiter = '\r\n', offset = 2); -Loading files from an authenticated S3 bucket +Loading Files from an Authenticated S3 Bucket --------------------------------------------------- .. code-block:: postgres COPY table_name FROM WRAPPER psv_fdw OPTIONS (location = 's3://secret-bucket/*.csv', DELIMITER = '\r\n', OFFSET = 2, AWS_ID = '12345678', AWS_SECRET = 'super_secretive_secret'); -Saving rejected rows to a file +Saving Rejected Rows to a File ---------------------------------- .. note:: When loading multiple files (e.g. with wildcards), this error threshold is for the entire transaction. @@ -474,14 +560,14 @@ Saving rejected rows to a file ); -Load CSV files from a set of directories +Loading CSV Files from a Set of Directories ------------------------------------------ .. code-block:: postgres COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/2019_08_*/*.csv'); -Rearrange destination columns +Rearranging Destination Columns --------------------------------- When the source of the files does not match the table structure, tell the ``COPY`` command what the order of columns should be @@ -492,7 +578,7 @@ When the source of the files does not match the table structure, tell the ``COPY .. note:: Any column not specified will revert to its default value or ``NULL`` value if nullable -Loading non-standard dates +Loading Non-Standard Dates ---------------------------------- If files contain dates not formatted as ``ISO8601``, tell ``COPY`` how to parse the column. After parsing, the date will appear as ``ISO8601`` inside SQream DB. @@ -504,5 +590,3 @@ In this example, ``date_col1`` and ``date_col2`` in the table are non-standard. .. code-block:: postgres COPY table_name FROM WRAPPER csv_fdw OPTIONS (location = '/tmp/*.csv', datetime_format = 'DMY'); - - diff --git a/reference/sql/sql_statements/dml_commands/copy_to.rst b/reference/sql/sql_statements/dml_commands/copy_to.rst index 3579dc154..bd32a325f 100644 --- a/reference/sql/sql_statements/dml_commands/copy_to.rst +++ b/reference/sql/sql_statements/dml_commands/copy_to.rst @@ -3,20 +3,23 @@ ********************** COPY TO ********************** +The **COPY TO** page includes the following sections: +.. contents:: + :local: + :depth: 1 + +Overview +========== ``COPY ... TO`` is a statement that can be used to export data from a SQream database table or query to a file on the filesystem. In general, ``COPY`` moves data between filesystem files and SQream DB tables. .. note:: To copy data from a file to a table, see :ref:`COPY FROM`. -Permissions -============= - -The role must have the ``SELECT`` permission on every table or schema that is referenced by the statement. - Syntax ========== +The following is the correct syntax for using the **COPY TO** statement: .. code-block:: postgres @@ -48,6 +51,11 @@ Syntax | AWS_ID = '{ AWS ID }' | AWS_SECRET = '{ AWS Secret }' + + | MAX_FILE_SIZE = '{ size_in_bytes }' + + | ENFORCE_SINGLE_FILE = { true | false } + delimiter ::= string @@ -57,8 +65,13 @@ Syntax AWS Secret ::= string +.. note:: The DELIMITER is applicable to the CSV format only. + +.. note:: In Studio, you must write the parameters using lower case letters. Using upper case letters generates an error. + Elements ============ +The following table shows the ``COPY_TO`` elements: .. list-table:: :widths: auto @@ -80,41 +93,332 @@ Elements - Specifies the character that separates fields (columns) within each row of the file. The default is a comma character (``,``). * - ``AWS_ID``, ``AWS_SECRET`` - Specifies the authentication details for secured S3 buckets + * - ``MAX_FILE_SIZE`` + - Sets the maximum file size (bytes). Default value: 16*2^20 (16MB). + * - ``ENFORCE_SINGLE_FILE`` + - Enforces the maximum file size (bytes). Permitted values: ``true`` - creates one file of unlimited size, ``false`` - permits creating several files together limited by the ``MAX_FILE_SIZE``. When set to ``true``, the single file size is not limited by the ``MAX_FILE_SIZE`` setting. When set to ``false``, the combined file sizes cannot exceed the ``MAX_FILE_SIZE``. Default value: ``FALSE``. -Usage notes +Usage Notes =============== +The **Usage Notes** describes the following: -Supported field delimiters +.. contents:: + :local: + :depth: 1 + +Supported Field Delimiters ------------------------------ +The **Supported Field Delimiters** section describes the following: -Printable characters -^^^^^^^^^^^^^^^^^^^^^ +.. contents:: + :local: + :depth: 1 +Printable ASCII Characters +^^^^^^^^^^^^^^^^^^^^^ Any printable ASCII character can be used as a delimiter without special syntax. The default CSV field delimiter is a comma (``,``). -A printable character is any ASCII character in the range 32 - 126. - -Non-printable characters +The following table shows the supported printable ASCII characters: + ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| **Character** | **Description** | **ASCII** | **Octal** | **Hex** | **Binary** | **HTML Code** | **HTML Name** | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| (Space) | Space | 32 | 40 | 20 | 100000 | | | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ! | Exclamation Mark | 33 | 41 | 21 | 100001 | ! | ! | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| # | Hash or Number | 35 | 43 | 23 | 100011 | # | # | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| $ | Dollar Sign | 36 | 44 | 24 | 100100 | $ | $ | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| % | Percentage | 37 | 45 | 25 | 100101 | % | % | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| & | Ampersand | 38 | 46 | 26 | 100110 | & | & | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ( | Left Parenthesis | 40 | 50 | 28 | 101000 | ( | ( | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ) | Right Parenthesis | 41 | 51 | 29 | 101001 | ) | ) | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| * | Asterisk | 42 | 52 | 2A | 101010 | * | * | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| + | Plus Sign | 43 | 53 | 2B | 101011 | + | + | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| , | Comma | 44 | 54 | 2C | 101100 | , | , | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| / | Slash | 47 | 57 | 2F | 101111 | / | / | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ; | Semicolon | 59 | 73 | 3B | 111011 | ; | ; | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| < | Less Than | 60 | 74 | 3C | 111100 | < | < | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| = | Equals Sign | 61 | 75 | 3D | 111101 | = | = | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| > | Greater Than | 62 | 76 | 3E | 111110 | > | > | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ? | Question Mark | 63 | 77 | 3F | 111111 | ? | ? | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| @ | At Sign | 64 | 100 | 40 | 1000000 | @ | @ | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| [ | Left Square Bracket | 91 | 133 | 5B | 1011011 | [ | [ | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| \ | Backslash | 92 | 134 | 5C | 1011100 | \ | \ | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ] | Right Square Bracket | 93 | 135 | 5D | 1011101 | ] | ] | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ^ | Caret or Circumflex | 94 | 136 | 5E | 1011110 | ^ | &hat; | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| _ | Underscore | 95 | 137 | 5F | 1011111 | _ | _ | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ` | Grave Accent | 96 | 140 | 60 | 1100000 | ` | ` | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| { | Left Curly Bracket | 123 | 173 | 7B | 1111011 | { | { | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| | | Vertical Bar | 124 | 174 | 7C | 1111100 | | | | | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| } | Right Curly Bracket | 125 | 175 | 7D | 1111101 | } | } | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ~ | Tilde | 126 | 176 | 7E | 1111110 | ~ | ˜ | ++---------------+----------------------+-----------+-----------+---------+------------+---------------+---------------+ + +Non-Printable ASCII Characters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -A non-printable character (1 - 31, 127) can be used in its octal form. - +The following table shows the supported non-printable ASCII characters: + ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| **Character** | **Description** | **Octal** | **ASCII** | **Hex** | **Binary** | **HTML Code** | **HTML Name** | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| NUL | Null | 0 | 0 | 0 | 0 | � | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| SOH | Start of Heading | 1 | 1 | 1 | 1 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| STX | Start of Text | 2 | 2 | 2 | 10 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ETX | End of Text | 3 | 3 | 3 | 11 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| EOT | End of Transmission | 4 | 4 | 4 | 100 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ENQ | Enquiry | 5 | 5 | 5 | 101 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ACK | Acknowledge | 6 | 6 | 6 | 110 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| BEL | Bell | 7 | 7 | 7 | 111 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| BS | Backspace | 10 | 8 | 8 | 1000 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| HT | Horizontal Tab | 11 | 9 | 9 | 1001 | | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| VT | Vertical Tab | 13 | 11 | 0B | 1011 | | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| FF | NP Form Feed, New Page | 14 | 12 | 0C | 1100 | | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| SO | Shift Out | 16 | 14 | 0E | 1110 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| SI | Shift In | 17 | 15 | 0F | 1111 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| DLE | Data Link Escape | 20 | 16 | 10 | 10000 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| DC1 | Device Control 1 | 21 | 17 | 11 | 10001 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| DC2 | Device Control 2 | 22 | 18 | 12 | 10010 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| DC3 | Device Control 3 | 23 | 19 | 13 | 10011 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| DC4 | Device Control 4 | 24 | 20 | 14 | 10100 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| NAK | Negative Acknowledge | 25 | 21 | 15 | 10101 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| SYN | Synchronous Idle | 26 | 22 | 16 | 10110 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ETB | End of Transmission Block | 27 | 23 | 17 | 10111 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| CAN | Cancel | 30 | 24 | 18 | 11000 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| EM | End of Medium | 31 | 25 | 19 | 11001 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| SUB | Substitute | 32 | 26 | 1A | 11010 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| ESC | Escape | 33 | 27 | 1B | 11011 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| FS | File Separator | 34 | 28 | 1C | 11100 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| GS | Group Separator | 35 | 29 | 1D | 11101 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| RS | Record Separator | 36 | 30 | 1E | 11110 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| US | Unit Separator | 37 | 31 | 1F | 11111 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ +| DEL | Delete | 177 | 127 | 7F | 1111111 |  | | ++---------------+---------------------------+-----------+-----------+---------+------------+---------------+---------------+ + A tab can be specified by escaping it, for example ``\t``. Other non-printable characters can be specified using their octal representations, by using the ``E'\000'`` format, where ``000`` is the octal value of the character. For example, ASCII character ``15``, known as "shift in", can be specified using ``E'\017'``. +.. note:: Delimiters are only applicable to the CSV file format. -Date format +Unsupported ASCII Field Delimiters +------------------------------ +The following table shows the unsupported ASCII field delimiters: + ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| **ASCII** | **Character** | **Description** | **Octal** | **Hex** | **Binary** | **HTML Code** | **HTML Name** | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 10 | LF | NL Line Feed, New Line | 12 | 0A | 1010 | | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 13 | CR | Carriage Return | 15 | 0D | 1101 | | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 34 | " | Double Quote | 42 | 22 | 100010 | " | " | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 39 | ' | Single Quote | 47 | 27 | 100111 | ' | ' | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 45 | - | Minus Sign | 55 | 2D | 101101 | - | − | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 46 | . | Period | 56 | 2E | 101110 | . | . | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 48 | 0 | Zero | 60 | 30 | 110000 | 0 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 49 | 1 | Number One | 61 | 31 | 110001 | 1 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 50 | 2 | Number Two | 62 | 32 | 110010 | 2 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 51 | 3 | Number Three | 63 | 33 | 110011 | 3 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 52 | 4 | Number Four | 64 | 34 | 110100 | 4 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 53 | 5 | Number Five | 65 | 35 | 110101 | 5 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 54 | 6 | Number Six | 66 | 36 | 110110 | 6 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 55 | 7 | Number Seven | 67 | 37 | 110111 | 7 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 56 | 8 | Number Eight | 70 | 38 | 111000 | 8 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 57 | 9 | Number Nine | 71 | 39 | 111001 | 9 | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 58 | : | Colon | 72 | 3A | 111010 | : | : | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 65 | A | Upper Case Letter A | 101 | 41 | 1000001 | A | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 66 | B | Upper Case Letter B | 102 | 42 | 1000010 | B | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 67 | C | Upper Case Letter C | 103 | 43 | 1000011 | C | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 68 | D | Upper Case Letter D | 104 | 44 | 1000100 | D | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 69 | E | Upper Case Letter E | 105 | 45 | 1000101 | E | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 70 | F | Upper Case Letter F | 106 | 46 | 1000110 | F | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 71 | G | Upper Case Letter G | 107 | 47 | 1000111 | G | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 72 | H | Upper Case Letter H | 110 | 48 | 1001000 | H | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 73 | I | Upper Case Letter I | 111 | 49 | 1001001 | I | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 74 | J | Upper Case Letter J | 112 | 4A | 1001010 | J | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 75 | K | Upper Case Letter K | 113 | 4B | 1001011 | K | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 76 | L | Upper Case Letter L | 114 | 4C | 1001100 | L | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 77 | M | Upper Case Letter M | 115 | 4D | 1001101 | M | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 78 | N | Upper Case Letter N | 116 | 4E | 1001110 | N | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 79 | O | Upper Case Letter O | 117 | 4F | 1001111 | O | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 80 | P | Upper Case Letter P | 120 | 50 | 1010000 | P | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 81 | Q | Upper Case Letter Q | 121 | 51 | 1010001 | Q | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 82 | R | Upper Case Letter R | 122 | 52 | 1010010 | R | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 83 | S | Upper Case Letter S | 123 | 53 | 1010011 | S | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 84 | T | Upper Case Letter T | 124 | 54 | 1010100 | T | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 85 | U | Upper Case Letter U | 125 | 55 | 1010101 | U | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 86 | V | Upper Case Letter V | 126 | 56 | 1010110 | V | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 87 | W | Upper Case Letter W | 127 | 57 | 1010111 | W | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 88 | X | Upper Case Letter X | 130 | 58 | 1011000 | X | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 89 | Y | Upper Case Letter Y | 131 | 59 | 1011001 | Y | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 90 | Z | Upper Case Letter Z | 132 | 5A | 1011010 | Z | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 97 | a | Lower Case Letter a | 141 | 61 | 1100001 | a | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 98 | b | Lower Case Letter b | 142 | 62 | 1100010 | b | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 99 | c | Lower Case Letter c | 143 | 63 | 1100011 | c | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 100 | d | Lower Case Letter d | 144 | 64 | 1100100 | d | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 101 | e | Lower Case Letter e | 145 | 65 | 1100101 | e | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 102 | f | Lower Case Letter f | 146 | 66 | 1100110 | f | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 103 | g | Lower Case Letter g | 147 | 67 | 1100111 | g | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 104 | h | Lower Case Letter h | 150 | 68 | 1101000 | h | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 105 | i | Lower Case Letter i | 151 | 69 | 1101001 | i | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 106 | j | Lower Case Letter j | 152 | 6A | 1101010 | j | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 107 | k | Lower Case Letter k | 153 | 6B | 1101011 | k | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 108 | l | Lower Case Letter l | 154 | 6C | 1101100 | l | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 109 | m | Lower Case Letter m | 155 | 6D | 1101101 | m | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 110 | n | Lower Case Letter n | 156 | 6E | 1101110 | n | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 111 | o | Lower Case Letter o | 157 | 6F | 1101111 | o | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 112 | p | Lower Case Letter p | 160 | 70 | 1110000 | p | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 113 | q | Lower Case Letter q | 161 | 71 | 1110001 | q | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 114 | r | Lower Case Letter r | 162 | 72 | 1110010 | r | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 115 | s | Lower Case Letter s | 163 | 73 | 1110011 | s | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 116 | t | Lower Case Letter t | 164 | 74 | 1110100 | t | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 117 | u | Lower Case Letter u | 165 | 75 | 1110101 | u | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 118 | v | Lower Case Letter v | 166 | 76 | 1110110 | v | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 119 | w | Lower Case Letter w | 167 | 77 | 1110111 | w | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 120 | x | Lower Case Letter x | 170 | 78 | 1111000 | x | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 121 | y | Lower Case Letter y | 171 | 79 | 1111001 | y | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ +| 122 | z | Lower Case Letter z | 172 | 7A | 1111010 | z | | ++-----------+---------------+------------------------+-----------+---------+------------+---------------+---------------+ + +Date Format --------------- - The date format in the output CSV is formatted as ISO 8601 (``2019-12-31 20:30:55.123``), regardless of how it was parsed initially with :ref:`COPY FROM date parsers`. +For more information on the ``datetime`` format, see :ref:`sql_data_types_date`. Examples =========== +The **Examples** section shows the following examples: + +.. contents:: + :local: + :depth: 1 -Export table to a CSV without HEADER +Exporting a Table to a CSV File without a HEADER Row ------------------------------------ +The following is an example of exporting a table to a CSV file without a HEADER row: .. code-block:: psql @@ -130,8 +434,9 @@ Export table to a CSV without HEADER Jonas Jerebko,Boston Celtics,8,PF,29,6-10,231,\N,5000000 Amir Johnson,Boston Celtics,90,PF,29,6-9,240,\N,12000000 -Export table to a CSV with a HEADER row +Exporting a Table to a CSV with a HEADER Row ----------------------------------------- +The following is an example of exporting a table to a CSV file with a HEADER row: .. code-block:: psql @@ -147,8 +452,9 @@ Export table to a CSV with a HEADER row R.J. Hunter,Boston Celtics,28,SG,22,6-5,185,Georgia State,1148640 Jonas Jerebko,Boston Celtics,8,PF,29,6-10,231,\N,5000000 -Export table to a TSV with a header row +Exporting a Table to TSV with a HEADER Row ----------------------------------------- +The following is an example of exporting a table to a TSV file with a HEADER row: .. code-block:: psql @@ -164,8 +470,9 @@ Export table to a TSV with a header row R.J. Hunter Boston Celtics 28 SG 22 6-5 185 Georgia State 1148640 Jonas Jerebko Boston Celtics 8 PF 29 6-10 231 \N 5000000 -Use non-printable ASCII characters as delimiter +Using Non-Printable ASCII Characters as Delimiters ------------------------------------------------------- +The following is an example of using non-printable ASCII characters as delimiters: Non-printable characters can be specified using their octal representations, by using the ``E'\000'`` format, where ``000`` is the octal value of the character. @@ -179,8 +486,9 @@ For example, ASCII character ``15``, known as "shift in", can be specified using COPY nba TO WRAPPER csv_fdw OPTIONS (LOCATION = '/tmp/nba_export.csv', DELIMITER = E'\011'); -- 011 is a tab character -Exporting the result of a query to a CSV +Exporting the Result of a Query to CSV File -------------------------------------------- +The following is an example of exporting the result of a query to a CSV file: .. code-block:: psql @@ -195,41 +503,46 @@ Exporting the result of a query to a CSV Charlotte Hornets,5222728 Chicago Bulls,5785558 -Saving files to an authenticated S3 bucket +Saving Files to an Authenticated S3 Bucket -------------------------------------------- +The following is an example of saving files to an authenticated S3 bucket: .. code-block:: psql COPY (SELECT "Team", AVG("Salary") FROM nba GROUP BY 1) TO WRAPPER csv_fdw OPTIONS (LOCATION = 's3://my_bucket/salaries/nba_export.csv', AWS_ID = 'my_aws_id', AWS_SECRET = 'my_aws_secret'); -Saving files to an HDFS path +Saving Files to an HDFS Path -------------------------------------------- +The following is an example of saving files to an HDFS path: .. code-block:: psql COPY (SELECT "Team", AVG("Salary") FROM nba GROUP BY 1) TO WRAPPER csv_fdw OPTIONS (LOCATION = 'hdfs://pp_namenode:8020/nba_export.csv'); - -Export table to a parquet file +Exporting a Table to a Parquet File ------------------------------ +The following is an example of exporting a table to a Parquet file: .. code-block:: psql COPY nba TO WRAPPER parquet_fdw OPTIONS (LOCATION = '/tmp/nba_export.parquet'); - -Export a query to a parquet file +Exporting a Query to a Parquet File -------------------------------- +The following is an example of exporting a query to a Parquet file: .. code-block:: psql COPY (select x,y from t where z=0) TO WRAPPER parquet_fdw OPTIONS (LOCATION = '/tmp/file.parquet'); - -Export table to a ORC file +Exporting a Table to an ORC File ------------------------------ +The following is an example of exporting a table to an ORC file: .. code-block:: psql COPY nba TO WRAPPER orc_fdw OPTIONS (LOCATION = '/tmp/nba_export.orc'); +Permissions +============= +The role must have the ``SELECT`` permission on every table or schema that is referenced by the statement. diff --git a/reference/sql/sql_statements/dml_commands/select.rst b/reference/sql/sql_statements/dml_commands/select.rst index f47ffaec1..299a83af1 100644 --- a/reference/sql/sql_statements/dml_commands/select.rst +++ b/reference/sql/sql_statements/dml_commands/select.rst @@ -163,15 +163,15 @@ Assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float + "Name" varchar(40), + "Team" varchar(40), + "Number" tinyint, + "Position" varchar(2), + "Age" tinyint, + "Height" varchar(4), + "Weight" real, + "College" varchar(40), + "Salary" float ); diff --git a/reference/sql/sql_statements/dml_commands/truncate_if_exists.rst b/reference/sql/sql_statements/dml_commands/truncate_if_exists.rst deleted file mode 100644 index fc1f4b85e..000000000 --- a/reference/sql/sql_statements/dml_commands/truncate_if_exists.rst +++ /dev/null @@ -1,84 +0,0 @@ -.. _truncate: - -********************** -TRUNCATE_IF_EXISTS -********************** - -``TRUNCATE_IF_EXISTS`` removes all rows from a table. It does not generate an error if the table does not exist. - -It is functionally identical to running a ``DELETE`` statement without a ``WHERE`` clause. - -Permissions -============= - -The role must have the ``DELETE`` permission at the table level. - -Syntax -========== - -.. code-block:: postgres - - truncate_table_statement ::= - TRUNCATE [ TABLE ] [schema_name.]table_name - [ RESTART IDENTITY | CONTINUE IDENTITY ] - ; - - table_name ::= identifier - - schema_name ::= identifier - -Parameters -============ - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Parameter - - Description - * - ``schema_name`` - - The name of the schema for the table. - * - ``table_name`` - - The name of the table to truncate. - * - ``RESTART IDENTITY`` - - Automatically restart sequences owned by columns of the truncated table. - * - ``CONTINUE IDENTITY`` - - Do not change the values of sequences. This is the default. - -Examples -=========== - -Truncating an existing table ---------------------------------------------- - -.. code-block:: psql - - farm=> SELECT * FROM cool_animals; - 1,Dog ,7 - 2,Possum ,3 - 3,Cat ,5 - 4,Elephant ,6500 - 5,Rhinoceros ,2100 - 6,\N,\N - - 6 rows - - farm=> SELECT TRUNCATE_IF_EXISTS('public', 'cool_animals'); - Status: 0 result rows - - farm=> SELECT * FROM cool_animals; - - 0 rows - - -Truncating a non-existing table ---------------------------------------------- -.. code-block:: psql - - farm=> SELECT * FROM cool_animals2; - - Status: Ended with errorError preparing statement: At row 1, col 15: table not found public.cool_animals2 - - farm=> SELECT TRUNCATE_IF_EXISTS('public', 'cool_animals2'); - - Status: 0 result rows \ No newline at end of file diff --git a/reference/sql/sql_statements/dml_commands/values.rst b/reference/sql/sql_statements/dml_commands/values.rst index d087e79ff..4e8e1f28b 100644 --- a/reference/sql/sql_statements/dml_commands/values.rst +++ b/reference/sql/sql_statements/dml_commands/values.rst @@ -9,7 +9,6 @@ VALUES .. tip:: * Use VALUES in conjunction with :ref:`INSERT` statements to insert a set of one or more rows. -.. note:: The maximum number of values in the ``VALUE`` clause is limited to 500. Permissions ============= @@ -81,4 +80,4 @@ Use ``AS`` to assign names to columns (3, 'horse'), (4, 'hippopotamus') ) AS t(id, name) - ); \ No newline at end of file + ); diff --git a/reference/sql/sql_statements/index.rst b/reference/sql/sql_statements/index.rst index 3218d5f92..9bc902115 100644 --- a/reference/sql/sql_statements/index.rst +++ b/reference/sql/sql_statements/index.rst @@ -1,66 +1,75 @@ .. _sql_statements: *************** -SQL statements +SQL Statements *************** +The **SQL Statements** page describes the following commands: + +.. contents:: + :local: + :depth: 1 SQream DB supports commands from ANSI SQL. .. _ddl_commands_list: -Data Definition commands (DDL) +Data Definition Commands (DDL) ================================ -.. list-table:: DDL commands +.. list-table:: DDL Commands :widths: auto :header-rows: 1 :name: ddl_commands * - Command - Usage - * - :ref:`ADD COLUMN` + * - :ref:`add_column` - Add a new column to a table - * - :ref:`ALTER DEFAULT SCHEMA` + * - :ref:`alter_default_schema` - Change the default schema for a role - * - :ref:`ALTER TABLE` + * - :ref:`alter_table` - Change the schema of a table - * - :ref:`CREATE DATABASE` + * - :ref:`cluster_by` + - Change clustering keys in a table + * - :ref:`create_database` - Create a new database - * - :ref:`CREATE EXTERNAL TABLE` + * - :ref:`create_external_table` - Create a new external table in the database (deprecated) - * - :ref:`CREATE FOREIGN TABLE` + * - :ref:`create_foreign_table` - Create a new foreign table in the database - * - :ref:`CREATE FUNCTION ` + * - :ref:`create_function` - Create a new user defined function in the database - * - :ref:`CREATE SCHEMA` + * - :ref:`create_schema` - Create a new schema in the database - * - :ref:`CREATE TABLE` + * - :ref:`create_table` - Create a new table in the database - * - :ref:`CREATE TABLE AS` + * - :ref:`create_table_as` - Create a new table in the database using results from a select query - * - :ref:`CREATE VIEW` + * - :ref:`create_view` - Create a new view in the database - * - :ref:`DROP COLUMN` + * - :ref:`drop_clustering_key` + - Drops all clustering keys in a table + * - :ref:`drop_column` - Drop a column from a table - * - :ref:`DROP DATABASE` + * - :ref:`drop_database` - Drop a database and all of its objects - * - :ref:`DROP FUNCTION` + * - :ref:`drop_function` - Drop a function - * - :ref:`DROP SCHEMA` + * - :ref:`drop_schema` - Drop a schema - * - :ref:`DROP TABLE` + * - :ref:`drop_table` - Drop a table and its contents from a database - * - :ref:`DROP VIEW` + * - :ref:`drop_view` - Drop a view - * - :ref:`RENAME COLUMN` + * - :ref:`rename_column` - Rename a column - * - :ref:`RENAME TABLE` + * - :ref:`rename_table` - Rename a table -Data manipulation commands (DML) +Data Manipulation Commands (DML) ================================ -.. list-table:: DML commands +.. list-table:: DML Commands :widths: auto :header-rows: 1 :name: dml_commands @@ -85,15 +94,21 @@ Data manipulation commands (DML) * - :ref:`VALUES` - Return rows containing literal values -Utility commands +Utility Commands ================== -.. list-table:: Utility commands +.. list-table:: Utility Commands :widths: auto :header-rows: 1 * - Command - Usage + * - :ref:`EXECUTE SAVED QUERY` + - Executes a previously saved query + * - :ref:`EXPLAIN` + - Returns a static query plan, which can be used to debug query plans + * - :ref:`SELECT GET_LICENSE_INFO` + - View a user's license information * - :ref:`SELECT GET_DDL` - View the ``CREATE TABLE`` statement for a table * - :ref:`SELECT GET_FUNCTION_DDL` @@ -104,61 +119,25 @@ Utility commands - Recreate a view after schema changes * - :ref:`SELECT DUMP_DATABASE_DDL` - View the ``CREATE TABLE`` statement for an current database - -Saved queries -=================== - -See more about :ref:`saved_queries` - -.. list-table:: Saved queries - :widths: auto - :header-rows: 1 - - * - Command - - Usage - * - :ref:`SELECT DROP_SAVED_QUERY` - - Drop a saved query - * - :ref:`SELECT EXECUTE_SAVED_QUERY` - - Executes a saved query - * - :ref:`SELECT LIST_SAVED_QUERIES` - - Returns a list of saved queries - * - :ref:`SELECT RECOMPILE_SAVED_QUERY` - - Recompiles a query that has been invalidated by a schema change - * - :ref:`SELECT SAVE_QUERY` - - Compiles and saves a query for re-use and sharing - * - :ref:`SELECT SHOW_SAVED_QUERY` - - Shows query text for a saved query - -Monitoring -=============== - -Monitoring statements allow a database administrator to execute actions in the system, such as aborting a query or get information about system processes. - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Command - - Usage - * - :ref:`explain` - - Returns a static query plan for a statement - * - :ref:`show_connections` - - Returns a list of jobs and statements on the current worker - * - :ref:`show_locks` - - Returns any existing locks in the database - * - :ref:`show_node_info` - - Returns a query plan for an actively running statement with timing information - * - :ref:`show_server_status` - - Shows running statements across the cluster - * - :ref:`show_version` - - Returns the version of SQream DB - * - :ref:`stop_statement` - - Stops a query (or statement) if it is currently running - -Workload management + * - :ref:`SHOW CONNECTIONS` + - Returns a list of active sessions on the current worker + * - :ref:`SHOW LOCKS` + - Returns a list of locks from across the cluster + * - :ref:`SHOW NODE INFO` + - Returns a snapshot of the current query plan, similar to ``EXPLAIN ANALYZE`` from other databases + * - :ref:`SHOW SERVER STATUS` + - Returns a list of active sessions across the cluster + * - :ref:`SHOW VERSION` + - Returns the system version for SQream DB + * - :ref:`STOP STATEMENT` + - Stops or aborts an active statement + + + +Workload Management ====================== -.. list-table:: +.. list-table:: Workload Management :widths: auto :header-rows: 1 @@ -167,14 +146,15 @@ Workload management * - :ref:`subscribe_service` - Add a SQream DB worker to a service queue * - :ref:`unsubscribe_service` - - Remove a SQream DB worker to a service queue + - Remove a SQream DB worker from a service queue * - :ref:`show_subscribed_instances` - Return a list of service queues and workers -Access control commands +Access Control Commands ================================ +The following table shows the Access Control commands: -.. list-table:: Access control commands +.. list-table:: Access Control Commands :widths: auto :header-rows: 1 @@ -188,6 +168,12 @@ Access control commands - Creates a roles, which lets a database administrator control permissions on tables and databases * - :ref:`drop_role` - Removes roles + * - :ref:`get_role_permissions` + - Returns all permissions granted to a role in table format + * - :ref:`get_role_global_ddl` + - Returns the definition of a global role in DDL format + * - :ref:`get_role_database_ddl` + - Returns the definition of a database role in DDL format * - :ref:`get_statement_permissions` - Returns a list of permissions required to run a statement or query * - :ref:`grant` @@ -195,18 +181,4 @@ Access control commands * - :ref:`revoke` - Revoke permissions from a role * - :ref:`rename_role` - - Rename a role - - -.. toctree:: - :maxdepth: 1 - :titlesonly: - :hidden: - :glob: - - ddl_commands/* - dml_commands/* - utility_commands/* - monitoring_commands/* - wlm_commands/* - access_control_commands/* \ No newline at end of file + - Rename a role \ No newline at end of file diff --git a/reference/sql/sql_statements/monitoring_commands/show_server_status.rst b/reference/sql/sql_statements/monitoring_commands/show_server_status.rst deleted file mode 100644 index f59f79ccc..000000000 --- a/reference/sql/sql_statements/monitoring_commands/show_server_status.rst +++ /dev/null @@ -1,108 +0,0 @@ -.. _show_server_status: - -******************** -SHOW_SERVER_STATUS -******************** - -``SHOW_SERVER_STATUS`` returns a list of active sessions across the cluster. - -To list active statements on the current worker only, see :ref:`show_connections`. - -Permissions -============= - -The role must have the ``SUPERUSER`` permissions. - -Syntax -========== - -.. code-block:: postgres - - show_server_status_statement ::= - SELECT SHOW_SERVER_STATUS() - ; - -Parameters -============ - -None - -Returns -========= - -This function returns a list of active sessions. If no sessions are active across the cluster, the result set will be empty. - -.. list-table:: Result columns - :widths: auto - :header-rows: 1 - - * - ``service`` - - The service name for the statement - * - ``instance`` - - The worker ID - * - ``connection_id`` - - Connection ID - * - ``serverip`` - - Worker end-point IP - * - ``serverport`` - - Worker end-point port - * - ``database_name`` - - Database name for the statement - * - ``user_name`` - - Username running the statement - * - ``clientip`` - - Client IP - * - ``statementid`` - - Statement ID - * - ``statement`` - - Statement text - * - ``statementstarttime`` - - Statement start timestamp - * - ``statementstatus`` - - Statement status (see table below) - * - ``statementstatusstart`` - - Last updated timestamp - -.. include from here: 66 - - -.. list-table:: Statement status values - :widths: auto - :header-rows: 1 - - * - Status - - Description - * - ``Preparing`` - - Statement is being prepared - * - ``In queue`` - - Statement is waiting for execution - * - ``Initializing`` - - Statement has entered execution checks - * - ``Executing`` - - Statement is executing - * - ``Stopping`` - - Statement is in the process of stopping - - -.. include until here 86 - -Notes -=========== - -* This utility shows the active sessions. Some sessions may be actively connected, but not running any statements. - -Examples -=========== - -Using ``SHOW_SERVER_STATUS`` to get statement IDs ----------------------------------------------------- - - -.. code-block:: psql - - t=> SELECT SHOW_SERVER_STATUS(); - service | instanceid | connection_id | serverip | serverport | database_name | user_name | clientip | statementid | statement | statementstarttime | statementstatus | statementstatusstart - --------+------------+---------------+--------------+------------+---------------+------------+-------------+-------------+-----------------------------+---------------------+-----------------+--------------------- - sqream | | 102 | 192.168.1.91 | 5000 | t | rhendricks | 192.168.0.1 | 128 | SELECT SHOW_SERVER_STATUS() | 24-12-2019 00:14:53 | Executing | 24-12-2019 00:14:53 - -The statement ID is ``128``, running on worker ``192.168.1.91``. diff --git a/reference/sql/sql_statements/utility_commands/execute_saved_query.rst b/reference/sql/sql_statements/utility_commands/execute_saved_query.rst index 6fe41fa08..b5e0283d3 100644 --- a/reference/sql/sql_statements/utility_commands/execute_saved_query.rst +++ b/reference/sql/sql_statements/utility_commands/execute_saved_query.rst @@ -8,7 +8,7 @@ EXECUTE_SAVED_QUERY Read more in the :ref:`saved_queries` guide. -See also: ref:`save_query`, :ref:`drop_saved_query`, ref:`show_saved_query`, ref:`list_saved_queries`. +See also: :ref:`save_query`, :ref:`drop_saved_query`, :ref:`show_saved_query`, :ref:`list_saved_queries`. Permissions ============= @@ -53,7 +53,7 @@ Notes * Query parameters can be used as substitutes for literal expressions. Parameters cannot be used to substitute identifiers, column names, table names, or other parts of the query. -* Query parameters of a string datatype (like ``VARCHAR``) must be of a fixed length, and can be used in equality checks, but not patterns (e.g. :ref:`like`, :ref:`rlike`, etc) +* Query parameters of a string datatype (like ``TEXT``) must be of a fixed length, and can be used in equality checks, but not patterns (e.g. :ref:`like`, :ref:`rlike`, etc) * Query parameters' types are inferred at compile time. @@ -66,14 +66,14 @@ Assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), + Name text, + Team text, Number tinyint, - Position varchar(2), + Position text, Age tinyint, - Height varchar(4), + Height text, Weight real, - College varchar(40), + College text, Salary float ); diff --git a/reference/sql/sql_statements/monitoring_commands/explain.rst b/reference/sql/sql_statements/utility_commands/explain.rst similarity index 100% rename from reference/sql/sql_statements/monitoring_commands/explain.rst rename to reference/sql/sql_statements/utility_commands/explain.rst diff --git a/reference/sql/sql_statements/utility_commands/get_license_info.rst b/reference/sql/sql_statements/utility_commands/get_license_info.rst new file mode 100644 index 000000000..8c28c0ea4 --- /dev/null +++ b/reference/sql/sql_statements/utility_commands/get_license_info.rst @@ -0,0 +1,72 @@ +.. _get_license_info: + +******************** +GET_LICENSE_INFO +******************** +``GET_LICENSE_INFO`` displays information related to data size limitations, expiration date, and type of license currently used by the SQream cluster. + +Permissions +============= +No special permissions are required. + +Syntax +========== +The following is the correct syntax for running the ``GET LICENSE INFO`` statement: + +.. code-block:: postgres + + get_license_info_statement ::= + SELECT GET_LICENSE_INFO() + ; + +Returns +========== +The following table shows the ``GET_LICENSE_INFO`` license information in the order that it is returned: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Parameter + - Description + - Example + * - ``compressed_cluster_size`` + - Shows the compressed size (GB) of the cluster. + - 10 + * - ``uncompressed_cluster_size`` + - Shows the uncompressed size (GB) of the cluster. + - 10 + * - ``compress_type`` + - Shows the compression type (``compressed``, ``uncompressed``). + - compressed + * - ``cluster_size_limit`` + - Shows the cluster size limit (GB) of the current license. + - 20 + * - ``expiration_date`` + - Shows the expiration date of the current license. + - 2024-03-18 + * - ``is_date_expired`` + - Shows whether the date has expired or not (``0`` - **No**, ``1`` - **Yes**). + - 0 + * - ``is_size_exceeded`` + - Shows whether the cluster size has exceeded the license size limit or not (``0`` - **No**, ``1`` - **Yes**). + - 0 + * - ``cluster_size_left`` + - Shows the remaining available cluster size (GB) in the license. + - 10 + +Example +=========== +The following is an example of the returned license information described in the **Returns** section above: + +.. code-block:: psql + + 10,100,compressed,20,2045-03-18,0,0,10 + +Parameters +============ +The ``GET_LICENSE_INFO`` command has no parameters. + +Notes +========= +If the license expires or exceeds quotas, contact a SQream representative to extend the license. diff --git a/reference/sql/sql_statements/utility_commands/get_role_database_ddl.rst b/reference/sql/sql_statements/utility_commands/get_role_database_ddl.rst new file mode 100644 index 000000000..e6b806244 --- /dev/null +++ b/reference/sql/sql_statements/utility_commands/get_role_database_ddl.rst @@ -0,0 +1,61 @@ +.. _get_role_database_ddl: + +******************** +GET_ROLE_DATABASE_DDL +******************** +The ``GET_ROLE_DATABASE_DDL`` statement returns the definition of a global role in DDL format. + +The ``GET_ROLE_DATABASE_DDL`` page describes the following: + +.. contents:: + :local: + :depth: 1 + +Syntax +========== +The following is the correct syntax for using the ``GET_ROLE_DATABASE_DDL`` statement: + +.. code-block:: postgres + + select get_role_database_ddl(<'role_name'>) + +Example +=========== +The following is an example of using the ``GET_ROLE_GLOBAL_DDL`` statement: + +.. code-block:: psql + + select get_role_global_ddl('public'); + +Parameters +============ +The following table shows the ``GET_ROLE_DATABASE_DDL`` parameters: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Parameter + - Description + * - ``role_name`` + - The definition of the database role in DDL format. + +Output +========== +The following is an example of the output of the ``GET_ROLE_DATABASE_DDL`` statement: + +.. code-block:: postgres + + grant create, usage on schema "public" to "public" ; alter default schema for "public" to "public"; alter default permissions for "public" for schemas grant superuser to creator_role ; alter default permissions for "public" for tables grant select, insert, delete, ddl to creator_role ; + +Permissions +============= +Using the ``GET_ROLE_DATABASE_DDL`` statement requires no special permissions. + +For more information, see the following: + +* :ref:`get_role_global_ddl` + + :: + +* :ref:`get_role_permissions` \ No newline at end of file diff --git a/reference/sql/sql_statements/utility_commands/get_role_global_ddl.rst b/reference/sql/sql_statements/utility_commands/get_role_global_ddl.rst new file mode 100644 index 000000000..3ba0255d3 --- /dev/null +++ b/reference/sql/sql_statements/utility_commands/get_role_global_ddl.rst @@ -0,0 +1,61 @@ +.. _get_role_global_ddl: + +******************** +GET_ROLE_GLOBAL_DDL +******************** +The ``GET_ROLE_GLOBAL_DDL`` statement returns the definition of a global role in DDL format. + +The ``GET_ROLE_GLOBAL_DDL`` page describes the following: + +.. contents:: + :local: + :depth: 1 + +Syntax +========== +The following is the correct syntax for using the ``GET_ROLE_GLOBAL_DDL`` statement: + +.. code-block:: postgres + + select get_role_global_ddl(<'role_name'>) + +Example +=========== +The following is an example of using the ``GET_ROLE_GLOBAL_DDL`` statement: + +.. code-block:: psql + + select get_role_global_ddl('public'); + +Parameters +============ +The following table shows the ``GET_ROLE_GLOBAL_DDL`` parameters: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Parameter + - Description + * - ``role_name`` + - The definition of the global role in DDL format. + +Output +========== +The following is an example of the output of the ``GET_ROLE_GLOBAL_DDL`` statement: + +.. code-block:: postgres + + create role "public"; + +Permissions +============= +Using the ``GET_ROLE_GLOBAL_DDL`` statement requires no special permissions. + +For more information, see the following: + +* :ref:`get_role_database_ddl` + + :: + +* :ref:`get_role_permissions` \ No newline at end of file diff --git a/reference/sql/sql_statements/utility_commands/get_role_permissions.rst b/reference/sql/sql_statements/utility_commands/get_role_permissions.rst new file mode 100644 index 000000000..8723f98c8 --- /dev/null +++ b/reference/sql/sql_statements/utility_commands/get_role_permissions.rst @@ -0,0 +1,74 @@ +.. _get_role_permissions: + +******************** +GET_ROLE_PERMISSIONS +******************** +The ``GET_ROLE_PERMISSIONS`` statement returns all permissions granted to a role in table format. + +The ``GET_ROLE_PERMISSIONS`` page describes the following: + +.. contents:: + :local: + :depth: 1 + +Syntax +========== +The following is the correct syntax for using the ``GET_ROLE_PERMISSIONS`` statement: + +.. code-block:: postgres + + select get_role_permissions() + +Example +=========== +The following is an example of using the ``GET_ROLE_PERMISSIONS`` statement: + +.. code-block:: psql + + select get_role_permissions(); + +Parameters +============ +The following table shows the ``GET_ROLE_PERMISSIONS`` parameters: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Parameter + - Description + * - ``()`` + - The permissions belonging to the role. + +Output +========== +The following is an example of the output of the ``GET_ROLE_PERMISSIONS`` statement: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Parameter + - Description + - Example + * - ``permission_type`` + - The permission type granted to the role. + - SUPERUSER + * - ``object_type`` + - The data object type. + - table + * - ``object_name`` + - The name of the object. + - master.public.nba + +Permissions +============= +Using the ``GET_ROLE_PERMISSIONS`` statement requires no special permissions. + +For more information, see the following: + +* :ref:`get_role_database_ddl` + + :: + +* :ref:`get_role_global_ddl` \ No newline at end of file diff --git a/reference/sql/sql_statements/utility_commands/save_query.rst b/reference/sql/sql_statements/utility_commands/save_query.rst index be34c33ed..d61d37fc3 100644 --- a/reference/sql/sql_statements/utility_commands/save_query.rst +++ b/reference/sql/sql_statements/utility_commands/save_query.rst @@ -70,15 +70,15 @@ Assume a table named ``nba``, with the following structure: CREATE TABLE nba ( - Name varchar(40), - Team varchar(40), - Number tinyint, - Position varchar(2), - Age tinyint, - Height varchar(4), - Weight real, - College varchar(40), - Salary float + "Name" varchar(40), + "Team" varchar(40), + "Number" tinyint, + "Position" varchar(2), + "Age" tinyint, + "Height" varchar(4), + "Weight" real, + "College" varchar(40), + "Salary" float ); diff --git a/reference/sql/sql_statements/monitoring_commands/show_connections.rst b/reference/sql/sql_statements/utility_commands/show_connections.rst similarity index 100% rename from reference/sql/sql_statements/monitoring_commands/show_connections.rst rename to reference/sql/sql_statements/utility_commands/show_connections.rst diff --git a/reference/sql/sql_statements/monitoring_commands/show_locks.rst b/reference/sql/sql_statements/utility_commands/show_locks.rst similarity index 100% rename from reference/sql/sql_statements/monitoring_commands/show_locks.rst rename to reference/sql/sql_statements/utility_commands/show_locks.rst diff --git a/reference/sql/sql_statements/monitoring_commands/show_node_info.rst b/reference/sql/sql_statements/utility_commands/show_node_info.rst similarity index 99% rename from reference/sql/sql_statements/monitoring_commands/show_node_info.rst rename to reference/sql/sql_statements/utility_commands/show_node_info.rst index 345d16440..9c1e1ec11 100644 --- a/reference/sql/sql_statements/monitoring_commands/show_node_info.rst +++ b/reference/sql/sql_statements/utility_commands/show_node_info.rst @@ -108,7 +108,7 @@ This is a full list of node types: - Compress data with both CPU and GPU schemes * - ``CpuDecompress`` - CPU - - Decompression operation, common for longer ``VARCHAR`` types + - Decompression operation, common for longer ``TEXT`` types * - ``CpuLoopJoin`` - CPU - A non-indexed nested loop join, performed on the CPU diff --git a/reference/sql/sql_statements/utility_commands/show_server_status.rst b/reference/sql/sql_statements/utility_commands/show_server_status.rst new file mode 100644 index 000000000..73902a046 --- /dev/null +++ b/reference/sql/sql_statements/utility_commands/show_server_status.rst @@ -0,0 +1,108 @@ +.. _show_server_status: + +******************** +SHOW_SERVER_STATUS +******************** +``SHOW_SERVER_STATUS`` returns a list of active sessions across the cluster. + +To list active statements on the current worker only, see :ref:`show_connections`. + +Syntax +========== +The following is the correct syntax when showing your server status: + +.. code-block:: postgres + + show_server_status_statement ::= + SELECT SHOW_SERVER_STATUS() + ; + +Parameters +============ +The Parameters section is not relevant for the ``SHOW_SERVER_STATUS`` statement. + +Returns +========= +The ``SHOW_SERVER_STATUS`` function returns a list of active sessions. If no sessions are active across the cluster, the result set will be empty. + +The following table shows the ``SHOW_SERVER_STATUS`` result columns; + +.. list-table:: Result Columns + :widths: auto + :header-rows: 1 + + * - service + - Statement Service Name + * - ``instance`` + - Shows the worker ID. + * - ``connection_id`` + - Shows the connection ID. + * - ``serverip`` + - Shows the worker end-point IP. + * - ``serverport`` + - Shows the worker end-point port. + * - ``database_name`` + - Shows the statement's database name. + * - ``user_name`` + - Shows the username running the statement. + * - ``clientip`` + - Shows the client IP. + * - ``statementid`` + - Shows the statement ID. + * - ``statement`` + - Shows the statement text. + * - ``statementstarttime`` + - Shows the statement start timestamp. + * - ``statementstatus`` + - Shows the statement status (see table below). + * - ``statementstatusstart`` + - Shows the most recently updated timestamp. + +.. include from here: 66 + +The following table shows the statement status values: + +.. list-table:: Statement Status Values + :widths: auto + :header-rows: 1 + + * - Status + - Description + * - ``Preparing`` + - The statement is being prepared. + * - ``In queue`` + - The statement is waiting for execution. + * - ``Initializing`` + - The statement has entered execution checks. + * - ``Executing`` + - The statement is executing. + * - ``Stopping`` + - The statement is in the process of stopping. + +.. include until here 86 + +Notes +=========== +This utility shows the active sessions. Some sessions may be actively connected, but not running any statements. + +Example +=========== + +Using SHOW_SERVER_STATUS to Get Statement IDs +---------------------------------------------------- +The following example shows how to use the ``SHOW_SERVER_STATUS`` statement to get statement IDs: + +.. code-block:: psql + + t=> SELECT SHOW_SERVER_STATUS(); + service | instanceid | connection_id | serverip | serverport | database_name | user_name | clientip | statementid | statement | statementstarttime | statementstatus | statementstatusstart + --------+------------+---------------+---------------+------------+---------------+------------------+---------------+-------------+-------------------------------------------------------------------------------------------------------+---------------------+-----------------+--------------------- + sqream | sqream_2 | 19 | 192.168.0.111 | 5000 | master | etl | 192.168.0.011 |2484923 | SELECT t1.account, t1.msisd from table a t1 join table b t2 on t1.id = t2.id where t1.msid='123123'; | 17-01-2022 16:19:31 | Executing | 17-01-2022 16:19:32 + sqream | sqream_1 | 2 | 192.168.1.112 | 5000 | master | etl | 192.168.1.112 |2484924 | select show_server_status(); | 17-01-2022 16:19:39 | Executing | 17-01-2022 16:19:39 + sqream | None | 248 | 192.168.1.112 | 5007 | master | maintenance_user | 192.168.1.112 |2484665 | select * from sqream_catalog.tables; | 17-01-2022 15:55:01 | In Queue | 17-01-2022 15:55:02 + +The statement ID is ``128``, running on worker ``192.168.1.91``. + +Permissions +============= +The role must have the ``SUPERUSER`` permissions. diff --git a/reference/sql/sql_functions/system_functions/show_version.rst b/reference/sql/sql_statements/utility_commands/show_version.rst similarity index 100% rename from reference/sql/sql_functions/system_functions/show_version.rst rename to reference/sql/sql_statements/utility_commands/show_version.rst diff --git a/reference/sql/sql_statements/monitoring_commands/stop_statement.rst b/reference/sql/sql_statements/utility_commands/stop_statement.rst similarity index 100% rename from reference/sql/sql_statements/monitoring_commands/stop_statement.rst rename to reference/sql/sql_statements/utility_commands/stop_statement.rst diff --git a/reference/sql/sql_syntax/index.rst b/reference/sql/sql_syntax/index.rst index 4b6b66aaa..9e0422db4 100644 --- a/reference/sql/sql_syntax/index.rst +++ b/reference/sql/sql_syntax/index.rst @@ -1,21 +1,19 @@ .. _sql_syntax: ********************** -SQL syntax features +SQL Syntax Features ********************** -SQream DB supports SQL from the ANSI 92 syntax. +SQream DB supports SQL from the ANSI 92 syntax and describes the following: -.. toctree:: - :maxdepth: 2 - :caption: SQL syntax topics - :glob: +.. hlist:: + :columns: 1 - keywords_and_identifiers - literals - scalar_expressions - joins - common_table_expressions - window_functions - subqueries - null_handling \ No newline at end of file + * :ref:`keywords_and_identifiers` + * :ref:`literals` + * :ref:`scalar_expressions` + * :ref:`joins` + * :ref:`common_table_expressions` + * :ref:`window_functions` + * :ref:`subqueries` + * :ref:`null_handling` \ No newline at end of file diff --git a/reference/sql/sql_syntax/joins.rst b/reference/sql/sql_syntax/joins.rst index 2527cec69..b12b08875 100644 --- a/reference/sql/sql_syntax/joins.rst +++ b/reference/sql/sql_syntax/joins.rst @@ -4,14 +4,14 @@ Joins *************************** -Joins combine results from two or more table expressions (tables, external tables, views) to form a new result set. +The ``JOIN`` clause combines results from two or more table expressions (tables, external tables, views) based on a related column or other condition. Performing a join outputs a new result set. For example, two tables containing one or more columns in common can be joined to match or correlate with rows from another table. -The combination of these table expressions ca nbe based on a set of conditions, such as equality of columns. + -Joins are useful when data in tables are related. For example, when two tables contain one or more columns in common, so that rows can be matched or correlated with rows from the other table. Syntax ========== +The following shows the correct syntax for creating a **join**: .. code-block:: postgres @@ -29,76 +29,145 @@ Syntax join_hint ::= MERGE | LOOP -Join types +Join Types ------------- +The **Join Types** section describes the following join types: + +* :ref:`Inner joins` +* :ref:`Left outer joins` +* :ref:`Right outer joins` +* :ref:`Cross joins` + +.. _inner_joins: -Inner join +Inner Joins ^^^^^^^^^^^^ +The following shows the correct syntax for creating an **inner join**: .. code-block:: postgres - left_side [ INNER ] JOIN right_side ON value_expr - left_side [ INNER ] JOIN right_side USING ( join_column [, ... ] ) + left_side [ INNER ] JOIN left_side ON value_expr + left_side [ INNER ] JOIN left_side USING ( join_column [, ... ] ) + +Inner joins are the default join type and return rows from the ``left_side`` and ``right_side`` based on a matching condition. + +An inner join can also be specified by listing several tables in the ``FROM`` clause, as shown below: + +.. code-block:: postgres -This is the default join type. -Rows from the ``left_side`` and ``right_side`` that match the condition are returned. + FROM table1, table2 + [ { INNER JOIN + | LEFT [OUTER] JOIN + | RIGHT [OUTER] JOIN + ON table1.column1 = table2.column1 ] -An inner join can also be specified by listing several tables in the ``FROM`` clause. -Omitting the ``ON`` or ``WHERE`` will result in a ``CROSS JOIN``, where every row of ``left_side`` is matched with every row on ``right_side``. +Omitting the ``ON`` or ``WHERE`` clause creates a ``CROSS JOIN``, where every ``left_side`` row is matched with every ``right_side`` row. -Left outer join +For more information about cross joins, see :ref:`cross joins` below. + +For an inner join example, see :ref:`Inner Join Example`. + +.. _left_outer_joins: + +Left Outer Joins ^^^^^^^^^^^^^^^^^^ +The following shows the correct syntax for creating an **left outer join**: .. code-block:: postgres left_side LEFT [ OUTER ] JOIN right_side ON value_expr left_side LEFT [ OUTER ] JOIN right_side USING ( join_column [, ... ] ) -Similar to the inner join - but for every row of ``left_side`` that does not match, ``NULL`` values are returned for columns on ``right_side`` +Left outer joins are similar to inner joins, except that for every ``left_side`` row without a matching condition, a ``NULL`` value is returned for the corresponding ``right_side`` column. + +For a left inner join example, see :ref:`Left Join Example`. + +.. _right_outer_joins: -Right outer join +Right Outer Joins ^^^^^^^^^^^^^^^^^^^ +The following shows the correct syntax for creating an **right outer join**: .. code-block:: postgres left_side RIGHT [ OUTER ] JOIN right_side ON value_expr left_side RIGHT [ OUTER ] JOIN right_side USING ( join_column [, ... ] ) -Similar to the inner join - but for every row of ``right_side`` that does not match, ``NULL`` values are returned for columns on ``left_side`` +Right outer joins are similar to inner joins, except that for every ``right_side`` row without a matching condition, a ``NULL`` value is returned for the corresponding ``left_side`` column. -Cross join +For a right outer join example, see :ref:`Right Join Example`. + + +.. _cross_joins: + +Cross Joins ^^^^^^^^^^^^^ +The following shows the correct syntax for creating an **cross join**: .. code-block:: postgres left_side CROSS JOIN right_side -A cartesian product - all rows match on all sides. For every row from ``left_side`` and ``right_side``, the result set will contain a row with all columns from ``left_side`` and all columns from ``right_side``. +Cross joins return all rows in all tables listed in a query, pairing each row in the first table with each row in the second table. A cross join is also known as a **Cartesian product**, and occurs when no relationship is defined between the two tables. + +The ``CROSS JOIN`` clause cannot have an ``ON`` clause, but the ``WHERE`` clause can be used to limit the result set. + +The following is an example of two tables that will be used as the basis for a cross join: + +.. image:: /_static/images/color_table.png + +The following is the output result of the cross join: + ++-----------+-------------+ +| **Color** | **Size** | ++===========+=============+ +| Red | Small | ++-----------+-------------+ +| Blue | Small | ++-----------+-------------+ +| Red | Medium | ++-----------+-------------+ +| Blue | Medium | ++-----------+-------------+ +| Red | Large | ++-----------+-------------+ +| Blue | Large | ++-----------+-------------+ +| Red | Extra Large | ++-----------+-------------+ +| Blue | Extra Large | ++-----------+-------------+ + + +For a cross join example, see :ref:`Cross Join Example`. + -The ``CROSS JOIN`` can't have an ``ON`` clause, but ``WHERE`` can be used to limit the result set. -Join conditions --------------------------------------------------------- -``ON value_expr`` -^^^^^^^^^^^^^^^^^^^^^^ -The ``ON`` condition is an expression that evaluates to a boolean to identify whether the rows match. +The ON Condition +------------- + +The ``ON`` condition is a value expression that generates a Boolean output to identify whether rows match. + +For example, the following is displayed when two name columns match: + + +.. code-block:: postgres -For example, ``ON left_side.name = right_side.name`` matches when both name columns match. + ON left_side.name = right_side.name -For ``LEFT`` and ``RIGHT`` joins, the ``ON`` clause is optional. However, if it is not specified, the result is a computationally intensive ``CROSS JOIN``. +The ``ON`` clause is optional for ``LEFT`` and ``RIGHT`` joins. However, excluding it results in a computationally intensive cross join. .. tip:: SQream DB does not support the ``USING`` syntax. However, queries can be easily rewritten. ``left_side JOIN right_side using (name)`` is equivalent to ``ON left_side.name = right_side.name`` -Examples +Join Type Examples ============= - -Assume a pair of tables with the following structure and content: +The examples in this section are based on a pair of tables with the following structure and content: .. code-block:: postgres @@ -108,10 +177,11 @@ Assume a pair of tables with the following structure and content: CREATE TABLE right_side (x INT); INSERT INTO right_side VALUES (2), (3), (4), (5), (6); -Inner join ------------- +.. _inner_join_example: -In inner joins, values that are not matched do not appear in the result set. +Inner Join Example +------------ +The following is an example of an inner join. .. code-block:: psql @@ -123,10 +193,13 @@ In inner joins, values that are not matched do not appear in the result set. 4 | 4 5 | 5 -Left join ------------- +Notice in the example above that values with no matching conditions do not appear in the result output. + +.. _left_join_example: -.. note:: Note the null values for ``1`` which were not matched. SQream DB outputs nulls last. +Left Join Example +------------ +The following is an example of a left join: .. code-block:: psql @@ -139,14 +212,17 @@ Left join 5 | 5 1 | \N -Right join ------------- +.. note:: Note the unmatched``NULL`` (``\N``) values for ``1``. SQream outputs ``NULL`` values last. + +.. _right_join_example: -.. note:: Note the null values for ``3``, ``6`` which were not matched. SQream DB outputs nulls last. +Right Join Example +------------ +The following is an example of a right join: .. code-block:: psql - t=> SELECT * FROM left_side AS l LEFT JOIN right_side AS r + t=> SELECT * FROM left_side AS l RIGHT JOIN right_side AS r . ON l.x = r.x; x | x0 ---+--- @@ -156,8 +232,13 @@ Right join \N | 3 \N | 6 -Cross join +.. note:: Note the unmatched ``NULL`` values for ``3`` and ``6``. SQream outputs ``NULL`` values last. + +.. _cross_join_example: + +Cross Join Example ------------- +The following is an example of a cross join: .. code-block:: psql @@ -185,7 +266,7 @@ Cross join 5 | 5 5 | 6 -Specifying multiple comma-separated tables is equivalent to a cross join, that can then be filtered with a ``WHERE`` clause.. +Specifying multiple comma-separated tables is equivalent to a cross join, which can be filtered with a ``WHERE`` clause as shown in the following example: .. code-block:: psql @@ -220,14 +301,12 @@ Specifying multiple comma-separated tables is equivalent to a cross join, that c 4 | 4 5 | 5 -Join hints +Join Hints ------------- -Join hints can be used to override the query compiler and choose a particular join algorithm. - -The available algorithms are ``LOOP`` (corresponding to non-indexed nested loop join algorithm), and ``MERGE`` (corresponding to sort merge join algorithm). +**Join hints** can be used to override the query compiler and choose a particular join algorithm. The available algorithms are ``LOOP`` (corresponding to non-indexed nested loop join algorithm), and ``MERGE`` (corresponding to sort merge join algorithm). If no algorithm is specified, a loop join is performed by default. -If no algorithm is specified, a loop join is performed by default. +The following is an example of using a join hint: .. code-block:: psql @@ -243,6 +322,4 @@ If no algorithm is specified, a loop join is performed by default. --+--- 2 | 2 4 | 4 - 5 | 5 - - + 5 | 5 \ No newline at end of file diff --git a/reference/sql/sql_syntax/keywords_and_identifiers.rst b/reference/sql/sql_syntax/keywords_and_identifiers.rst index 231b22e10..bc2cb1de6 100644 --- a/reference/sql/sql_syntax/keywords_and_identifiers.rst +++ b/reference/sql/sql_syntax/keywords_and_identifiers.rst @@ -1,158 +1,72 @@ .. _keywords_and_identifiers: *************************** -Keywords and Identifiers -*************************** - -SQL statements are made up of three components: - -* Keywords: Keywords have specific meaning in SQL, like ``SELECT``, ``CREATE``, ``WHERE``, etc. -* Identifiers: Names for things like tables, columns, databases, etc. - -.. _keywords: - -Keywords -=========== - -Keywords are special words that make up SQream DB's SQL syntax, and have a meaning in a statement. - -Keywords are not allowed as identifiers. :ref:`A full list of reserved keywords ` can be found at the end of this document. - -.. _identifiers: - Identifiers -============= - -Identifiers are typically used as database objects names, such as databases, tables, views or columns. Because of their use, these are also often also called "names". - -Identifiers can also be used to change a column name in the result (column alias) in a ``SELECT`` statement. - -Identifier rules ------------------- - -An identifier can be either quoted or unquoted, and are maximum 128 characters long. - -Regular identifiers follow these rules: - -#. Does not contain any special characters, apart for an underscore (``_``) -#. Case insensitive. SQream DB converts all identifiers to lowercase unless quoted. -#. Does not equal any :ref:`keyword `, like ``SELECT``, ``OR``, ``AND``, etc. - - -To bypass these limitations, use a quoted identifier by surrounding the identifier with double quotes (``"``). - -Quoted identifiers follow these rules: - -* Surrounded with double quotes ("). -* May contain any ASCII character, except ``@``, ``$`` or ``"``. -* Quoted identifiers are stored case-sensitive and must be referenced with double quotes. - -.. _reserved_keywords: - -Reserved keywords -================== - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Keyword - * - ``ALL`` - * - ``ANALYSE`` - * - ``ANALYZE`` - * - ``AND`` - * - ``ANY`` - * - ``ARRAY`` - * - ``AS`` - * - ``ASC`` - * - ``AUTHORIZATION`` - * - ``BINARY`` - * - ``BOTH`` - * - ``CASE`` - * - ``CAST`` - * - ``CHECK`` - * - ``COLLATE`` - * - ``COLUMN`` - * - ``CONCURRENTLY`` - * - ``CONSTRAINT`` - * - ``CREATE`` - * - ``CROSS`` - * - ``CURRENT_CATALOG`` - * - ``CURRENT_ROLE`` - * - ``CURRENT_TIME`` - * - ``CURRENT_USER`` - * - ``DEFAULT`` - * - ``DEFERRABLE`` - * - ``DESC`` - * - ``DISTINCT`` - * - ``DO`` - * - ``ELSE`` - * - ``END`` - * - ``EXCEPT`` - * - ``FALSE`` - * - ``FETCH`` - * - ``FOR`` - * - ``FREEZE`` - * - ``FROM`` - * - ``FULL`` - * - ``GRANT`` - * - ``GROUP`` - * - ``HASH`` - * - ``HAVING`` - * - ``ILIKE`` - * - ``IN`` - * - ``INITIALLY`` - * - ``INNER`` - * - ``INTERSECT`` - * - ``INTO`` - * - ``IS`` - * - ``ISNULL`` - * - ``JOIN`` - * - ``LEADING`` - * - ``LEFT`` - * - ``LIKE`` - * - ``LIMIT`` - * - ``LOCALTIME`` - * - ``LOCALTIMESTAMP`` - * - ``LOOP`` - * - ``MERGE`` - * - ``NATURAL`` - * - ``NOT`` - * - ``NOTNULL`` - * - ``NULL`` - * - ``OFFSET`` - * - ``ON`` - * - ``ONLY`` - * - ``OPTION`` - * - ``OR`` - * - ``ORDER`` - * - ``OUTER`` - * - ``OVER`` - * - ``OVERLAPS`` - * - ``PLACING`` - * - ``PRIMARY`` - * - ``REFERENCES`` - * - ``RETURNING`` - * - ``RIGHT`` - * - ``RLIKE`` - * - ``SELECT`` - * - ``SESSION_USER`` - * - ``SIMILAR`` - * - ``SOME`` - * - ``SYMMETRIC`` - * - ``SYMMETRIC`` - * - ``TABLE`` - * - ``THEN`` - * - ``TO`` - * - ``TRAILING`` - * - ``TRUE`` - * - ``UNION`` - * - ``UNIQUE`` - * - ``USER`` - * - ``USING`` - * - ``VARIADIC`` - * - ``VERBOSE`` - * - ``WHEN`` - * - ``WHERE`` - * - ``WINDOW`` - * - ``WITH`` \ No newline at end of file +*************************** +**Identifiers** are names given to SQL entities, such as tables, columns, databases, and variables. Identifiers must be unique so that entities can be correctly identified during the execution of a program. Identifiers can also be used to change a column name in the result (column alias) in a ``SELECT`` statement. Identifiers can be either quoted or unquoted and a maximum 128 characters long. + +Identifiers are sometimes referred to as "names". + +Regular identifiers must follow these rules: + +* Must not contain any special characters except for underscores (``_``). +* Must be case-insensitive. SQream converts all identifiers to lowercase unless quoted. +* Does not equal any keywords, such as ``SELECT``, ``OR``, or ``AND``, etc. + +To bypass the rules above you can surround an identifier with double quotes (``"``) or square brackets (``[]``). + +Quoted identifiers must follow these rules: + +* Must be surrounded with double quotes (``"``) or square brackets (``[]``). +* May contain any ASCII character except ``@``, ``$`` or ``"``. +* Must be case-sensitive and referenced with double quotes or square brackets (``[]``). + +Identifiers are different than **keywords**, which are predefined words reserved with specific meanings in a statement. Some examples of keywords are ``SELECT``, ``CREATE``, and ``WHERE``. Note that keywords **cannot** be used as identifiers. + +The following table shows a full list of the reserved keywords: + ++-------------------------------------------------------------------------------------------------+ +| **Keywords** | ++-------------------+---------------------+--------------------+------------------+---------------+ +| **A - C** | **C - G** | **H - N** | **N - S** | **S - W** | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``ALL`` | ``CURRENT_CATALOG`` | ``HASH`` | ``NOT`` | ``SIMILAR`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``ANALYSE`` | ``CURRENT_ROLE`` | ``HAVING`` | ``NOTNULL`` | ``SOME`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``ANALYZE`` | ``CURRENT_TIME`` | ``ILIKE`` | ``NULL`` | ``SYMMETRIC`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``AND`` | ``CURRENT_USER`` | ``IN`` | ``OFFSET`` | ``TABLE`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``ANY`` | ``DEFAULT`` | ``INITIALLY`` | ``ON`` | ``THEN`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``ARRAY`` | ``DEFERRABLE`` | ``INNER`` | ``ONLY`` | ``TO`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``AS`` | ``DESC`` | ``INTERSECT`` | ``OPTION`` | ``TRAILING`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``ASC`` | ``DISTINCT`` | ``INTO`` | ``OR`` | ``TRUE`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``AUTHORIZATION`` | ``DO`` | ``IS`` | ``ORDER`` | ``UNION`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``BINARY`` | ``ELSE`` | ``ISNULL`` | ``OUTER`` | ``UNIQUE`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``BOTH`` | ``END`` | ``JOIN`` | ``OVER`` | ``USER`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``CASE`` | ``EXCEPT`` | ``LEADING`` | ``OVERLAPS`` | ``USING`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``CAST`` | ``FALSE`` | ``LEFT`` | ``PLACING`` | ``VARIADIC`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``CHECK`` | ``FETCH`` | ``LIKE`` | ``PRIMARY`` | ``VERBOSE`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``COLLATE`` | ``FOR`` | ``LIMIT`` | ``REFERENCES`` | ``WHEN`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``COLUMN`` | ``FREEZE`` | ``LOCALTIME`` | ``RETURNING`` | ``WHERE`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``CONCURRENTLY`` | ``FROM`` | ``LOCALTIMESTAMP`` | ``RIGHT`` | ``WINDOW`` | ++-------------------+---------------------+--------------------+------------------+---------------+ +| ``CONSTRAINT`` | ``FULL`` | ``LOOP`` | ``RLIKE`` | ``WITH`` | ++-------------------+---------------------+--------------------+------------------+ | +| ``CREATE`` | ``GRANT`` | ``MERGE`` | ``SELECT`` | | ++-------------------+---------------------+--------------------+------------------+ | +| ``CROSS`` | ``GROUP`` | ``NATURAL`` | ``SESSION_USER`` | | ++-------------------+---------------------+--------------------+------------------+---------------+ diff --git a/reference/sql/sql_syntax/literals.rst b/reference/sql/sql_syntax/literals.rst index 5ec0e6512..906684590 100644 --- a/reference/sql/sql_syntax/literals.rst +++ b/reference/sql/sql_syntax/literals.rst @@ -4,21 +4,23 @@ Literals *************************** + Literals represent constant values. -SQream DB contains several types of literals: +SQream DB contains the following types of literals: -* :ref:`Number literals` - define numbers such as ``1.3``, ``-5`` +* :ref:`Numeric literals` - define numbers such as ``1.3``, ``-5`` * :ref:`String literals` - define text values like ``'Foxes are cool'``, ``'1997-01-01'`` * :ref:`Typed literals` - define values with explicit types like ``(3.0 :: float)`` -* :ref:`Other constants` - predfined values like ``NULL`` or ``TRUE`` +* :ref:`Boolean literals` - define values that include ``true`` and ``false`` +* :ref:`Other constants` - predefined values like ``NULL`` or ``TRUE`` -.. _number_literals: +.. _numeric_literals: -Number literals +Numeric Literals =================== -A number literal can be expressed in the following way: +**Numeric literals** can be expressed as follows: .. code-block:: postgres @@ -55,16 +57,19 @@ Examples For example, any number containing the decimal point will be considered ``FLOAT`` by default. Any whole number will considered ``INT``, unless the value is larger than the :ref:`maximum value`, in which case the type will become a ``BIGINT``. +.. note:: + A numeric literal that contains neither a decimal point nor an exponent is considered ``INT`` by default if its value fits in type ``INT`` (32 bits). If not, it is considered ``BIGINT`` by default if its value fits in type ``BIGINT`` (64 bits). If neither are true, it is considered ``FLOAT``. Literals that contain decimal points and/or exponents are always considered ``FLOAT``. + .. _string_literals: -String literals +String Literals ================== -String literals are string (text) values, encoded either in ASCII or UTF-8. +**String literals** are string (text) values, encoded either in ASCII or UTF-8. -A string literal is quoted with single quotes (``'``) or dollars (``$$``) +String literals are surrounded by single quotes (``'``) or dollars (``$$``) -.. tip:: To use a single quote in a string, repeat the single quote twice. See examples below. +.. tip:: To use a single quote in a string, use a repeated single quote. Examples @@ -86,24 +91,131 @@ The actual data type of the value changes based on context, the format used, and .. code-block:: postgres INSERT INTO cool_dates(date_col, reason) VALUES ('1955-11-05', 'Doc Brown discovers flux capacitor'); + +This section describes the following types of literals: + +* :ref:`Regular string literals` +* :ref:`Dollar-quoted string literals` +* :ref:`Escaped string literals` + + +.. _regular_literals: + +Regular String Literals +----------------------- +In SQL, a **regular string literal** is a sequence of zero or more characters bound by single quotes (``'``): + +.. code-block:: postgres + + 'This is a string'. + +You can include a single-quote character in a string literal with two consecutive single quotes (``''``): + +.. code-block:: postgres + + 'Dianne''s horse'. + +Note that two adjacent single quotes is not the same as a double-quote character (``"``). + +Examples +------------ + +The following are some examples of regular string literals: + +.. code-block:: postgres + + '123' + + 'אבג' + + 'a''b' + + '' + +.. _dollar_quoted_string_literals: + +Dollar-Quoted String Literals +----------------------- +**Dollar-quoted string literals** consist of a dollar sign (``$``), an optional "tag" of zero or more characters, another dollar sign, an arbitrary sequence of characters that make up the string content, a dollar sign, the same tag at the beginning of the dollar quote, and another dollar sign. + + +Examples +------------ + +For example, below are two different ways to specify the string ``Dianne's horse`` using dollar-quoted string literals: + +.. code-block:: postgres + + $$Dianne's horse$$ + $$Dianne's horse$$ + +Note that you can use single quotes inside the dollar-quoted string without an escape. Because the string is always written literally, you do not need to escape any characters inside a dollar-quoted string. Backslashes and dollar signs indicate no specific functions unless they are part of a sequence matching the opening tag. + +Any used tags in a dollar-quoted string follow the same rules as for unquoted identifiers, except that they cannot contain a dollar sign. + +In addition, because tags are case sensitive, ``$$String content$$`` is correct, but ``$$String content$$`` is incorrect. + +A dollar-quoted string that follows a keyword or identifier must be separated from it by whitespace (such as spaces, tabs, or newlines). If you do not separate them with whitespace, the dollar-quoting delimiter is taken as part of the preceding identifier. + +.. _escaped_string_literals: + +Escaped String Literals +----------------------- +Because regular string literals do not support inserting special characters (such as new lines), the **escaped string literals** syntax was added to support inserting special characters with an escaping syntax. + +In addition to being enclosed by single quotes (e.g. 'abc'), escaped string literals are preceded by a capital ``E``. + +.. code-block:: postgres + + E'abc' + +The character sequence inside the single quotes can contain escaped characters in addition to regular characters, shown below: + +.. list-table:: + :widths: 25 85 + :header-rows: 1 + + * - Sequence + - Interpretation + * - ``\b`` + - Inserts a backspace. + * - ``\f`` + - Inserts a form feed. + * - ``\n`` + - Inserts a newline. + * - ``\r`` + - Inserts a carriage return. + * - ``\t`` + - Inserts a tab. + * - ``\o``, ``\oo``, ```\ooo``` (``o`` = ``0`` - ``7``) + - Inserts an octal byte value. This sequence is currently not supported. + * - ``\xh``, ``\xhh`` (``h`` = ``0`` - ``9``, ``A`` - ``F``) + - Inserts a hexadecimal byte value. This sequence is currently not supported. + * - ``\uxxxx``, ``\Uxxxxxxxx`` + - Inserts a 16 or 32-bit hexadecimal unicode character value (x = 0 - 9, A - F). + +Excluding the characters in the table above, escaped string literals take all other characters following a backslash literally. To include a backslash character, use two consecutive backslashes (``\\``). You can use a single quote in an escape string by writing ``\'``, in addition to the normal method (``''``). .. _typed_literals: -Typed literals +Typed Literals ================ -A typed literal allows the creation of any data type using either of the following syntaxes: +**Typed literals** allow you to create any data type using either of the following syntaxes: .. code-block:: postgres CAST(literal AS type_name) - -- or + +.. code-block:: postgres + literal :: type_name See also :ref:`cast` for more information about supported casts. -Syntax reference +Syntax Reference ------------------- +The following is a syntax reference for typed literals: .. code-block:: postgres @@ -143,12 +255,25 @@ Examples CAST(42 :: FLOAT) +.. _boolean_literals: + +Boolean Literals +=================== +**Boolean literals** include the keywords ``true`` or ``false``. + +Example +---------- + +.. code-block:: postgres + + INSERT INTO animals VALUES ('fox',true), ('cat',true), ('kiwi',false); + .. _constants: -Other constants +Other Constants ================ -* ``TRUE`` and ``FALSE`` are interpreted as values of type ``BOOL``. - -* ``NULL`` - which has no type by itself. The type is inferred from context during query compilation. +The following other constants can be used: +* ``TRUE`` and ``FALSE`` - interpreted as values of type ``BOOL``. +* ``NULL`` - which has no type of its own. The type is inferred from context during query compilation. diff --git a/reference/sql/sql_syntax/nba-t10.csv b/reference/sql/sql_syntax/nba-t10.csv index 024530355..c3d41e729 100644 --- a/reference/sql/sql_syntax/nba-t10.csv +++ b/reference/sql/sql_syntax/nba-t10.csv @@ -1,10 +1,9 @@ -Name,Team,Number,Position,Age,Height,Weight,College,Salary -Avery Bradley,Boston Celtics,0.0,PG,25.0,6-2,180.0,Texas,7730337.0 -Jae Crowder,Boston Celtics,99.0,SF,25.0,6-6,235.0,Marquette,6796117.0 -John Holland,Boston Celtics,30.0,SG,27.0,6-5,205.0,Boston University, -R.J. Hunter,Boston Celtics,28.0,SG,22.0,6-5,185.0,Georgia State,1148640.0 -Jonas Jerebko,Boston Celtics,8.0,PF,29.0,6-10,231.0,,5000000.0 -Amir Johnson,Boston Celtics,90.0,PF,29.0,6-9,240.0,,12000000.0 -Jordan Mickey,Boston Celtics,55.0,PF,21.0,6-8,235.0,LSU,1170960.0 -Kelly Olynyk,Boston Celtics,41.0,C,25.0,7-0,238.0,Gonzaga,2165160.0 -Terry Rozier,Boston Celtics,12.0,PG,22.0,6-2,190.0,Louisville,1824360.0 +Avery Bradley,Boston Celtics,0,PG,25,2-Jun,180,Texas,7730337 +Jae Crowder,Boston Celtics,99,SF,25,6-Jun,235,Marquette,6796117 +John Holland,Boston Celtics,30,SG,27,5-Jun,205,Boston University, +R.J. Hunter,Boston Celtics,28,SG,22,5-Jun,185,Georgia State,1148640 +Jonas Jerebko,Boston Celtics,8,PF,29,10-Jun,231,,5000000 +Amir Johnson,Boston Celtics,90,PF,29,9-Jun,240,,12000000 +Jordan Mickey,Boston Celtics,55,PF,21,8-Jun,235,LSU,1170960 +Kelly Olynyk,Boston Celtics,41,C,25,Jul-00,238,Gonzaga,2165160 +Terry Rozier,Boston Celtics,12,PG,22,2-Jun,190,Louisville,1824360 diff --git a/reference/sql/sql_syntax/window_functions.rst b/reference/sql/sql_syntax/window_functions.rst index 4c07f5a5d..cb87e085e 100644 --- a/reference/sql/sql_syntax/window_functions.rst +++ b/reference/sql/sql_syntax/window_functions.rst @@ -1,7 +1,7 @@ .. _window_functions: ******************** -Window functions +Window Functions ******************** Window functions are functions applied over a subset (known as a window) of the rows returned by a :ref:`select` query. @@ -30,6 +30,13 @@ Syntax | RANK | ROW_NUMBER | SUM + | FIRST_VALUE + | LAST_VALUE + | NTH_VALUE + | DENSE_RANK + | PERCENT_RANK + | CUME_DIST + | NTILE frame_clause ::= @@ -68,38 +75,62 @@ Arguments - An expression or column reference to order by -Supported window functions +Supported Window Functions =========================== -.. list-table:: Window function aggregations - :widths: auto +.. list-table:: Window Aggregation Functions + :widths: 16 200 :header-rows: 1 * - Function + - Description * - :ref:`avg` + - Returns the average of numeric values. * - :ref:`count` + - Returns the count of numeric values, or only the distinct values. * - :ref:`max` + - Returns the maximum values. * - :ref:`min` + - Returns the minimum values. * - :ref:`sum` + - Returns the sum of numeric values, or only the distinct values. + + -.. versionchanged:: 2020.1 - :ref:`count` and :ref:`avg` are supported in window functions from v2020.1. -.. list-table:: Ranking functions - :widths: auto +.. list-table:: Ranking Functions + :widths: 15 200 :header-rows: 1 - + * - Function + - Description * - :ref:`lag` + - Returns a value from a previous row within the partition of a result set. * - :ref:`lead` - * - :ref:`rank` + - Returns a value from a subsequent row within the partition of a result set. * - :ref:`row_number` - -.. versionchanged:: 2020.2 - :ref:`lag` and :ref:`lead` are supported from v2020.2. - - -How window functions work + - Returns the row number of each row within the partition of a result set. + * - :ref:`rank` + - Returns the rank of each row within the partition of a result set. + * - :ref:`first_value` + - Returns the value in the first row of a window. + * - :ref:`last_value` + - Returns the value in the last row of a window. + * - :ref:`nth_value` + - Returns the value in a specified (``n``) row of a window. + * - :ref:`dense_rank` + - Returns the rank of the current row with no gaps. + * - :ref:`percent_rank` + - Returns the relative rank of the current row. + * - :ref:`cume_dist` + - Returns the cumulative distribution of rows. + * - :ref:`ntile` + - Returns an integer ranging between ``1`` and the argument value, dividing the partitions as equally as possible. + + + + +How Window Functions Work ============================ A window function operates on a subset ("window") of rows. @@ -137,8 +168,7 @@ Without ``ORDER BY``, rows are processed in an unspecified order. Frames ------- -.. versionchanged:: 2020.1 - Frames are supported from v2020.1. + .. note:: Frames and frame exclusions have been tested extensively, but are a complex feature. They are released as a preview in v2020.1 pending longer-term testing. @@ -171,17 +201,14 @@ Restrictions For example ``RANGE BETWEEN CURRENT ROW AND 7 PRECEDING`` is not allowed. However, while ``ROWS BETWEEN 7 PRECEDING AND 8 PRECEDING`` is allowed, it would never select any rows. -Frame exclusion +Frame Exclusion ----------------- The ``frame_exclusion`` option allows rows around the current row to be excluded from the frame, even if they would be included according to the frame start and frame end options. ``EXCLUDE CURRENT ROW`` excludes the current row from the frame. ``EXCLUDE GROUP`` excludes the current row and its ordering peers from the frame. ``EXCLUDE TIES`` excludes any peers of the current row from the frame, but not the current row itself. ``EXCLUDE NO OTHERS`` simply specifies explicitly the default behavior of not excluding the current row or its peers. Limitations ================== - -* At this phase, text columns are not supported in window function expressions. - -* Window function calls are permitted only in the :ref:`select` list. +Window functions do not support the Numeric data type. @@ -213,7 +240,7 @@ Here's a peek at the table contents (:download:`Download nba.csv TO )``, to convert a value from one type to another. For example, ``CAST('1997-01-01' TO DATE)``, ``CAST(3.45 TO SMALLINT)``, ``CAST(some_column TO VARCHAR(30))``. -* `` :: ``, a shorthand for the ``CAST`` syntax. For example, ``'1997-01-01' :: DATE``, ``3.45 :: SMALLINT``, ``(3+5) :: BIGINT``. -* See the :ref:`SQL functions reference ` for additional functions that convert from a specific value which is not an SQL type, such as :ref:`from_unixts`, etc. - -More details about the supported casts for each type in the following section. - -Data type reference -====================== - - -Boolean (``BOOL``) -------------------- -A ``BOOL`` datatype is designed to store Boolean values of ``true`` or ``false``. - -Syntax -^^^^^^^^ - -A ``BOOL`` type can accept either ``true`` or ``false`` (case insensitive). - -When loading from CSV, ``BOOL`` columns can accept ``0`` as ``false`` and ``1`` as ``true``. - -Size -^^^^^^ - -A ``BOOL`` type is 1 byte, but resulting average data sizes could be lower after compression. - -Examples -^^^^^^^^^^ - -.. code-block:: postgres - - CREATE TABLE animals (name VARCHAR(15), is_angry BOOL); - - INSERT INTO animals VALUES ('fox',true), ('cat',true), ('kiwi',false); - - SELECT name, CASE WHEN is_angry THEN 'Is really angry!' else 'Is not angry' END FROM animals; - -.. code-block:: text - - "fox","Is really angry!" - "cat","Is really angry!" - "kiwi","Is not angry" - -Casts and conversions -^^^^^^^^^^^^^^^^^^^^^^^ - -A ``BOOL`` value can be converted to: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Type - - Details - * - ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT`` - - ``true`` → ``1``, ``false`` → ``0`` - * - ``REAL``, ``DOUBLE`` - - ``true`` → ``1.0``, ``false`` → ``0.0`` - - - - - - - - - -Integers (``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT``) ------------------------------------------------------------- -Integer datatypes are designed to store whole numbers. - -For information about identity sequences (sometimes called auto-increment or auto-numbers), see :ref:`identity`. - -Integer types -^^^^^^^^^^^^^^^^^^^ -.. list-table:: Integer types - :widths: auto - :header-rows: 1 - - * - Name - - Details - - Data size (not null, uncompressed) - - Example - * - ``TINYINT`` - - Unsigned integer (0 - 255) - - 1 byte - - ``5`` - * - ``SMALLINT`` - - Integer (-32,768 - 32,767) - - 2 bytes - - ``-155`` - * - ``INT`` - - Integer (-2,147,483,648 - 2,147,483,647) - - 4 bytes - - ``1648813`` - * - ``BIGINT`` - - Integer (-9,223,372,036,854,775,808 - 9,223,372,036,854,775,807) - - 8 bytes - - ``36124441255243`` - -Syntax -^^^^^^^^ - -An integer can be entered as a regular literal, such as ``12``, ``-365``. - -Size -^^^^^^ - -Integer types range between 1, 2, 4, and 8 bytes - but resulting average data sizes could be lower after compression. - -Examples -^^^^^^^^^^ - -.. code-block:: postgres - - CREATE TABLE cool_numbers (a INT NOT NULL, b TINYINT, c SMALLINT, d BIGINT); - - INSERT INTO cool_numbers VALUES (1,2,3,4), (-5, 127, 32000, 45000000000); - - SELECT * FROM cool_numbers; - -.. code-block:: text - - 1,2,3,4 - -5,127,32000,45000000000 - -Casts and conversions -^^^^^^^^^^^^^^^^^^^^^^^ - -Integer values can be converted to: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Type - - Details - * - ``REAL``, ``DOUBLE`` - - ``1`` → ``1.0``, ``-32`` → ``-32.0`` - * - ``VARCHAR(n)`` (All numberic values must fit in the string length) - - ``1`` → ``'1'``, ``2451`` → ``'2451'`` - - -Floating point (``REAL``, ``DOUBLE``) ------------------------------------------------- -``REAL`` and ``DOUBLE`` are inexact floating point data types, designed to store up to 9 or 17 digits of precision respectively. - -The floating point representation is based on `IEEE 754 `_. - -Floating point types -^^^^^^^^^^^^^^^^^^^^^^ -.. list-table:: Floating point types - :widths: auto - :header-rows: 1 - - * - Name - - Details - - Data size (not null, uncompressed) - - Example - * - ``REAL`` - - Single precision floating point (inexact) - - 4 bytes - - ``3.141`` - * - ``DOUBLE`` - - Double precision floating point (inexact) - - 8 bytes - - ``0.000003`` - -Aliases -^^^^^^^^^^ - -``DOUBLE`` is also known as ``FLOAT``. - - -Syntax -^^^^^^^^ - -A double precision floating point can be entered as a regular literal, such as ``3.14``, ``2.718``, ``.34``, ``2.71e-45``. - -To enter a ``REAL`` floating point number, cast the value. For example, ``(3.14 :: REAL)``. - -Size -^^^^^^ - -Floating point types are either 4 or 8 bytes, but size could be lower after compression. - -Examples -^^^^^^^^^^ - -.. code-block:: postgres - - CREATE TABLE cool_numbers (a REAL NOT NULL, b DOUBLE); - - INSERT INTO cool_numbers VALUES (1,2), (3.14159265358979, 2.718281828459); - - SELECT * FROM cool_numbers; - -.. code-block:: text - - 1.0,2.0 - 3.1415927,2.718281828459 - -.. note:: Most SQL clients control display precision of floating point numbers, and values may appear differently in some clients. - -Casts and conversions -^^^^^^^^^^^^^^^^^^^^^^^ - -Floating point values can be converted to: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Type - - Details - * - ``BOOL`` - - ``1.0`` → ``true``, ``0.0`` → ``false`` - * - ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT`` - - ``2.0`` → ``2``, ``3.14159265358979`` → ``3``, ``2.718281828459`` → ``2``, ``0.5`` → ``0``, ``1.5`` → ``1`` - * - ``VARCHAR(n)`` (n > 6 recommended) - - ``1`` → ``'1.0000'``, ``3.14159265358979`` → ``'3.1416'`` - -.. note:: As shown in the above examples, when casting ``real`` to ``int``, we round down. - -String types (``TEXT``, ``VARCHAR``) ------------------------------------------------- -``TEXT`` and ``VARCHAR`` are types designed for storing text or strings of characters. - -SQream DB separates ASCII (``VARCHAR``) and UTF-8 representations (``TEXT``). - -.. note:: The data type ``NVARCHAR`` has been deprecated by ``TEXT`` as of version 2020.1. - -String types -^^^^^^^^^^^^^^^^^^^^^^ -.. list-table:: String types - :widths: auto - :header-rows: 1 - - * - Name - - Details - - Data size (not null, uncompressed) - - Example - * - ``TEXT [(n)]``, ``NVARCHAR (n)`` - - Varaiable length string - UTF-8 unicode. ``NVARCHAR`` is synonymous with ``TEXT``. - - Up to ``4*n`` bytes - - ``'キウイは楽しい鳥です'`` - * - ``VARCHAR (n)`` - - Variable length string - ASCII only - - ``n`` bytes - - ``'Kiwis have tiny wings, but cannot fly.'`` - -Length -^^^^^^^^^ - -When using ``TEXT``, specifying a size is optional. If not specified, the text field carries no constraints. - -To limit the size of the input, use ``VARCHAR(n)`` or ``TEXT(n)``, where n is the number of characters allowed. - -* If the data exceeds the column length limit on ``INSERT`` or ``COPY`` operations, SQream DB will return an error. - -* When casting or converting, the string has to fit in the target. For example, ``'Kiwis are weird birds' :: VARCHAR(5)`` will return an error. Use ``SUBSTRING`` to truncate the length of the string. - -* ``VARCHAR`` strings are padded with spaces. - -Syntax -^^^^^^^^ - -String types can be written with standard SQL string literals, which are enclosed with single quotes, such as -``'Kiwi bird'``. To include a single quote in the string, repeat the quote twice: ``'Kiwi bird''s wings are tiny'``. - -String literals can also be dollar-quoted with the dollar sign ``$``. For example: ``$$Kiwi bird's wings are tiny$$`` is the same as ``'Kiwi bird''s wings are tiny'``. - -Size -^^^^^^ - -``VARCHAR(n)`` can occupy up to *n* bytes, whereas ``TEXT(n)`` can occupy up to *4*n* bytes. -However, the size of strings is variable and is compressed by SQream DB. - -Examples -^^^^^^^^^^ - -.. code-block:: postgres - - CREATE TABLE cool_strings (a VARCHAR(25) NOT NULL, b TEXT); - - INSERT INTO cool_strings VALUES ('hello world', 'Hello to kiwi birds specifically'); - - INSERT INTO cool_strings VALUES ('This is ASCII only', 'But this column can contain 中文文字'); - - SELECT * FROM cool_strings; - -.. code-block:: text - - hello world ,Hello to kiwi birds specifically - This is ASCII only,But this column can contain 中文文字 - -.. note:: Most clients control display precision of floating point numbers, and values may appear differently in some clients. - -Casts and conversions -^^^^^^^^^^^^^^^^^^^^^^^ - -String values can be converted to: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Type - - Details - * - ``BOOL`` - - ``'true'`` → ``true``, ``'false'`` → ``false`` - * - ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT`` - - ``'2'`` → ``2``, ``'-128'`` → ``-128`` - * - ``REAL``, ``DOUBLE`` - - ``'2.0'`` → ``2.0``, ``'3.141592'`` → ``3.141592`` - * - ``DATE``, ``DATETIME`` - - Requires a supported format, such as ``'1955-11-05`` → ``date '1955-11-05'``, ``'1955-11-05 01:24:00.000'`` → ``'1955-11-05 01:24:00.000'`` - - - -Date types (``DATE``, ``DATETIME``) ------------------------------------------------- - -``DATE`` is a type designed for storing year, month, and day. - -``DATETIME`` is a type designed for storing year, month, day, hour, minute, seconds, and milliseconds in UTC with 1 millisecond precision. - - -Date types -^^^^^^^^^^^^^^^^^^^^^^ -.. list-table:: Date types - :widths: auto - :header-rows: 1 - - * - Name - - Details - - Data size (not null, uncompressed) - - Example - * - ``DATE`` - - Date - - 4 bytes - - ``'1955-11-05'`` - * - ``DATETIME`` - - Date and time pairing in UTC - - 8 bytes - - ``'1955-11-05 01:24:00.000'`` - -Aliases -^^^^^^^^^^ - -``DATETIME`` is also known as ``TIMESTAMP``. - - -Syntax -^^^^^^^^ - -``DATE`` values are formatted as string literals. For example, ``'1955-11-05'`` or ``date '1955-11-05'``. - -``DATETIME`` values are formatted as string literals conforming to `ISO 8601 `_, for example ``'1955-11-05 01:26:00'``. - -SQream DB will attempt to guess if the string literal is a date or datetime based on context, for example when used in date-specific functions. - -Size -^^^^^^ - -A ``DATE`` column is 4 bytes in length, while a ``DATETIME`` column is 8 bytes in length. - -However, the size of these values is compressed by SQream DB. - -Examples -^^^^^^^^^^ - -.. code-block:: postgres - - CREATE TABLE important_dates (a DATE, b DATETIME); - - INSERT INTO important_dates VALUES ('1997-01-01', '1955-11-05 01:24'); - - SELECT * FROM important_dates; - -.. code-block:: text - - 1997-01-01,1955-11-05 01:24:00.0 - -.. code-block:: postgres - - SELECT a :: DATETIME, b :: DATE FROM important_dates; - -.. code-block:: text - - 1997-01-01 00:00:00.0,1955-11-05 - - -.. warning:: Some client applications may alter the ``DATETIME`` value by modifying the timezone. - -Casts and conversions -^^^^^^^^^^^^^^^^^^^^^^^ - -``DATE`` and ``DATETIME`` values can be converted to: - -.. list-table:: - :widths: auto - :header-rows: 1 - - * - Type - - Details - * - ``VARCHAR(n)`` - - ``'1997-01-01'`` → ``'1997-01-01'``, ``'1955-11-05 01:24'`` → ``'1955-11-05 01:24:00.000'`` diff --git a/reference/sql_feature_support.rst b/reference/sql_feature_support.rst index 6e537bcdd..8337724ca 100644 --- a/reference/sql_feature_support.rst +++ b/reference/sql_feature_support.rst @@ -11,12 +11,12 @@ To understand which ANSI SQL and other SQL features SQream DB supports, use the :local: -Data types and values +Data Types and Values ========================= Read more about :ref:`supported data types`. -.. list-table:: Value +.. list-table:: Data Types and Values :widths: auto :header-rows: 1 @@ -24,46 +24,46 @@ Read more about :ref:`supported data types`. - Supported - Further information * - ``BOOL`` - - ✓ + - Yes - Boolean values * - ``TINTINT`` - - ✓ + - Yes - Unsigned 1 byte integer (0 - 255) * - ``SMALLINT`` - - ✓ + - Yes - 2 byte integer (-32,768 - 32,767) * - ``INT`` - - ✓ + - Yes - 4 byte integer (-2,147,483,648 - 2,147,483,647) * - ``BIGINT`` - - ✓ + - Yes - 8 byte integer (-9,223,372,036,854,775,808 - 9,223,372,036,854,775,807) * - ``REAL`` - - ✓ + - Yes - 4 byte floating point * - ``DOUBLE``, ``FLOAT`` - - ✓ + - Yes - 8 byte floating point * - ``DECIMAL``, ``NUMERIC`` - - Planned - - Fixed-point numbers. Use ``DOUBLE`` instead + - Yes + - Fixed-point numbers. * - ``VARCHAR`` - - ✓ + - Yes - Variable length string - ASCII only - * - ``TEXT``, ``NVARCHAR`` - - ✓ + * - ``TEXT`` + - Yes - Variable length string - UTF-8 encoded * - ``DATE`` - - ✓ + - Yes - Date * - ``DATETIME``, ``TIMESTAMP`` - - ✓ + - Yes - Date and time * - ``NULL`` - - ✓ + - Yes - ``NULL`` values * - ``TIME`` - - ✗ + - No - Can be stored as a text string or as part of a ``DATETIME`` @@ -78,13 +78,13 @@ Contraints - Supported - Further information * - Not null - - ✓ + - Yes - ``NOT NULL`` * - Default values - - ✓ + - Yes - ``DEFAULT`` * - ``AUTO INCREMENT`` - - ✓ Different name + - Yes Different name - ``IDENTITY`` @@ -107,10 +107,10 @@ SQream DB does not support explicit indexing, but does support clustering keys. Read more about :ref:`clustering keys` and our :ref:`metadata system`. -Schema changes +Schema Changes ================ -.. list-table:: Schema changes +.. list-table:: Schema Changes :widths: auto :header-rows: 1 @@ -118,43 +118,43 @@ Schema changes - Supported - Further information * - ``ALTER TABLE`` - - ✓ + - Yes - :ref:`alter_table` - Add column, alter column, drop column, rename column, rename table, modify clustering keys * - Rename database - - ✗ + - No - * - Rename table - - ✓ + - Yes - :ref:`rename_table` * - Rename column - - ✓ + - Yes - :ref:`rename_column` * - Add column - - ✓ + - Yes - :ref:`add_column` * - Remove column - - ✓ + - Yes - :ref:`drop_column` * - Alter column data type - - ✗ + - No - * - Add / modify clustering keys - - ✓ + - Yes - :ref:`cluster_by` * - Drop clustering keys - - ✓ + - Yes - :ref:`drop_clustering_key` * - Add / Remove constraints - - ✗ + - No - * - Rename schema - - ✗ + - No - * - Drop schema - - ✓ + - Yes - :ref:`drop_schema` * - Alter default schema per user - - ✓ + - Yes - :ref:`alter_default_schema` @@ -169,28 +169,28 @@ Statements - Supported - Further information * - SELECT - - ✓ + - Yes - :ref:`select` * - CREATE TABLE - - ✓ + - Yes - :ref:`create_table` * - CREATE FOREIGN / EXTERNAL TABLE - - ✓ + - Yes - :ref:`create_foreign_table` * - DELETE - - ✓ + - Yes - :ref:`delete_guide` * - INSERT - - ✓ + - Yes - :ref:`insert`, :ref:`copy_from` * - TRUNCATE - - ✓ + - Yes - :ref:`truncate` * - UPDATE - - ✗ + - No - * - VALUES - - ✓ + - Yes - :ref:`values` Clauses @@ -204,25 +204,25 @@ Clauses - Supported - Further information * - ``LIMIT`` / ``TOP`` - - ✓ + - Yes - * - ``LIMIT`` with ``OFFSET`` - - ✗ + - No - * - ``WHERE`` - - ✓ + - Yes - * - ``HAVING`` - - ✓ + - Yes - * - ``OVER`` - - ✓ + - Yes - -Table expressions +Table Expressions ==================== -.. list-table:: Table expressions +.. list-table:: Table Expressions :widths: auto :header-rows: 1 @@ -230,28 +230,28 @@ Table expressions - Supported - Further information * - Tables, Views - - ✓ + - Yes - * - Aliases, ``AS`` - - ✓ + - Yes - * - ``JOIN`` - ``INNER``, ``LEFT [ OUTER ]``, ``RIGHT [ OUTER ]``, ``CROSS`` - - ✓ + - Yes - * - Table expression subqueries - - ✓ + - Yes - * - Scalar subqueries - - ✗ + - No - -Scalar expressions +Scalar Expressions ==================== Read more about :ref:`scalar_expressions`. -.. list-table:: Scalar expressions +.. list-table:: Scalar Expressions :widths: auto :header-rows: 1 @@ -259,34 +259,34 @@ Read more about :ref:`scalar_expressions`. - Supported - Further information * - Common functions - - ✓ + - Yes - ``CURRENT_TIMESTAMP``, ``SUBSTRING``, ``TRIM``, ``EXTRACT``, etc. * - Comparison operators - - ✓ + - Yes - ``<``, ``<=``, ``>``, ``>=``, ``=``, ``<>, !=``, ``IS``, ``IS NOT`` * - Boolean operators - - ✓ + - Yes - ``AND``, ``NOT``, ``OR`` * - Conditional expressions - - ✓ + - Yes - ``CASE .. WHEN`` * - Conditional functions - - ✓ + - Yes - ``COALESCE`` * - Pattern matching - - ✓ + - Yes - ``LIKE``, ``RLIKE``, ``ISPREFIXOF``, ``CHARINDEX``, ``PATINDEX`` * - REGEX POSIX pattern matching - - ✓ + - Yes - ``RLIKE``, ``REGEXP_COUNT``, ``REGEXP_INSTR``, ``REGEXP_SUBSTR``, * - ``EXISTS`` - - ✗ + - No - * - ``IN``, ``NOT IN`` - Partial - Literal values only - * - Bitwise arithemtic - - ✓ + * - Bitwise arithmetic + - Yes - ``&``, ``|``, ``XOR``, ``~``, ``>>``, ``<<`` @@ -304,24 +304,24 @@ Read more about :ref:`access_control` in SQream DB. - Supported - Further information * - Roles as users and groups - - ✓ + - Yes - * - Object default permissions - - ✓ + - Yes - * - Column / Row based permissions - - ✗ + - No - * - Object ownership - - ✗ + - No - -Extra functionality +Extra Functionality ====================== -.. list-table:: Extra functionality +.. list-table:: Extra Functionality :widths: auto :header-rows: 1 @@ -329,20 +329,20 @@ Extra functionality - Supported - Further information * - Information schema - - ✓ + - Yes - :ref:`catalog_reference` * - Views - - ✓ + - Yes - :ref:`create_view` * - Window functions - - ✓ + - Yes - :ref:`window_functions` * - CTEs - - ✓ + - Yes - :ref:`common_table_expressions` * - Saved queries, Saved queries with parameters - - ✓ + - Yes - :ref:`saved_queries` * - Sequences - - ✓ + - Yes - :ref:`identity` diff --git a/releases/2019.2.1.rst b/releases/2019.2.1.rst index 7de2d2e8d..c9c96b59b 100644 --- a/releases/2019.2.1.rst +++ b/releases/2019.2.1.rst @@ -1,7 +1,7 @@ .. _2019.2.1: ****************************** -What's new in 2019.2.1 +Release Notes 2019.2.1 ****************************** * 250 bugs fixed. Thanks to all of our customers and an unprecedented number of deployments for helping us find and fix these! @@ -29,10 +29,10 @@ Improvements * SQream DB CLI’s history can be disabled, by passing ``./ClientCmd --no-history`` -Behaviour changes +Behaviour Changes ===================== -* SQream DB no longer applies an implicit cast from a long text column to a shorter text column (``VARCHAR``/``NVARCHAR``). This means some ``INSERT``/``COPY`` operations will now error instead of truncating the text. This is intended to prevent accidental truncation of text columns. If you want the old truncation behaviour, you can use the ``SUBSTRING`` function to truncate the text. +* SQream DB no longer applies an implicit cast from a long text column to a shorter text column (``VARCHAR``/``TEXT``). This means some ``INSERT``/``COPY`` operations will now error instead of truncating the text. This is intended to prevent accidental truncation of text columns. If you want the old truncation behaviour, you can use the ``SUBSTRING`` function to truncate the text. Operations @@ -45,20 +45,20 @@ Operations * When upgrading from SQream DB v3.2 or lower, the storage version must be upgraded using the :ref:`upgrade_storage_cli_reference` utility: ``./bin/upgrade_storage /path/to/storage/sqreamdb/`` -Known Issues & Limitations +Known Issues and Limitations =================================== -* NVARCHAR columns cannot be used as a ``GROUP BY`` key when there are multiple ``COUNT (DISTINCT …)`` operations in a query +* TEXT columns cannot be used as a ``GROUP BY`` key when there are multiple ``COUNT (DISTINCT …)`` operations in a query -* NVARCHAR columns cannot be used in a statement containing window functions +* TEXT columns cannot be used in a statement containing window functions -* NVARCHAR is not supported as a join key +* TEXT is not supported as a join key -* The following functions are not supported on ``NVARCHAR`` column types: ``chr``, ``min``, ``max``, ``patindex``, ``to_binary``, ``to_hex``, ``rlike``, ``regexp_count``, ``regexp_instr``, ``regexp_substr`` +* The following functions are not supported on ``TEXT`` column types: ``chr``, ``min``, ``max``, ``patindex``, ``to_binary``, ``to_hex``, ``rlike``, ``regexp_count``, ``regexp_instr``, ``regexp_substr`` -* SQream DB dashboard: Only works with a HA clustered installation +* SQream Dashboard: Only works with a HA clustered installation -* SQream DB editor: External tables and UDFs don’t appear in the DB Tree but do appear in the relevant sqream_catalog entries. +* SQream Editor: External tables and UDFs don’t appear in the DB Tree but do appear in the relevant sqream_catalog entries. Fixes @@ -68,7 +68,7 @@ Fixes * Variety of performance improvements: -* Improved performance of ``NVARCHAR`` by up to 315% for a variety of scenarios, including ``COPY FROM``, ``INNER JOIN``, ``LEFT JOIN``. +* Improved performance of ``TEXT`` by up to 315% for a variety of scenarios, including ``COPY FROM``, ``INNER JOIN``, ``LEFT JOIN``. * Improved load performance from previous versions diff --git a/releases/2020.1.rst b/releases/2020.1.rst index ff261925e..e4928855e 100644 --- a/releases/2020.1.rst +++ b/releases/2020.1.rst @@ -1,7 +1,7 @@ .. _2020.1: ************************** -What's new in 2020.1 +Release Notes 2020.1 ************************** SQream DB v2020.1 contains lots of new features, improved performance, and bug fixes. @@ -185,4 +185,4 @@ Upgrading to v2020.1 Versions are available for IBM POWER9, RedHat (CentOS) 7, Ubuntu 18.04, and other OSs via Docker. -Contact your account manager to get the latest release of SQream DB. \ No newline at end of file +Contact your account manager to get the latest release of SQream DB. diff --git a/releases/2020.2.rst b/releases/2020.2.rst index 9fa249b78..3dc25b78a 100644 --- a/releases/2020.2.rst +++ b/releases/2020.2.rst @@ -1,30 +1,30 @@ .. _2020.2: ************************** -What's new in 2020.2 +Release Notes 2020.2 ************************** -SQream DB v2020.2 contains some new features, improved performance, and bug fixes. +SQream v2020.2 contains some new features, improved performance, and bug fixes. This version has new window ranking function and a new editor UI to empower data users to analyze more data with less friction. -As always, the latest release improves reliability and performance, and makes getting more data into SQream DB easier than ever. +As always, the latest release improves reliability and performance, and makes getting more data into SQream easier than ever. -New features +New Features ================ UI ---------- -* New :ref:`sqream_studio` replaces the previous Statement Editor. Try it out today! +* New :ref:`sqream_studio` replaces the previous Statement Editor. Integrations ----------------- -* Our :ref:`Python driver (pysqream)` now has an SQLAlchemy dialect. Customers can write high-performance Python applications that make full use of SQream DB - connect, query, delete, and insert data. Data scientists can use pysqream with Pandas, Numpy, and AI/ML frameworks like TensorFlow for direct queries of huge datasets. +* Our :ref:`Python driver (pysqream)` now has an SQLAlchemy dialect. Customers can write high-performance Python applications that make full use of SQream - connect, query, delete, and insert data. Data scientists can use pysqream with Pandas, Numpy, and AI/ML frameworks like TensorFlow for direct queries of huge datasets. -SQL support +SQL Support --------------- * Added :ref:`lag`/:ref:`lead` ranking functions to our :ref:`window_functions` support. We will have more features coming in the next version. @@ -32,7 +32,7 @@ SQL support * New syntax preview for :ref:`external_tables`. Foreign tables replace external tables, with improved functionality. - You can keep using the existing external table syntax for now, but it may be deprecated in the future. + You can keep using the existing foreign table syntax for now, but it may be deprecated in the future. .. code-block:: postgres @@ -47,10 +47,10 @@ SQL support ( LOCATION = 'hdfs://hadoop-nn.piedpiper.com:8020/demo-data/example.orc' ); -Improvements and fixes +Improvements and Fixes ======================== -SQream DB v2020.2 includes hundreds of small new features and tunable parameters that improve performance, reliability, and stability. +SQream v2020.2 includes hundreds of small new features and tunable parameters that improve performance, reliability, and stability. * ~100 bug fixes, including: @@ -67,7 +67,7 @@ SQream DB v2020.2 includes hundreds of small new features and tunable parameters Operations ======================== -* When upgrading from a previous version of SQream DB (for example, v2019.2), the storage version must be upgraded using the :ref:`upgrade_storage_cli_reference` utility: ``./bin/upgrade_storage /path/to/storage/sqreamdb/`` +* When upgrading from a previous version of SQream (for example, v2019.2), the storage version must be upgraded using the :ref:`upgrade_storage_cli_reference` utility: ``./bin/upgrade_storage /path/to/storage/sqreamdb/`` * A change in memory allocation behaviour in this version sees the introduction of a new setting, ``limitQueryMemoryGB``. This is an addition to the previous ``spoolMemoryGB`` setting. @@ -97,7 +97,7 @@ Operations -Known Issues & Limitations +Known Issues and Limitations ================================ * An invalid formatted CSV can cause an ``insufficient memory`` error on a :ref:`copy_from` statement if a quote isn't closed and the file is much larger than system memory. @@ -107,9 +107,9 @@ Known Issues & Limitations * ``TEXT`` columns can't be used with an outer join together with an inequality check (``!= , <>``) -Upgrading to v2020.2 +Upgrading to Version 2020.2 ======================== Versions are available for IBM POWER9, RedHat (CentOS) 7, Ubuntu 18.04, and other OSs via Docker. -Contact your account manager to get the latest release of SQream DB. \ No newline at end of file +Contact your account manager to get the latest release of SQream. diff --git a/releases/2020.3.1.rst b/releases/2020.3.1.rst index 8bf24dd55..0667306d7 100644 --- a/releases/2020.3.1.rst +++ b/releases/2020.3.1.rst @@ -1,59 +1,72 @@ .. _2020.3.1: ************************** -What's new in 2020.3.1 +Release Notes 2020.3.1 ************************** +The 2020.3.1 release notes were released on October 8, 2020 and describe the following: -SQream DB v2020.3.1 contains minor features, many performance improvements and bug fixes. +.. contents:: + :local: + :depth: 1 -New or expanded features -========================= + +New Features +------------- +The following list describes the new features: + * TEXT data type: - * Full support for MIN and MAX aggregate functions on TEXT columns in GROUP BY queries - * Support Text-type as window partition keys (e.g., select distinct name, max(id) over (partition by name) from textTable;) - * Support Text-type fields in windows order by keys - * Support join on Text columns (e.g., t1.x = t2.y where x and y are columns of type Text) - * Complete the implementation of LIKE on TEXT columns (previously limited to prefix & suffix) - * Support for cast fromm TEXT to REAL/FLOAT - * New string function - REPEAT which repeats a string value for a specified number of times + * Full support for ``MIN`` and ``MAX`` aggregate functions on ``TEXT`` columns in ``GROUP BY`` queries. + * Support Text-type as window partition keys (e.g., select distinct name, max(id) over (partition by name) from ``textTable;``). + * Support Text-type fields in windows order by keys. + * Support join on ``TEXT`` columns (such as ``t1.x = t2.y`` where ``x`` and ``y`` are columns of type ``TEXT``). + * Complete the implementation of ``LIKE`` on ``TEXT`` columns (previously limited to prefix and suffix). + * Support for cast fromm ``TEXT`` to ``REAL/FLOAT``. + * New string function - ``REPEAT`` for repeating a string value for a specified number of times. -* Allow mapping DECIMAL ORC columns to SQream's floating-point types +* Support mapping ``DECIMAL ORC`` columns to SQream's floating-point types. -* Support LIKE on non-literal patterns (i.e., columns and complex expressions) +* Support ``LIKE`` on non-literal patterns (such as columns and complex expressions). -* Catch OS signals and save the signal along with the stack trace in the SQream debug log +* Catch OS signals and save the signal along with the stack trace in the SQream debug log. -* Support equijoins conditions on columns with different types (i.e., tinyint, smallint, int & bigint) +* Support equijoin conditions on columns with different types (such as ``tinyint``, ``smallint``, ``int`` and ``bigint``). -* `DUMP_DATABASE_DDL` now includes foreign tables in the output +* ``DUMP_DATABASE_DDL`` now includes foreign tables in the output. -* New utility function - TRUNCATE_IF_EXISTS +* New utility function - ``TRUNCATE_IF_EXISTS``. Performance Enhancements -========================= -* Introduced the "MetaData on Demand" feature which results in signicant proformance improvements +------------- +The following list describes the performance enhancements: + + +* Introduced the "MetaData on Demand" feature which results in signicant proformance improvements. + +* Implemented regex functions (``RLIKE``, ``REGEXP_COUNT``, ``REGEXP_INSTR``, ``REGEXP_SUBSTR``, ``PATINDEX``) for ``TEXT`` columns on GPU. + -* Implement regex functions (RLIKE, REGEXP_COUNT, REGEXP_INSTR, REGEXP_SUBSTR, PATINDEX) for TEXT columns on GPU +Resolved Issues +------------- +The following list describes the resolved issues: -Bug fixes -============ -* Multiple distinct aggregates no longer need to be used with developerMode flag -* In some scenrios, the statement_id and connection_id values are incorrectly recorded as -1 in the log -* NOT RLIKE is not supported for TEXT in the compiler -* Cast from TEXT to date/datetime returns an error when the TEXT column contains NULL +* Multiple distinct aggregates no longer need to be used with developerMode flag. +* In some scenarios, the ``statement_id`` and ``connection_id values`` are incorrectly recorded as ``-1`` in the log. +* ``NOT RLIKE`` is not supported for ``TEXT`` in the compiler. +* Casting from ``TEXT`` to ``date/datetime`` returns an error when the ``TEXT`` column contains ``NULL``. -Known Issues & Limitations -================================ +Known Issues and Limitations +------------- +No known issues and limitations. Upgrading to v2020.3.1 -======================== +---------------- Versions are available for IBM POWER9, RedHat (CentOS) 7, Ubuntu 18.04, and other OSs via Docker. -Contact your account manager to get the latest release of SQream DB. +Contact your account manager to get the latest release of SQream DB. \ No newline at end of file diff --git a/releases/2020.3.2.1.rst b/releases/2020.3.2.1.rst new file mode 100644 index 000000000..3c551b636 --- /dev/null +++ b/releases/2020.3.2.1.rst @@ -0,0 +1,31 @@ +.. _2020.3.2.1: + +************************** +Release Notes 2020.3.2.1 +************************** +The 2020.3.2.1 release notes were released on October 8, 2020 and describe the following: + +.. contents:: + :local: + :depth: 1 + + +Overview +----------------- +SQream DB v2020.3.2.1 contains major performance improvements and some bug fixes. + +Performance Enhancements +------------- +* Metadata on Demand optimization resulting in reduced latency and improved overall performance. + + +Known Issues and Limitations +------------- +* Multiple count distinct operations is enabled for all data types. + +Upgrading to v2020.3.2.1 +------------- + +Versions are available for IBM POWER9, RedHat (CentOS) 7, Ubuntu 18.04, and other OSs via Docker. + +Contact your account manager to get the latest release of SQream DB. \ No newline at end of file diff --git a/releases/2020.3.2.rst b/releases/2020.3.2.rst new file mode 100644 index 000000000..66bd0db68 --- /dev/null +++ b/releases/2020.3.2.rst @@ -0,0 +1,28 @@ +.. _2020.3.2: + +************************** +What's New in 2020.3.2 +************************** + +SQream DB v2020.3.2 contains major performance improvements and some bug fixes. + +Performance Enhancements +========================= +* Metadata on Demand optimization resulting in reduced latency and improved overall performance + + +Known Issues & Limitations +================================ +* Bug with STDDEV_SAMP,STDDEV_POP and STDEV functions +* Window function query returns wrong results +* rank() in window function sometimes returns garbage +* Window function on null value could have bad result +* Window function lead() on varchar can have garbage results +* Performance degradation when using "groupby" or outer_join + +Upgrading to v2020.3.2 +======================== + +Versions are available for IBM POWER9, RedHat (CentOS) 7, Ubuntu 18.04, and other OSs via Docker. + +Contact your account manager to get the latest release of SQream DB. diff --git a/releases/2020.3.rst b/releases/2020.3.rst index f4264a19e..52bf63991 100644 --- a/releases/2020.3.rst +++ b/releases/2020.3.rst @@ -1,18 +1,28 @@ .. _2020.3: ************************** -What's new in 2020.3 +Release Notes 2020.3 ************************** +The 2020.3 release notes were released on October 8, 2020 and describe the following: -SQream DB v2020.3 contains new features, improved performance, and bug fixes. +.. contents:: + :local: + :depth: 1 -New features -============ +Overview +------------ +SQream DB v2020.3 contains new features, performance enhancements, and resolved issues. + + +New Features +---------- +The following list describes the new features: + * Parquet and ORC files can now be exported to local storage, S3, and HDFS with :ref:`copy_to` and foreign data wrappers. -* We've introduced new error tolerance features when loading data with foreign data wrappers +* New error tolerance features when loading data with foreign data wrappers. * ``TEXT`` is ramping up with new features (previously only available with VARCHARs): @@ -24,50 +34,56 @@ New features * :ref:`sqream_studio` v5.1 - * New log viewer helps you track and debug what's going on in SQream DB + * New log viewer helps you track and debug what's going on in SQream DB. - * Dashboard now also available for non-k8s deployments + * Dashboard now also available for non-k8s deployments. - * The editor contains a new query concurrency tool for date and numeric ranges + * The editor contains a new query concurrency tool for date and numeric ranges. -Enhancements -============ - -* Error handling for CSV FDW -* Enable logging errors - ORC, Parquet, CSV -* Add limit and offset options to csv_fdw import -* Enable logging errors to an external file when skipping CSV, Parquet, and ORC errors -* Option to specify date format to the CSV FDW -* Support all existing Varchar functions with Text on GPU -* Support INSERT INTO + ORDER BY optimization for non-clustered tables -* Performance improvements with I/O - -Bug fixes -============ - -* Better error message when passing the max errors limit -* showFullExceptionInfo is no longer restricted to developer mode -* StreamAggregateA reduction error when performing aggregation on a null column -* Insert into query fails with ""Error at Sql phase during Stages ""rewriteSqlQuery"" -* Cast from varchar to nvarchar doesn't remove the spaces -* Internal Runtime Error t1.size() == t2.size() when querying sqream_catalog.delete_predicates -* spoolMemoryGB and limitQueryMemoryGB show incorrectly in the runtime global section of show_conf -* Casting empty text to int causes illegal memory access -* Copy from TEXT field is 1.5x slower than the VARCHAR equivalent -* TPCDS 10TB - Internal runtime error (std::bad_alloc: out of memory) on 2020.1.0.2 -* Not equal join on not existing TEXT crashes the system -* Internal runtime time error when using TEXT (tpcds) -* Copying CSV with a quote in the middle of a field to TEXT field does not produce an error -* Can't monitor long network insert loads with SQream -* Upper & like performance on Nvarchar -* Insert into from 4 instances get stuck (hanging) -* An invalid formatted CSV can cause an insufficient memory error on a COPY FROM statement if a quote isn’t closed and the file is much larger than system memory. -* TEXT columns can’t be used with an outer join together with an inequality check (!= , <>) - -Known Issues & Limitations -================================ +Performance Enhancements +---------- +The following list describes the performance enhancements: + + +* Error handling for CSV FDW. +* Enable logging errors - ORC, Parquet, CSV. +* Add limit and offset options to ``csv_fdw`` import. +* Enable logging errors to an external file when skipping CSV, Parquet, and ORC errors. +* Option to specify date format to the CSV FDW. +* Support all existing ``VARCHAR`` functions with ``TEXT`` on GPU. +* Support ``INSERT INTO`` + ``ORDER BY`` optimization for non-clustered tables. +* Performance improvements with I/O. + +Resolved Issues +--------------- +The following list describes the resolved issues: + + +* Better error message when passing the max errors limit. This was fixed. +* ``showFullExceptionInfo`` is no longer restricted to Developer Mode. This was fixed. +* An ``StreamAggregateA`` reduction error occured when performing aggregation on a ``NULL`` column. This was fixed. +* Insert into query fails with ""Error at Sql phase during Stages ""rewriteSqlQuery"". This was fixed. +* Casting from ``VARCHAR`` to ``TEXT`` does not remove the spaces. This was fixed. +* An ``Internal Runtime Error t1.size() == t2.size()`` occurs when querying the ``sqream_catalog.delete_predicates``. This was fixed. +* ``spoolMemoryGB`` and ``limitQueryMemoryGB`` show incorrectly in the **runtime global** section of ``show_conf.`` This was fixed. +* Casting empty text to ``int`` causes illegal memory access. This was fixed. +* Copying from the ``TEXT`` field is 1.5x slower than the ``VARCHAR`` equivalent. This was fixed. +* ``TPCDS 10TB - Internal runtime error (std::bad_alloc: out of memory)`` occurs on 2020.1.0.2. This was fixed. +* An unequal join on non-existing ``TEXT`` caused a system crash. This was fixed. +* An ``Internal runtime time error`` occured when using ``TEXT (tpcds)``. This was fixed. +* Copying CSV with a quote in the middle of a field to a ``TEXT`` field does not produce the required error. This was fixed. +* Cannot monitor long network insert loads with SQream. This was fixed. +* Upper and like performance on ``TEXT``. This was fixed. +* Insert into from 4 instances would get stuck (hanging). This was fixed. +* An invalid formatted CSV would cause an insufficient memory error on a ``COPY FROM`` statement if a quote was not closed and the file was much larger than system memory. This was fixed. +* ``TEXT`` columns cannot be used with an outer join together with an inequality check (!= , <>). This was fixed. + +Known Issues And Limitations +---------- +The following list describes the known issues and limitations: + * Cast from ``TEXT`` to a ``DATE`` or ``DATETIME`` errors when the ``TEXT`` column contains ``NULL`` @@ -79,8 +95,8 @@ Known Issues & Limitations Upgrading to v2020.3 -======================== +---------- Versions are available for IBM POWER9, RedHat (CentOS) 7, Ubuntu 18.04, and other OSs via Docker. -Contact your account manager to get the latest release of SQream DB. +Contact your account manager to get the latest release of SQream. diff --git a/releases/2020.3_index.rst b/releases/2020.3_index.rst new file mode 100644 index 000000000..28a0fb4bf --- /dev/null +++ b/releases/2020.3_index.rst @@ -0,0 +1,19 @@ +.. _2020.3_index: + +************************** +Release Notes 2020.3 +************************** +The 2020.3 Release Notes describe the following releases: + +.. contents:: + :local: + :depth: 1 + +.. toctree:: + :maxdepth: 1 + :glob: + + 2020.3.2.1 + 2020.3.2 + 2020.3.1 + 2020.3 \ No newline at end of file diff --git a/releases/index.rst b/releases/index.rst index 65468dd27..22bcdce89 100644 --- a/releases/index.rst +++ b/releases/index.rst @@ -1,25 +1,24 @@ .. _releases: ********** -Releases +Release Notes ********** + .. list-table:: :widths: auto :header-rows: 1 + * - Version - Release Date - * - :ref:`2020.3.1` - - January 11, 2021 * - :ref:`2020.3` - October 8, 2020 * - :ref:`2020.2` - July 22, 2020 * - :ref:`2020.1` - January 15, 2020 - * - :ref:`2019.2.1` - - July 25, 2019 + .. toctree:: @@ -27,8 +26,6 @@ Releases :glob: :hidden: - 2020.3.1 - 2020.3 + 2020.3_index 2020.2 2020.1 - 2019.2.1 diff --git a/requirements.txt b/requirements.txt index 609b0b4b1..806fe2730 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,7 @@ +# File: docs/requirements.txt + +# Defining the exact version will make sure things don't break +sphinx==3.5.3 +sphinx_rtd_theme==0.5.2 sphinx-notfound-page -Pygments>=2.4.0 \ No newline at end of file +Pygments>=2.4.0 diff --git a/sqream_studio_5.4.3/configuring_your_instance_of_sqream.rst b/sqream_studio_5.4.3/configuring_your_instance_of_sqream.rst new file mode 100644 index 000000000..2b5a80e8a --- /dev/null +++ b/sqream_studio_5.4.3/configuring_your_instance_of_sqream.rst @@ -0,0 +1,23 @@ +.. _configuring_your_instance_of_sqream: + +**************************** +Configuring Your Instance of SQream +**************************** +The **Configuration** section lets you edit parameters from one centralized location. While you can edit these parameters from the **worker configuration file (config.json)** or from your CLI, you can also modify them in Studio in an easy-to-use format. + +Configuring your instance of SQream in Studio is session-based, which enables you to edit parameters per session on your own device. +Because session-based configurations are not persistent and are deleted when your session ends, you can edit your required parameters while avoiding conflicts between parameters edited on different devices at different points in time. + +Editing Your Parameters +------------------------------- +When configuring your instance of SQream in Studio you can edit parameters for the **Generic** and **Admin** parameters only. + +Studio includes two types of parameters: toggle switches, such as **flipJoinOrder**, and text fields, such as **logSysLevel**. After editing a parameter, you can reset each one to its previous value or to its default value individually, or revert all parameters to their default setting simultaneously. Note that you must click **Save** to save your configurations. + +You can hover over the **information** icon located on each parameter to read a short description of its behavior. + +Exporting and Importing Configuration Files +------------------------- +You can also export and import your configuration settings into a .json file. This allows you to easily edit your parameters and to share this file with other users if required. + +For more information about configuring your instance of SQream, see `Configuration `_. \ No newline at end of file diff --git a/sqream_studio_5.4.3/creating_assigning_and_managing_roles_and_permissions.rst b/sqream_studio_5.4.3/creating_assigning_and_managing_roles_and_permissions.rst new file mode 100644 index 000000000..9f0f7f39a --- /dev/null +++ b/sqream_studio_5.4.3/creating_assigning_and_managing_roles_and_permissions.rst @@ -0,0 +1,98 @@ +.. _creating_assigning_and_managing_roles_and_permissions: + +.. _roles_5.4.3: + +**************************** +Creating, Assigning, and Managing Roles and Permissions +**************************** +The **Creating, Assigning, and Managing Roles and Permissions** describes the following: + +.. contents:: + :local: + :depth: 1 + +Overview +--------------- +In the **Roles** area you can create and assign roles and manage user permissions. + +The **Type** column displays one of the following assigned role types: + +.. list-table:: + :widths: 15 75 + :header-rows: 1 + + * - Role Type + - Description + * - Groups + - Roles with no users. + * - Enabled users + - Users with log-in permissions and a password. + * - Disabled users + - Users with log-in permissions and with a disabled password. An admin may disable a user's password permissions to temporary disable access to the system. + +.. note:: If you disable a password, when you enable it you have to create a new one. + +:ref:`Back to Creating, Assigning, and Managing Roles and Permissions` + + +Viewing Information About a Role +-------------------- +Clicking a role in the roles table displays the following information: + + * **Parent Roles** - displays the parent roles of the selected role. Roles inherit all roles assigned to the parent. + + :: + + * **Members** - displays all members that the role has been assigned to. The arrow indicates the roles that the role has inherited. Hovering over a member displays the roles that the role is inherited from. + + :: + + * **Permissions** - displays the role's permissions. The arrow indicates the permissions that the role has inherited. Hovering over a permission displays the roles that the permission is inherited from. + +:ref:`Back to Creating, Assigning, and Managing Roles and Permissions` + + +Creating a New Role +-------------------- +You can create a new role by clicking **New Role**. + + + +An admin creates a **user** by granting login permissions and a password to a role. Each role is defined by a set of permissions. An admin can also group several roles together to form a **group** to manage them simultaneously. For example, permissions can be granted to or revoked on a group level. + +Clicking **New Role** lets you do the following: + + * Add and assign a role name (required) + * Enable or disable log-in permissions for the role. + * Set a password. + * Assign or delete parent roles. + * Add or delete permissions. + * Grant the selected user with superuser permissions. + +From the New Role panel you view directly and indirectly (or inherited) granted permissions. Disabled permissions have no connect permissions for the referenced database and are displayed in gray text. You can add or remove permissions from the **Add permissions** field. From the New Role panel you can also search and scroll through the permissions. In the **Search** field you can use the **and** operator to search for strings that fulfill multiple criteria. + +When adding a new role, you must select the **Enable login for this role** and **Has password** check boxes. + +:ref:`Back to Creating, Assigning, and Managing Roles and Permissions` + +Editing a Role +-------------------- +Once you've created a role, clicking the **Edit Role** button lets you do the following: + + * Edit the role name. + * Enable or disable log-in permissions. + * Set a password. + * Assign or delete parent roles. + * Assign a role **administrator** permissions. + * Add or delete permissions. + * Grant the selected user with superuser permissions. + +From the Edit Role panel you view directly and indirectly (or inherited) granted permissions. Disabled permissions have no connect permissions for the referenced database and are displayed in gray text. You can add or remove permissions from the **Add permissions** field. From the Edit Role panel you can also search and scroll through the permissions. In the **Search** field you can use the **and** operator to search for strings that fulfill multiple criteria. + +:ref:`Back to Creating, Assigning, and Managing Roles and Permissions` + +Deleting a Role +----------------- +Clicking the **delete** icon displays a confirmation message with the amount of users and groups that will be impacted by deleting the role. + +:ref:`Back to Creating, Assigning, and Managing Roles and Permissions` \ No newline at end of file diff --git a/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst b/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst new file mode 100644 index 000000000..25d2be354 --- /dev/null +++ b/sqream_studio_5.4.3/executing_statements_and_running_queries_from_the_editor.rst @@ -0,0 +1,492 @@ +.. _executing_statements_and_running_queries_from_the_editor: + +.. _editor_top_5.4.3: + +**************************** +Executing Statements and Running Queries from the Editor +**************************** +The **Editor** is used for the following: + +* Selecting an active database and executing queries. +* Performing statement-related operations and showing metadata. +* Executing pre-defined queries. +* Writing queries and statements and viewing query results. + +The following is a brief description of the Editor panels: + + +.. list-table:: + :widths: 10 34 56 + :header-rows: 1 + + * - No. + - Element + - Description + * - 1 + - :ref:`Toolbar` + - Used to select the active database you want to work on, limit the number of rows, save query, etc. + * - 2 + - :ref:`Database Tree and System Queries panel` + - Shows a hierarchy tree of databases, views, tables, and columns + * - 3 + - :ref:`Statement panel` + - Used for writing queries and statements + * - 4 + - :ref:`Results panel` + - Shows query results and execution information. + + + +.. _top_5.4.3: + +.. _studio_5.4.3_editor_toolbar: + +Executing Statements from the Toolbar +================ +You can access the following from the Toolbar pane: + +* **Database dropdown list** - select a database that you want to run statements on. + + :: + +* **Service dropdown list** - select a service that you want to run statements on. The options in the service dropdown menu depend on the database you select from the **Database** dropdown list. + + :: + +* **Execute** - lets you set which statements to execute. The **Execute** button toggles between **Execute** and **Stop**, and can be used to stop an active statement before it completes: + + * **Statements** - executes the statement at the location of the cursor. + * **Selected** - executes only the highlighted text. This mode should be used when executing subqueries or sections of large queries (as long as they are valid SQLs). + * **All** - executes all statements in a selected tab. + +* **Format SQL** - Lets you reformat and reindent statements. + + :: + +* **Download query** - Lets you download query text to your computer. + + :: + +* **Open query** - Lets you upload query text from your computer. + + :: + +* **Max Rows** - By default, the Editor fetches only the first 10,000 rows. You can modify this number by selecting an option from the **Max Rows** dropdown list. Note that setting a higher number may slow down your browser if the result is very large. This number is limited to 100,000 results. To see a higher number, you can save the results in a file or a table using the :ref:`create_table_as` command. + + +For more information on stopping active statements, see the :ref:`STOP_STATEMENT` command. + +:ref:`Back to Executing Statements and Running Queries from the Editor` + + +.. _studio_5.4.3_editor_db_tree: + +Performing Statement-Related Operations from the Database Tree +================ +From the Database Tree you can perform statement-related operations and show metadata (such as a number indicating the amount of rows in the table). + + + + + +The database object functions are used to perform the following: + +* The **SELECT** statement - copies the selected table's **columns** into the Statement panel as ``SELECT`` parameters. + + :: + +* The **copy** feature |icon-copy| - copies the selected table's **name** into the Statement panel. + + :: + +* The **additional operations** |icon-dots| - displays the following additional options: + + +.. |icon-user| image:: /_static/images/studio_icon_user.png + :align: middle + +.. |icon-dots| image:: /_static/images/studio_icon_dots.png + :align: middle + +.. |icon-editor| image:: /_static/images/studio_icon_editor.png + :align: middle + +.. |icon-copy| image:: /_static/images/studio_icon_copy.png + :align: middle + +.. |icon-select| image:: /_static/images/studio_icon_select.png + :align: middle + +.. |icon-dots| image:: /_static/images/studio_icon_dots.png + :align: middle + +.. |icon-filter| image:: /_static/images/studio_icon_filter.png + :align: middle + +.. |icon-ddl-edit| image:: /_static/images/studio_icon_ddl_edit.png + :align: middle + +.. |icon-run-optimizer| image:: /_static/images/studio_icon_run_optimizer.png + :align: middle + +.. |icon-generate-create-statement| image:: /_static/images/studio_icon_generate_create_statement.png + :align: middle + +.. |icon-plus| image:: /_static/images/studio_icon_plus.png + :align: middle + +.. |icon-close| image:: /_static/images/studio_icon_close.png + :align: middle + +.. |icon-left| image:: /_static/images/studio_icon_left.png + :align: middle + +.. |icon-right| image:: /_static/images/studio_icon_right.png + :align: middle + +.. |icon-format-sql| image:: /_static/images/studio_icon_format.png + :align: middle + +.. |icon-download-query| image:: /_static/images/studio_icon_download_query.png + :align: middle + +.. |icon-open-query| image:: /_static/images/studio_icon_open_query.png + :align: middle + +.. |icon-execute| image:: /_static/images/studio_icon_execute.png + :align: middle + +.. |icon-stop| image:: /_static/images/studio_icon_stop.png + :align: middle + +.. |icon-dashboard| image:: /_static/images/studio_icon_dashboard.png + :align: middle + +.. |icon-expand| image:: /_static/images/studio_icon_expand.png + :align: middle + +.. |icon-scale| image:: /_static/images/studio_icon_scale.png + :align: middle + +.. |icon-expand-down| image:: /_static/images/studio_icon_expand_down.png + :align: middle + +.. |icon-add| image:: /_static/images/studio_icon_add.png + :align: middle + +.. |icon-add-worker| image:: /_static/images/studio_icon_add_worker.png + :align: middle + +.. |keep-tabs| image:: /_static/images/studio_keep_tabs.png + :align: middle + + +.. list-table:: + :widths: 30 70 + :header-rows: 1 + + * - Function + - Description + * - Insert statement + - Generates an `INSERT `_ statement for the selected table in the editing area. + * - Delete statement + - Generates a `DELETE `_ statement for the selected table in the editing area. + * - Create Table As statement + - Generates a `CREATE TABLE AS `_ statement for the selected table in the editing area. + * - Rename statement + - Generates an `RENAME TABLE AS `_ statement for renaming the selected table in the editing area. + * - Adding column statement + - Generates an `ADD COLUMN `_ statement for adding columns to the selected table in the editing area. + * - Truncate table statement + - Generates a `TRUNCATE_IF_EXISTS `_ statement for the selected table in the editing area. + * - Drop table statement + - Generates a ``DROP`` statement for the selected object in the editing area. + * - Table DDL + - Generates a DDL statement for the selected object in the editing area. To get the entire database DDL, click the |icon-ddl-edit| icon next to the database name in the tree root. See `Seeing System Objects as DDL `_. + * - DDL Optimizer + - The `DDL Optimizer `_ lets you analyze database tables and recommends possible optimizations. + +Optimizing Database Tables Using the DDL Optimizer +----------------------- +The **DDL Optimizer** tab analyzes database tables and recommends possible optimizations according to SQream's best practices. + +As described in the previous table, you can access the DDL Optimizer by clicking the **additional options icon** and selecting **DDL Optimizer**. + +The following table describes the DDL Optimizer screen: + +.. list-table:: + :widths: 15 75 + :header-rows: 1 + + * - Element + - Description + * - Column area + - Shows the column **names** and **column types** from the selected table. You can scroll down or to the right/left for long column lists. + * - Optimization area + - Shows the number of rows to sample as the basis for running an optimization, the default setting (1,000,000) when running an optimization (this is also the overhead threshold used when analyzing ``VARCHAR`` fields), and the default percent buffer to add to ``VARCHAR`` lengths (10%). Attempts to determine field nullability. + * - Run Optimizer + - Starts the optimization process. + +Clicking **Run Optimizer** adds a tab to the Statement panel showing the optimized results of the selected object. + +For more information, see `Optimization and Best Practices `_. + +Executing Pre-Defined Queries from the System Queries Panel +--------------- +The **System Queries** panel lets you execute predefined queries and includes the following system query types: + +* **Catalog queries** - Used for analyzing table compression rates, users and permissions, etc. + + :: + +* **Admin queries** - Queries useful for SQream database management. + +Clicking an item pastes the query into the Statement pane, and you can undo a previous operation by pressing **Ctrl + Z**. + +.. _studio_5.4.3_editor_statement_area: + +Writing Statements and Queries from the Statement Panel +============== +The multi-tabbed statement area is used for writing queries and statements, and is used in tandem with the toolbar. When writing and executing statements, you must first select a database from the **Database** dropdown menu in the toolbar. When you execute a statement, it passes through a series of statuses until completed. Knowing the status helps you with statement maintenance, and the statuses are shown in the **Results panel**. + +The auto-complete feature assists you when writing statements by suggesting statement options. + +The following table shows the statement statuses: + +.. list-table:: + :widths: 45 160 + :header-rows: 1 + + * - Status + - Description + * - Pending + - The statement is pending. + * - In queue + - The statement is waiting for execution. + * - Initializing + - The statement has entered execution checks. + * - Executing + - The statement is executing. + * - Statement stopped + - The statement has been stopped. + +You can add and name new tabs for each statement that you need to execute, and Studio preserves your created tabs when you switch between databases. You can add new tabs by clicking |icon-plus| , which creates a new tab to the right with a default name of SQL and an increasing number. This helps you keep track of your statements. + +You can also rename the default tab name by double-clicking it and typing a new name and write multiple statements in tandem in the same tab by separating them with semicolons (``;``).If too many tabs to fit into the Statement Pane are open at the same time, the tab arrows are displayed. You can scroll through the tabs by clicking |icon-left| or |icon-right|, and close tabs by clicking |icon-close|. You can also close all tabs at once by clicking **Close all** located to the right of the tabs. + +.. tip:: If this is your first time using SQream, see `Getting Started `_. + + +.. Keyboard shortcuts +.. ^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. :kbd:`Ctrl` +: kbd:`Enter` - Execute all queries in the statement area, or just the highlighted part of the query. + +.. :kbd:`Ctrl` + :kbd:`Space` - Auto-complete the current keyword + +.. :kbd:`Ctrl` + :kbd:`↑` - Switch to next tab. + +.. :kbd:`Ctrl` + :kbd:`↓` - Switch to previous tab + +.. _studio_editor_results_5.4.3: + +:ref:`Back to Executing Statements and Running Queries from the Editor` + +.. _studio_5.4.3_editor_results: + +.. _results_panel_5.4.3: + +Viewing Statement and Query Results from the Results Panel +============== +The results panel shows statement and query results. By default, only the first 10,000 results are returned, although you can modify this from the :ref:`studio_editor_toolbar`, as described above. By default, executing several statements together opens a separate results tab for each statement. Executing statements together executes them serially, and any failed statement cancels all subsequent executions. + +.. image:: /_static/images/results_panel.png + +The following is a brief description of the Results panel views highlighted in the figure above: + +.. list-table:: + :widths: 45 160 + :header-rows: 1 + + * - Element + - Description + * - :ref:`Results view` + - Lets you view search query results. + * - :ref:`Execution Details view` + - Lets you analyze your query for troubleshooting and optimization purposes. + * - :ref:`SQL view` + - Lets you see the SQL view. + + +.. _results_view_5.4.3: + +:ref:`Back to Executing Statements and Running Queries from the Editor` + +Searching Query Results in the Results View +---------------- +The **Results view** lets you view search query results. + +From this view you can also do the following: + +* View the amount of time (in seconds) taken for a query to finish executing. +* Switch and scroll between tabs. +* Close all tabs at once. +* Enable keeping tabs by selecting **Keep tabs**. +* Sort column results. + +Saving Results to the Clipboard +^^^^^^^^^^^^ +The **Save results to clipboard** function lets you save your results to the clipboard to paste into another text editor or into Excel for further analysis. + +.. _save_results_to_local_file_5.4.3: + +Saving Results to a Local File +^^^^^^^^^^^^ +The **Save results to local file** functions lets you save your search query results to a local file. Clicking **Save results to local file** downloads the contents of the Results panel to an Excel sheet. You can then use copy and paste this content into other editors as needed. + +In the Results view you can also run parallel statements, as described in **Running Parallel Statements** below. + +.. _running_parallel_statements_5.4.3: + +Running Parallel Statements +^^^^^^^^^^^^ +While Studio's default functionality is to open a new tab for each executed statement, Studio supports running parallel statements in one statement tab. Running parallel statements requires using macros and is useful for advanced users. + +The following shows the syntax for running parallel statements: + +.. code-block:: console + + $ @@ parallel + $ $$ + $ select 1; + $ select 2; + $ select 3; + $ $$ + + +:ref:`Back to Viewing Statement and Query Results from the Results Panel` + +.. _execution_details_view_5.4.3: + +.. _execution_tree_5.4.3: + +Execution Details View +-------------- +The **Execution Details View** section describes the following: + +.. contents:: + :local: + :depth: 1 + +Overview +^^^^^^^^^^^^ +Clicking **Execution Details View** displays the **Execution Tree**, which is a chronological tree of processes that occurred to execute your queries. The purpose of the Execution Tree is to analyze all aspects of your query for troubleshooting and optimization purposes, such as resolving queries with an exceptionally long runtime. + +.. note:: The **Execution Details View** button is enabled only when a query takes longer than five seconds. + +From this screen you can scroll in, out, and around the execution tree with the mouse to analyze all aspects of your query. You can navigate around the execution tree by dragging or by using the mini-map in the bottom right corner. + +.. image:: /_static/images/execution_tree_1.png + +You can also search for query data by pressing **Ctrl+F** or clicking the search icon |icon-search| in the search field in the top right corner and typing text. + +.. image:: /_static/images/search_field.png + +Pressing **Enter** takes you directly to the next result matching your search criteria, and pressing **Shift + Enter** takes you directly to the previous result. You can also search next and previous results using the up and down arrows. + +.. |icon-search| image:: /_static/images/studio_icon_search.png + :align: middle + +The nodes are color-coded based on the following: + +* **Slow nodes** - red +* **In progress nodes** - yellow +* **Completed nodes** - green +* **Pending nodes** - white +* **Currently selected node** - blue +* **Search result node** - purple (in the mini-map) + +The execution tree displays the same information as shown in the plain view in tree format. + +The Execution Tree tracks each phase of your query in real time as a vertical tree of nodes. Each node refers to an operation that occurred on the GPU or CPU. When a phase is completed, the next branch begins to its right until the entire query is complete. Joins are displayed as two parallel branches merged together in a node called **Join**, as shown in the figure above. The nodes are connected by a line indicating the number of rows passed from one node to the next. The width of the line indicates the amount of rows on a logarithmic scale. + +Each node displays a number displaying its **node ID**, its **type**, **table name** (if relevant), **status**, and **runtime**. The nodes are color-coded for easy identification. Green nodes indicate **completed nodes**, yellow indicates **nodes in progress**, and red indicates **slowest nodes**, typically joins, as shown below: + +.. image:: /_static/images/nodes.png + +Viewing Query Statistics +^^^^^^^^^^^^ +The following statistical information is displayed in the top left corner, as shown in the figure above: + +* **Query Statistics**: + + * **Elapsed** - the total time taken for the query to complete. + * **Result rows** - the amount of rows fetched. + * **Running nodes completion** + * **Total query completion** - the amount of the total execution tree that was executed (nodes marked green). + +* **Slowest Nodes** information is displayed in the top right corner in red text. Clicking the slowest node centers automatically on that node in the execution tree. + +You can also view the following **Node Statistics** in the top right corner for each individual node by clicking a node: + +.. list-table:: + :widths: 45 160 + :header-rows: 1 + + * - Element + - Description + * - Node type + - Shows the node type. + * - Status + - Shows the execution status. + * - Time + - The total time taken to execute. + * - Rows + - Shows the number of produced rows passed to the next node. + * - Chunks + - Shows number of produced chunks. + * - Average rows per chunk + - Shows the number of average rows per chunk. + * - Table (for **ReadTable** and joins only) + - Shows the table name. + * - Write (for joins only) + - Shows the total date size written to the disk. + * - Read (for **ReadTable** and joins only) + - Shows the total data size read from the disk. + +Note that you can scroll the Node Statistics table. You can also download the execution plan table in .csv format by clicking the download arrow |icon-download| in the upper-right corner. + +.. |icon-download| image:: /_static/images/studio_icon_download.png + :align: middle + +Using the Plain View +^^^^^^^^^^^^ +You can use the **Plain View** instead of viewing the execution tree by clicking **Plain View** |icon-plain| in the top right corner. The plain view displays the same information as shown in the execution tree in table format. + +.. |icon-plain| image:: /_static/images/studio_icon_plain.png + :align: middle + + + + +The plain view lets you view a query’s execution plan for monitoring purposes and highlights rows based on how long they ran relative to the entire query. + +This can be seen in the **timeSum** column as follows: + +* **Rows highlighted red** - longest runtime +* **Rows highlighted orange** - medium runtime +* **Rows highlighted yellow** - shortest runtime + +:ref:`Back to Viewing Statement and Query Results from the Results Panel` + +.. _sql_view_5.4.3: + +Viewing Wrapped Strings in the SQL View +------------------ +The SQL View panel allows you to more easily view certain queries, such as a long string that appears on one line. The SQL View makes it easier to see by wrapping it so that you can see the entire string at once. It also reformats and organizes query syntax entered in the Statement panel for more easily locating particular segments of your queries. The SQL View is identical to the **Format SQL** feature in the Toolbar, allowing you to retain your originally constructed query while viewing a more intuitively structured snapshot of it. + +.. _save_results_to_clipboard_5.4.3: + +:ref:`Back to Viewing Statement and Query Results from the Results Panel` + +:ref:`Back to Executing Statements and Running Queries from the Editor` diff --git a/sqream_studio_5.4.3/getting_started.rst b/sqream_studio_5.4.3/getting_started.rst new file mode 100644 index 000000000..85747b30b --- /dev/null +++ b/sqream_studio_5.4.3/getting_started.rst @@ -0,0 +1,61 @@ +.. _getting_started: + +**************************** +Getting Started with SQream Acceleration Studio 5.4.3 +**************************** +Setting Up and Starting Studio +---------------- +Studio is included with all `dockerized installations of SQream DB `_. When starting Studio, it listens on the local machine on port 8080. + +Logging In to Studio +--------------- +**To log in to SQream Studio:** + +1. Open a browser to the host on **port 8080**. + + For example, if your machine IP address is ``192.168.0.100``, insert the IP address into the browser as shown below: + + .. code-block:: console + + $ http://192.168.0.100:8080 + +2. Fill in your SQream DB login credentials. These are the same credentials used for :ref:`sqream sql` or JDBC. + + When you sign in, the License Warning is displayed. + +Navigating Studio's Main Features +------------- +When you log in, you are automatically taken to the **Editor** screen. The Studio's main functions are displayed in the **Navigation** pane on the left side of the screen. + +From here you can navigate between the main areas of the Studio: + +.. list-table:: + :widths: 10 90 + :header-rows: 1 + + * - Element + - Description + * - :ref:`Dashboard` + - Lets you monitor system health and manage queues and workers. + * - :ref:`Editor` + - Lets you select databases, perform statement operations, and write and execute queries. + * - :ref:`Logs` + - Lets you view usage logs. + * - :ref:`Roles` + - Lets you create users and manage user permissions. + * - :ref:`Configuration` + - Lets you configure your instance of SQream. + +By clicking the user icon, you can also use it for logging out and viewing the following: + +* User information +* Connection type +* SQream version +* SQream Studio version +* License expiration date +* License storage capacity +* Log out + +.. _back_to_dashboard_5.4.3: + +.. _studio_dashboard_5.4.3: diff --git a/sqream_studio_5.4.3/index.rst b/sqream_studio_5.4.3/index.rst new file mode 100644 index 000000000..023fa265a --- /dev/null +++ b/sqream_studio_5.4.3/index.rst @@ -0,0 +1,19 @@ +.. _sqream_studio_5.4.3: + +********************************** +SQream Acceleration Studio 5.4.3 +********************************** +The SQream Acceleration Studio is a web-based client for use with SQream. Studio provides users with all functionality available from the command line in an intuitive and easy-to-use format. This includes running statements, managing roles and permissions, and managing SQream clusters. + +This section describes how to use the SQream Accleration Studio version 5.4.3: + +.. toctree:: + :maxdepth: 1 + :glob: + + getting_started + monitoring_workers_and_services_from_the_dashboard + executing_statements_and_running_queries_from_the_editor + viewing_logs + creating_assigning_and_managing_roles_and_permissions + configuring_your_instance_of_sqream \ No newline at end of file diff --git a/sqream_studio_5.4.3/monitoring_workers_and_services_from_the_dashboard.rst b/sqream_studio_5.4.3/monitoring_workers_and_services_from_the_dashboard.rst new file mode 100644 index 000000000..9966137fa --- /dev/null +++ b/sqream_studio_5.4.3/monitoring_workers_and_services_from_the_dashboard.rst @@ -0,0 +1,265 @@ +.. _monitoring_workers_and_services_from_the_dashboard: + +.. _back_to_dashboard_5.4.3: + +**************************** +Monitoring Workers and Services from the Dashboard +**************************** +The **Dashboard** is used for the following: + +* Monitoring system health. +* Viewing, monitoring, and adding defined service queues. +* Viewing and managing worker status and add workers. + +The following is an image of the Dashboard: + +.. image:: /_static/images/dashboard.png + +You can only access the Dashboard if you signed in with a ``SUPERUSER`` role. + +The following is a brief description of the Dashboard panels: + +.. list-table:: + :widths: 10 25 65 + :header-rows: 1 + + * - No. + - Element + - Description + * - 1 + - :ref:`Services panel` + - Used for viewing and monitoring the defined service queues. + * - 2 + - :ref:`Workers panel` + - Monitors system health and shows each Sqreamd worker running in the cluster. + * - 3 + - :ref:`License information` + - Shows the remaining amount of days left on your license. + + +.. _data_storage_panel_5.4.3: + + + +:ref:`Back to Monitoring Workers and Services from the Dashboard` + +.. _services_panel_5.4.3: + +Subscribing to Workers from the Services Panel +-------------------------- +Services are used to categorize and associate (also known as **subscribing**) workers to particular services. The **Service** panel is used for viewing, monitoring, and adding defined `service queues `_. + + + +The following is a brief description of each pane: + +.. list-table:: + :widths: 10 90 + :header-rows: 1 + + * - No. + - Description + * - 1 + - Adds a worker to the selected service. + * - 2 + - Shows the service name. + * - 3 + - Shows a trend graph of queued statements loaded over time. + * - 4 + - Adds a service. + * - 5 + - Shows the currently processed queries belonging to the service/total queries for that service in the system (including queued queries). + +Adding A Service +^^^^^^^^^^^^^^^^^^^^^ +You can add a service by clicking **+ Add** and defining the service name. + +.. note:: If you do not associate a worker with the new service, it will not be created. + +You can manage workers from the **Workers** panel. For more information about managing workers, see the following: + +* :ref:`Managing Workers from the Workers Panel` +* `Workers `_ + +:ref:`Back to Monitoring Workers and Services from the Dashboard` + +.. _workers_panel_5.4.3: + +Managing Workers from the Workers Panel +------------ +From the **Workers** panel you can do the following: + +* :ref:`View workers ` +* :ref:`Add a worker to a service` +* :ref:`View a worker's active query information` +* :ref:`View a worker's execution plan` + +.. _view_workers_5.4.3: + +Viewing Workers +^^^^^^^^ +The **Worker** panel shows each worker (``sqreamd``) running in the cluster. Each worker has a status bar that represents the status over time. The status bar is divided into 20 equal segments, showing the most dominant activity in that segment. + +From the **Scale** dropdown menu you can set the time scale of the displayed information +You can hover over segments in the status bar to see the date and time corresponding to each activity type: + +* **Idle** – the worker is idle and available for statements. +* **Compiling** – the worker is compiling a statement and is preparing for execution. +* **Executing** – the worker is executing a statement after compilation. +* **Stopped** – the worker was stopped (either deliberately or due to an error). +* **Waiting** – the worker was waiting on an object locked by another worker. + +.. _add_worker_to_service_5.4.3: + +Adding A Worker to A Service +^^^^^^^^^^^^^^^^^^^^^ +You can add a worker to a service by clicking the **add** button. + + + +Clicking the **add** button shows the selected service's workers. You can add the selected worker to the service by clicking **Add Worker**. Adding a worker to a service does not break associations already made between that worker and other services. + + +.. _view_worker_query_information_5.4.3: + +Viewing A Worker's Active Query Information +^^^^^^^^^^^^^^^^^^^^^ +You can view a worker's active query information by clicking **Queries**, which displays them in the selected service. + + +Each statement shows the **query ID**, **status**, **service queue**, **elapsed time**, **execution time**, and **estimated completion status**. In addition, each statement can be stopped or expanded to show its execution plan and progress. For more information on viewing a statement's execution plan and progress, see :ref:`Viewing a Worker's Execution Plan ` below. + +Viewing A Worker's Host Utilization +^^^^^^^^^^^^^^^^^^^^^ + +While viewing a worker's query information, clicking the **down arrow** expands to show the host resource utilization. + + + +The graphs show the resource utilization trends over time, and the **CPU memory** and **utilization** and the **GPU utilization** values on the right. You can hover over the graph to see more information about the activity at any point on the graph. + +Error notifications related to statements are displayed, and you can hover over them for more information about the error. + + +.. _view_worker_execution_plan_5.4.3: + +Viewing a Worker's Execution Plan +^^^^^^^^^^^^^^^^^^^^^ + +Clicking the ellipsis in a service shows the following additional options: + +* **Stop Query** - stops the query. +* **Show Execution Plan** - shows the execution plan as a table. The columns in the **Show Execution Plan** table can be sorted. + +For more information on the current query plan, see `SHOW_NODE_INFO `_. For more information on checking active sessions across the cluster, see `SHOW_SERVER_STATUS `_. + +.. include:: /reference/sql/sql_statements/monitoring_commands/show_server_status.rst + :start-line: 67 + :end-line: 84 + +Managing Worker Status +^^^^^^^^^^^^^^^^^^^^^ + +In some cases you may want to stop or restart workers for maintenance purposes. Each Worker line has a :kbd:`⋮` menu used for stopping, starting, or restarting workers. + + +Starting or restarting workers terminates all queries related to that worker. When you stop a worker, its background turns gray. + + + + +.. |icon-user| image:: /_static/images/studio_icon_user.png + :align: middle + +.. |icon-dots| image:: /_static/images/studio_icon_dots.png + :align: middle + +.. |icon-editor| image:: /_static/images/studio_icon_editor.png + :align: middle + +.. |icon-copy| image:: /_static/images/studio_icon_copy.png + :align: middle + +.. |icon-select| image:: /_static/images/studio_icon_select.png + :align: middle + +.. |icon-dots| image:: /_static/images/studio_icon_dots.png + :align: middle + +.. |icon-filter| image:: /_static/images/studio_icon_filter.png + :align: middle + +.. |icon-ddl-edit| image:: /_static/images/studio_icon_ddl_edit.png + :align: middle + +.. |icon-run-optimizer| image:: /_static/images/studio_icon_run_optimizer.png + :align: middle + +.. |icon-generate-create-statement| image:: /_static/images/studio_icon_generate_create_statement.png + :align: middle + +.. |icon-plus| image:: /_static/images/studio_icon_plus.png + :align: middle + +.. |icon-close| image:: /_static/images/studio_icon_close.png + :align: middle + +.. |icon-left| image:: /_static/images/studio_icon_left.png + :align: middle + +.. |icon-right| image:: /_static/images/studio_icon_right.png + :align: middle + +.. |icon-format-sql| image:: /_static/images/studio_icon_format.png + :align: middle + +.. |icon-download-query| image:: /_static/images/studio_icon_download_query.png + :align: middle + +.. |icon-open-query| image:: /_static/images/studio_icon_open_query.png + :align: middle + +.. |icon-execute| image:: /_static/images/studio_icon_execute.png + :align: middle + +.. |icon-stop| image:: /_static/images/studio_icon_stop.png + :align: middle + +.. |icon-dashboard| image:: /_static/images/studio_icon_dashboard.png + :align: middle + +.. |icon-expand| image:: /_static/images/studio_icon_expand.png + :align: middle + +.. |icon-scale| image:: /_static/images/studio_icon_scale.png + :align: middle + +.. |icon-expand-down| image:: /_static/images/studio_icon_expand_down.png + :align: middle + +.. |icon-add| image:: /_static/images/studio_icon_add.png + :align: middle + +.. |icon-add-worker| image:: /_static/images/studio_icon_add_worker.png + :align: middle + +.. |keep-tabs| image:: /_static/images/studio_keep_tabs.png + :align: middle + +:ref:`Back to Monitoring Workers and Services from the Dashboard` + + + +.. _license_information_5.4.3: + +License Information +---------------------- +The license information section shows the following: + + * The amount of time in days remaining on the license. + * The license storage capacity. + + + + +:ref:`Back to Monitoring Workers and Services from the Dashboard` \ No newline at end of file diff --git a/sqream_studio_5.4.3/viewing_logs.rst b/sqream_studio_5.4.3/viewing_logs.rst new file mode 100644 index 000000000..7f98bea5e --- /dev/null +++ b/sqream_studio_5.4.3/viewing_logs.rst @@ -0,0 +1,122 @@ +.. _viewing_logs: + +.. _logs_top_5.4.3: + +**************************** +Viewing Logs +**************************** +The **Logs** screen is used for viewing logs and includes the following elements: + +.. list-table:: + :widths: 15 75 + :header-rows: 1 + + * - Element + - Description + * - :ref:`Filter area` + - Lets you filter the data shown in the table. + * - :ref:`Query tab` + - Shows basic query information logs, such as query number and the time the query was run. + * - :ref:`Session tab` + - Shows basic session information logs, such as session ID and user name. + * - :ref:`System tab` + - Shows all system logs. + * - :ref:`Log lines tab` + - Shows the total amount of log lines. + + +.. _filter_5.4.3: + +Filtering Table Data +------------- +From the Logs tab, from the **FILTERS** area you can also apply the **TIMESPAN**, **ONLY ERRORS**, and additional filters (**Add**). The **Timespan** filter lets you select a timespan. The **Only Errors** toggle button lets you show all queries, or only queries that generated errors. The **Add** button lets you add additional filters to the data shown in the table. The **Filter** button applies the selected filter(s). + +Other filters require you to select an item from a dropdown menu: + +* INFO +* WARNING +* ERROR +* FATAL +* SYSTEM + +You can also export a record of all of your currently filtered logs in Excel format by clicking **Download** located above the Filter area. + +.. _queries_5.4.3: + +:ref:`Back to Viewing Logs` + + +Viewing Query Logs +---------- +The **QUERIES** log area shows basic query information, such as query number and the time the query was run. The number next to the title indicates the amount of queries that have been run. + +From the Queries area you can see and sort by the following: + +* Query ID +* Start time +* Query +* Compilation duration +* Execution duration +* Total duration +* Details (execution details, error details, successful query details) + +In the Queries table, you can click on the **Statement ID** and **Query** items to set them as your filters. In the **Details** column you can also access additional details by clicking one of the **Details** options for a more detailed explanation of the query. + +:ref:`Back to Viewing Logs` + +.. _sessions_5.4.3: + +Viewing Session Logs +---------- +The **SESSIONS** tab shows the sessions log table and is used for viewing activity that has occurred during your sessions. The number at the top indicates the amount of sessions that have occurred. + +From here you can see and sort by the following: + +* Timestamp +* Connection ID +* Username +* Client IP +* Login (Success or Failed) +* Duration (of session) +* Configuration Changes + +In the Sessions table, you can click on the **Timestamp**, **Connection ID**, and **Username** items to set them as your filters. + +:ref:`Back to Viewing Logs` + +.. _system_5.4.3: + +Viewing System Logs +---------- +The **SYSTEM** tab shows the system log table and is used for viewing all system logs. The number at the top indicates the amount of sessions that have occurred. Because system logs occur less frequently than queries and sessions, you may need to increase the filter timespan for the table to display any system logs. + +From here you can see and sort by the following: + +* Timestamp +* Log type +* Message + +In the Systems table, you can click on the **Timestamp** and **Log type** items to set them as your filters. In the **Message** column, you can also click on an item to show more information about the message. + +:ref:`Back to Viewing Logs` + +.. _log_lines_5.4.3: + +Viewing All Log Lines +---------- +The **LOG LINES** tab is used for viewing the total amount of log lines in a table. From here users can view a more granular breakdown of log information collected by Studio. The other tabs (QUERIES, SESSIONS, and SYSTEM) show a filtered form of the raw log lines. For example, the QUERIES tab shows an aggregation of several log lines. + +From here you can see and sort by the following: + +* Timestamp +* Message level +* Worker hostname +* Worker port +* Connection ID +* Database name +* User name +* Statement ID + +In the **LOG LINES** table, you can click on any of the items to set them as your filters. + +:ref:`Back to Viewing Logs` \ No newline at end of file diff --git a/troubleshooting/core_dumping_related_issues.rst b/troubleshooting/core_dumping_related_issues.rst new file mode 100644 index 000000000..ace7c8787 --- /dev/null +++ b/troubleshooting/core_dumping_related_issues.rst @@ -0,0 +1,58 @@ +.. _core_dumping_related_issues: + +*********************** +Core Dumping Related Issues +*********************** + +The **Core Dumping Related Issues** page describes the troubleshooting procedure to be followed if all parameters have been configured correctly, but the cores have not been created. + +**To troubleshoot core dumping:** + +1. Reboot the server. + +2. Verify that you have folder permissions: + + .. code-block:: console + + $ sudo chmod -R 777 /tmp/core_dumps + +3. Verify that the limits have been set correctly: + + .. code-block:: console + + $ ulimit -c + + If all parameters have been configured correctly, the correct output is: + + .. code-block:: console + + $ unlimited + +4. If all parameters have been configured correctly, but running **ulimit -c** outputs **0**, run the following: + + .. code-block:: console + + $ sudo vim /etc/profile + +5. Search for line and tag it with the **hash** symbol: + + .. code-block:: console + + $ ulimit -S -c 0 > /dev/null 2>&1 + + +6. If the line is not found in **/etc/profile** directory, do the following: + + a. Run the following command: + + .. code-block:: console + + $ sudo vim /etc/init.d/functions + + b. Search for the following: + + .. code-block:: console + + $ ulimit -S -c ${DAEMON_COREFILE_LIMIT:-0} >/dev/null 2>&1 + + c. If the line is found, tag it with the **hash** symbol and reboot the server. diff --git a/troubleshooting/examining_logs.rst b/troubleshooting/examining_logs.rst new file mode 100644 index 000000000..9b5a5fb79 --- /dev/null +++ b/troubleshooting/examining_logs.rst @@ -0,0 +1,6 @@ +.. _examining_logs: + +*********************** +Examining Logs +*********************** +See the :ref:`collecting_logs` section of the :ref:`information_for_support` guide for information about collecting logs for support. \ No newline at end of file diff --git a/troubleshooting/identifying_configuration_issues.rst b/troubleshooting/identifying_configuration_issues.rst new file mode 100644 index 000000000..25708cda6 --- /dev/null +++ b/troubleshooting/identifying_configuration_issues.rst @@ -0,0 +1,32 @@ +.. _identifying_configuration_issues: + +*********************** +Identifying Configuration Issues +*********************** + +The **Troubleshooting Common Issues** page describes how to troubleshoot the following common issues: + +.. toctree:: + :maxdepth: 2 + :glob: + :titlesonly: + + +Starting a SQream DB temporarily (not as part of a cluster, with default settings) can be helpful in identifying configuration issues. + +Example: + +.. code-block:: console + + $ sqreamd /home/rhendricks/raviga_database 0 5000 /home/sqream/.sqream/license.enc + +.. tip:: + + * Using ``nohup`` and ``&`` sends SQream DB to run in the background. + + * + It is safe to stop SQream DB at any time using ``kill``. No partial data or data corruption should occur when using this method to stop the process. + + .. code-block:: console + + $ kill -9 $SQREAM_PID \ No newline at end of file diff --git a/troubleshooting/index.rst b/troubleshooting/index.rst new file mode 100644 index 000000000..efbcdd412 --- /dev/null +++ b/troubleshooting/index.rst @@ -0,0 +1,25 @@ +.. _troubleshooting: + +*********************** +Troubleshooting +*********************** + +The **Troubleshooting** page describes solutions to the following issues: + +.. toctree:: + :maxdepth: 1 + :glob: + + remedying_slow_queries + resolving_common_issues + examining_logs + identifying_configuration_issues + lock_related_issues + sas_viya_related_issues + tableau_related_issues + solving_code_126_odbc_errors + log_related_issues + node_js_related_issues + core_dumping_related_issues + sqream_sql_installation_related_issues + information_for_support \ No newline at end of file diff --git a/guides/operations/information_for_support.rst b/troubleshooting/information_for_support.rst similarity index 93% rename from guides/operations/information_for_support.rst rename to troubleshooting/information_for_support.rst index 62f79b227..16dcb4174 100644 --- a/guides/operations/information_for_support.rst +++ b/troubleshooting/information_for_support.rst @@ -1,7 +1,7 @@ .. _information_for_support: ******************************************* -Gathering information for SQream support +Gathering Information for SQream Support ******************************************* .. What do we want to look into a performance issue @@ -12,7 +12,7 @@ Gathering information for SQream support `SQream Support `_ is ready to answer any questions, and help solve any issues with SQream DB. -Getting support and reporting bugs +Getting Support and Reporting Bugs ======================================= When contacting `SQream Support `_, we recommend reporting the following information: @@ -28,7 +28,7 @@ When possible, please attach as many of the following: * :ref:`Log files` * Screen captures if relevant -How SQream debugs issues +How SQream Debugs Issues =================================== Reproduce @@ -65,7 +65,7 @@ Your SQream account manager will keep you up-to-date about the status of the iss .. _reproducible_statement: -Collecting a reproducible example of a problematic statement +Collecting a Reproducible Example of a Problematic Statement =============================================================== SQream DB contains an SQL utility that can help SQream support reproduce a problem with a query or statement. @@ -107,7 +107,7 @@ Example .. _collecting_logs: -Collecting logs and metadata database +Collecting Logs and Metadata Database ============================================= SQream DB comes bundled with a data collection utility and an SQL utility intended for collecting logs and additional information that can help SQream support drill down into possible issues. @@ -132,7 +132,8 @@ Write an archive to ``/home/rhendricks``, containing log files and metadata data ; -Using the command line utility: +Using the Command Line Utility: +============================================= .. code-block:: console diff --git a/troubleshooting/lock_related_issues.rst b/troubleshooting/lock_related_issues.rst new file mode 100644 index 000000000..1a15858ec --- /dev/null +++ b/troubleshooting/lock_related_issues.rst @@ -0,0 +1,29 @@ +.. _lock_related_issues: + +*********************** +Lock Related Issues +*********************** +Sometimes, a rare situation can occur where a lock is never freed. + +The workflow for troubleshooting locks is: + +#. Identify which statement has obtained locks +#. Understand if the statement is itself stuck, or waiting for another statement +#. Try to abort the offending statement +#. Force the stale locks to be removed + +For example, we will assume that the statement from the previous example is stuck (statement #\ ``287``). We can attempt to abort it using :ref:`stop_statement`: + +.. code-block:: psql + + t=> SELECT STOP_STATEMENT(287); + executed + +If the locks still appear in the :ref:`show_locks` utility, we can force remove the stale locks: + +.. code-block:: psql + + t=> SELECT RELEASE_DEFUNCT_LOCKS(); + executed + +.. warning:: This operation can cause some statements to fail on the specific worker on which they are queued. This is intended as a "last resort" to solve stale locks. \ No newline at end of file diff --git a/troubleshooting/log_related_issues.rst b/troubleshooting/log_related_issues.rst new file mode 100644 index 000000000..a260f59d5 --- /dev/null +++ b/troubleshooting/log_related_issues.rst @@ -0,0 +1,133 @@ +.. _log_related_issues: + +*********************** +Log Related Issues +*********************** +The **Log Related Issues** page describes how to resolve the following common issues: + +.. toctree:: + :maxdepth: 2 + :glob: + :titlesonly: + +Loading Logs with Foreign Tables +--------------------------------------- +Assuming logs are stored at ``/home/rhendricks/sqream_storage/logs/``, a database administrator can access the logs using the :ref:`external_tables` concept through SQream DB. + +.. code-block:: postgres + + CREATE FOREIGN TABLE logs + ( + start_marker VARCHAR(4), + row_id BIGINT, + timestamp DATETIME, + message_level TEXT, + thread_id TEXT, + worker_hostname TEXT, + worker_port INT, + connection_id INT, + database_name TEXT, + user_name TEXT, + statement_id INT, + service_name TEXT, + message_type_id INT, + message TEXT, + end_message VARCHAR(5) + ) + WRAPPER csv_fdw + OPTIONS + ( + LOCATION = '/home/rhendricks/sqream_storage/logs/**/sqream*.log', + DELIMITER = '|', + CONTINUE_ON_ERROR = true + ) + ; + +For more information, see `Loading Logs with Foreign Tables `_. + + + + + +Counting Message Types +------------------------------ + +.. code-block:: psql + + t=> SELECT message_type_id, COUNT(*) FROM logs GROUP BY 1; + message_type_id | count + ----------------+---------- + 0 | 9 + 1 | 5578 + 4 | 2319 + 10 | 2788 + 20 | 549 + 30 | 411 + 31 | 1720 + 32 | 1720 + 100 | 2592 + 101 | 2598 + 110 | 2571 + 200 | 11 + 500 | 136 + 1000 | 19 + 1003 | 19 + 1004 | 19 + 1010 | 5 + +Finding Fatal Errors +---------------------- + +.. code-block:: psql + + t=> SELECT message FROM logs WHERE message_type_id=1010; + Internal Runtime Error,open cluster metadata database:IO error: lock /home/rhendricks/sqream_storage/leveldb/LOCK: Resource temporarily unavailable + Internal Runtime Error,open cluster metadata database:IO error: lock /home/rhendricks/sqream_storage/leveldb/LOCK: Resource temporarily unavailable + Mismatch in storage version, upgrade is needed,Storage version: 25, Server version is: 26 + Mismatch in storage version, upgrade is needed,Storage version: 25, Server version is: 26 + Internal Runtime Error,open cluster metadata database:IO error: lock /home/rhendricks/sqream_storage/LOCK: Resource temporarily unavailable + +Countng Error Events Within a Certain Timeframe +--------------------------------------------------- + +.. code-block:: psql + + t=> SELECT message_type_id, + . COUNT(*) + . FROM logs + . WHERE message_type_id IN (1010,500) + . AND timestamp BETWEEN '2019-12-20' AND '2020-01-01' + . GROUP BY 1; + message_type_id | count + ----------------+------ + 500 | 18 + 1010 | 3 + + +.. _tracing_errors: + +Tracing Errors to Find Offending Statements +------------------------------------------------- + +If we know an error occured, but don't know which statement caused it, we can find it using the connection ID and statement ID. + +.. code-block:: psql + + t=> SELECT connection_id, statement_id, message + . FROM logs + . WHERE message_level = 'ERROR' + . AND timestamp BETWEEN '2020-01-01' AND '2020-01-06'; + connection_id | statement_id | message + --------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------ + 79 | 67 | Column type mismatch, expected UByte, got INT64 on column Number, file name: /home/sqream/nba.parquet + +Use the ``connection_id`` and ``statement_id`` to narrow down the results. + +.. code-block:: psql + + t=> SELECT database_name, message FROM logs + . WHERE connection_id=79 AND statement_id=67 AND message_type_id=1; + database_name | message + --------------+-------------------------- + master | Query before parsing + master | SELECT * FROM nba_parquet \ No newline at end of file diff --git a/troubleshooting/node_js_related_issues.rst b/troubleshooting/node_js_related_issues.rst new file mode 100644 index 000000000..b3b95b2ed --- /dev/null +++ b/troubleshooting/node_js_related_issues.rst @@ -0,0 +1,54 @@ +.. _node_js_related_issues: + +*********************** +Node.js Related Issues +*********************** +The **Node.js Related Issues** page describes how to resolve the following common issues: + +.. toctree:: + :maxdepth: 2 + :glob: + :titlesonly: + +Preventing Heap Out of Memory Errors +-------------------------------------------- + +Some workloads may cause Node.JS to fail with the error: + +.. code-block:: none + + FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory + +To prevent this error, modify the heap size configuration by setting the ``--max-old-space-size`` run flag. + +For example, set the space size to 2GB: + +.. code-block:: console + + $ node --max-old-space-size=2048 my-application.js + +Providing Support for BIGINT Data Type +------------------------ + +The Node.JS connector supports fetching ``BIGINT`` values from SQream DB. However, some applications may encounter an error when trying to serialize those values. + +The error that appears is: +.. code-block:: none + + TypeError: Do not know how to serialize a BigInt + +This is because JSON specification do not support BIGINT values, even when supported by Javascript engines. + +To resolve this issue, objects with BIGINT values should be converted to string before serializing, and converted back after deserializing. + +For example: + +.. code-block:: javascript + + const rows = [{test: 1n}] + const json = JSON.stringify(rows, , (key, value) => + typeof value === 'bigint' + ? value.toString() + : value // return everything else unchanged + )); + console.log(json); // [{"test": "1"}] \ No newline at end of file diff --git a/troubleshooting/remedying_slow_queries.rst b/troubleshooting/remedying_slow_queries.rst new file mode 100644 index 000000000..8a109f0c0 --- /dev/null +++ b/troubleshooting/remedying_slow_queries.rst @@ -0,0 +1,79 @@ +.. _remedying_slow_queries: + +*********************** +Remedying Slow Queries +*********************** + +The **Remedying Slow Queries** page describes how to troubleshoot the causes of slow queries. + +The following table is a checklist you can use to identify the cause of your slow queries: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Step + - Description + - Results + * - 1 + - A single query is slow + - + If a query isn't performing as you expect, follow the :ref:`Query best practices` part of the :ref:`sql_best_practices` guide. + + If all queries are slow, continue to step 2. + * - 2 + - All queries on a specific table are slow + - + #. If all queries on a specific table aren't performing as you expect, follow the :ref:`Table design best practices` part of the :ref:`sql_best_practices` guide. + #. Check for active delete predicates in the table. Consult the :ref:`delete_guide` guide for more information. + + If the problem spans all tables, continue to step 3. + * - 3 + - Check that all workers are up + - + Use ``SELECT show_cluster_nodes();`` to list the active cluster workers. + + If the worker list is incomplete, follow the :ref:`cluster troubleshooting` section below. + + If all workers are up, continue to step 4. + * - 4 + - Check that all workers are performing well + - + #. Identify if a specific worker is slower than others by running the same query on different workers. (e.g. by connecting directly to the worker or through a service queue) + #. If a specific worker is slower than others, investigate performance issues on the host using standard monitoring tools (e.g. ``top``). + #. Restart SQream DB workers on the problematic host. + + If all workers are performing well, continue to step 5. + * - 5 + - Check if the workload is balanced across all workers + - + #. Run the same query several times and check that it appears across multiple workers (use ``SELECT show_server_status()`` to monitor) + #. If some workers have a heavier workload, check the service queue usage. Refer to the :ref:`workload_manager` guide. + + If the workload is balanced, continue to step 6. + * - 6 + - Check if there are long running statements + - + #. Identify any currently running statements (use ``SELECT show_server_status()`` to monitor) + #. If there are more statements than available resources, some statements may be in an ``In queue`` mode. + #. If there is a statement that has been running for too long and is blocking the queue, consider stopping it (use ``SELECT stop_statement()``). + + If the statement does not stop correctly, contact SQream support. + + If there are no long running statements or this does not help, continue to step 7. + * - 7 + - Check if there are active locks + - + #. Use ``SELECT show_locks()`` to list any outstanding locks. + #. If a statement is locking some objects, consider waiting for that statement to end or stop it. + #. If after a statement is completed the locks don't free up, refer to the :ref:`concurrency_and_locks` guide. + + If performance does not improve after the locks are released, continue to step 8. + * - 8 + - Check free memory across hosts + - + #. Check free memory across the hosts by running ``$ free -th`` from the terminal. + #. If the machine has less than 5% free memory, consider **lowering** the ``limitQueryMemoryGB`` and ``spoolMemoryGB`` settings. Refer to the :ref:`configuration` guide. + #. If the machine has a lot of free memory, consider **increasing** the ``limitQueryMemoryGB`` and ``spoolMemoryGB`` settings. + + If performance does not improve, contact SQream support for more help. \ No newline at end of file diff --git a/troubleshooting/resolving_common_issues.rst b/troubleshooting/resolving_common_issues.rst new file mode 100644 index 000000000..fd90472d3 --- /dev/null +++ b/troubleshooting/resolving_common_issues.rst @@ -0,0 +1,60 @@ +.. _resolving_common_issues: + +*********************** +Resolving Common Issues +*********************** + +The **Resolving Common Issues** page describes how to resolve the following common issues: + +.. toctree:: + :maxdepth: 2 + :glob: + :titlesonly: + + +Troubleshooting Cluster Setup and Configuration +----------------------------------------------------- + +#. Note any errors - Make a note of any error you see, or check the :ref:`logs` for errors you might have missed. + +#. If SQream DB can't start, start SQream DB on a new storage cluster, with default settings. If it still can't start, there could be a driver or hardware issue. :ref:`Contact SQream support`. + +#. Reproduce the issue with a standalone SQream DB - starting up a temporary, standalone SQream DB can isolate the issue to a configuration issue, network issue, or similar. + +#. Reproduce on a minimal example - Start a standalone SQream DB on a clean storage cluster and try to replicate the issue if possible. + + +Troubleshooting Connectivity Issues +----------------------------------- + +#. Verify the correct login credentials - username, password, and database name. + +#. Verify the host name and port + +#. Try connecting directly to a SQream DB worker, rather than via the load balancer + +#. Verify that the driver version you're using is supported by the SQream DB version. Driver versions often get updated together with major SQream DB releases. + +#. Try connecting directly with :ref:`the built in SQL client`. If you can connect with the local SQL client, check network availability and firewall settings. + +Troubleshooting Query Performance +------------------------------------ + +#. Use :ref:`show_node_info` to examine which building blocks consume time in a statement. If the query has finished, but the results are not yet materialized in the client, it could point to a problem in the application's data buffering or a network throughput issue.. + +#. If a problem occurs through a 3\ :sup:`rd` party client, try reproducing it directly with :ref:`the built in SQL client`. If the performance is better in the local client, it could point to a problem in the application or network connection. + +#. Consult the :ref:`sql_best_practices` guide to learn how to optimize queries and table structures. + + +Troubleshooting Query Behavior +--------------------------------- + +#. Consult the :ref:`sql` reference to verify if a statement or syntax behaves correctly. SQream DB may have some differences in behavior when compared to other databases. + +#. If a problem occurs through a 3\ :sup:`rd` party client, try reproducing it directly with :ref:`the built in SQL client`. If the problem still occurs, file an issue with SQream support. + +File an issue with SQream support +------------------------------------ + +To file an issue, follow our :ref:`information_for_support` guide. \ No newline at end of file diff --git a/troubleshooting/sas_viya_related_issues.rst b/troubleshooting/sas_viya_related_issues.rst new file mode 100644 index 000000000..6661dec95 --- /dev/null +++ b/troubleshooting/sas_viya_related_issues.rst @@ -0,0 +1,55 @@ +.. _sas_viya_related_issues: + +*********************** +SAS Viya Related Issues +*********************** + +This section describes the following best practices and troubleshooting procedures when connecting to SQream using SAS Viya: + +.. contents:: + :local: + +Inserting Only Required Data +------ +When using Tableau, SQream recommends using only data that you need, as described below: + +* Insert only the data sources you need into SAS Viya, excluding tables that don’t require analysis. + + :: + + +* To increase query performance, add filters before analyzing. Every modification you make while analyzing data queries the SQream database, sometimes several times. Adding filters to the datasource before exploring limits the amount of data analyze and increases query performance. + + +Creating a Separate Service for SAS Viya +------ +SQream recommends creating a separate service for SAS Viya with the DWLM. This reduces the impact that Tableau has on other applications and processes, such as ETL. In addition, this works in conjunction with the load balancer to ensure good performance. + +Locating the SQream JDBC Driver +------ +In some cases, SAS Viya cannot locate the SQream JDBC driver, generating the following error message: + +.. code-block:: text + + java.lang.ClassNotFoundException: com.sqream.jdbc.SQDriver + +**To locate the SQream JDBC driver:** + +1. Verify that you have placed the JDBC driver in a directory that SAS Viya can access. + + :: + + +2. Verify that the classpath in your SAS program is correct, and that SAS Viya can access the file that it references. + + :: + + +3. Restart SAS Viya. + +For more troubleshooting assistance, see the `SQream Support Portal `_. + + +Supporting TEXT +------ +In SAS Viya versions lower than 4.0, casting ``TEXT`` to ``CHAR`` changes the size to 1,024, such as when creating a table including a ``TEXT`` column. This is resolved by casting ``TEXT`` into ``CHAR`` when using the JDBC driver. diff --git a/troubleshooting/solving_code_126_odbc_errors.rst b/troubleshooting/solving_code_126_odbc_errors.rst new file mode 100644 index 000000000..2e652b113 --- /dev/null +++ b/troubleshooting/solving_code_126_odbc_errors.rst @@ -0,0 +1,14 @@ +.. _solving_code_126_odbc_errors: + +*********************** +Solving "Code 126" ODBC Errors +*********************** +After installing the ODBC driver, you may experience the following error: + +.. code-block:: none + + The setup routines for the SQreamDriver64 ODBC driver could not be loaded due to system error + code 126: The specified module could not be found. + (c:\Program Files\SQream Technologies\ODBC Driver\sqreamOdbc64.dll) + +This is an issue with the Visual Studio Redistributable packages. Verify you've correctly installed them, as described in the :ref:`Visual Studio 2015 Redistributables ` section above. diff --git a/troubleshooting/sqream_sql_installation_related_issues.rst b/troubleshooting/sqream_sql_installation_related_issues.rst new file mode 100644 index 000000000..8225a2f18 --- /dev/null +++ b/troubleshooting/sqream_sql_installation_related_issues.rst @@ -0,0 +1,33 @@ +.. _sqream_sql_installation_related_issues: + +*********************** +SQream SQL Installation Related Issues +*********************** + +The **SQream SQL Installation Related Issues** page describes how to resolve SQream SQL installation related issues. + +Upon running sqream sql for the first time, you may get an error ``error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory``. + +Solving this error requires installing the ncruses or libtinfo libraries, depending on your operating system. + +* Ubuntu: + + #. Install ``libtinfo``: + + ``$ sudo apt-get install -y libtinfo`` + #. Depending on your Ubuntu version, you may need to create a symbolic link to the newer libtinfo that was installed. + + For example, if ``libtinfo`` was installed as ``/lib/x86_64-linux-gnu/libtinfo.so.6.2``: + + ``$ sudo ln -s /lib/x86_64-linux-gnu/libtinfo.so.6.2 /lib/x86_64-linux-gnu/libtinfo.so.5`` + +* CentOS / RHEL: + + #. Install ``ncurses``: + + ``$ sudo yum install -y ncurses-libs`` + #. Depending on your RHEL version, you may need to create a symbolic link to the newer libtinfo that was installed. + + For example, if ``libtinfo`` was installed as ``/usr/lib64/libtinfo.so.6``: + + ``$ sudo ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so.5`` \ No newline at end of file diff --git a/troubleshooting/tableau_related_issues.rst b/troubleshooting/tableau_related_issues.rst new file mode 100644 index 000000000..99b4a04dd --- /dev/null +++ b/troubleshooting/tableau_related_issues.rst @@ -0,0 +1,73 @@ +.. _tableau_related_issues: + +*********************** +Tableau Related Issues +*********************** +This section describes the following best practices and troubleshooting procedures when connecting to Tableau: + +.. contents:: + :local: + +Inserting Only Required Data +~~~~~~~~~~~~~~~~~~ +When using Tableau, SQream recommends using only data that you need, as described below: + +* Insert only the data sources you need into Tableau, excluding tables that don't require analysis. + + :: + +* To increase query performance, add filters before analyzing. Every modification you make while analyzing data queries the SQream database, sometimes several times. Adding filters to the datasource before exploring limits the amount of data analyze and increases query performance. + +Using Tableau's Table Query Syntax +~~~~~~~~~~~~~~~~~~~ +Dragging your desired tables into the main area in Tableau builds queries based on its own syntax. This helps ensure increased performance, while using views or custom SQL may degrade performance. In addition, SQream recommends using the :ref:`create_view` to create pre-optimized views, which your datasources point to. + +Creating a Separate Service for Tableau +~~~~~~~~~~~~~~~~~~~ +SQream recommends creating a separate service for Tableau with the DWLM. This reduces the impact that Tableau has on other applications and processes, such as ETL. In addition, this works in conjunction with the load balancer to ensure good performance. + +Error Saving Large Quantities of Data as Files +~~~~~~~~~~~~~~~~~~~ +An **FAB9A2C5** error can when saving large quantities of data as files. If you receive this error when writing a connection string, add the ``fetchSize`` parameter to ``1``, as shown below: + +.. code-block:: text + + jdbc:Sqream:///;user=;password=sqream;[; fetchSize=1...] + +For more information on troubleshooting error **FAB9A2C5**, see the `Tableau Knowledge Base `_. + +Troubleshooting Workbook Performance Before Deploying to the Tableau Server +~~~~~~~~~~~~~~~~~~~ +Tableau has a built-in `performance recorder `_ that shows how time is being spent. If you're seeing slow performance, this could be the result of a misconfiguration such as setting concurrency too low. + +Use the Tableau Performance Recorder for viewing the performance of queries run by Tableau. You can use this information to identify queries that can be optimized by using views. + +Troubleshooting Error Codes +~~~~~~~~~~~~~~~~~~~ +Tableau may be unable to locate the SQream JDBC driver. The following message is displayed when Tableau cannot locate the driver: + +.. code-block:: console + + Error Code: 37CE01A3, No suitable driver installed or the URL is incorrect + +**To troubleshoot error codes:** + +If Tableau cannot locate the SQream JDBC driver, do the following: + + 1. Verify that the JDBC driver is located in the correct directory: + + * **Tableau Desktop on Windows:** *C:\Program Files\Tableau\Drivers* + * **Tableau Desktop on MacOS:** *~/Library/Tableau/Drivers* + * **Tableau on Linux**: */opt/tableau/tableau_driver/jdbc* + + 2. Find the file path for the JDBC driver and add it to the Java classpath: + + * **For Linux** - ``export CLASSPATH=;$CLASSPATH`` + + :: + + * **For Windows** - add an environment variable for the classpath: + + + +If you experience issues after restarting Tableau, see the `SQream support portal `_.