4.3 #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

TaliaSQream wants to merge 892 commits into master from 4.3

.readthedocs.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,28 @@
+    # .readthedocs.yaml
+    # Read the Docs configuration file
+    # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+    # Required
+    version: 2
+    # Set the version of Python and other tools you might need
+    build:
+      os: ubuntu-22.04
+      tools:
+        python: "3.11"
+        # You can also specify other tool versions:
+        # nodejs: "19"
+        # rust: "1.64"
+        # golang: "1.19"
+    # Build documentation in the docs/ directory with Sphinx
+    sphinx:
+       configuration: conf.py
+    # If using Sphinx, optionally build your docs in additional formats such as PDF
+    formats: [htmlzip,pdf]
+    # Optionally declare the Python requirements required to build your docs
+    python:
+       install:
+       - requirements: requirements.txt

404.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,18 +1,14 @@
  
    :orphan:

    *********************************

     Couldn't find the page - 404

    *********************************

    **************************

    Page Cannot Be Found - 404

    **************************

    Unfortunately we could not find this page.

    Use the **search bar**, or use the navigation sidebar to find what you're looking for.

    .. rubric:: Looking for the old documentation?

    If you're looking for an older version of the documentation, versions 1.10 through 2019.2.1 are available at http://previous.sqream.com .

    Use the **Search docs** bar, or use the navigation sidebar to find what you're looking for.

    .. rubric:: Need help?

    If you couldn't find what you're looking for, we're always happy to help. Visit `SQream's support portal <https://support.sqream.com>`_ for additional support.

    If you couldn't find what you're looking for, we're always happy to help. 

    Visit the `SQreamDB support portal <https://sqream.atlassian.net/servicedesk/customer/portal/2/group/8/create/26>`_ for additional help.

_static/images/3c9c4e8b-49bd-44a8-9829-81d1772ed962.gif

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/New_Dark_Gray.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/SAP_BO.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/SAP_BO_2.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/SQream_logo_without background-15.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/Third_Party_Connectors/pentaho/connection_tested_successfully_2.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/chunks_and_extents.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/color_table.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/kafka_flow.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/new.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/new_2022.1.1.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/new_2022.1.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/new_dark_gray_2022.1.1.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/new_gray_2022.1.1.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/sqream_db_internals.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/sqream_db_table_crop.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/storage_organization.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

_static/images/table_columns_storage.png

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

architecture/concurrency_and_scaling_in_sqream.rst

-Original file line number
+Diff line change
@@ -0,0 +1,144 @@
+    .. _concurrency_and_scaling_in_sqream:
+    ******
+    Sizing
+    ******
+    Concurrency and Scaling in SQreamDB
+    ===================================
+    A SQreamDB cluster can execute one statement per worker process while also supporting the concurrent operation of multiple workers. Utility functions with minimal resource requirements, such as :ref:`show_server_status`, :ref:`show_locks`, and :ref:`show_node_info` will be executed regardless of the workload.
+    Minimum Resource Required Per Worker:
+    .. list-table::
+       :widths: auto
+       :header-rows: 1
+       * - Component
+         - CPU Cores
+         - RAM (GB)
+         - Local Storage (GB)
+       * - Worker
+         - 8
+         - 128
+         - 10
+       * - Metadata Server
+         - 16 cores per 100 Workers
+         - 20 GB RAM for every 1 trillion rows
+         - 	10
+       * - SqreamDB Acceleration Studio
+         - 16
+         - 16
+         - 	50
+       * - Server Picker
+         - 1
+         - 2
+         -
+    Lightweight queries, such as :ref:`copy_to` and :ref:`Clean-Up<delete_guide>` require 64 RAM (GB).
+    Maximum Workers Per GPU:
+    .. list-table::
+       :widths: auto
+       :header-rows: 1
+       * - GPU
+         - Workers
+       * - NVIDIA Turing T4 (16GB)
+         - 1
+       * - NVIDIA Volta V100 (32GB)
+         - 2
+       * - NVIDIA Ampere A100 (40GB)
+         - 3
+       * - NVIDIA Ampere A100 (80GB)
+         - 6
+       * - NVIDIA Hopper H100 (80GB)
+         - 6
+       * - L40S Ada Lovelace (48GB)
+         - 4
+    .. tip:: Your GPU is not on the list? Visit `SQreamDB Support <https://sqream.atlassian.net/servicedesk/customer/portal/2/group/8/create/26>`_ for additional information.
+    Scaling When Data Sizes Grow
+    ----------------------------
+    For many statements, SQreamDB scales linearly when adding more storage and querying on large data sets. It uses optimized 'brute force' algorithms and implementations, which don't suffer from sudden performance cliffs at larger data sizes.
+    Scaling When Queries Are Queuing
+    --------------------------------
+    SQreamDB scales well by adding more workers, GPUs, and nodes to support more concurrent statements.
+    What To Do When Queries Are Slow
+    --------------------------------
+    Adding more workers or GPUs does not boost the performance of a single statement or query.
+    To boost the performance of a single statement, start by examining the :ref:`best practices<sql_best_practices>` and ensure the guidelines are followed.
+    Adding additional RAM to nodes, using more GPU memory, and faster CPUs or storage can also sometimes help.
+    .. _spooling:
+    Spooling Configuration
+    ======================
+    :math:`limitQueryMemoryGB=\frac{\text{Total RAM - Internal Operation - metadata Server - Server picker}}{\text{Number of Workers}}`
+    :math:`spoolMemoryGB=limitQueryMemoryGB - 50GB`
+    The ``limitQueryMemoryGB`` flag is the total memory you’ve allocated for processing queries. In addition, the ``limitQueryMemoryGB`` defines how much total system memory is used by each worker. Note that ``spoolMemoryGB`` must bet set to less than the ``limitQueryMemoryGB``.
+    Example
+    -------
+    Setting Spool Memory
+    ~~~~~~~~~~~~~~~~~~~~
+    The provided examples assume a configuration with 2T of RAM, 8 workers running on 2 A100(80GB) GPUs, with 200 GB allocated for Internal Operations, Metadata Server, Server Picker, and UI.
+    Configuring the ``limitQueryMemoryGB`` using the Worker configuration file:
+    .. code-block:: console
+       {
+           “cluster”: “/home/test_user/sqream_testing_temp/sqreamdb”,
+           “gpu”:  0,
+           “licensePath”: “home/test_user/SQream/tests/license.enc”,
+           “machineIP”: “127.0.0.1”,
+           “metadataServerIp”: 127.0.0.1,
+           “metadataServerPort”: 3105,
+           “port”: 5000,
+           “useConfigIP”: true,
+           “limitQueryMemoryGB" : 225,
+       }
+    Configuring the ``spoolMemoryGB`` using the legacy configuration file:
+    .. code-block:: console
+    	{
+    		"diskSpaceMinFreePercent": 10,
+    		"enableLogDebug": false,
+    		"insertCompressors": 8,
+    		"insertParsers": 8,
+    		"isUnavailableNode": false,
+    		"logBlackList": "webui",
+    		"logDebugLevel": 6,
+    		"nodeInfoLoggingSec": 60,
+    		"useClientLog": true,
+    		"useMetadataServer": true,
+    		"spoolMemoryGB": 175,
+    		"waitForClientSeconds": 18000,
+    		"enablePythonUdfs": true
+    	}
+    .. rubric:: Need help?
+    Visit `SQreamDB Support <https://sqream.atlassian.net/servicedesk/customer/portal/2/group/8/create/26>`_ for additional information.

architecture/filesystem_and_filesystem_usage.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,33 +1,33 @@
  
    .. _filesystem_and_filesystem_usage:

    *******************************

    Filesystem and usage

    *******************************

    ********************

    Filesystem and Usage

    ********************

    SQream DB writes and reads data from disk.

    SQreamDB writes and reads data from disk.

    The SQream DB storage directory, sometimes refered to as a **storage cluster** is a collection of database objects, metadata database, and logs.

    The SQreamDB storage directory, sometimes referred to as a **storage cluster** is a collection of database objects, metadata database, and logs.

    Each SQream DB worker and the metadata server must have access to the storage cluster in order to function properly.

    Each SQreamDB worker and the metadata server must have access to the storage cluster in order to function properly.

    .. _storage_cluster:

    Directory organization

    ============================

    ======================

    .. figure:: /_static/images/storage_organization.png

    The **cluster root** is the directory in which all data for SQream DB is stored.

    The **cluster root** is the directory in which all data for SQreamDB is stored.

    .. contents:: SQream DB storage cluster directories

    .. contents:: SQreamDB storage cluster directories

       :local:

    ``databases``

    ----------------

    -------------

    The databases directory houses all of the actual data in tables and columns.

    Each database is stored as it's own directory. Each table is stored under it's respective database, and columns are stored in their respective table.

    Each database is stored as its own directory. Each table is stored under its respective database, and columns are stored in their respective table.

    .. figure:: /_static/images/table_columns_storage.png

    @@ -63,27 +63,27 @@ Each column directory will contain extents, which are collections of chunks.
  
    .. figure:: /_static/images/chunks_and_extents.png

    ``metadata`` or ``leveldb``

    ----------------------------

    ``metadata`` or ``rocksdb``

    ---------------------------

    SQream DB's metadata is an embedded key-value store, based on LevelDB. LevelDB helps SQream DB ensure efficient storage for keys, handle atomic writes, snapshots, durability, and automatic recovery.

    SQreamDB's metadata is an embedded key-value store, based on RocksDB. RocksDB helps SQreamDB ensure efficient storage for keys, handle atomic writes, snapshots, durability, and automatic recovery.

    The metadata is where all database objects are stored, including roles, permissions, database and table structures, chunk mappings, and more.

    ``temp``

    ----------------

    --------

    The ``temp`` directory is where SQream DB writes temporary data.

    The ``temp`` directory is where SQreamDB writes temporary data.

    The directory to which SQream DB writes temporary data can be changed to any other directory on the filesystem. SQream recommends remapping this directory to a fast local storage to get better performance when executing intensive larger-than-RAM operations like sorting. SQream recommends an SSD or NVMe drive, in mirrored RAID 1 configuration.

    The directory to which SQreamDB writes temporary data can be changed to any other directory on the filesystem. SQreamDB recommends remapping this directory to a fast local storage to get better performance when executing intensive larger-than-RAM operations like sorting. SQreamDB recommends an SSD or NVMe drive, in mirrored RAID 1 configuration.

    If desired, the ``temp`` folder can be redirected to a local disk for improved performance, by setting the ``tempPath`` setting in the :ref:`configuration<configuration>` file.

    If desired, the ``temp`` folder can be redirected to a local disk for improved performance, by setting the ``tempPath`` setting in the :ref:`legacy configuration<current_method_modification_methods>` file.

    ``logs``

    ----------------

    --------

    The logs directory contains logs produced by SQream DB.

    The logs directory contains logs produced by SQreamDB.

    See more about the logs in the :ref:`logging` guide.

architecture/index.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,21 +1,18 @@
  
    .. _architecture:

    ***********************

    System Architecture

    ***********************

    ************

    Architecture

    ************

    This topic includes guides that walk an end-user, database administrator, or system architect through the main ideas behind SQream DB.

    The :ref:`internals_architecture`, :ref:`concurrency_and_scaling_in_sqream`, and :ref:`filesystem_and_filesystem_usage` guides are walk-throughs for end-users, database administrators, and system architects who wish to get familiarized with the SQreamDB system and its unique capabilities. 

    While SQream DB has many similarities to other database management systems, it has some unique and additional capabilities.

    Explore the guides below for information about SQream DB's architecture.

    .. figure:: /_static/images/sqream_db_table_crop.png

       :scale: 60 %

    .. toctree::

       :maxdepth: 2

       :caption: In this section:

       :glob:

       :titlesonly:

       :hidden:

       internals_architecture

       xxprocesses_and_network_architecture

       filesystem_and_filesystem_usage

       concurrency_and_scaling_in_sqream