[Init]Init project from 0.1.0 dev

.github/workflows/matchers/ruff.json

@@ -0,0 +1,17 @@
+{
+    "problemMatcher": [
+      {
+        "owner": "ruff",
+        "pattern": [
+          {
+            "regexp": "^(.+?):(\\d+):(\\d+): (\\w+): (.+)$",
+            "file": 1,
+            "line": 2,
+            "column": 3,
+            "code": 4,
+            "message": 5
+          }
+        ]
+      }
+    ]
+  }

.github/workflows/ruff.yaml

@@ -0,0 +1,31 @@
+name: ruff
+
+on:
+  pull_request:
+    branches:
+      - 'main'
+      - '*-dev'
+
+jobs:
+  ruff:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.12"]
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@8d9ed9ac5c53483de85588cdf95a591a75ab9f55 # v5.5.0
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-lint.txt
+      - name: Analysing the code with ruff
+        run: |
+          echo "::add-matcher::.github/workflows/matchers/ruff.json"
+          ruff check --output-format github .
+      - name: Run isort
+        run: |
+          isort . --check-only

.github/workflows/yapf.yaml

@@ -0,0 +1,31 @@
+name: yapf
+
+on:
+  pull_request:
+    branches:
+      - 'main'
+      - '*-dev'
+    paths:
+      - "**/*.py"
+      - .github/workflows/yapf.yml
+
+jobs:
+  yapf:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10"]
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@8d9ed9ac5c53483de85588cdf95a591a75ab9f55 # v5.5.0
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install toml
+          pip install yapf==0.32.0
+      - name: Running yapf
+        run: |
+          yapf --diff --recursive .

CHANGELOG.md

@@ -0,0 +1,19 @@
+# Release Note
+
+## v0.0.1 
+
+### Highlights
+- Initial release of the project.
+- Added core functionality for interacting with Elasticsearch indices.
+
+### Core
+- Implemented `search` command to query Elasticsearch indices.
+- Added `check` command to filter commit IDs based on existing records in Elasticsearch.
+- Introduced `delete` command to remove specific documents by ID from Elasticsearch.
+- Added support for filtering results by specific fields using the `--filter` option in the `search` command.
+
+### Other
+- Added basic logging functionality for better debugging and monitoring.
+- Included `CONTRIBUTING.md` to guide contributors.
+- Added `pytest` support for running unit tests.
+- Improved documentation for all commands and usage examples.

CONTRIBUTING.md

@@ -0,0 +1,50 @@
+# Contributing to This Project
+
+Thank you for your interest in contributing to this project! We welcome all contributions, including bug reports, feature requests, documentation improvements, and code contributions.
+
+## How to Contribute
+
+### 1. Reporting Issues
+If you encounter a bug or have a question, please open an issue on the [Issues](https://github.com/Potabk/elastic-tool/issues) page. When reporting an issue, please include:
+- A clear and detailed description of the problem.
+- Steps to reproduce the issue.
+- Expected and actual behavior.
+- Environment details (e.g., operating system, Python version, etc.).
+
+### 2. Requesting Features
+If you have a feature request, please open an issue on the [Issues](https://github.com/Potabk/elastic-tool/issues) page. Describe your idea clearly and explain why it would be useful.
+
+### 3. Improving Documentation
+Documentation is a critical part of this project. If you find any errors or areas for improvement, feel free to submit a Pull Request.
+
+### 4. Submitting Code
+If you'd like to contribute code, please follow these steps:
+1. Fork this repository.
+2. Create a new branch:
+```bash
+git checkout -b feature/your-feature-name
+```
+
+3. Make your changes and commit them：
+```
+git commit -m "Add your commit message"
+```
+4. Push your changes to your fork:
+```
+git push origin feature/your-feature-name
+```
+5. Open a Pull Request to the main repository.
+
+## Code Guidelines
+- Ensure your code is clean, readable, and adheres to the project's coding style.
+- Run all tests before submitting your code to ensure it works as expected.
+- If you add a new feature, update the relevant documentation and tests.
+
+## Community Guidelines
+Please adhere to our Code of Conduct. We strive to maintain a friendly and inclusive community.
+
+## Contact Us
+If you have any questions or suggestions, feel free to:
+- Open an issue
+- Email us at wangli858794774@gmail.com
+Thank you for contributing!

README.md

@@ -17,11 +17,24 @@ pip install -e .
 
 ## Quick Start
 
-✅ Check if CLI is available
+### Check if CLI is available
 
 ```
 escli --help
 ```
+### Add domain and token
+A domain and token is necessary to interact with es, you can follow the next step to ensure elastic is accessible.
+There are currently two ways to add
+- Environment variables:
+set `ES_OM_DOMAIN` and `ES_OM_AUTHORIZATION` to have a login
+
+- Keyring:
+ Login through the command line, we will automatically store it in the operating system's keyring, In comparison, this method is safer and protects the secrets from being leaked.
+ The login command looks like:
+```
+escli login --domain "your domain" --token "your token"
+```
+
 
 ## Features
 Create and delete Elasticsearch indexes
@@ -32,4 +45,6 @@ Perform search queries (DSL or keyword-based)
 
 Support both inline JSON and file-based input
 
-Environment config via ES_OM_DOMAIN and ES_OM_AUTHORIZATION env vars
+Environment config via ES_OM_DOMAIN and ES_OM_AUTHORIZATION env vars
+
+Check and filter _doc not existed in es

escli_tool/__init__.py

@@ -0,0 +1,7 @@
+import importlib
+import pkgutil
+
+import escli_tool.processor
+
+for _, module_name, _ in pkgutil.iter_modules(escli_tool.processor.__path__):
+    importlib.import_module(f"escli_tool.processor.{module_name}")

escli_tool/cli.py

@@ -1,17 +1,25 @@
 # escli_tool/cli.py
 
 import argparse
-from escli_tool.commands import search, create, delete, update, login
+
+from escli_tool.commands import add, check, create, delete, login, search
+from escli_tool.utils import get_logger
+
+logger = get_logger()
+
 
 def main():
-    parser = argparse.ArgumentParser(prog="escli", description="Elastic CLI 工具")
+    parser = argparse.ArgumentParser(prog="escli",
+                                     description="Elastic CLI 工具")
     subparsers = parser.add_subparsers(dest="command")
 
     # register subcommand
     create.register_subcommand(subparsers)
     search.register_subcommand(subparsers)
     login.register_subcommand(subparsers)
-
+    add.register_subcommand(subparsers)
+    delete.register_subcommand(subparsers)
+    check.register_subcommand(subparsers)
 
     args = parser.parse_args()
 

escli_tool/commands/add.py

@@ -1,31 +1,44 @@
 # escli_tool/commands/create.py
-import json
-import os
-
-from escli_tool.handler import DataHandler
+from escli_tool.registry import get_class
 
 
 def register_subcommand(subparsers):
-    parser = subparsers.add_parser("add", help="创建索引或插入文档")
-    parser.add_argument("--index", required=True, help="索引名称")
-    parser.add_argument("--file_name", help="创建索引的字段映射(json 字符串或文件)")
-    #parser.add_argument("--all", action="store_true", help="是否选择结果路径中的全部文件")
+    parser = subparsers.add_parser("add",
+                                   help="Insert a new _id in the given index")
+    parser.add_argument("--tag", default=None, help="Which version to save")
+    parser.add_argument("--res_dir",
+                        help="Result dir which include json files")
+    parser.add_argument("--processor",
+                        help="Processor selected to process json files")
+    parser.add_argument("--commit_id", help="Commit hash")
+    parser.add_argument("--commit_title", help="Commit massage")
+    parser.add_argument("--model_name", help="Model test on")
+    parser.add_argument("--created_at",
+                        help="What time current commit is submitted")
     parser.set_defaults(func=run)
 
-def run(args):
-    handler = DataHandler()
-    handler.index_name = args.index
-
-    if args.mapping:
-        mapping = _load_json(args.mapping)
-        handler.add_single_data(mapping)
 
-    if args.doc:
-        doc = _load_json(args.doc)
-        handler.insert_document(doc)
+def run(args):
+    """
+    Insert a document loading from local dir, need to provide a processor to process the specific data.
+    For example, if you want to insert performance benchmark result(which saved as json files), you need
+    to provide a benchmark processor to process the json files. and the processor should process the data
+    into a data format that es can accept.
+    If the processor is not provided, the default processor will be used.
+    """
+    processor_name = args.processor
+    if not processor_name:
+        # Set default processor to benchmark
+        processor_name = 'benchmark'
+    # TODO: do not only read data from local dir, but also read dict user customized
+    if not args.res_dir:
+        raise ValueError("Result dir is required")
 
-def _load_json(content):
-    if os.path.exists(content):
-        with open(content) as f:
-            return json.load(f)
-    return json.loads(content)
+    processor = get_class(processor_name)(
+        args.commit_id,
+        args.commit_title,
+        args.created_at,
+        args.model_name,
+        args.tag,
+    )
+    processor.send_to_es(args.res_dir, )