A SurrealDB powered data format swiss army knife.
Explore the SurrealQL docs »

Report Bug · Request Feature

Table of Contents

1. About
   • Built With
2. Getting Started
   • Prerequisites
   • Build
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About

SQX is a WIP tool to query, aggregate, filter, and convert structured data formats on disk. Leveraging SurrealDB's flexible in memory database allows the user to easily express complicated queries on current-gen datasets containing heavily nested and relational data.

(back to top)

Built With

(back to top)

Getting Started

Linux binaries can be found on the Actions Page.

Prerequisites

You'll need git, cargo and rustup.

https://rustup.rs/

Build

1. Clone the repo

   git clone https://github.com/poisson-fish/sqx.git
   cd sqx

2. Build release

   cargo build --release

3. Copy sqx binary to your preferred runpath.

(back to top)

Usage

(WIP)

• SQX is in infant stages and functionality is limited (for now!!)
• SQX can ingest only JSON via stdin or file blob, more formats to come (CSV, TSV, Apache Arrow via Pola.rs) .
• SQX only outputs in visual table format, both input and output format flags do nothing.
• SQX takes quoted SurrealQL queries with the -s flag.

To query on stdin use the 'stdin' table:

ps auxw | jc --ps | ./sqx -s "SELECT command,mem_percent FROM stdin ORDER BY mem_percent DESC LIMIT 10;"

Gives:

╭──────────────────────────────────┬─────────────╮
│ command                          │ mem_percent │
├──────────────────────────────────┼─────────────┤
│ "/home/twin/.vscode-server/exten │ "2.6"       │
│ "/home/twin/.vscode-server/bin/9 │ "0.2"       │
│ "/home/twin/.vscode-server/bin/9 │ "0.1"       │
│ "/home/twin/.vscode-server/bin/9 │ "0.1"       │
│ "/home/twin/.vscode-server/bin/9 │ "0.1"       │
│ "./sqx -s SELECT command,mem_per │ "0.0"       │
│ "/usr/lib/systemd/systemd-journa │ "0.0"       │
│ "/usr/bin/dbus-daemon --system - │ "0.0"       │
│ "/home/twin/.vscode-server/bin/9 │ "0.0"       │
│ "/usr/bin/zsh -i"                │ "0.0"       │
╰──────────────────────────────────┴─────────────╯

To query on file input you use the 'filein' table. Specify a space separated list of file blobs with an options postfixed double dash:

./sqx -vv -s "SELECT * FROM filein;" -- testdata/* 

For more specific and powerful query language examples, please refer to the SurrealQL Documentation

(back to top)

Roadmap

• Stdin input
• Blob filepath input
• Multiformat ingest support
  • JSON
  • TSV
  • CSV
  • Arrow
  • Raw posix tool output parsers
• Multiformat export support
  • JSON
  • TSV
  • CSV
  • Arrow
  • SQL
  • Tabled
• Tables
  • Basic table view
  • Optional
• Tests

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

twin - hyperviridian@gmail.com

Project Link: https://github.com/poisson-fish/sqx

(back to top)

Acknowledgments

Big credit to SurrealDB for building the next generation database in Rust!

(back to top)