Organize datasets

[aousterh]

Create an EBS volume to hold all sample datasets and their transformations:

• zq-sample-data (link → z-dataset-sample-data volume)
• zeek dataset (link: shared privately → z-dataset-zeek)
• suricata dataset (link: shared privately → z-dataset-suricata)
• baseball dataset (link → z-dataset-baseball); format lineage:
  • csv → zng → ndjson, zst
  • csv → parquet
  • elastic

#HowTos:

Volume creation:

• Create EBS volume on EC2 console
• Pick SSD gp2
• Pick a size based on the dataset size/planned transformations/...
• Availability zone: us-west-2a
• Do not encrypt the volume
• Add a name to the volume (in the Volumes page): z-dataset-[DATASET_NAME]

Use the volume:

• Attach the volume to your instance
• Run lsblk on the instance to identify the device
• Create a directory on the instance (e.g., /data) and mount the device to it;
  (More details can be found on EBS doc)

Note: remember to umount the volume (if it is mounted somewhere) before detach on the EC2 console!

Done!

[autumnli11]

todo: wget all the zeeks files but error: 404 not found