Connecting notebooks to different kernels?

[Montana]

Hey folks,

The current issue at hand I’m wrestling with is memory, although I’d like to get more into parallelizing my code and (e2e testing). To be honest, I don’t need the scale most of the time, but there are times (maybe 10-20%) when it would be helpful. That’s why I’ve been wanting to find some sort of EC2 on demand set up that meshes relatively seamlessly with my development environment.

As of now, I can connect different notebooks to different kernels or conda environments. I’ve been intrigued by the cloudJHub implementation, that can kind of scale, and I've also looked into dask and that seems like it could work, but cloudJHub seems like the more proper solution, when switching instances the script I could write would read something like this:

#!/bin/bash
alias s3='aws s3'
alias ec2='aws ec2'

alias instance=awsInstance
function awsInstance(){
  aws ec2 describe-instances --instance-ids $1 --output table
}

alias instances=awsInstances
function awsInstances(){
  aws ec2 describe-instances --query 'Reservations[].Instances[].[ [Tags[?Key==`Name`].Value][0][0],[Tags[?Key==`Environment`].Value][0][0],PrivateIpAddress,InstanceId,State.Name,Placement.AvailabilityZone, SecurityGroups.GroupName ]' --output table
}

alias ami=awsAMI
function awsAMI(){
  aws ec2 describe-images --owner self --query 'Images[].[ImageId, Name, VirtualizationType]' --output table
}

Another thought I had was, say I select the node, in theory I could make the instance type to make it bigger if I stop the machine. My question is do you think cloudJHub would be a good substitute for say something like dask in someways?

Thank you,
Montana Mendy

[farassadek]

Hi Montana,
Frankly I do not have enough knowledge about how desk work to give you an answer. But I will be more than happy to explain the technical details of cloudJHub if you like.
Faras