[Feature] Add HuaweiCloud provider

[kiwik]

If CloudTik can support huaweicloud provider, it's great. We would like using CloudTik to launch Spark/ML Cluster with OAP enhancement on HuaweiCloud, and I can help to implement it.

https://www.huaweicloud.com/

[jerrychenhf]

Hi @kiwik, it's great that you can contribute to CloudTik.
If you want to implement support for HuaweiCloud, you need to implement:

1. the NodeProvider: Implement the functions to creating or terminating nodes with tags.
2. the WorkspaceProvider: Implement the workspace resources provisioning (the VPN, identities and roles, cloud storage) given a workspace name.

You can submit a PR if you have some code.

[kiwik]

@jerrychenhf thank you reminding the key message, I just start to read CloudTik document in these days, don't design and coding yet. I will discuss with my colleagues about it, maybe commit design in following months.

Our team is intersting in Big Data and AI technoloy and enhancement, we can build cooperation with oap-project team in severl open source projects.

[kiwik]

Task list:

• Huaweicloud credentials
  • Update workspace-schema.json
  • Check login and testing
• Update WorkspaceProvider
  • Huaweicloud ECS Client
  • VPC API
  • Subnet API
  • ...
• Update NodeProvider
  • Create Node
  • Terminate Node
  • Update XXX API
  • Info API
• Templates and examples

[jerrychenhf]

@kiwik Great to hear that!

As to the tasks, it's exactly the right sequence. You can start with implement the workspace create, delete and status which implement the VPC design (VPC and subnets, firewalls, ... with or without public IP, with VPC peering or not), identity and roles for instance authentication and authorization, and managed cloud storage for workspace.

Once the workspace is implemented, it's ready to implement the Node Provider to create or delete instances, tagging, get instance information and so on.

For Spark (and other workload) to access the cloud storage, there some lightweight implementation in Runtime configuration steps. But this can leave to the last step.

CloudTik aslo support K8S provider with integration with Cloud (mostly related OIDC provider integration for identity and roles). If Huawei Cloud have a K8S engine to integrate with Huawei cloud resources, a integration layer can be developed for Cloud Kubernetes Provider.

[kiwik]

Thank you @jerrychenhf to append so many details, it help to make whole workflow clear.

On high level plan, I have started to implement HuaweiCloud ECS provider first, that based on virtual machine, and HuaweiCloud support K8S engine too, named CCE service, I will implement CCE provider after ECS provider is ready.

I will commit the workspace related functions for ECS provider in this week. I plan to split whole HuaweiCloud provider code to a series of patch sets that foucs on a certain feature, like: workspace, node provider and so on. Hopefully, this small-scale PR will make code reviewing a little easier.

[jerrychenhf]

@kiwik The plan looks great! The workspace functions has been committed.
The next step will be the node provider implementation which is the core for bring the cluster up and running at Huawei Cloud.

While you are implementing, I will be able to help in some aspects when I free up. A few notes for helping your implementation of this part:

1. The key connection between the Workspace and cluster (node provider) is the bootstrap process. node_provider.bootstrap_config. The bootstrap_config process will configure all the necessary parameters from the workspace and cluster options into node_config for each node types (head and worker). An idea design of the parameters under the node_config is the identical config_map that is acceptable to Huawei Cloud run_instances API (or other similar name). the bootstrap information includes:
   a. Set the right IAM role based on workspace for the instance (head or worker) to node_config
   b. Generate and create the key pair for the cluster if user doesn't specify one and set to the key pair name to node_config
   c. Set the managed cloud storage information of the workspace to cluster config under storage section.
   d. Set the VPC and subnets information of the workspace to node_config
   e. Set the security group id of the workspace to node_config for creating instances.
   f. Set the latest ubuntu_20_04 image as the image id to use to node_config if user doesn't specify one.
   h. Set the spot preference based on the common option in provider to node_config if user doesn't override in node_config
   You can refer to the existing provider bootstrap implementation and modify.
2. The node_provider implementation. Assume that node_config has almost every options that needed to create instance. Add tag handling. You can start with the existing provider and modify. It will be straight forward.

Look forward to your next patch for this. I can help to start with the Spark runtime and fuse support for Huawei cloud storage.

One best practice is to follow an existing node provider (the most close one to HuaweiCloud API). Avoid changes without specific reason so that the new implementation will more likely not breaking any assumptions.

[jerrychenhf]

@kiwik I merged the code for Huawei Cloud Hadoop to integrate obs storage. #1138

A few issues found in Hadoop Huawei project (https://github.com/huaweicloud/obsa-hdfs)
huaweicloud/obsa-hdfs#14

I made some modifications to the source code and compiled Hadoop 3.3.1 version of hadoop-huawei jar.
Maybe you can push huawei team to fix the issue in the repo.

[jerrychenhf]

@kiwik I committed a PR for Huawei cloud node provider to integrate work runtime storage configurations. #1139

Please have a check. You need to implement the remaining node provider methods for instance operations. (See TODO in node_provider and TODO in bootstrap_huaweicloud)

[jerrychenhf]

@kiwik #1140 PR made some improvements to the make_xxx_client so that the credential handling logic is shared and can be improved more easily in the future.
One additional note:
I add optional region if the region for creating client with region other than the region in the provider config. This is the case for _get_current_vpc call of the working node (instance) because the working node region may be different than the region set in the provider (workspace region) when VPC peering is used. So the working node vpc is in the region different than than region of the workspace VPC (this is allowed) and Peering is established between the two VPC in different region (Huawei Cloud allow this? I think so).

So the following code:

`
def _get_current_vpc(config, vpc_client=None):

vm_loca_ip_url = HWC_VM_METADATA_URL + 'local-ipv4'
response = requests.get(vm_loca_ip_url)
vm_local_ip = response.text
ecs_client = make_ecs_client(config)
response = ecs_client.list_servers_details(
    ListServersDetailsRequest(ip=vm_local_ip))

`

ecs_client = make_ecs_client(config) shoud be ->

ecs_client = make_ecs_client(config, region_of_working_node)

[kiwik]

I ping Huawei Cloud obsa-hdfs maintainer, hope him can help. ^

And so nice to you to help add framework of Node provider, it's a entry point, let me starting to solve out whole code path.
I will continue to implement the PR, after rebase on you commits, thank you very much.

[jerrychenhf]

@kiwik It's great!
I was checking fuse mount feature for a forward looking thing.
I found obsfs (https://support.huaweicloud.com/fstg-obs/obs_12_0001.html). It mentioned "obsfs只支持挂载OBS并行文件系统，不支持挂载对象存储桶。", what does this mean? Can we use it use the current managed OBS bucket you created in CloudTik?

mount cloud fs to local is very useful feature especially for ML/DL cases.

Additionally, obsfs seemed to be quite old and doesn't mention to support newer Linux versions such as 20.04 (we use).

[kiwik]

So the working node vpc is in the region different than than region of the workspace VPC (this is allowed) and Peering is established between the two VPC in different region (Huawei Cloud allow this? I think so).

It's different in Huawei Cloud, for the concept VPC Peering in Huawei Cloud only support to connect VPCs in same region, and concept Cloud Connect for VPCs cross regions, see following refer. They apply different API and SDK, so I perfer to keep it simple for CloudTik in the first Huawei Cloud support release, support VPC Peering in same region right now, then support Cloud Connect cross region in the future, maybe we can add some describe and limitation into CloudTik document for Huawei Cloud provider, something like Huawei Cloud only support VPC Peering between different VPCs in the same region. What do you think?

VPC Peering: https://support.huaweicloud.com/usermanual-vpc/zh-cn_topic_0046655036.html
Cloud Connect: https://support.huaweicloud.com/function-cc/index.html