Skip to content

cloud-e2e-security.md

Latest commit

 

History

History
478 lines (383 loc) · 31.3 KB

cloud-e2e-security.md

File metadata and controls

478 lines (383 loc) · 31.3 KB
subcollection solution-tutorials
copyright
years
2018-2021
lastupdated 2021-01-07
lasttested 2020-12-07
content-type tutorial
services containers, cloud-object-storage, Activity-Tracker-with-LogDNA, Registry, certificate-manager, appid, Cloudant, key-protect, Log-Analysis-with-LogDNA
account-plan paid
completion-time 2h

{:step: data-tutorial-type='step'} {:shortdesc: .shortdesc} {:new_window: target="_blank"} {:codeblock: .codeblock} {:screen: .screen} {:tip: .tip} {:pre: .pre}

Apply end to end security to a cloud application

{: #cloud-e2e-security} {: toc-content-type="tutorial"} {: toc-services="containers, cloud-object-storage, Activity-Tracker-with-LogDNA, Registry, certificate-manager, appid, Cloudant, key-protect, Log-Analysis-with-LogDNA"} {: toc-completion-time="2h"}

This tutorial may incur costs. Use the Cost Estimator to generate a cost estimate based on your projected usage. {: tip}

This tutorial walks you through key security services available in the {{site.data.keyword.cloud}} catalog and how to use them together. An application that provides file sharing will put security concepts into practice. {:shortdesc}

No application architecture is complete without a clear understanding of potential security risks and how to protect against such threats. Application data is a critical resource which can not be lost, compromised or stolen. Additionally, data should be protected at rest and in transit through encryption techniques. Encrypting data at rest protects information from disclosure even when it is lost or stolen. Encrypting data in transit (e.g. over the Internet) through methods such as HTTPS, SSL, and TLS prevents eavesdropping and so called man-in-the-middle attacks.

Authenticating and authorizing users' access to specific resources is another common requirement for many applications. Different authentication schemes may need to be supported: customers and suppliers using social identities, partners from cloud-hosted directories, and employees from an organization’s identity provider.

Objectives

{: #cloud-e2e-security-objectives}

  • Encrypt content in storage buckets with your own encryption keys
  • Require users to authenticate before accessing an application
  • Monitor and audit security-related API calls and other actions across cloud services

The tutorial features a sample application that enables groups of users to upload files to a common storage pool and to provides access to those files via shareable links. The application is written in Node.js and deployed as a container to the {{site.data.keyword.containershort_notm}}. It leverages several security-related services and features to improve the application's security posture.

This tutorial will work with a Kubernetes cluster running in Classic Infrastructure or VPC Infrastructure.

Architecture

  1. User connects to the application.
  2. If using a custom domain and a TLS certificate, the certificate is managed by and deployed from the {{site.data.keyword.cloudcerts_short}}.
  3. {{site.data.keyword.appid_short}} secures the application and redirects the user to the authentication page. Users can also sign up.
  4. The application runs in a Kubernetes cluster from an image stored in the {{site.data.keyword.registryshort_notm}}. This image is automatically scanned for vulnerabilities.
  5. Uploaded files are stored in {{site.data.keyword.cos_short}} with accompanying metadata stored in {{site.data.keyword.cloudant_short_notm}}.
  6. File storage buckets leverage a user-provided key to encrypt data.
  7. Application management activities are logged by {{site.data.keyword.at_full_notm}}.

Before you begin

{: #cloud-e2e-security-prereqs}

This tutorial requires:

  • {{site.data.keyword.cloud_notm}} CLI,
    • {{site.data.keyword.containerfull_notm}} plugin (kubernetes-service),
    • {{site.data.keyword.registryshort_notm}} plugin (container-registry),
  • kubectl to interact with Kubernetes clusters,
  • git to clone source code repository.

You will find instructions to download and install these tools for your operating environment in the Getting started with tutorials guide.

Note: To avoid the installation of these tools you can use the {{site.data.keyword.cloud-shell_short}} from the {{site.data.keyword.cloud_notm}} console.

Create services

{: #cloud-e2e-security-setup} {: step}

In the next section, you are going to create the services used by the application.

Decide where to deploy the application

{: #cloud-e2e-security-4}

  1. Identify the location and resource group where you will deploy the application and its resources.

Capture user and application activities

{: #cloud-e2e-security-activity-tracker }

The {{site.data.keyword.at_full_notm}} service records user-initiated activities that change the state of a service in {{site.data.keyword.Bluemix_notm}}. At the end of this tutorial, you will review the events that were generated by completing the tutorial's steps.

  1. Access the {{site.data.keyword.cloud_notm}} catalog and create an instance of {{site.data.keyword.at_full_notm}}. Note that there can only be one instance of {{site.data.keyword.at_short}} per region. Set the Service name to secure-file-storage-activity-tracker.
  2. Ensure you have the right permissions assigned to manage the service instance by following these instructions.

Create a cluster for the application

{: #cloud-e2e-security-6}

{{site.data.keyword.containershort_notm}} provides an environment to deploy highly available apps in containers that run in Kubernetes clusters.

Skip this section if you have an existing cluster you want to reuse with this tutorial, throughout the remainder of this tutorial the cluster name is referenced as secure-file-storage-cluster, simply substitute with the name of your cluster. Note the minimum required Kubernetes version of 1.19. {: tip}

A minimal cluster with one (1) zone, one (1) worker node and the smallest available size (Flavor) is sufficient for this tutorial. A minimum Kubernetes version of 1.19 is required. Make sure to select an appropriate version when creating the cluster.

  • Set the cluster name to secure-file-storage-cluster.
  • For Kubernetes on VPC infrastructure, you are required to create a VPC and subnet(s) prior to creating the Kubernetes cluster. You may follow the instructions provided under the Creating a standard VPC Gen 2 compute cluster in the console.
    • Make sure to attach a Public Gateway for each of the subnets that you create as it is required for App ID.
  • For Kubernetes on Classic infrastructure follow the Creating a standard classic cluster instructions.

While the cluster is being provisioned, you will create the other services required by the tutorial.

Use your own encryption keys

{: #cloud-e2e-security-7}

{{site.data.keyword.keymanagementserviceshort}} helps you provision encrypted keys for apps across {{site.data.keyword.Bluemix_notm}} services. {{site.data.keyword.keymanagementserviceshort}} and {{site.data.keyword.cos_full_notm}} work together to protect your data at rest. In this section, you will create one root key for the storage bucket.

  1. Create an instance of {{site.data.keyword.keymanagementserviceshort}}.
    • Set the name to secure-file-storage-kp.
    • Select the resource group where to create the service instance.
  2. Under Manage, click the Add Key button to create a new root key. It will be used to encrypt the storage bucket content.
    • Set the name to secure-file-storage-root-enckey.
    • Set the key type to Root key.
    • Then Create key.

Bring your own key (BYOK) by importing an existing root key. {: tip}

Setup storage for user files

{: #cloud-e2e-security-8}

The file sharing application saves files to a {{site.data.keyword.cos_short}} bucket. The relationship between files and users is stored as metadata in a {{site.data.keyword.cloudant_short_notm}} database. In this section, you'll create and configure these services.

A bucket for the content

{: #cloud-e2e-security-9}

  1. Create an instance of {{site.data.keyword.cos_short}}.
    • Set the name to secure-file-storage-cos.
    • Use the same resource group as for the previous services.
  2. Under Service credentials, create a New credential.
    • Set the name to secure-file-storage-cos-acckey.
    • Set Role to Writer.
    • Under Advanced options, check Include HMAC Credential. This is required to generate pre-signed URLs.
    • Click Add.
    • Make note of the credentials. You will need them in a later step.
  3. Click Endpoint from the menu: set Resiliency to Regional and set the Location to the target location:
    • Classic infrastructure: Copy the Private service endpoint. It will be used later in the configuration of the application.
    • VPC infrastructure: Copy the Direct service endpoint. It will be used later in the configuration of the application.

Before creating the bucket, you will grant the {{site.data.keyword.cos_short}} service instance access to the root key stored in the {{site.data.keyword.keymanagementserviceshort}} service instance.

  1. Go to Manage > Access (IAM) > Authorizations in the {{site.data.keyword.cloud_notm}} console.
  2. Click the Create button.
  3. In the Source service menu, select Cloud Object Storage.
  4. In the Source service instance menu, select the {{site.data.keyword.cos_short}} service instance previously created.
  5. In the Target service menu, select Key Protect.
  6. In the Target service instance menu, select the {{site.data.keyword.keymanagementserviceshort}} service instance created earlier.
  7. Enable the Reader role.
  8. Click the Authorize button.

Finally create the bucket.

  1. Access the {{site.data.keyword.cos_short}} service instance from the Resource List.
  2. Click Create bucket and then Custom bucket.
    1. Set the name to a unique value, such as <your-initials>-secure-file-upload.
    2. Set Resiliency to Regional.
    3. Set Location to the same location where you created the {{site.data.keyword.keymanagementserviceshort}} service instance.
    4. Set Storage class to Standard
  3. Under Key Management Services, select the checkbox Key Protect.
    1. Select the {{site.data.keyword.keymanagementserviceshort}} service instance created earlier.
    2. Select secure-file-storage-root-enckey as the key.
  4. Enable {{site.data.keyword.at_short}} events to be recorded under Additional Services.
    1. After clicking the checkmark the service information for the previously created {{site.data.keyword.at_short}} instance should be shown.
    2. Now, enable Track Data events and select read & write as Data Events.
  5. Click Create bucket.

A database map relationships between users and their files

{: #cloud-e2e-security-10}

The {{site.data.keyword.cloudant_short_notm}} database will contain metadata for all files uploaded from the application.

  1. Create an instance of {{site.data.keyword.cloudant_short_notm}}.
    • Set the name to secure-file-storage-cloudant.
    • Set the location.
    • Use the same resource group as for the previous services.
    • Set Available authentication methods to Use only IAM.
    • Click Create.
  2. Back to the Resource List, locate the newly created service and click on it. (Note: You will need to wait until the status changes to Provisioned)
    • Under Service credentials, create New credential.
    • Set the name to secure-file-storage-cloudant-acckey.
    • Set Role to Manager.
    • Keep the default values for the the remaining fields.
    • Add.
  3. Expand the newly created credentials and make note of the values. You will need them in a later step.
  4. Under Manage, launch the Cloudant dashboard.
  5. Click Create Database to create a Non-partitioned database named secure-file-storage-metadata.

{{site.data.keyword.cloudant_short_notm}} instances on dedicated hardware allow private endpoints. Instances with dedicated service plans allow to define a list of allowed IP addresses. See {{site.data.keyword.cloudant_short_notm}} Secure access control for details. {: tip}

Authenticate users

{: #cloud-e2e-security-11}

With {{site.data.keyword.appid_short}}, you can secure resources and add authentication to your applications. {{site.data.keyword.appid_short}} integrates with {{site.data.keyword.containershort_notm}} to authenticate users accessing applications deployed in the cluster.

  1. Create an instance of {{site.data.keyword.appid_short}}.
    • Select the Graduated tier as plan.
    • Set the Service name to secure-file-storage-appid.
    • Use the same location and resource group as for the previous services.
  2. Under Manage Authentication, in the Authentication Settings tab, add a web redirect URL pointing to the domain you will use for the application. For example, if your cluster Ingress subdomain is mycluster-1234-d123456789.us-south.containers.appdomain.cloud, the redirect URL will be https://secure-file-storage.mycluster-1234-d123456789.us-south.containers.appdomain.cloud/oauth2-secure-file-storage-appid/callback. {{site.data.keyword.appid_short}} requires the web redirect URL to be https. You can view your Ingress subdomain in the cluster dashboard or with ibmcloud ks cluster get --cluster <cluster-name>.
  3. In the same tab under Authentication Settings under Runtime Activity enable capturing events in {{site.data.keyword.at_short}}.

You should customize the identity providers used as well as the login and user management experience in the {{site.data.keyword.appid_short}} dashboard. This tutorial uses the defaults for simplicity. For a production environment, consider to use Multi-Factor Authentication (MFA) and advanced password rules. {: tip}

Deploy the app

{: #cloud-e2e-security-deploy} {: step}

All services have been configured. In this section you will deploy the tutorial application to the cluster.

Get the code

{: #cloud-e2e-security-13}

  1. Get the application's code: 
    git clone https://github.com/IBM-Cloud/secure-file-storage
    {: codeblock}
  2. Go to the secure-file-storage directory: 
    cd secure-file-storage
    {: codeblock}

Build the container image

{: #cloud-e2e-security-14}

To build the container image in {{site.data.keyword.registryshort_notm}}:

  1. Identify your {{site.data.keyword.registryshort_notm}} URL, such as us.icr.io or uk.icr.io: 
    ibmcloud cr region
    {:pre}
  2. Pick one of your existing registry namespaces or create a new one. To list existing namespaces, use: 
    ibmcloud cr namespaces
    {:pre} To create a new namespace: 
    ibmcloud cr namespace-add <your-namespace>
    {:pre}
  3. Build the image with a unique name such as secure-file-storage : 
    ibmcloud cr build -t <your-registry-url>/<your-namespace>/<your-image-name>:latest .
    {: codeblock}

Fill in credentials and configuration settings

{: #cloud-e2e-security-15}

  1. Copy credentials.template.env to credentials.env: 
    cp credentials.template.env credentials.env
    {: codeblock}
  2. Edit credentials.env and fill in the blanks with these values:
    • the {{site.data.keyword.cos_short}} service regional endpoint, the bucket name, the credentials created for the {{site.data.keyword.cos_short}} service,
    • and the credentials for secure-file-storage-cloudant.
  3. Copy secure-file-storage.template.yaml to secure-file-storage.yaml: 
    cp secure-file-storage.template.yaml secure-file-storage.yaml
    {: codeblock}
  4. Edit secure-file-storage.yaml and replace the placeholders ($IMAGE_PULL_SECRET, $IMAGE_REPOSITORY, $TARGET_NAMESPACE, $INGRESS_SUBDOMAIN, $INGRESS_SECRET, $BASENAME) with the correct values. As example, assuming the application is deployed to the default Kubernetes namespace:
Variable Value Description
$IMAGE_PULL_SECRET Keep the lines commented in the .yaml A secret to access the registry.
$IMAGE_REPOSITORY us.icr.io/namespace/image-name The URL-like identifier for the built image based on the registry URL, namespace and image name from the previous section.
$TARGET_NAMESPACE default the Kubernetes namespace where the app will be pushed.
$INGRESS_SUBDOMAIN mycluster-1234-d123456789.us-south.containers.appdomain.cloud Retrieve from the cluster overview page or with ibmcloud ks cluster get --cluster <your-cluster-name>.
$INGRESS_SECRET secure-file-stora-123456 Retrieve from the cluster overview page or with ibmcloud ks cluster get --cluster <your-cluster-name>.
$BASENAME secure-file-storage The prefix used to identify resources.

$IMAGE_PULL_SECRET is only needed if you want to use another Kubernetes namespace than the default one. This requires additional Kubernetes configuration (e.g. creating a container registry secret in the new namespace). {: tip}

Deploy to the cluster

{: #cloud-e2e-security-16}

  1. Gain access to your cluster as described on the Access tab of your cluster.
  2. If not present, enable the ALB OAuth Proxy add-on in your cluster. 
    ibmcloud ks cluster addon enable alb-oauth-proxy --cluster <your-cluster-name>
    {: codeblock} You can check for existing add-ons with this command: 
       ibmcloud ks cluster addon ls --cluster <your-cluster-name>
    {: codeblock}
  3. Only if deploying to a non-default namespace, ensure that the Ingress secret is available in that namespace. First, get the CRN of the Ingress secret for your custom domain or default Ingress subdomain. It should be named similar to your cluster. 
    ibmcloud ks ingress secret ls -c <your-cluster-name>
    {: codeblock}
    Now use its name and CRN to create a secret in the namespace: 
    ibmcloud ks ingress secret create -c <your-cluster-name> -n <your-namespace> --cert-crn <crn-shown-in-the-output-above> --name <secret-name-shown-above>
    {: codeblock}
  4. Create the secret used by the application to obtain service credentials: 
    kubectl create secret generic secure-file-storage-credentials --from-env-file=credentials.env
    {: codeblock}
  5. Bind the {{site.data.keyword.appid_short_notm}} service instance to the cluster. If you have several services with the same name the command will fail. You should pass the service GUID instead of its name. To find the GUID of a service, use ibmcloud resource service-instance <service-name>. Replace default namespace if using a different namespace. 
    ibmcloud ks cluster service bind --cluster <your-cluster-name> --namespace default --service secure-file-storage-appid
    {: codeblock}
  6. Deploy the app. 
    kubectl apply -f secure-file-storage.yaml
    {: codeblock}

Test the application

{: #cloud-e2e-security-5} {: step}

The application can be accessed at https://secure-file-storage.<your-cluster-ingress-subdomain>/.

  1. Go to the application's home page. You will be redirected to the {{site.data.keyword.appid_short_notm}} default login page.
  2. Sign up for a new account with a valid email address.
  3. Wait for the email in your inbox to verify the account.
  4. Login.
  5. Choose a file to upload. Click Upload.
  6. Use the Share action on a file to generate a pre-signed URL that can be shared with others to access the file. The link is set to expire after 5 minutes.

Authenticated users have their own spaces to store files. While they can not see each other files, they can generate pre-signed URLs to grant temporary access to a specific file.

You can find more details about the application in the source code repository.

Review Security Events

{: #cloud-e2e-security-18} {: step}

Now that the application and its services have been successfully deployed, you can review the security events generated by that process. All the events are centrally available in {{site.data.keyword.at_short}} instance.

  1. From the Observability dashboard, locate the {{site.data.keyword.at_short}} instance for the region where your application is deployed and click View LogDNA.
  2. Review all logs sent to the service as you were provisioning and interacting with resources.

Optional: Use a custom domain and encrypt network traffic

{: #cloud-e2e-security-19} {: step} By default, the application is accessible on a generic hostname at a subdomain of containers.appdomain.cloud. However, it is also possible to use a custom domain with the deployed app. For continued support of https, access with encrypted network traffic, either a certificate for the desired hostname or a wildcard certificate needs to be provided. In the following section, you will either upload an existing certificate or order a new certificate in the {{site.data.keyword.cloudcerts_short}} and deploy it to the cluster. You will also update the app configuration to use the custom domain.

For secured connection, you can either obtain a certificate from Let's Encrypt as described in the following {{site.data.keyword.cloud}} blog or through {{site.data.keyword.cloudcerts_long}}. {: tip}

  1. Create an instance of {{site.data.keyword.cloudcerts_short}}
    • Set the name to secure-file-storage-certmgr.
    • Use the same location and resource group as for the other services.
  2. Click on Import Certificate to import your existing certificate.
    • Set name to SecFileStorage and description to Certificate for e2e security tutorial.
    • Upload the certificate file using the Browse button.
    • Click Import to complete the import process.
  3. Locate the entry for the imported certificate and expand it
    • Verify the certificate entry, e.g., that the domain name matches your custom domain. If you uploaded a wildcard certificate, an asterisk is included in the domain name.
    • Click the copy symbol next to the certificate's crn.
  4. Switch to the command line to deploy the certificate information as a secret to the cluster. Execute the following command after copying in the crn from the previous step. 
    ibmcloud ks alb cert deploy --secret-name secure-file-storage-certificate --cluster <your-cluster-name> --cert-crn <the copied crn from previous step>
    {: codeblock} Verify that the cluster knows about the certificate by executing the following command. 
    ibmcloud ks alb certs --cluster <your-cluster-name>
    {: codeblock}
  5. Edit the file secure-file-storage.yaml.
    • Find the section for Ingress.
    • Uncomment and edit the lines covering custom domains and fill in your domain and host name. The CNAME entry for your custom domain needs to point to the cluster. Check this guide on mapping custom domains in the documentation for details. {: tip}
  6. Apply the configuration changes to the deployed: 
    kubectl apply -f secure-file-storage.yaml
    {: codeblock}
  7. Switch back to the browser. In the {{site.data.keyword.Bluemix_notm}} Resource List locate the previously created and configured {{site.data.keyword.appid_short}} service and launch its management dashboard.
    • Go to Manage under the Identity Providers, then to Settings.
    • In the Add web redirect URLs form add https://secure-file-storage.<your custom domain>/appid_callback as another URL.
  8. Everything should be in place now. Test the app by accessing it at your configured custom domain https://secure-file-storage.<your custom domain>.

Security: Rotate service credentials

{: #cloud-e2e-security-20} {: step} To maintain security, service credentials, passwords and other keys should be replaced (rotated) a regular basis. Many security policies have a requirement to change passwords and credentials every 90 days or with similar frequency. Moreover, in the case an employee leaves the team or in (suspected) security incidents, access privileges should be changed immediately.

In this tutorial, services are utilized for different purposes, from storing files and metadata over securing application access to managing container images. Rotating the service credentials typically involves

  • renaming the existing service keys,
  • creating a new set of credentials with the previously used name,
  • replacing the access data in existing Kubernetes secrets and applying the changes,
  • and, after verification, deactivating the old credentials by deleting the old service keys.

Expand the tutorial

{: #cloud-e2e-security-21}

Security is never done. Try the below suggestions to enhance the security of your application.

  • Replace {{site.data.keyword.keymanagementservicelong_notm}} by {{site.data.keyword.hscrypto}} for even greater security and control over encryption keys.

Share resources

{: #cloud-e2e-security-22}

If you want to work with others on resources of this solution tutorial, you can share all or only some of the components. {{site.data.keyword.cloud_notm}} Identity and Access Management (IAM) enables the authentication of users and service IDs and the access control to cloud resources. For granting access to a resource, you can assign predefined access roles to either a user, a service ID, or to an access group. An access group can be created to organize a set of users and service IDs into a single entity. It makes it easy for you to assign access. You can assign a single policy to the group instead of assigning the same access multiple times per individual user or service ID. Thus, you can organize groups for roles on your development project and align security and project management.

You can find information on the individual services and their available IAM access roles here:

To get started, check out the best practices for access management and how to define access groups.

Remove resources

{: #cloud-e2e-security-23} {:removeresources}

To remove the resource, delete the deployed container and then the provisioned services.

If you share an account with other users, always make sure to delete only your own resources. {: tip}

  1. Delete the deployed container: 
    kubectl delete -f secure-file-storage.yaml
    {: codeblock}
  2. Delete the secrets for the deployment: 
    kubectl delete secret secure-file-storage-credentials
    {: codeblock}
  3. Remove the container image from the container registry: 
    ibmcloud cr image-rm <your-registry-url>/<your-namespace>/<your-image-name>:latest
    {: codeblock}
  4. In the {{site.data.keyword.Bluemix_notm}} Resource List locate the resources that were created for this tutorial. Use the search box and secure-file-storage as pattern. Delete each of the services by clicking on the context menu next to each service and choosing Delete Service. Note that the {{site.data.keyword.keymanagementserviceshort}} service can only be removed after the key has been deleted. Click on the service instance to get to the related dashboard and to delete the key.

Related content

{: #cloud-e2e-security-12} {:related}