diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 6ba9b6ec..28b9a785 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -21,6 +21,7 @@ .Operations * xref:operations:auth.adoc[] * xref:operations:functions.adoc[] +* xref:operations:functions-transform.adoc[] * xref:operations:io-connectors.adoc[] * xref:operations:scale-cluster.adoc[] * xref:operations:troubleshooting.adoc[] \ No newline at end of file diff --git a/modules/components/pages/admin-console-tutorial.adoc b/modules/components/pages/admin-console-tutorial.adoc index ace206e4..5d8db57f 100644 --- a/modules/components/pages/admin-console-tutorial.adoc +++ b/modules/components/pages/admin-console-tutorial.adoc @@ -2,14 +2,6 @@ The *{company} Admin Console for {pulsar-reg}* is a web-based UI from {company} that administers topics, namespaces, sources, sinks, and various aspects of {pulsar} features. -* xref:components:admin-console-tutorial.adoc#getting-started[] -* xref:components:admin-console-tutorial.adoc#features[] -* xref:components:admin-console-tutorial.adoc#send-receive[] -* xref:components:admin-console-tutorial.adoc#create-topics[] -* xref:components:admin-console-tutorial.adoc#code-samples[] -* xref:components:admin-console-tutorial.adoc#connect-to-pulsar[] -* xref:components:admin-console-tutorial.adoc#video[] - [#getting-started] == Getting Started in {pulsar-short} Admin Console @@ -151,13 +143,6 @@ You can get the token from the {pulsar-short} Admin Console's *Credentials* page Alternatively, you can save the URL authentication parameters in your `client.conf` file. -[#video] -== Admin console video - -You can also follow along with this video from our *Five Minutes About {pulsar-short}* series to get started with the admin console. - -video::1IwblLfPiPQ[youtube, list=PL2g2h-wyI4SqeKH16czlcQ5x4Q_z-X7_m] - == Next steps For more on building and running a standalone {pulsar-short} Admin console, see the xref:admin-console-vm.adoc[Admin Console on Server/VM] or the https://github.com/datastax/pulsar-admin-console#dev[{pulsar-short} Admin console README]. \ No newline at end of file diff --git a/modules/components/pages/admin-console-vm.adoc b/modules/components/pages/admin-console-vm.adoc index 799034a7..f6264a5a 100644 --- a/modules/components/pages/admin-console-vm.adoc +++ b/modules/components/pages/admin-console-vm.adoc @@ -4,16 +4,6 @@ The Admin Console is a VueJS application that runs in a browser. It also includes a web server that serves up the files for the Admin Console as well as providing configuration and authentication services. -This document covers: - -* <> - -* <> - -* <> - -* <> - [#install] == Install {pulsar-short} Admin Console diff --git a/modules/install-upgrade/pages/quickstart-helm-installs.adoc b/modules/install-upgrade/pages/quickstart-helm-installs.adoc index 9cca2842..d6320e9a 100644 --- a/modules/install-upgrade/pages/quickstart-helm-installs.adoc +++ b/modules/install-upgrade/pages/quickstart-helm-installs.adoc @@ -76,13 +76,10 @@ For a component like BookKeeper, which requires stateful storage, we need to ove There are two ways to override `default_storage`: -* Leave `existingStorageClassName` blank and specify the storage class parameters below. +* Leave `existingStorageClassName` blank and specify the storage class parameters for your cloud provider: + -[tabs] -==== -GKE:: +** GKE: + --- [source,yaml] ---- volumes: @@ -97,11 +94,9 @@ GKE:: extraParams: replication-type: none ---- --- -EKS:: +** EKS: + --- [source,yaml] ---- volumes: @@ -116,11 +111,9 @@ EKS:: extraParams: iopsPerGB: "10" ---- --- -AKS:: +** AKS: + --- [source,yaml] ---- volumes: @@ -135,8 +128,6 @@ AKS:: extraParams: replication-type: none ---- --- -==== * Create a custom storage configuration as a `yaml` file (https://github.com/datastax/pulsar-helm-chart/blob/master/helm-chart-sources/pulsar/templates/bookkeeper/bookkeeper-storageclass.yaml[like the {company} example]) and tell the Helm chart to use that storage configuration when it creates the BookKeeper PVCs. + @@ -505,13 +496,6 @@ Once you have created the secrets that store the certificate info (or specified `enableTls: yes` -[#video] -== Getting started with Kubernetes video - -Follow along with this video from our *Five Minutes About {pulsar-short}* series to get started with a Helm installation. - -video::hEBP_IVQqQM[youtube, list=PL2g2h-wyI4SqeKH16czlcQ5x4Q_z-X7_m] - == Next steps To learn about installing Luna Streaming for Bare Metal/VM, see xref:install-upgrade:quickstart-server-installs.adoc[Quickstart for Bare Metal/VM Installs]. \ No newline at end of file diff --git a/modules/operations/pages/auth.adoc b/modules/operations/pages/auth.adoc index c602b9ec..7c9dd022 100644 --- a/modules/operations/pages/auth.adoc +++ b/modules/operations/pages/auth.adoc @@ -128,8 +128,6 @@ pulsar@pulsar-bastion-85c9b777f6-gt9ct:/pulsar$ curl -d "client_id=test-client" ---- + .Results -[%collapsible] -==== [source,json] ---- { @@ -141,7 +139,6 @@ pulsar@pulsar-bastion-85c9b777f6-gt9ct:/pulsar$ curl -d "client_id=test-client" "scope":"email profile" } ---- -==== . Copy the `access_token` contents and use it in the `pulsar-admin` command's `--auth-params` option: + diff --git a/modules/operations/pages/functions-transform.adoc b/modules/operations/pages/functions-transform.adoc new file mode 100644 index 00000000..e86dedbf --- /dev/null +++ b/modules/operations/pages/functions-transform.adoc @@ -0,0 +1,55 @@ += {product} {pulsar-short} transform functions configuration reference +:navtitle: Transform functions configuration reference + +include::common:streaming:partial$transform-functions/intro.adoc[] + +To deploy transform functions, see xref:operations:functions.adoc[Deploy and manage functions]. + +== Configure transform functions + +include::common:streaming:partial$transform-functions/config.adoc[] + +[#expression-language] +=== Expression language + +include::common:streaming:partial$transform-functions/expression-language.adoc[] + +[#conditional-steps] +=== Conditional steps + +include::common:streaming:partial$transform-functions/when.adoc[] + +[#cast] +== Cast + +include::common:streaming:partial$transform-functions/cast.adoc[] + +[#compute] +== Compute + +include::common:streaming:partial$transform-functions/compute.adoc[] + +[#drop] +== Drop + +include::common:streaming:partial$transform-functions/drop.adoc[] + +[#drop-fields] +== Drop fields + +include::common:streaming:partial$transform-functions/drop-fields.adoc[] + +[#flatten] +== Flatten + +include::common:streaming:partial$transform-functions/flatten.adoc[] + +[#merge-keyvalue] +== Merge KeyValue + +include::common:streaming:partial$transform-functions/merge-keyvalue.adoc[] + +[#unwrap-keyvalue] +== Unwrap KeyValue + +include::common:streaming:partial$transform-functions/unwrap-keyvalue.adoc[] \ No newline at end of file diff --git a/modules/operations/pages/functions.adoc b/modules/operations/pages/functions.adoc index 9cd3b7e7..846f43bd 100644 --- a/modules/operations/pages/functions.adoc +++ b/modules/operations/pages/functions.adoc @@ -1,15 +1,28 @@ -= Luna Streaming Functions += Deploy and manage {product} {pulsar-short} functions +:navtitle: Deploy and manage {pulsar-short} functions -Functions are lightweight compute processes that enable you to process each message received on a topic or multiple topics. You can apply custom logic to that message, transforming or enriching it, and then output it to a different topic. +Functions are lightweight compute processes that you can run on each message a topic receives. +You can apply custom logic to a message, transforming or enriching it, and then output it to a different topic. -Functions run inside Luna Streaming and are therefore serverless. Write the code for your function in Java, Python, or Go, then upload the code to the {pulsar-short} cluster and deploy the function. The function will be automatically run for each message published to the specified input topic. See https://pulsar.apache.org/docs/en/functions-overview/[{pulsar-short} Functions overview] for more information about {pulsar-reg} functions. +Functions run inside {product}, which makes them serverless. +You write the code for your function in Java, Python, or Go, and then upload the code to the {pulsar-short} cluster and deploy the function. +The function automatically runs for each message published to the specified input topic. + +Functions are implemented using https://pulsar.apache.org/docs/en/functions-overview/[{pulsar-reg} functions]. + +https://github.com/datastax/pulsar[Luna Streaming 2.10 or later] is required to deploy custom functions in {pulsar-short}. == Manage functions using {pulsar-short} Admin CLI -Add functions using the {pulsar-short} Admin CLI. Create a new Python function to consume a message from one topic, add an exclamation point, and publish the results to another topic. +There are multiple ways to add functions using the {pulsar-short} Admin CLI. -. Create the following Python function in `function.py`: +=== Deploy custom function code + +. Create a file that contains your function code. ++ +For example purposes, you can use the following sample function that consumes a message from one topic, adds an exclamation point, and then publishes the result to another topic. + +.function.py [source,python] ---- from pulsar import Function @@ -22,52 +35,143 @@ class ExclamationFunction(Function): return input + '!' ---- -. Deploy `function.py` to your {pulsar-short} cluster using the {pulsar-short} Admin CLI: +. Deploy your function file to your {pulsar-short} cluster using the {pulsar-short} Admin CLI: + [source,bash] ---- -$ ./pulsar-admin functions create \ +./pulsar-admin functions create \ --py function.py \ --classname function.ExclamationFunction \ - --tenant \ - --namespace default \ + --tenant $TENANT \ + --namespace $NAMESPACE \ --name exclamation \ - --inputs persistent:///default/ - --output persistent:///default/ + --inputs persistent://$TENANT/$NAMESPACE/$INPUT_TOPIC \ + --output persistent://$TENANT/$NAMESPACE/$OUTPUT_TOPIC +---- ++ +If the function is set up and ready to accept messages, then the output is `Created Successfully`. + +. Confirm the function was created by listing the functions in the tenant: ++ +[source,bash] +---- +./pulsar-admin functions list --tenant $TENANT +---- + +=== Deploy {company} transform functions + +Transform functions are built-in functions that perform common data transformation tasks, such as casting data types, dropping fields, flattening nested structures, and merging or unwrapping KeyValue pairs. +For more information, see the xref:operations:functions-transform.adoc[transform functions configuration reference]. + +You can find the transform function `.nar` in the `/functions` directory of your {pulsar-short} deployment. + +==== Deploy transform functions in {pulsar-short} standalone + +To deploy the built-in transform function locally in {pulsar-short} standalone, do the following: + +. Start {pulsar-short} standalone: ++ +[source,shell] +---- +./bin/pulsar standalone +---- + +. Create a transform function in `localrun` mode: ++ +[source,bash] +---- +./bin/pulsar-admin functions localrun \ + --jar functions/pulsar-transformations-2.0.1.nar \ + --classname com.datastax.oss.pulsar.functions.transforms.TransformFunction \ + --inputs $INPUT_TOPIC \ + --output $OUTPUT_TOPIC \ + --user-config '{"steps": [{"type": "drop-fields", "fields": "password"}, {"type": "merge-key-value"}, {"type": "unwrap-key-value"}, {"type": "cast", "schema-type": "STRING"}]}' ---- + -If the function is set up and ready to accept messages, you should see "Created Successfully!" +For the `user-config` parameters, see the xref:operations:functions-transform.adoc[transform functions configuration reference]. -. Use `./pulsar-admin functions list --tenant ` to list the functions in your tenant and confirm your new function was created. +==== Deploy transform functions in a {pulsar-short} cluster -=== Trigger with CLI +To deploy a built-in transform function to a {pulsar-short} cluster, do the following: + +. Create a built-in transform function with the {pulsar-short} CLI: ++ +[source,bash] +---- +./bin/pulsar-admin functions create \ + --tenant $TENANT \ + --namespace $NAMESPACE \ + --name transform-function \ + --inputs persistent://$TENANT/$NAMESPACE/$INPUT_TOPIC \ + --output persistent://$TENANT/$NAMESPACE/$OUTPUT_TOPIC \ + --classname com.datastax.oss.pulsar.functions.transforms.TransformFunction \ + --jar functions/pulsar-transformations-2.0.1.nar +---- ++ +This option deployed the functions using the transform function `.nar`. +You can also deploy individual functions by passing the function configuration in the `--user-config` parameter. +For more information, see the xref:operations:functions-transform.adoc[transform functions configuration reference]. + +. Confirm the function was created by listing the functions in the tenant: ++ +[source,bash] +---- +./pulsar-admin functions list --tenant $TENANT +---- + +=== Deploy a function in a {pulsar-short} sink + +By default, functions modify data either after it is written to a topic by a source connector, or before it is read from a topic by a sink connector. +This requires either a custom connector or an intermediate topic (with additional storage, IO, and latency). + +Alternatively, you can deploy functions in a sink to apply preprocessing to sink topic writes (outgoing messages from the sink). + +To deploy a function in a sink, include the function path and configuration when you create the sink connector. +For example: + +[source,bash] +---- +pulsar-admin sinks create \ + --sink-type elastic-sink \ + --inputs my-input-topic \ + --tenant public \ + --namespace default \ + --name my-sink \ + --transform-function "builtin://transforms" \ + --transform-function-config '{"steps": [{"type": "drop-fields", "fields": "password"}, {"type": "merge-key-value"}, {"type": "unwrap-key-value"}' +---- + +== Trigger functions with the {pulsar-short} CLI Triggering a function is a convenient way to test that the function is working. When you trigger a function, you are publishing a message on the function's input topic, which triggers the function to run. -To test a function with the {pulsar-short} CLI, send a test value with {pulsar-short} CLI's `trigger`. +To test a function with the {pulsar-short} CLI, send a test value with {pulsar-short} CLI's `functions trigger` command: . Listen for messages on the output topic: + [source,bash] ---- -bin/pulsar-client consume persistent:///default/ \ +bin/pulsar-client consume persistent://$TENANT/$NAMESPACE/$TOPIC \ --subscription-name my-subscription --num-messages 0 # Listen indefinitely ---- -. Test your exclamation function with `trigger`: +. Test the function with the `functions trigger` command and a test message. ++ +The following example triggers a function named `exclamation`: + [source,bash] ---- ./pulsar-admin functions trigger \ --name exclamation \ - --tenant \ - --namespace default \ + --tenant $TENANT \ + --namespace $NAMESPACE \ --trigger-value "Hello world" ---- -+ -The trigger sends the string `Hello world` to your exclamation function. -Your function should output `Hello world!` to your consumed output. + +. Make sure the output shows that the messages was processed as expected based on your function logic. +For example, the `exclamation` sample function adds an exclamation point to the input string. +Therefore, the test message `Hello world` should output `Hello world!`. == Add Functions using {pulsar-short} Admin Console @@ -148,6 +252,6 @@ Enter your message in the *Message to Send* field, and select the output topic. In this case, the trigger sends the string `Hello world!` to your exclamation function with no output function. If the function has an output topic and the function returns data to the output topic, it is printed. -== Next steps +== See also -For more information, see the https://pulsar.apache.org/docs/en/functions-develop/[{pulsar-short} documentation on developing functions]. \ No newline at end of file +* https://pulsar.apache.org/docs/en/functions-develop/[{pulsar-short} documentation on developing functions] \ No newline at end of file diff --git a/modules/operations/pages/troubleshooting.adoc b/modules/operations/pages/troubleshooting.adoc index 9f6d2cfa..8109119a 100644 --- a/modules/operations/pages/troubleshooting.adoc +++ b/modules/operations/pages/troubleshooting.adoc @@ -1,11 +1,5 @@ = Troubleshooting -To troubleshoot Luna Streaming, look at these potential issues: - -* xref:troubleshooting.adoc#checkquotas[**Insufficient Quota**] -* xref:troubleshooting.adoc#containerspending[**Containers in Pending State**] -* xref:troubleshooting.adoc#bastionpod[**Using the Bastion Pod**] - [#checkquotas] == Check quotas