VERSION

Features

  • Use SQS source connector to sync data to Pulsar

Tags

Pulsar IO

SQS

Source

Enterprise Support

StreamNative supported

Author

StreamNative

SQS Source

The AWS Simple Queue Service (SQS) source connector feeds data from Amazon AWS SQS and writes data to Pulsar topics.

How to get

You can get the SQS source connector using one of the following methods:

  • Download the NAR package from here.

  • Build it from the source code.

    1. Clone the source code to your machine.

      git clone https://github.com/streamnative/pulsar-io-sqs
    2. Assume that PULSAR_IO_SQS_HOME is the home directory for the pulsar-io-sqs repo. Build the connector in the ${PULSAR_IO_SQS_HOME} directory.

      mvn clean install -DskipTests

      After the connector is successfully built, a NAR package is generated under the target directory.

      ls target
      pulsar-io-sqs-2.7.0.nar

How to configure

Before using the SQS source connector, you need to configure it.

You can create a configuration file (JSON or YAML) to set the following properties.

Name Type Required Default Description
awsEndpoint String false "" (empty string) AWS SQS end-point URL. It can be found at here.
awsRegion String true " " (empty string) Supported AWS region. For example, us-west-1, us-west-2.
awsCredentialPluginName String false " " (empty string) Fully-qualified class name of implementation of AwsCredentialProviderPlugin.
awsCredentialPluginParam String true " " (empty string) JSON parameter to initialize AwsCredentialsProviderPlugin.
queueName String true " " (empty string) Name of the SQS queue that messages should be read from or written to.

Example

  • JSON

    {
         "tenant": "public",
         "namespace": "default",
         "name": "sqs-source",
         "topicName": "test-queue-pulsar",
         "archive": "connectors/pulsar-io-sqs-2.7.0.nar",
         "parallelism": 1,
         "configs":
         {
             "awsEndpoint": "https://sqs.us-west-2.amazonaws.com",
             "awsRegion": "us-west-2",
             "queueName": "test-queue",
             "awsCredentialPluginName": "",
             "awsCredentialPluginParam": '{"accessKey":"myKey","secretKey":"my-Secret"}'
         }
     }
  • YAML

    tenant: "public"
    namespace: "default"
    name: "sqs-source"
    topicName: "test-queue-pulsar"
    archive: "connectors/pulsar-io-sqs-2.7.0.nar"
    parallelism: 1
    
    configs:
       awsEndpoint: "https://sqs.us-west-2.amazonaws.com"
       awsRegion: "us-west-2"
       queueName: "test-queue"
       awsCredentialPluginName: ""
       awsCredentialPluginParam: '{"accessKey":"myKey","secretKey":"my-Secret"}'

How to use

You can use the SQS source connector as a non built-in connector or a built-in connector as below.

Use as non built-in connector

If you already have a Pulsar cluster, you can use the SQS source connector as a non built-in connector directly.

This example shows how to create an SQS source connector on a Pulsar cluster using the pulsar-admin sources create command.

PULSAR_HOME/bin/pulsar-admin sources create \
--archive pulsar-io-sqs-2.7.0.nar \
--source-config-file sqs-source-config.yaml \
--classname org.apache.pulsar.ecosystem.io.sqs.SQSSource \
--name sqs-source

Use as built-in connector

You can make the SQS source connector as a built-in connector and use it on a standalone cluster, on-premises cluster, or K8S cluster.

Standalone cluster

This example describes how to use the SQS source connector to feed data from SQS and write data to Pulsar topics in the standalone mode.

  1. Prepare SQS service.

    For more information, see Getting Started with Amazon SQS.

  2. Copy the NAR package to the Pulsar connectors directory.

    cp pulsar-io-sqs-2.7.0.nar PULSAR_HOME/connectors/pulsar-io-sqs-2.7.0.nar
  3. Start Pulsar in standalone mode.

    PULSAR_HOME/bin/pulsar standalone
  4. Run the SQS source connector locally.

    PULSAR_HOME/bin/pulsar-admin sources localrun --source-type sqs  --source-config-file sqs-source-config.yaml
  5. Consume the message from the Pulsar topic.

    PULSAR_HOME/bin/pulsar-client consume -s "sub-products" public/default/test-queue-pulsar -n 0
  6. Send a message to the SQS queue using the AWS SQS CLI tool.

    aws sqs send-message --queue-url ${QUEUE_URL} --message-body "Hello From SQS"

    Now you can see the message "Hello From SQS" from the Pulsar consumer.

On-premises cluster

This example explains how to create an SQS source connector in an on-premises cluster.

  1. Copy the NAR package of the SQS connector to the Pulsar connectors directory.

    cp pulsar-io-sqs-2.7.0.nar $PULSAR_HOME/connectors/pulsar-io-sqs-2.7.0.nar
  2. Reload all built-in connectors.

    PULSAR_HOME/bin/pulsar-admin sources reload
  3. Check whether the SQS source connector is available on the list or not.

    PULSAR_HOME/bin/pulsar-admin sources available-sources
  4. Create an SQS source connector on a Pulsar cluster using the pulsar-admin sources create command.

    PULSAR_HOME/bin/pulsar-admin sources create \
    --source-type sqs \
    --source-config-file sqs-source-config.yaml \
    --name sqs-source

K8S cluster

This example demonstrates how to create an SQS source connector on a K8S cluster.

  1. Build a new image based on the Pulsar image with the SQS source connector and push the new image to your image registry. This example tags the new image as streamnative/pulsar-sqs:2.7.0.

    FROM apachepulsar/pulsar-all:2.7.0
    RUN curl https://github.com/streamnative/pulsar-io-sqs/releases/download/v2.7.0/pulsar-io-sqs-2.7.0.nar -o /pulsar/connectors/pulsar-io-sqs-2.7.0.nar
  2. Extract the previous --set arguments from K8S to the pulsar.yaml file.

    helm get values <release-name> > pulsar.yaml
  3. Replace the images section in the pulsar.yaml file with the images section of streamnative/pulsar-sqs:2.7.0.

  4. Upgrade the K8S cluster with the pulsar.yaml file.

    helm upgrade <release-name> streamnative/pulsar \
        --version <new version> \
        -f pulsar.yaml

    Tip

    For more information about how to upgrade a Pulsar cluster with Helm, see Upgrade Guide.

  5. Create an SQS source connector on a Pulsar cluster using the pulsar-admin sources create command.

    PULSAR_HOME/bin/pulsar-admin sources create \
    --source-type sqs 
    --source-config-file sqs-source-config.yaml 
    --name sqs-source