2016

Archive for 2016

Using Web Services Client adapter in Apama

Software AG Apama provides a SOAP-based Web Services Client adapter to invoke web services from your Apama application. In this post we will see how to use this adapter in your application.

We will use a free web service available at  CDYNE (link) for our application. This service has multiple operations, out of which we will use GetCityWeatherByZIP operation.

Below are the sample request and response messages for this operation, which we will use to describe our event definition in Apama.

Request XML
<soap:envelope 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soap:body>
    <GetCityWeatherByZip xmlns="http://ws.cdyne.com/WeatherWS/">
      <Zip>43209</Zip>
    </GetCityWeatherByZip>
  </soap:body>
</soap:envelope>

Response XML
<soap:Envelope 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
 xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
 xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetCityWeatherByZIPResponse 
    xmlns="http://ws.cdyne.com/WeatherWS/">
      <GetCityWeatherByZIPResult>
        <Success>true</Success>
        <ResponseText>City Found</ResponseText>
        <State>OH</State>
        <City>Columbus</City>
        <WeatherStationCity>Columbus</WeatherStationCity>
        <WeatherID>14</WeatherID>
        <Description>Cloudy</Description>
        <Temperature>63</Temperature>
        <RelativeHumidity>83</RelativeHumidity>
        <Wind>CALM</Wind>
        <Pressure>29.93R</Pressure>
        <Visibility/>
        <WindChill/>
        <Remarks/>
      </GetCityWeatherByZIPResult>
    </GetCityWeatherByZIPResponse>
  </soap:Body>
</soap:Envelope>

Let's begin by creating a new Apama project. We will name it WeatherWebServiceDemo.

While creating the project using 'New Apama Project Wizard', select 'Web Services Client Adapter' bundle



Now let's describe the event definitions. We will define events such that their structure corresponds directly to the XML structure of the web service message shown above. Such type of mapping is called as 'Convention based' mapping where Apama automatically converts the event instance to the request XML structure of the web service, and also converts the web service response XML to an event instance.

The event definition will look like this:

event GetCityWeatherByZIPType {
    string ZIP;
}

event GetCityWeatherByZIPResultType {
    string Success;
    string ResponseText;
    string State;
    string City;
    string WeatherStationCity;
    string WeatherID;
    string Description;
    string Temperature;
    string RelativeHumidity;
    string Wind;
    string Pressure;
    string Visibility;
    string WindChill;
    string Remarks;    
}

event GetCityWeatherByZIPResponseType {
    GetCityWeatherByZIPResultType GetCityWeatherByZIPResult;
}

event MyRequest {
    GetCityWeatherByZIPType GetCityWeatherByZIP;
}

event MyResponse {
    GetCityWeatherByZIPResponseType GetCityWeatherByZIPResponse;
}

event MyZIP {
    string ZIP;
}

Out of these the MyRequest and MyResponse event definitions are wrapper types required for Convention based mapping, whereas MyZIP is our custom event definition which we will use to request weather information for a zip code.

After describing the event definitions, we can import the WSDL and map the event definitions to web service input and output. To do that, open the adapter instance file located in Adapters->WebServices Client Adapter -> instance1 in your project and click on + (plus) to add the operation name, it will bring up the following screen to configure your web service.  



In this screen click on 'Add' operation, and then click on 'Create New' to bring the following screen to import the WSDL and select the operation that you want to use.


We will only import GetCityWeatherByZIP operation for our application as shown below.


After importing the operation select the input and output event types corresponding to the input and output of the operation


Once input and output event for the web service is selected, you can perform the mapping of fields to web service parameters

In the input mapping it's important to select the 'Convert to XML' transformation to convert the event structure to XML. You can select this transformation by using 'Add computed node' option in your input mapping.

The output mapping is straightforward as shown below.


Once the mapping is completed you can write your monitor script (EPL) program to call the weather service and then identify the pattern in the data.

Following simple monitor script program will just print the weather information received from web service.

In this script, whenever the correlator engine detects MyZIP event in the queue, it will route the MyRequest event. MyRequest event will in turn call the web service to get the weather information using web service client adapter and receive the response in the form of MyResponse event. Whenever MyResponse event is detected, our script prints the weather information in the log.

monitor WeatherServiceMonitor {
    MyRequest myRequest;
    MyResponse myResponse;
    MyZIP myZIP;

    action onload() {

        on all MyZIP(): myZIP {
            print "Received request for ZIP code : " + myZIP.ZIP;
            print "Preparing to call the service " ;
            myRequest.GetCityWeatherByZIP.ZIP := myZIP.ZIP;
            route myRequest;            
        }

        on all MyResponse(): myResponse {
            print 
                myResponse.GetCityWeatherByZIPResponse.
                GetCityWeatherByZIPResult.ResponseText 
            + " " 
            +     myResponse.GetCityWeatherByZIPResponse.
                GetCityWeatherByZIPResult.City
            + " " 
            +     myResponse.GetCityWeatherByZIPResponse.
                GetCityWeatherByZIPResult.Temperature
            + " " 
            +     myResponse.GetCityWeatherByZIPResponse.
                GetCityWeatherByZIPResult.Description
            + " " 
            +     myResponse.GetCityWeatherByZIPResponse.
                GetCityWeatherByZIPResult.Wind; 
        }
    }
}

 Let's fetch the weather for zip code 43209
C:\SoftwareAG\Apama\bin>engine_send
MyZIP("43209");



We can invoke the web service periodically to fetch the weather information using 'on all wait() ' as shown below. This code snippet will route the MyZIP event every minute (60 seconds) to fetch the weather information.

on all wait (60.0) {
MyZIP("43209"); 
}

Instead of logging the information to a file we can actually use Apama Dashboard to visually monitor the weather fluctuation and any patterns in the data. I will explain how to do that in another post.


Using OAuth 2.0 with webMethods Integration Server

webMethods Integration Server supports OAuth 2.0. IS can be used as an OAuth client, an authorization server or a resource server. This post describes how to use OAuth 2.0 with Integration Server in a simplified format.

For the ease of understanding we will consider IS as an authorization server and resource server in this post. However the same concept can be used to set up any role that you may want IS to perform in your architecture.

Integration Server supports two types of OAuth grant types
  1. Authorization Code:
    This is a secure way of obtaining OAuth token, where client needs to authorize himself with the authorization server. The authorization server responds with the authorization code, which the client uses to obtain the access token. This approach is typically used by clients hosted on web server based applications where the application code runs on the server.

  2. Implicit:
    This approach is used by mobile apps or browser based applications. In this approach the client does not need to authorize himself with the authorization server. The access token is passed through the browser, as a fragment in the URL. A simple javascript code can be used to extract the access token from the fragment. 

We will see how to implement both the approaches in the integration server. For both the approaches you need to do some basic setup on the IS console.

  • Client Configuration: 
A client must register himself and obtain client_id from the integration server.  In the IS console go to Security->OAuth->Client Registration page and click on 'Register Client'. Enter the information as shown below.
Redirect URL should be the URL, where IS will redirect the page after authorizing the request. I have created a simple RESTful service on the IS as a redirect URL.



Remember that the Client ID will be generated when you click Save Changes.


Your client will use this 'Client ID' to authorize himself and request an access token.

  •  Scope Management
The scope indicates the resources the client can access on behalf of  resource owner. You need to indicate one or more folders or services on the IS in the scope. Once a client is granted access to a scope, he can access the folders and services included in that scope.

Go to Security->Oauth->Scope Management page and click on 'Add Scope'. Once the page is open, mention the scope details as shown below

Here we have included one service 'OAuthDemo' in the scope. This is an URL Alias for /rest/Sandbox.OAuthDemo folder on the server



Once the scope is added, map the scope with the client using 'Associate Scope to Clients' option in the 'Scope Management' page.



Once this setup is done your client can request the access token either for Authorization Code approach or Implicit approach.

  •  Authorization Code:
    Requesting token for authorization code approach is a 2 step process. In the first step the client authorizes himself with the authorization server using pub.auth:authorize service and receives the authorization code. Once the authorization code is received, it can be used to obtain the access token using pub.auth:getAccessToken service.


    When the client initiates the pub.oauth:authorize request, it brings up a page for resource owner to either approve the access request or reject it.

    When the resource owner approves the request, the integration server generates the authorization code and redirects the page to the 'redirect URL' specified in the client configuration. 

    The service hosted at redirect URL passes the authorization code to pub.oauth:getAccessToken service to exchange authorization code for an access token as shown below

    The client application can now use the access token to access the resource described in the Scope.
  • Implicit:
    Requesting token for implicit approach is one step process.
    When the client initiates the pub.oauth:authorize request, it indicates the implicit approach by mentioning the response type as 'token' in the input to pub.oauth:authorize service. The service brings up a page for resource owner to either approve the access request or reject it. 


    When the  resource owner approves the request the integration server generates the access token and includes it as a fragment in the redirect URI, which client application can extract it using simple javascript code.


     Integration Server administrator can verify all the tokens in the IS console


    Client application can use the token as bearer token to access the resource on the server. 


    If the client application tries to use invalid token or tries to access service which is not in the scope, it will get an error




Using webMethods Integration Server with MySQL database

I feel every webMethods integration developer should have a local installation of MySQL database as it is fairly lightweight and free to use. I use XAMPP to setup MySQL on my development system. It also provides an HTTP server and very usable admin console to get things done quickly. This helps a lot in creating proofs of concept or simply trying out new things on the database without bothering DBAs in your organization.

webMethods Integration Server supports connectivity with MySQL database using JDBC Adapter.  The configuration is as described below.

Copy the JAR
  1. Download mysql connector jar (mysql-connector-java-5.1.33-bin.jar) from MySQL site
  2. Add this jar to <IntegrationServer>/lib/jars folder
  3. Restart the Integration Server, go to 'About' page and make sure that you see the jar listed in the server classpath     

Configure JDBC Adapter

  • Configure JDBC adapter connection as shown below. 



  • Enable the connection

Once JDBC adapter connection is enabled you can use it to create your adapter services using webMethods designer.


Integration Server class loading for custom jars

If a specific functionality is not available out of box from the integration server, you often need to use external jars or java libraries in IS. e.g. If you want to generate a PDF document or excel spreadsheet on your IS, you need to use your preferred libraries like IText or Apache POI.

It is not recommended to place your custom jars in the IntegrationServer/lib/jars folder. But Integration Server provides you couple of places to place these jars depending on how you want them to be loaded.

1. packages/<package name>/code/jars or packages/<package name>/code/classes:  The jar file placed in these folders are only accessible to java services inside the same package and the packages dependent on this package. If there is any change in the jar, simply reloading of the package is sufficient to have the classes loaded in memory.

2. packages/<package name>/code/jars/static are accessible to all java services across the entire IS. If there is any change in the jar, a restart of IS is required to activate any additions or changes to these.

Implementing WS Security Policies for your services on webMethods Integration Server - Username Over Transport Policy

Before implementing the policies for your services make sure that you have completed the prerequisite steps described in the earlier post.

The Username Over Transport policy users a Username token to provide client authentication with transport binding and includes timestamp token to guard against replay attacks. The entire message is secured by HTTPS protocol. The policy does not enforce signature or encryption.

a. Go to the provider descriptor in the server machine. Open the policies tab and select Username over transport policy from the list. Click on Ok and save the descriptor.



b. Now if you try to invoke the connector service you will get the error as 'Invalid Security', because the provider web service expects the client to pass the necessary 'WS-Security' information in the header.

c. Attach the same policy on the consumer web service descriptor so that it will send the WS-Security information in the SOAP header while invoking the service. Make sure that you pass the 'user' and 'password' in the 'auth/message' document. This information will be used while constructing the username token.

Now when you run the service you should see a successful response.


 

Implementing WS Security Policies for your services on webMethods Integration Server - Username, Signature and Encryption Policy

Before implementing the policies for your services make sure that you have completed the prerequisite steps described in the earlier post.

The Username, Signature and Encryption policy uses a username token to provide client authentication, users symmetric binding to sign messages to ensure message integrity, uses symmetric binding to encrypt messages to ensure message confidentiality and includes a timestamp token to guard against replay attacks.

a. Go to the provider descriptor in the server machine. Open the policies tab and select Username, Signature and Encryption policy from the list. Click on Ok and save the descriptor.



b. Now if you try to invoke the connector service you will get the error as 'Invalid Security', because the provider web service expects the client to pass the necessary 'WS-Security' information in the header.

c. Attach the same policy on the consumer web service descriptor so that it will send the WS-Security information in the SOAP header while invoking the service. Make sure that you pass the 'user' and 'password' in the 'auth/message' document. This information will be used while constructing the username token.

Now when you run the service you should see a successful response.

d. Open the TCPMon screen and see the message exchange, you will notice that the timestamp has been included in the SOAP header. Also the message body itself is encrypted, so only sender and receivers are able to interpret it.



Implementing WS Security Policies for your services on webMethods Integration Server - Username and Signature Policy

Before implementing the policies for your services make sure that you have completed the prerequisite steps described in the earlier post.

The Username_Signature policy uses a Username token to provide client authentication, uses symmetric binding to sign messages to ensure message integrity, and includes a timestamp token to guard against replay attacks. Because this policy uses symmetric binding, the sender of an outbound message does not need a private key. Instead, the client generates a symmetric key.
This policy does not enforce encryption.

a. Go to the provider descriptor in the server machine. Open the policies tab and select Username and Signature policy from the list. Click on ok and save the descriptor.



b.If you try to invoke the connector service now you will get error as 'invalid security', because the provider web service expects the client to pass the necessary WS-Security information in the header.

c. We will attach the same policy on the consumer web service descriptor so it will send the WS-Security information in the SOAP header while invoking the service.

d. Attach the Username_Signature policy on the consumer web service descriptor as explained in the step above. Save the descriptor and run the connector service. Make sure that you pass the 'user' and 'password' in the 'auth/message' document. This information will be used while constructing the username token. Make sure that you receive successful response.

e. See the message exchange on the TCPMon screen. You will notice that the timestamp has been included in the SOAP header however the message is not encrypted.


Implementing WS Security Policies for your services on webMethods Integration Server - Username and Encryption Policy

SoftwareAG provides a few out of box WS-Security policies to secure the messages to protect the authenticity and integrity of the messages. You can also create your own policies as long as the security assertions used in your policy are supported by SoftwareAG Integration Server.

Before implementing the policies for your services make sure that you have completed the prerequisite steps described in the earlier post.

The Username_Encryption policy uses a username token to provide client authentication, uses symmetric binding to encrypt messages to ensure message confidentiality, and includes a timestamp token to guard against replay attacks.

Perform following steps to implement the Username_Encryption policy for your service

a. Go to the provider descriptor in the server machine. Open the policies tab and select Username and Encryption policy from the list. Click on Ok. and save the descriptor.


b. Now if you try to invoke the connector service you will get the error as 'Invalid Security', because the provider web service expects the client to pass the necessary 'WS-Security' information in the header.

c. Attach the same policy on the consumer web service descriptor so that it will send the WS-Security information in the SOAP header while invoking the service. Make sure that you pass the 'user' and 'password' in the 'auth/message' document. This information will be used while constructing the username token.

Now when you run the service you should see a successful response.

d. Open the TCPMon screen and see the message exchange, you will notice that the timestamp has been included in the SOAP header. Also the message body itself is encrypted, so only sender and receivers are able to interpret it.




Implementing WS Security Policies for your services on webMethods Integration Server - Prerequisites

SoftwareAG webMethods Integration Server provides a few out of box WS-Security policies to secure the messages to protect the authenticity and integrity of the messages. You can also create your own policies as long as the security assertions used in your policy are supported by SoftwareAG Integration Server.

To test the SoftwareAG support for WS-Security policies we need to do some basic setup which we will discuss in this post. Please bear with me as there are multiple steps to be executed to complete the setup.

1. Create a simple WSDL (HelloWorld.wsdl)
2. Using OpenSSL tool create the key stores and trust stores for client and server

(Do not use JDK keytool to generate the key pairs. It only generates v1 certs whereas SoftwareAG policies require v3 certificates

Setup on the Server side machine

We will designate one IS as a 'Server Machine'. This IS will host the HelloWorld service and act as a provider of the service. The service provides a 'sayHello' operation for the clients to invoke. The sayHello operation will simply respond to client by prefixing 'Hello' to the input string provided by client.

a.  On the 'Server machine' create a provider descriptor for the wsdl and implement the 'sayHello' operation


 b.  Create the keystore and truststore aliases on the server machine


c.  Create provider web service end point alias on the server machine

d.  Associate the provider web service alias to the provider descriptor in the 'binding' tab



Setup on the Client side machine

We will designate another IS as a 'Client machine'. This machine will consume the WSDL and invoke the 'sayHello' operation provided by 'HelloWorld' service hosted on the 'Server machine'

a.  On the 'Client Machine' create a consumer service descriptor for the WSDL


b.  Create the keystore and truststore aliases on the client machine


c.  Create consumer web service end point alias on the client machine
d.  Associate the consumer web service alias to the consumer descriptor in the binding tab


e.  Verify that the connector is invoking the service hosted on the server and receiving the response as expected


Setting up TCPMon

We will setup Apache TCPMon to monitor the traffic between client and server. This tool is very useful for any webservice or integration service developer.

Apache TCPMon can be downloaded from http://ws.apache.org/commons/tcpmon/

a. In the TCPMon admin tab create a new listener



 b.  In the connector service on the client machine, use the TCPMon endpoint (instead of hitting the server machine endpoint) so that the traffic is routed through TCPMon.



c.  Invoke the connector service and verify that TCPMon is able to monitor the traffic


In the screen above you can see complete request and response exchanged between client and server.

As seen in the TCPMon screen, if the message exchanges between client and server are not secure, you can ‘tap’ the message, modify and resend it, or it can even be used for ‘replay’ attacks.

This completes our setup.

Writing first Apama program in EPL

SoftwareAG Apama is a complex event processing platform to monitor rapidly moving event data feeds. It helps you detect and analyze important patterns of events and promptly acts on critical events.

Apama program written in EPL (event processing language) consists of following important components
  • event type definitions 
  • monitors
  • actions
  • listeners
  • variables
  • packages

Writing your first program in Apama is the best way to get started and understand these components.

1. Start Apama studio and create a new project


2. In the project name field type name of your project as 'HelloWorldProject' and click finish


3.  You will see following folders created in the project view. If you don't see all the folders shown below, click on the spectacle icon and they should be visible. We will not go into the significance of each folder for this simple exercise.

 

4. Right click on eventdefinitions folder and create a new monitorscript file.




You can create the event definition as shown in the code sample below

    
 event HelloWorld {
   string message;
 }


5. Once you have created event, you can start looking it by writing a monitor.  We will just print a simple message on console whenever this event is detected by Apama correlator engine. Right click on the monitors folder and create a new monitorscript file HelloWorldMonitor.mon

You can create the monitor script as shown in the code sample below
 
 monitor HelloWorldMonitor {
   action onload() {
     HelloWorld hw;
            
       on all HelloWorld():hw {
         print "Hello World, I got the message - " + hw.message;
       }
   }
 }


6. Start the correlator and send the message to it to test your first Apama program




Nice!