webMethods Elastic ESB using Command Central (v9.5)

Introduction

In version 9.x of webMethods, +Software AG have come up with a new method of scaling Integration Servers (see official blog here - http://goo.gl/79OJOP), using a product called Command Cental. In the current 9.5 version, it allows for Command Central to use scripts to clone an existing product stack to another server. The official blog actually talks about scaling in the cloud to bring up instances when the load is too much.

This article will go through how to provision a new installation

Pre-requisites

• webMethods product image (wm9_5_win64.zip)
• Existing webMethods product installation including CommandCentral and SPM (SoftwareAG Platform Manager)

Provisioning Process

Initial Setup

1. Startup CommandCentral and open the browser to the instance (default port is 8090)
2. Create a new environment (I called mine dev)
3. On the second tab, click on Add
4. It should prompt you to enter some information:
• Logical node name - dev-master
• hostname
• port - 8092 (this is the SPM port number)
5. If all the details are correct, you should see a list of all the products installed along with version numbers
6. Now you'll need to install only SPM on the target server using the product image (make note of the port number)
7. Once you have done that go back to CommandCentral and add a new environment
8. Repeat step 3-5 (this time you should only see the SPM product)

Now that all the initial steps are completed, it's time to begin the provisioning

Process

1. Open up a command/shell window
2. Navigate to the CommandCentral/client/bin folder
3. We are going to check that everything is in order before provisioning a new instance
1. Check nodes in landscape 
• cc list landscape nodes -e ONLINE -p <password>
• You should see both nodes from the initial setup online
2. Check IS is installed on source node 
• cc list inventory products IS-Local -e integrationServer -w 0 -p <password>
• You should see something like IS-Local IS-Local-integrationServer integrationServer Integration Server | Server
4. Now it's time to add our image to the repository of the target
1. cc add repository products path=D:/sag/webMethods/wm9_5_win64.zip -p <password>
5. Check that the repository was added successfully
• cc list repository products -p <password>
• You should see something like REPOSITORY-wm9_5_win64.zip      product false
6. Now we are going to create a template from the source node (file can be downloaded and then replace the alias and nodeAlias with your own)
• cc create templates -i D:/createTemplate.xml -p <password>
7. List the templates on the source node
• cc list templates -p <password>
• You should see something like local-source    IS-Local        Administrator
8. Apply the product template to target (again replace alias and nodeAlias as target this time)
• cc exec templates apply -i D:/applyProducts.xml -p <password>
• It should return a job number if working. Make note of this.This will take around 5-10 minutes to complete. If you want to monitor the job, run this:
• cc get jobmanager jobs -e "DONE|WARNING|ERROR" -w 180 -p <password>

Conclusion

Although not fully polished off, this method of spawning instances of the product is quite handy and can be modelled using BPM to automatically kick off when certain SLA's are breached.

In the upcoming versions of webMethods, I'm sure this will be provided from the GUI itself, making it easier to change ports and other configurations.

Handling JSON in the ESB

Introduction

In version 9.x of webMethods, JSON support is out of the box, however for the older versions, you need to write your own handler.

This article will provide sample code on how to use GSON (Google's JSON parser) to handle JSON strings and arrays in the ESB.
We are assuming your IData objects are of type key (String) and value (Object)

IS Java Services

JSONToDocument

public static final void JSONToDocument(IData pipeline) throws ServiceException {
// pipeline
IDataCursor pipelineCursor = pipeline.getCursor();
String jsonString = IDataUtil.getString(pipelineCursor, "jsonString");
pipelineCursor.destroy();

Gson gson = new Gson();
Type type = new TypeToken<Map<String, Object>>() {}.getType();
Map<String, Object> jsonMapList = gson.fromJson(jsonString, type);
IData document = convertToIData(jsonMapList);
// pipeline
IDataCursor pipelineCursor_1 = pipeline.getCursor();
IDataUtil.put(pipelineCursor_1, "document", document);
pipelineCursor_1.destroy(); }

JSONToDocumentList

public static final void JSONToDocumentList(IData pipeline) throws ServiceException {
// pipeline
IDataCursor pipelineCursor = pipeline.getCursor();
String jsonString = IDataUtil.getString(pipelineCursor, "jsonString");
pipelineCursor.destroy();

Gson gson = new Gson();
Type type = new TypeToken<List<Map<String, Object>>>() {}.getType();
List<Map<String, Object>> jsonMapList = gson.fromJson(jsonString, type);
List<IData> docList = new ArrayList<IData>()
for(Map<String, Object> map : jsonMapList) {
docList.add(convertToIData(map));
}
IData[] documentList = new IData[docList.size()];
docList.toArray(documentList);
// pipeline
IDataCursor pipelineCursor_1 = pipeline.getCursor();
IDataUtil.put(pipelineCursor_1, "documentList", documentList);
pipelineCursor_1.destroy();
}


DocumentToJSON

public static final void documentToJSON(IData pipeline) throws ServiceException {

// pipeline

IDataCursor pipelineCursor = pipeline.getCursor();

IData document = IDataUtil.getIData(pipelineCursor, "document");

String callback = IDataUtil.getString(pipelineCursor, "callback");

if(document == null) {

return;

}

Map result = convertToMap(document, new HashMap());

Gson gson = new Gson();

String jsonString = gson.toJson(result);

if(callback != null && !callback.isEmpty()) {

jsonString = callback + "(" + jsonString + ")";

}

IDataUtil.put(pipelineCursor, "jsonString", jsonString);

pipelineCursor.destroy();

}

DocumentListToJSON

public static final void documentListToJSON(IData pipeline)

throws ServiceException {

// pipeline

IDataCursor pipelineCursor = pipeline.getCursor();

IData[] documentList = IDataUtil.getIDataArray(pipelineCursor, "documentList");

String callback = IDataUtil.getString(pipelineCursor, "callback");

if(documentList == null) {

return;

}

List

for(IData data : documentList) {

result.add(convertToMap(data, new HashMap()));

}

Gson gson = new Gson();

String jsonString = gson.toJson(result);

if(callback != null && !callback.isEmpty()) {

jsonString = callback + "(" + jsonString + ")";

}

IDataUtil.put(pipelineCursor, "jsonString", jsonString);

pipelineCursor.destroy();

}

Shared Code

public static Map convertToMap(IData data, Map map) {
IDataCursor dataCursor = data.getCursor();
while(dataCursor.next()) {
Object key = dataCursor.getKey();
Object val = dataCursor.getValue();
if (val instanceof IData[]) {
IData[] ida = (IData[]) val;
List
for (int l = 0; l < ida.length; l++) {
valueList.add(convertToMap(ida[l], new HashMap()));
}
map.put(String.valueOf(key),valueList);
} else if (val instanceof IData) {
map.put(String.valueOf(key),convertToMap((IData)val, new HashMap()));
} else {
map.put(String.valueOf(key), val);
}
}
dataCursor.destroy();
return map;
}

public static IData convertToIData(Map map) {
IData doc = IDataFactory.create();
IDataCursor docCursor = doc.getCursor();
for(Entry entry : map.entrySet()) {
if(entry.getValue() instanceof List) {
List
List docList = new ArrayList();
for(Map m : list) {
docList.add(convertToIData(m));
}
IData[] docArray = new IData[docList.size()];
IDataUtil.put(docCursor, entry.getKey(), docList.toArray(docArray));
} else {
IDataUtil.put(docCursor, entry.getKey(), entry.getValue());
}
}
docCursor.destroy();
return doc;
}

Integration Patterns: Internal vs External Applications

Introduction

I've been working on a lot of projects that deal with both internal & external system integration and the same question always arises - "What interface scheme should we use to provide/consume data from these systems?" Speaking with a lot of clients and their internal IT team, you hear a lot of different methods that have been used before. They all will potentially work, but sometimes things are overly architected. With the industry slowly moving away from your standard SDLC and more towards AGILE, the way we develop our solutions also need to change.

External Systems

My idea is that when integrating with an external system, you should make your interfaces strict. What I mean by this is that there should be some sort of schema to validate the request with strong data types. These interfaces shouldn't be changing very often and since there are other organisations that could be using them, you want to cause minimal impact.
I think using WSDLs or a canonical built of an XSD is the way to go with this approach. This caters for both SOAP and other communication means like JMS.

Internal Systems

For internal system integration, you will usually have databases, CRMs, finance systems etc. Most of these systems will have their own API's/methods for communication, but if you are going to be the provider of the service, I'd recommend using RESTful services with XML or JSON, if the end system can communicate via HTTP, or JMS. Both are lightweight and work well with an AGILE methodology. Internally you will usually have more changes due to the business not knowing exactly what they want.
These aren't strict principles that I follow and each implementation is different based on the requirements.
Let me know in the comments on your thoughts and opinions.

Apache Camel Tutorial on EIP, Routes and Testing

Interesting blog about Apache Camel. I'll be definitely playing around with this on the weekend and post my findings about it.

Please read it at the following link - http://www.kai-waehner.de/blog/2012/05/04/apache-camel-tutorial-introduction/

Merging IS document to LiveCycle PDF

Introduction

The following post outlines what is required to merge data from an IS document to an Adobe LiveCycle Static PDF form.

Prerequisites

• Adobe Livecycle Static PDF Form
• iText jars in package
• Names used in Static form are the same as in IS Document, otherwise data won't be merged

IS Code

Inputs/Outputs:

• Document - IS document containing data
• pdfFormTemplate - full file path of the Adobe Livecycle Static PDF form
• pdfBytes - byte[] containing merged data


import com.wm.data.*;
import com.wm.util.Values;
import com.wm.app.b2b.server.Service;
import com.wm.app.b2b.server.ServiceException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.HashMap;
import java.util.Set;
import com.itextpdf.text.pdf.AcroFields;
import com.itextpdf.text.pdf.PdfReader;
import com.itextpdf.text.pdf.PdfStamper;
import com.itextpdf.text.pdf.RandomAccessFileOrArray;
import com.wm.data.IData;
import com.wm.data.IDataCursor;
import com.wm.data.IDataUtil;
import com.wm.util.coder.IDataCodable;

public final class documentToPdfBytes_SVC

{

 /**
  * The primary method for the Java service
  *
  * @param pipeline
  * The IData pipeline
  * @throws ServiceException
  */
 public static final void documentToPdfBytes(IData pipeline)
   throws ServiceException {
 
  // pipeline
  IDataCursor pipelineCursor = pipeline.getCursor();
  IData document = IDataUtil.getIData(pipelineCursor, "document");
  if (document == null) {
   throw new ServiceException("No document provided");
  }
  HashMap<String,String> fieldMap = new HashMap<String, String>();
  String pdfFormTemplate = IDataUtil.getString( pipelineCursor, "pdfFormTemplate" );
  byte[] pdfBytes = null;
  try {
   populateHashMap(document, fieldMap);
   pdfBytes = createPdfData(pdfFormTemplate, fieldMap);
  } catch(Exception e) {
   throw new ServiceException(e);
  }
  // pipeline
  IDataUtil.put(pipelineCursor, "pdfBytes", pdfBytes);
  pipelineCursor.destroy();
   
 }
 
 // --- <<IS-BEGIN-SHARED-SOURCE-AREA>> ---
 
 /**
  * Fill out a form the "traditional way".
  * @param pdfTemplate the original PDF
  * @param outputFile the final signed output PDF file
  * @param fieldMap {@link HashMap} of IData key/value pairs in the input document
  * @throws Exception
  */
 private static byte[] createPdfData(String pdfTemplate, HashMap<String, String> fieldMap) throws Exception {
  RandomAccessFileOrArray raf = new RandomAccessFileOrArray(pdfTemplate);
  PdfReader reader = new PdfReader(raf,null);
  
  File tmpFile = File.createTempFile("cpv", ".pdf");
  tmpFile.deleteOnExit();
  PdfStamper stamper = new PdfStamper(reader, new FileOutputStream(tmpFile));
 
  AcroFields form = stamper.getAcroFields();
  form.removeXfa();
  Set<String> fields = form.getFields().keySet();
  if(fields.size() == 0) {
   throw new ServiceException("No AcroFields in this pdf template "+ pdfTemplate);
  }
  for(String s : fields) {
   for(String key : fieldMap.keySet()) {
    if(s.contains(key)) {
     form.setField(s, fieldMap.get(key));
    }
   }
  }
  stamper.setFormFlattening(true);
  stamper.close();
 
  byte[] fileBytes = getBytesFromFile(tmpFile);
  return fileBytes;
 }
 
 // Returns the contents of the file in a byte array.
 private static byte[] getBytesFromFile(File file) throws IOException {
  InputStream is = new FileInputStream(file);
 
  // Get the size of the file
  long length = file.length();
 
  // You cannot create an array using a long type.
  // It needs to be an int type.
  // Before converting to an int type, check
  // to ensure that file is not larger than Integer.MAX_VALUE.
  if (length > Integer.MAX_VALUE) {
  // File is too large
  }
  // Create the byte array to hold the data
  byte[] bytes = new byte[(int)length];
 
  // Read in the bytes
  int offset = 0;
  int numRead = 0;
  while (offset < bytes.length && (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
  offset += numRead;
  }
  // Ensure all the bytes have been read in
  if (offset < bytes.length) {
  throw new IOException("Could not completely read file "+file.getName());
  }
  // Close the input stream and return bytes
  is.close();
  return bytes;
 }
 
 /**
  * @param in {@link IData} pipeline document
  * @param fieldMap {@link HashMap} map of document fields
  * @param indent
  * @throws ServiceException
  */
 private static void populateHashMap(IData in, HashMap<String,String> fieldMap) throws ServiceException {
  IDataCursor idc = in.getCursor();
  for (int i = 0; idc.next(); i++) {
   Object key = idc.getKey();
   Object val = idc.getValue();
   if (val instanceof IData[]) {
    IData[] ida = (IData[]) val;
    for (int l = 0; l < ida.length; l++) {
     populateHashMap(ida[l], fieldMap);
    }
   } else if (val instanceof IData) {
    populateHashMap((IData)val, fieldMap);
   } else if (val instanceof IDataCodable[]) {
    IDataCodable[] ida = (IDataCodable[]) val;
    for (int l = 0; l < ida.length; l++) {
     populateHashMap(ida[l].getIData(), fieldMap);
    }
   } else {
    if(val != null) {
     fieldMap.put(key.toString(), val.toString());
    }
   }
  }
  idc.destroy();
 }
 
 private static void logMessage(Object msg) {
  IData input = IDataFactory.create();
  IDataCursor inputCursor = input.getCursor();
  IDataUtil.put(inputCursor, "message", "field: "+msg.toString());
  try {
   Service.doInvoke("pub.flow","debugLog", input);
  } catch (Exception e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
  inputCursor.destroy();
 }
  
 
 // --- <<IS-END-SHARED-SOURCE-AREA>> ---
}


Comparing Open Source ESBs

I've working heavily with webMethods (http://goo.gl/z69Nu) for the last 7 or so years, I'm always thinking about what open source alternatives are there and how mature they have become compared to more proprietary EBS's like webMethods, Tibco, Oracle Fusion, IBM WebSphere etc.

I've been playing around recently with Apache Camel (http:/camel.apache.org) and Mule ESB (http://www.mulesoft.org) to get a better idea of how easy it is to use these open source ESBs

I've found that Apache Camel has quite a lot of the standard endpoints used in most organisations, however, the GUI for quick mappings and orchestrations is lacking. The tutorials are good and very detailed, especially with providing sample code, but doesn't feel like it's complete. Due to it's open source nature and with many collaborators working on Apache projects, there is a big talent pool out there to help make this a solid alternative to a paid ESB.

Mule, on the other hand, was originally driven through a XML file that had all the endpoint configurations. When dealing with configuring XML for endpoints, it was quite painful. With Mule Studio GA releasing, it has made development and integration a whole lot easier with a GUI that allows for orchestrations and quick integration projects.
One of my good friends has done quite a bit of Mule integration and with the new GUI, he's been able to reduce the amount of time taken to do his projects.

I've been looking to use Mule for my home projects and eventual integration to websites, but with the very small nature of those projects, I might just stick with using some simpler Java based solution.

This was just my experience with some of the commonly used open source ESB's compared to paid alternatives. I'll be using them more in the future and will be keeping track of them as well as any newer ones that pop up.

Communication between MWS & ESB with certificates

Introduction

The following article guides you through setting up your MWS and IS instances to encrypt data/decrypt data 

MWS

Installing the IS certificate into the MWS truststore

1. On your MWS server download a copy of InstallCert.java
2. Open the file and change the line 72 to reflect your install path of MWS
3. Compile the code javac InstallCert.java
4. Run the code as follows:
• java -cp . InstallCert <ISHostName:httpsPortNum> <glueTrustStore.jksPassword>
5. This will try to download the certificate from the IS server and install it in the MWS truststore

Setup environment variables in your CAF application

Configure your CAF application to have the following environment entries:

String store = "<MWS_HOME>/server/<server_Instance>/config/glue/glueTrustStore.jks";

String sPass = passphrase_for_file_above;

String alias = alias_of_IS_key;

Write a method that takes in your data to be encrypted with the Certificate, plus the above parameters:

private static byte[] encryptData(String store, String sPass, String alias, String data) throws Exception {
    KeyStore ks = KeyStore.getInstance("JKS");
    FileInputStream fis = new FileInputStream(store);
    ks.load(fis, sPass.toCharArray());
    fis.close();
    java.security.cert.Certificate cert = ks.getCertificate(alias);
    PublicKey key = cert.getPublicKey();
    Cipher cipher = Cipher.getInstance(key.getAlgorithm());  
    cipher.init(Cipher.ENCRYPT_MODE, key);
    return cipher.doFinal(data.getBytes());
}

Bind the byte[] to the input provided by the ESB WSD.

ESB

Create a service that takes in a byte[] input from above and retrieves the privateKey of the IS as well as decryption of the payload (using the following flow service to get the privateKey):
pub.security.keystore:getKeyAndChain

Here is the Java code for the decryption service:

public static final void decryptData(IData pipeline) throws ServiceException {
    // pipeline
    IDataCursor pipelineCursor = pipeline.getCursor();
    PrivateKey privateKey = (PrivateKey) IDataUtil.get(pipelineCursor, "privateKey");
    byte[] encryptedData = (byte[]) IDataUtil.get(pipelineCursor, "encryptedData");
    pipelineCursor.destroy();

    Cipher cipher = null;
    String decryptedData = null;
    try {
        cipher = Cipher.getInstance(privateKey.getAlgorithm());
        cipher.init(Cipher.DECRYPT_MODE, privateKey);
        decryptedData = new String(cipher.doFinal(encryptedData));
    } catch (Exception e) {
        throw new ServiceException(e);
    }
    // pipeline
    IDataCursor pipelineCursor_1 = pipeline.getCursor();
    IDataUtil.put(pipelineCursor_1, "decryptedData",decryptedData);
    pipelineCursor_1.destroy();
}

For details on setting up the certificates on the IS, please refer to the Administration Guide.