Web Processing Service

Preface

A WPS can be configured to offer any sort of GIS functionality to clients across a network, including access to pre-programmed calculations and/or computation models that operate on spatially referenced data. A WPS may offer calculations as simple as subtracting one set of spatially referenced numbers from another (e.g., determining the difference in influenza cases between two different seasons), or as complicated as a global climate change model. The data required by the WPS can be delivered across a network, or available at the server. (snippet from the OGC website)

Introduction

This implementation was build for the research inside the ITC Generalization group. The focus of this implementation was to build up a extensible framework to provide algorithms for generalization on the web. The research was inspired by the work of the group of Moritz Neun at the University of Zurich, who presented some XML-RPC based architecture for generalization. According to the current development of the OGC in setting up a standardized Web Processing Service (WPS), this was the signal for the group to make some experiments on it. The results of this experiment are presented below and are hosted at 52n.

To avoid some redundancy of words, the specification of the WPS will not be introduced in detail on this site. So please refer to the current document of the OGC. The implementation is based on the current document of the OGC with the release name: 05-007r7 .

To have an overview about the architecture of the implementation, please have a look here at the implementation figure.

Also a good source of information is the website on the 52n main portal about 52N WPS.

The documentation for the WPS according version 0.4.0 is still available in the history of the wiki under: https://incubator.52north.org/twiki/bin/view/Processing/52nWebProcessingService?rev=51

Features

• Full java-based Open Source implementation
• Supports all features of WPS spec version 1.0.0
• Pluggable framework for algorithms and XML data handling and processing frameworks
• Build up on robust libraries (JTS, geotools, xmlBeans, servlet API, derby)
• Supports all mandatory operations of WPS 1.0.0 (plus beta WSDL support)
• Supports full logging of service activity
• Supports exception handling according to the spec
• Basic client implementation for accessing the WPS (including the complete XML encoding)
• Plug-in for uDig and JUMP (Java Unified Mapping Platform) (fast hack, but give it a try)
• Storing of execution results
• Full GML2 support for ComplexData (i.e. FeatureCollections)
• Asynchronous processing
• Raw data support
• All responses are stored on the filesystem, but can also be stored in any database such as Apache derby
• New WPS Client based on uDig for seamless web service-based processing of distributed geodata. Click here for more information.

Not supported at this time, but perhaps desireable

• Web GUI to maintain the service
• raster data processing

Our view on asynchronous processing of the WPS

TBD

New additions

Since release 52NWPS1.0.0-20071204-CVS

• initial version supporting 1.0.0
• major bug fixing in the datahandlers
• Raw data support
• Architecture extended to support easily processing frameworks such as GRASS (i.e. RepositoryManager^?)
• Database is now again optional. All stored responses are written to the filesystem directly by default
• HSQLDB database is supported as an alternative for faster database performance
• WSDL support
• The input data can be forwarded to the process (and back) without creating java object
• The process can declare what parser or generator to use (by default these are tied to the schema)
• Minor changes for convenient use

Upcoming releases

• Raster support empowered by GDAL.
• Include more tests. Maybe already 52n QM approved.

Third party libraries

There are some third party libraries used within the WPS package. These libraries are also included in the downloadable archives, except the java servlet library (you only need for source compilation of the service) and HSQLDB (downloadable from the website). The list below informs you about the purpose and the license of each library.

Library name	Website	Usage	Version	License
jump-1.1.2.jar	Java Unified Mapping Platform	Basis for Client	1.1.2	GPL
jts-1.7.jar	Java Topology Suite	Used to parse geometries from GML	1.7	LGPL
xbean.jar	Xmlbeans	Used to handle the XML	2.2	Apache license 2.0
commons-collections-3.2.jar, commons-io-1.3.1.jar, commons-lang-2.1.jar, commons-discovery-0.2.jar	Jakarta Commons	Provides a case insensitive HashMap for robust Request Parameter access etc.	Several	Apache license 2.0
derby.jar	Apache derby	Provides an embedded database approach to store the responses	10.X.X.X	Apache license 2.0
hsqldb-1.8.0.7.jar	HSQLDB	Alternative for Apache Derby	1.8.0.7	hsqldb_lic.txt
log4j-1.2.13.jar	Apache Logging	Just logging	1.2.13	Apache license 2.0
bsh-2.0b1.jar, Jama-1.0.1.jar, jdom.jar, xercesImpl.jar	part of JUMP		Several
geoapi-nongenerics-2.1-M7.jar	GeoAPI	Provides feature representation and XML handling	2.1-M7	LGPL
gt2-api.jar, gt2-main.jar, gt2-referencing.jar	GeoTools	Provides feature representation and XML handling	2.3.1	LGPL
gt2-imageio-asciigrid-2.3.1-SNAPSHOT.jar, gt2-imageio-streams-2.3.1-SNAPSHOT.jar	GeoTools	TBD	2.3.1-SNAPSHOT	LGPL
jsr173_1.0.jar	Streaming API for XML	Provides feature representation and XML handling	1.0	LGPL

Besides these libraries, one or more packages include some self-made jars, namely: 52n-wps-server.jar, 52n-wps-xml.jar, 52n-wps-client.jar, gmlpacket-2.0-0.4.jar, 52n-wps-client-jump.jar.

Screenshots & Screencasts

• Here are some screenshots of the Client, taken by the Bloggers @ GeoLabs.
• Short demo flash-movie of the WPS client. Captured by Timon ter Braak.

Installation

You can download the binaries as ready-to-use packages (file wps.zip and wpsClient.zip, located at the bottom of this site) or directly at the sourceforge CVS. The sources are located at the CVS-module: WPS

If you have any problems accessing the CVS, please check the information about the CVS on sourceforge.

All applications require the J2SE 5!

In order to use the service you have to set up a Apache Tomcat 5.5 or 6 instance. The web-application has to be accessible through the /wps path. In case of using the zip file, you should just unzip it in your webapp directory.

If you have any problems running your WPS, please have a look at this hint.

Installation of the service (CVS and maven instructions)

A short manual about how to set up WPS as a multi module project in eclipse using Maven 2 is linked here.

Please look at the Tomcat documentation how to deal with installing and compiling web applications in general. In the end the service has to be accessible through http://localhost:[YOUR_PORT]/wps/WebProcessingService.

As the WPS is organized as a multi module project using Maven, you have to get the source from the CVS and use Maven to compile it. Information about Apache Maven 2 can be found here: maven.apache.org.

The 52north remote repository is located at:

http://incubator.52north.org/maven/maven-repo/releases

As you have to set this repository in your settings.xml your configuration could look somehow like this:

  <profile>
  <id>52n-start</id>
  <repositories>
    <repository>
      <id>n52-releases</id>
      <name>52n Releases</name>
      <url>http://incubator.52north.org/maven/maven-repo/releases</url>
    </repository>
    <repository>
    <id>geotools</id>
    <name>Geotools repository</name>
    <url>http://maven.geotools.fr/repository</url>
     </repository>
    <repository>
    <id>Refractions</id>
    <name>Refractions repository</name>
    <url>http://lists.refractions.net/m2</url>
     </repository> 
<repository>
    <id>Apache</id>
    <name>Apache repository</name>
    <url>http://repo1.maven.org/maven2</url>
     </repository>     
       </repositories>
</profile>
...
  <activeProfiles>
  <activeProfile>52n-start</activeProfile>
  </activeProfiles>

Please note, that there are two repositories for geotools libraries, as the synchronization of these repositories does not seem to work that well. So just refer to both repositories.

Basic commands to compile your sources are mvn compile or mvn install. The resulting war in the 52n-wps-webapp/target dir, can then be dropped into the tomcat webapp. We are aware of the fact, that some libraries have to be installed manually into the local maven repository, but we try to set up the remote repository for the 52N WPS asap.

Installation of the service (continue)

For further configuration you have to take a look at the conf folder in the root directory of the wps webapp, where the wpsSkeleton.xml file is located. There you may add metadata about the service provider, so not a big issue. The OperationsMetadata and the ProcessOfferings will be updated automatically (part of the plug-in mechanism)! For further configuration you have to edit the mentioned wps.properties file. Please take care, that you have the latest version of the properties file in your classes folder. At this time there are some parameters to check:

Property key	Occurence	Advanced	Description	Notes	Default value
hostname	Mandatory	No	This name will be used to maintain the entry URLs in the OperationsMetadata section in the capabilities.	You have to name here your maschine name under which the service should be accessible	localhost
hostport	Mandatory	No	Is used in the GetCapabilities-operations responses OperationsMetadata section to tell to the client the correct port for operation requests.		8080
includeDataInputsInResponse	Mandatory	Yes	This flag is used to indicate, if the server should always include the complete request in the response.		false
registeredLocalAlgorithms	Mandatory	Yes		You do not have to change here anything. In case, that you want to add a new algorithm, please just write the full class name in that comma-seperated list.
registeredParsers	Mandatory	Yes		If you are using another schema than the gmlpacket.xsd and the GML2 schema, please add here your parser implementation in the comma-seperated list.
registeredGenerators	Mandatory	Yes		If you are using another schema than the gmlpacket.xsd and the GML2 schema, please add here your generator implementation in the comma-seperated list.
registeredAlgorithmRepositories	Mandatory	Yes		If you have another AlgorithmRepository to be used within WPS.	LocalAlgorithmRepository
webappPath	Optional	No		If you are running your WPS instance on another webapp path. This might be the case, if you want to run two WPS instances on one tomcat in parallel.	wps
computationTimeoutSeconds	Optional	No	Sleeping time used in RequestHandler.		0
databaseName	Optional	No	Name of the database		wps
databaseClass	Optional	No	Defining the class implementing the database.	Possible database classes: org.n52.wps.server.database.HSQLDatabase, org.n52.wps.server.database.DerbyDatabase and org.n52.wps.server.database.FlatFileDatabase	org.n52.wps.server.database.FlatFileDatabase
database	Optional	No	Name used to identify the database type in the directory path.	This value is only used if a specific path is not given to the database. Could be same for all different drivers, but to keep the directory paths logical a different name should be used for different database drivers (for example DERBY or HSQL).
databasePath	Optional	No	Defines a specific absolute database path to be used.
hsqlProtocol	Optional	No	Defines a protocol identifier for HSQLDB if it is used.	Possible HSQLDB protocol identifiers: mem, file, res, hsql, http, hsqls, https

Important to note: The Advanced values, can be left as they are, but should not be removed from the properties file. The optional values, might be set or removed.

The service will log into the catalina_home/logs/wps.log by default. Please feel free to change that location, by adjusting the log4j.properties in the classes root.

After restarting your tomcat the logs in the mentioned log file should somehow look like this:

2006-01-23 17:51:35,193 INFO  org.n52.wps.server.WebProcessingService  - WebProcessingService initializing...
2006-01-23 17:51:35,208 INFO  org.n52.wps.server.WebProcessingService  - Initialization of wps properties successful!
2006-01-23 17:51:38,521 INFO  org.n52.wps.server.AlgorithmRepository  - Algorithm class registered: org.n52.wps.server.SimpleBufferAlgorithm
2006-01-23 17:51:38,552 INFO  org.n52.wps.server.AlgorithmRepository  - Algorithms registered!
2006-01-23 17:51:38,552 INFO  org.n52.wps.server.WebProcessingService  - Algorithms initialized
2006-01-23 17:51:38,599 INFO  org.n52.wps.server.WebProcessingService  - XML Parsers initialized successfully
2006-01-23 17:51:38,630 INFO  org.n52.wps.server.WebProcessingService  - XML Generators initialized successfully
2006-01-23 17:51:38,708 INFO  org.n52.wps.server.WebProcessingService  - !WPS up and running!

Now you should get a http://localhost:[YOUR_WEBAPPS_PORT]/wps/WebProcessingService?Request=GetCapabilities&Service=WPS

If the tomcat runs under port 8080 you can access a very simple test site on http://localhost:8080/wps.

When you are getting the capabilities, feel free to test the DescribeProcess and the Execute Operation. If you would like to do that in a comfortable way, continue reading about how to use the JUMP Client.

There is also a test instance running of the server at, feel free to use it as well: http://geoserver.itc.nl:8080/wps100/WebProcessingService

To have a look at the capabilities of the WPS instance, just try this: http://geoserver.itc.nl:8080/wps100/WebProcessingService?REQUEST=GetCapabilities&SERVICE=WPS

Installation hints

It has been reported by a user, that the WPS (end november release) also requires xalan.jar and xml-apis.jar. By now I cannot investigate the reason for that. It might be related to the GML2.X generator and a java, tomcat version dependency. So please keep this hint in mind.

Installation of the JUMP client

The client is used to test the service and to visualize the processing results. You can simply download the binaries of client on the wiki below or check the sources in the CVS. If you are doing the first, just unzip the wpsClient.zip. Please make sure, that the java.exe is available in the windows shell. Otherwise you have to add the JAVA_HOME/bin to the Path variable of windows. And now just run the JUMPworkbench.bat. That s it!

Running the service

Some test requests for your service can be found at http://YOURMASCHINE:PORT/WEBAPP_PATH/testDocs. A test client for plain request execution can be found at YOUR_WEBAPP_PATH/test.html.

Migrating from 0.4.0 to 1.0.0

If you have already a WPS running based on 0.4.0 the only thing you have to ensure is, that your processes are compliant to the java interfaces (AbstractAlgorithm), as they have changed slightly and to ensure that your DescribeProcess^? skeletons are compliant to 1.0.0. Please read the new specification carefully and use the existing DescribeProcess documents for the SimpleBufferAlgorithm.xml as a guideline (location: classes/org/n52/wps/server/algorithm).

WPS JUMP client GUI

The Client consists of two plug-ins. Both are located under the menu WPS Client:

• Add WPS: adds a WPS to the current session.
• Execute Process: Provides an overview over the logged WPS and their capabilities and a form to fill in the Parameter values.

How to use the JUMP client

You will execute the client via the JUMPWorkbench.bat. First get familiar with JUMP at all. Visit the JUMP website for further documentation.

Now let's start using the WPS Client (Access the menu point WPS Client). Log in a service with the Add WPS Plugin (just type the URL of the desired WPS in the text field of the dialog and hit OK), you should see a confirmation message in the lower left corner. Load some data, or just make your own data in JUMP. Now use the Execute Process Plugin to just have an overview of the logged services and their operations. Choose one service on the left tree structure and fill in the parameters in the Execute tab on the right. Then press Execute Process. The result of the execution will be displayed as a new layer in JUMP. If an error occures, you will be informed, on the status bar in the lower left. A detailed exception message should be available under the menu view -> Log...

Please mind, that this client is very experimental. I was really fighting with the Listeners to keep track of user changes in the Execute Process window. In some situations I was not successful.

For testing purposes you can also use our own instance under http://geoserver.itc.nl:8080/wps/WebProcessingService

Extending the service

The service can be extended very easily in two ways.

• Adding support for other XML schemas for ComplexValue and ComplexValueReference
• Adding more processes to the WPS

Adding support for other XML schemas

The service handles XML included in the ComplexValue and provided by the ComplexValueReference element in two ways: first to parse the XML data input to a FeatureCollection object and second to generate XML out of a FeatureCollection. It is important to note, that this approach can be easily extended to other object representations (e.g. for raster support), as the parsers and generators make use of the generic java.lang.Object class.

However, to use this feature and to enable the WPS for your own XML schema, you have just to extend the abstract class org.n52.wps.xml.AbstractXMLParser (XML to Object) or the abstract class org.n52.wps.xml.AbstractXMLGenerator (Object to XML). To make your own parsers/generators accessible to the service, just place the full java class name in the comma seperated list of the parameter registeredGenerators/registeredParsers in the wps.properties file, located in classes/org/n52/wps/server. At this time there is not such a easy plug in mechanism available to extend the WPSClient XMLHandling capabilities. You can inform the Client by writing some code in the org.n52.wps.client.jump.WPSClientExtension class. If there is a big need of the plug-in mechanism at the client side, I will add this feature instantly.

Adding your own processes to the server

The service provides also a plug-in mechanism to deploy your own algorithms to the service. The deployment of such algorithms consists of three steps:

• Extending the abstract class AbstractAlgorithm(recommended) or implementing the IAlgorithm interface
• Writing a ProcessDescription file according to the describeProcess.xsd of the WPS
• Deploying the class and the description file to the service

The algorithm has to implement the interface org.n52.wps.server.IAlgorithm, which consists of mainly two methods run() and getDescription(). If you are heading to have results in a fast way, I recommend to just extend the org.n52.wps.server.AbstractAlgorithm, which provides a facility to generate the description dynamically, so you only have to implement run().

Additionally you have to write a DescribeProcess file, which has to be located in the folder of the compiled algorithm with the same file name (for instance you have the algorithm org.myproject.MyAlgorithm than the file should be at org/myproject/MyAlgorithm.xml). The identifiers in the IOValueType have to correspond to the names used to retrieve and put objects out/in the passed java.util.HashMap of IAlgorithm.run(). Also the identifier of dataTypes in the LiteralData have to correspnd to the basicXMLDataTypes. (For an example of how to configure such algorithm please take a look at the org.n52.wps.server.SimpleBufferAlgorithm, which is shipped with the service package by default.)

After that you have to add your compiled algorithm with the XML file to the classes or lib (in case you made a jar file) folder of the wps webapp. Finally you have to add the full class name of your Algorithm to the registeredAlgorithm parameter in the wps.properties located at classes/org/n52/wps/server. Just reload your webapp context and you will have an updated service capabilities with a new entry in the ProcessOfferings Element and a new entry in the log file.

WPS client API"> Using the WPS client API

Both WPS clients use the same core API. The current source for the client API is available through the incubator CVS of sourceforge under the module WPSClient.

The WPS client API provides you some facilities to communicate with more than one WPS instance. It provides a sophisticated handling of ExceptionReports, which were returned by a WPS and the complete java object model to directly access and manipulate the WPS-Documents (ProcessDescriptions, Execute and ExecuteResponse).

The API consists mainly of two packages:

• org.n52.wps.client - client facilities
• org.n52.wps.xml - java representation of WPS Documents (this however is a commonly used package of all WPS components and can be found in the core sourceforge CVS under the CVS-module WPS).

The main entry to access the API is to instantiate a WPSClientSession object. This can be assigned to a session in case of using it for servlet or struts programming, or just initialize it at the start of your application. The WPSClientSession has the following methods:

• connect()
• disconnect()
• execute()
• describeProcess()

The WPSClientSession also provides a simple caching method, to just request the ProcessDescriptions of all available processes on connect of a service. This increases the speed, while working with the processes. Please consult javaDoc for further information.

WPS projects"> Further WPS projects

• Coupling 52n WPS with geoserver
• Udig WPS Client

Contact

If there are any problems or any other comments, please feel free to write an email to the Geoprocessing working group mailing list or to Theodor Foerster.

Download section

The current release of WPS plus the clients is available through http://sourceforge.net/project/showfiles.php?group_id=183259.

Other downloads (presentation, figures, etc.):

• IntroWPS.ppt: Introduction to the WPS - presentation

• implArchitecture.jpg: General overview of the implemented architecture

• 20060905_Foss4G_WPS_foerster.ppt: Slides of the presentation at Foss4G 2006 in Lausanne

• jump_wps_demo.swf: Short demo movie of the WPS client

• Sequence diagram for WPS (already implementing 1.0.0):

• WPS_overview_NLS_finland_September_2007.ppt: Presentation at the National Land Survey of Finland about WPS

• Setting_up_the_WPS_as_a_Multiple_Module_Eclipse_Project_svn.pdf: WPS development install

• WPS_Tutorial.pdf: WPS_Tutorial.pdf

• 52n_WPS_Grid_Computing_Hands_On.pdf: 52n_WPS_Grid_Computing_Hands_On.pdf