Welcome to SWSRAT,
a Semantic Web Service Relevance Assessment Tool.

What is it all about?
SWSRAT aims to continuously improve the OWLS-TC and SAWSDL-TC test collections.

Both these test collections provide semantic web service descriptions for service offers and some sample service requests. Additionally human relevance assessments identifying relevant service offers for the given sample requests are given.

SWSRAT aims to expand and improve these relevance judgements on service request/offer pairs. It provides a simple web interface to assess the relevance of service offers to service requests given in the test collections.

To reflect the diversity of human relevance judgements two different graded relevance scales are supported (i.e. a 4 grade scale and a 3 grade scale).
Historically relevance is treated as a binary attribute in information retrieval. So to be compatible with binary evaluation measures, the graded relevance assessments can be projected to binary relevance sets. Both, graded ratings as well as binary relevance sets, can be exported in an xml file for external evaluation.
How does it work?
SWSRAT offers two ways to assess service request/service offer pairs. Either you select a pool and a rating goal. Then the tool will automatically select service request/offer pairs to be assessed by you according to your rating goal until the rating goal is satisfied (i.e. you have rated all pairs specified by the goal). This automated rating is the preferable method to assess substantial amounts of request/offer pairs in limited time.
On the other hand, you can browse through all service request/offer pairs (of a certain pool) and manually select the pair to be assessed. This method is suggested to assess small numbers of selected request/offer pairs or to reevaluate assessments you made at an earlier point.
How can I use it?
You need to register before you can contribute relevance assessments or extract ratings. Then you can log in with your email address and password.
When you are logged in, you can submit relevance assessments and extract your own ratings as well as those of other assessors and use them freely on your own, e.g. to evaluate your matchmaker.

If you encounter any problems, please make sure your browser is up-to-date, Cookies are allowed and JavaScript is enabled.

For the evaluation of your own matchmaker you may also want to download the actual test collections OWLS-TC and SAWSDL-TC from SemWebCentral.org .