Website Parse Template

Website Parse Template
Filename extension	`.icdl`
Developed by	OMFICA
Type of format	Website Parse Template
Container for	ICDL Crawling
Extended from	XML
Website	WPT

Website Parse Template (WPT) is an XML-based open format which provides HTML structure description of website pages. WPT format allows web crawlers to generate Semantic Web’s RDFs for web pages. WPT is compatible with existing Semantic Web concepts defined by W3C (RDF and OWL) and UNL specifications.

WPT Syntax

Website Parse Template's Visual Representation

Website Parse Template consists of following sections:

Ontology, where publisher defines concepts and relations which are used in the website.
Templates, where publisher provides templates for groups of web pages which are similar by their content category and structure. Publisher provides the HTML elements’ XPath or TagIDs and links with website Ontology concepts.
URLa, where publisher provides URL Patterns which collect the group of web pages linking them to "Parse Template". In the URLa section publisher can separate form URLs the part as a concept and link to website Ontology.

Website Parse Template begins with opening <icdl> tag and ends with closing </icdl> tag. Single Website Parse Template is referred to the same host, while single host may have several Website Parse Templates describing its HTML structure. It is required to specify the host for Website Parse Template at the beginning in <icdl> tag:

<icdl host="http://music.yahoo.com"> 
. . . . . . . . . . . . . . . . . . .
</icdl>

WPT ontology

Ontology section contains enumeration and definition of all concepts used in website. Listed concepts must be enclosed within <ontology> </ontology> tags. It is required to specify the ontology name (any rational string) and indicate supported language ("icdl:ontology", "owl" or "unl:uws") which is used to specify the concepts.

Example 1. Concepts used in Yahoo! Music for "artist" object

<ontology name="general" language="icdl:ontology">  
     <concept name="Qbye Music">
          <inherit concept="person"></inherit>
          <has object="name"></has>
          <has object="album"></has>
          <has object="track"></has>
          <has object="image"></has>
          <has object="bio"></has>
          <has object="video"></has>
          <has object="id"></has>
          <has object="fullname"></has>
     </concept>
     <concept name="Logo"></concept>
     <concept name="Menu"></concept>
     <concept name="Advertisement"> </concept>
</ontology>

Each concept's definition should start with <concept> tag and ends with </concept> tag. <inherit> tag shows inheritance relations and <has> tag shows attributable relations between two concepts. Either of defined concepts has default attribute - object identifier (id) to be used by web crawlers to co-ordinate the same object's attributes used in different pages of the same website.

Website Parse Template foresees several predefined concepts that are general for all kind of websites:

“Menu” - navigation bar/menu
“Logo” - design element/logo
“Content” - element that contains main textual content of the page
“Advertisement” – advertisement/banner
“External Link” – element that contains external links

WPT templates

Templates section contains number of templates for groups of similarly structured web pages. Either of those templates refers to a single group of similarly structured web pages. HTML elements’ XPath references or TagIDs are used for linking structured content with defined concepts. The template description starts with opening <template> tags and ends with closing </template> tag. In <template> tag it is required to specify template name and language used for templates description. As a template name can be chosen any string, but for the language it is necessary to indicate supported language type, e.g. "icdl:template", "rdf" or "unl:expression".

Example 2. Simple template for single artist page on Yahoo! Music

<template name="Artist page on Yahoo! Music" language="icdl:template">
     <html_tag tagid="yent-uhdr" content="Menu"/>
     <html_tag xpath="/html/body/div[2]/div/div/div[3]/div/a/span" content="Logo"/>
     <html_tag xpath="/html/body/div/div" content="Advertisement"/>
     <html_tag xpath="/html/body/div[3]/table/tbody/tr/td[2]/div/h1" content="artist.name"/>
     <html_tag tagid="art_img" content="artist.image"/>
     <html_tag tagid="biography" content="artist.bio" reference="Artist Bio"/>
     <html_tag xpath="/html/body/div[3]/table/tbody/tr/td[2]/table/tbody/tr[22]" content="artist.album"/>
     <html_tag xpath="/html/body/div[3]/table/tbody/tr/td[2]/table/tbody/tr[10]" content="artist.track"/>
     <html_tag xpath="/html/body/div[3]/table/tbody/tr/td[2]/table/tbody/tr[13]" content="artist.video"/>
</template>

The web page may contain structured repeatable content (<repeatable block>) included in one main HTML element (<container>) that are specified as follows:

Example 3. Repeatable content representation

<template name="Artist page on Yahoo! Music" language="icdl:template">
    . . . . . . . . . . . . . . . 
     <container container_xpath="/html/body/div[3]/table/tbody/tr[10]/td/table"> 
         <repeatable_block block_xpath="/html/body/div[3]/table/tbody/tr[10]/td/table/tbody/tr/td[4]">
         <html_tag xpath="/html/body/div[3]/table/tbody/tr[10]/td/table/tbody/tr/td[4]" content="artist.track"/>
         </repeatable_block>
     </container>
    . . . . . . . . . . . . . . .
</template>

In case of specified complex HTML element is already described by another template the <reference>tag can be used to point to that template block. It makes possible to create hierarchic relations between WPT templates so that web crawlers can use specified reference(s) to identify the same object in different pages of a given website.

Example 4. Hierarchic relations between WPT Templates

<template name="Artist page on Yahoo! Music" language="icdl:template">
     . . . . . . . . . . . . . . .
     <html_tag tagid="biography" content="artist.bio" reference="Artist Bio"/>
     . . . . . . . . . . . . . . .
</template>
<template name="Artist Bio" language="icdl:template">
     <html_tag xpath="/html/body/div[3]/table/tbody/tr/td[2]/div/h1" content="artist.name"/>
     <html_tag tagid="art_img" content="artist.image"/>
     <html_tag xpath="/html/body/div[3]/table/tbody/tr/td[2]/table/tbody/tr/td" content="artist.bio"/>
</template>

URLs section

This section defines the URLs/URL patterns that are corresponding to groups of similarly structured web pages described in Templates section. In accordance with Templates section URLs section also may consist of several blocks and either of those blocks should start with <urls> tag and ends with </urls> tag.

Example 5. URLs/URL patterns

<urls name="Artist page on Yahoo! Music" template="Artist page on Yahoo! Music">
     <url url="http://music.yahoo.com/ar-8206256---Amy-Winehouse"/>
     <url url="http://music.yahoo.com/ar-([artist.id[0-9]*])---(artist.full name[[A-Z,a-z,-,0-9]*])"/>
</urls>

As a URLs block name can be chosen any string, but for the template it is necessary to indicate certain template name described in previous section. The URL pattern provided in Example 5 also include the represented real URL. RegExp specifications are used for URL patterns descriptions. The concepts necessary for URL pattern definition (such as "id" and "fullname") are to be defined previously in Ontology section.

External links

OMFICA. Archived from the original on August 6, 2008. Retrieved March 1, 2015.

Internet search

Types	Web search engine (List) Metasearch engine Collaborative search engine Human flesh search engine

Tools	Local search Vertical search Search engine marketing Search engine optimization Search oriented architecture Selection-based search Social search Document retrieval Text mining Web crawler Multisearch Federated search Search aggregator Index/Web indexing Focused crawler Spider trap Robots exclusion standard Distributed web crawling Web archiving Website mirroring software Web search query Voice search Natural language search engine Web query classification

Applications	Image search Video search engine Enterprise search Semantic search

Protocols and standards	Z39.50 Search/Retrieve Web Service Search/Retrieve via URL OpenSearch Representational State Transfer Website Parse Template Wide area information server

See also	Search engine Desktop search Online search

Template:Web Crawler

This article is issued from Wikipedia - version of the Thursday, April 09, 2015. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.