Keyword Services Platform
From Wikipedia, the free encyclopedia
The Keyword Services Platform (KSP), originally conceived by ZhaoHui Tang, Dylan Huang and Wayne Guan at Microsoft adCenter Labs in May 2006 aims to provide a core set of data and technology to empower search engine marketing efforts and keyword research efforts. The KSP uniquely delivers a standardized set of keyword technologies in an easily consumable manner through a web services model accessible via both API and an excel add-in (see the adCenter Add-in for Excel). Many of the keyword technologies delivered in the KSP come from adCenter Labs. Keyword Services Platform
Contents |
[edit] Architecture
At the core of the KSP, there is a set of Shared Services, which includes a crawler, in-memory data structures, word stemming, etc. These shared services are used by different providers and executed by stored procedures. The Service Container contains a set of providers that provides various keyword technologies. There is a Provider Plug-In Framework that allows researchers to plug in new algorithms or mining models inside the Service Container. Each provider brings to the KSP architecture a specific keyword technology, for instance, keyword association, keyword extraction, or keyword classification. There is also a Server Object Model, which allows developers to leverage different services. A Stored Procedure (sproc) is another important feature of the KSP; it consolidates and centralizes logic behind applications and makes selected sets of procedures available to users. Developers may use .NET programming languages to write procedures that combine the usage of different providers, or code additional business logic processing based on the output from a provider. The Services Container feature hosts all the Providers and sprocs, allowing them to be executed in parallel. The Security component handles permissions of calling different provider methods, as well as the sproc implementation. The Keyword API layer provides a set of standard web services for various keyword tasks. These services are based on Windows Communication Foundations and can be consumed by client applications (for example the Excel Add-in developed for the KSP) or mash-ups.
[edit] Keyword API
The KSP has defined a set of APIs for each class of keyword services. These services include keyword extraction (ITermExtraction), keyword categorization (ITermCategorization), keyword suggestion (ITermSuggestion), keyword forecast (ITermForecast), keyword monetization (ITermMonetization) and more. The API defines the signature of each web service.
[edit] Keyword API Samples
Several sample API calls follow for reference. To find the five (5) most closely related keywords to the keyword “BMW” you might use the following query: GetTermSuggestion(“BMW”, 5). The query result is in the following table, and by default, sorted by confidence:
To view not only the suggested five (5) Terms, but also the confidence Score it is possible to add the parameter ‘True’ to indicate interest in including statistics: GetTermSuggestion(“BMW”, 5, true). The query result is displayed in the following table along with a Score and a Support column; similar to Data Mining Extension in SQL (DMX) syntax. The Score represents the confidence, or probability. The Support represents the number of cases supporting the rule in the training dataset.
If you want to return only those Terms with a high confidence Score it is possible to set a filter on the Score column with the following query: GetTermSuggestion(“BMW”, 5, true, “Score>0.8”). The query result is displayed in the following table. It only returns four (4) rows as these are the only Terms with the qualifying Score.
In the case where the input is a table of terms possibly including thousands of keywords it is possible to use the batch query syntax: GetTermSuggestion(myInputTermTable, 2). The query result is in the following table. For each Original Term in myInputTermTable, it returns the two (2) most relevant Terms.
The samples reviewed so far have explored the web services of ITermSuggestion exclusively; there are many more services available for different keyword related tasks.
The following query tells you the demographic distribution of the term “Minivan” using ITermDemographics web services: GetTermDemographics(“minvan”). It returns the following table:
The interface ITermMonetization provides a set of web services to query keywords’ monetization values specific to paid search. The following query returns the KPIs for the keyword “Online bank” based on last week’s paid search data, in the third position of sponsored listings: GetTermKPIs("online bank", TimeInterval.LastWeek, 3).
The result of the query is shown below, containing the input keyword, the number of Clicks in the sponsored link for “Online bank”, overall Impressions for the keyword, Position, average Click Thru Rate, and average Cost per Click.
The following query extracts the most relevant keywords from a web page using the ITermExtraction web service: GetTermExtraction(“autos.msn.com”, 8, true). It returns the following table with the column Score representing the relevance of the extracted keyword to the page content, while Support means the number of occurrences of the keyword on the page.
[edit] Sample Code to Consumer KSP API Through KSP.net
A sample of code to connect to the KSP server and call the term forecast web services of the KeywordForecastProvider follows. KSP connection is enabled through few lines of code, enabling easy access to the web services.
using (KeywordServer server = new KeywordServer("https://ksp.microsoft.com"))
{
server.UserName = "username";
server.Password = "********";
ITermForecast provider = null;
try
{
server.Open();
// context can be set if needed. It will remain during the following calls
provider = server.GetProviderByImplementation<ITermForecast>("Microsoft.adCenterLabs.Providers.KeywordForecastProvider");
if (provider != null)
{
// single mode API
DataTable result = provider.GetTermForecast(term, -5, 3);
DisplayResults(result);
// batch mode API
result = provider.GetTermForecast(terms, -5, 3);
DisplayResults(result);
}
}
catch (FaultException)
{
// handle fault returned from calling the proxy method
}
catch (CommunicationException)
{
// hanlde lost network connection error
}
catch (TimeoutException)
{
// handle time out error
}
finally
{
if (provider != null)
server.ReleaseService(provider);
}
}
[edit] Providers
Each KSP provider supplies a specific kind of intelligent keyword technology and implements one class of keyword interface (ITermSuggestion, ITermForecast, ITermExtraction, etc…). The API defines the signature of each web service, and the format of the returned data. The Keyword Service Provider is a server-side object encapsulating a particular implementation of a keyword technology and exposes its functionalities through service contracts in the Windows Communication Foundation (WCF); the WCF is Microsoft’s unified programming model for building service-oriented applications. It enables developers to build secure, reliable, transacted solutions that integrate across platforms and interoperate with existing investments. To enable seamless integration of a Keyword Service Provider into the KSP, and correspondingly seamless integration with third party tools and applications a number of conditions have to be met by providers:
- Custom configuration settings stored in configuration files, not in the code;
- Standard .NET tracing and message logging used to enable service monitoring and diagnostics;
- Standard Windows Management Interface performance counters implemented for performance monitoring; and
- Document service contracts included using service description language for better understanding and testing of the Keyword Service Provider.
These goals help to simplify the task of developing well integrated management applications, allowing partners to produce best-of-breed, enterprise-scalable management solutions.
[edit] Stored Procedures
The KSP server hosts the Common Language Runtime compilation (CLR). Developers can write stored procedures using any .NET language and these procedures are executed server-side on the KSP. Similar to a database sproc, a KSP sproc is designed to enable developers to code numerous types of business logic on the server side after retrieving result data from providers. KSP Stored Procedures require no configuration management and no setup requirements. Two types of stored procedures are supported: Managed Assembly Stored Procedure (MASP) and Common Language Runtime Stored Procedure (CLRSP). A MASP consists of a compiled .NET assembly containing a public interface exposed through the KSP as well as any dependant files. Once the MASP is uploaded to the KSP through its management interface, it becomes callable by KSP client programs. A CLRSP consists of a source file written in one of the supported CLR programming languages (C#, VB.Net, managed C++, and others). The functionalities of the CLRSPs are exposed through a public interface defined in the source file. Once the CLRSP is deployed to KSP through its management interface, it is compiled on demand by KSP and becomes callable by KSP client programs. Compared to database stored procedures, KSP stored procedures and KSP sprocs are both object-oriented. A sproc may contain a set of related functions, even same name functions with different signatures.
[edit] Server Object Model and Shared Services
KSP Server Object Models and Shared Services enable KSP Service Providers and stored procedure developers to access server-side objects and functionalities easily and consistently. The object model consists of the following four collections:
- Service Providers: This collection enables callers to access server side service provider objects by name, service provider implementation interface, and/or class name. Once callers obtain the service provider object, all of the functionalities of the service provider are accessible through its public interface.
- Stored Procedures: This collection enables callers to access server-side stored procedure objects by name, stored procedure implementation interface, and/or class name. Once callers obtain the stored procedure object, all of the functionalities of the stored procedure are accessible through its public interface.
- Services: This collection enables callers to access server-side shared services by name, by service implementation interface, or class name. Once callers obtain the shared service object, all of the functionalities of the shared service provider are accessible through its public interface.
[edit] Cloud Server Model
The adCenter KSP server farm provides a scalable platform for keyword technologies. Each server in the farm can have different configuration to suit a variety of service providers and stored procedures. A Dynamic Service Load Balancer server, “Cloud”, is the hub of the KSP server farm, see [cloud computing]. When a KSP server is added to the server farm via the Cloud Server, all available keyword service providers and stored procedures are dynamically discovered and registered with the Cloud Server. Any changes in the availability of the KSP server (as well as all its running service providers and stored procedures) are automatically discovered and registered with the Cloud Server.
The Cloud Server distributes accesses to services running on a KSP server farm through its load balancer provider. The default implementation of the load balancer provider uses a round robin approach. Over time, the Cloud Server builds up usage patterns and statistics of various service providers and stored procedures running on each KSP server in the farm. This information will be used by the Cloud Server to determine how to automatically deploy additional service providers and stored procedures. For example, if the ‘Keyword Forecast’ provider is being used heavily in the server farm and the providers running on machine A are used lightly, the Cloud Server will automatically deploy the ‘Keyword Forecast’ provider to machine A and route requests to machine A to balance the load for the ‘Keyword Forecast’ provider.
When a client application calls a service provider or stored procedure through the Cloud Server, a KSP server with a matching service provider or stored procedure is selected by the load balancer provider in the Cloud Server and the request is routed to the appropriate KSP server. If a server, service provider, or stored procedure in the KSP server farm is offline it will be taken out of rotation by the load balancer automatically.
[edit] Data Mart
Many KSP providers require real-time database access. The database may contain a list of reference keywords, their corresponding traffic, latest click-through data, and mining model contents. This data is updated on a regular basis based on the provider’s requirements.
[edit] Technology Transfer
KSP’s architecture permits agile development and quick technology transfer by providing a platform for researchers to rapidly ship their research results to a live system. The API defines the standard contract between the research models and developers. Researchers simply need to implement providers and deploy the providers into the selected set of KSP Cloud Server machines. The scope is limited, and thus very easy to use for live testing. Once the provider is live tested and proven, KSP can switch to the default provider without any changes at the application side. This infrastructure enables researchers at Microsoft and other academic settings to speed up innovation in keyword technology and deploy the latest research results to KSP consumers.
[edit] KSP Data Access Enabled via Excel 2007
Capitalizing on the web services model of the KSP and enabling users to consume the data directly via Excel rather than through the API, Microsoft released an add-in: the Microsoft adCenter Add-in for Excel. This tool taps much of the keyword technology available in the API directly in Excel; essentially it is an example of the type of mash-up and creative use of data that can be found associated with the KSP. The add-in delivers features such as keyword extraction, suggestion, forecasting, monetization and more; utilizing the same data source as the KSP API.
[edit] KSP API Access
KSP API beta access is available for researchers and developers upon request from the Keyword Services Platform feedback link.
[edit] Applications of the KSP
The KSP incorporates keyword technologies from adCenter Labs and other Microsoft Research groups. Keyword APIs can be consumed by third party business applications from paid search, content ads, behavioral targeting, presale business intelligence apps, and more. A few usage scenarios follow:
- Campaign Creation and Management Tools –
- The Keyword Association provider can help advertisers generate a set of the most relevant keywords for a campaign leading to more efficient planning and improved ROI.
- The Keyword Forecasting Provider can help advertisers to understand traffic history and trends, and eventually help to manage an integrated campaign budget that makes seasonal allowances.
- The Keyword Extraction provider can extract the important keywords on a publisher’s web page; helping to identify what ads should be served for that page and facilitating landing page analysis.
- Behavioral Targeting and Display Ads –
- The Keyword Demographic and Geographic Distribution can help advertisers understand various customer segments and their keyword usage patterns leading to more effectively targeted advertising and a decreased overall spend.
- Keyword Association can help to expand existing customer segments to include other customers with similar interests based on language patterns.
[edit] References
- Microsoft Announces New Keyword Platform at SIS
- adCenter Add-in Released Review of keyword technologies from the KSP
[edit] Further Reading
- Wen-tau Yih, Joshua Goodman, Vitor R. Carvalho: Finding advertising keywords on web pages. WWW 2006: 213-222
- Ning Liu, Shuzhen Nong, Jun Yan, Benyu Zhang, Zheng Chen, Ying Li: Similarity of Temporal Query Logs Based on ARIMA Model. ICDM 2006: 975-979
- Honghua (Kathy) Dai, Lingzhi Zhao, Zaiqing Nie, Ji-Rong Wen, Lee Wang, Ying Li: Detecting online commercial intention (OCI). WWW 2006: 829-837
- Lee Wang, Chuang Wang, Xing Xie, Josh Forman, Yansheng Lu, Wei-Ying Ma, Ying Li: Detecting dominant locations from search queries. SIGIR 2005: 424-431
- ZhaoHui Tang, Jamie Maclennan, Pyungchul (Peter) Kim: Building data mining solutions with OLE DB for DM and XML for analysis. SIGMOD Record 34(2): 80-85 (2005)
- ZhaoHui Tang, Jamie Maclennan: Data Mining with SQL Server 2008, Wiley, 2008.
