Wikipedia:Creating a bot

From Wikipedia, the free encyclopedia

This article may require cleanup to meet Wikipedia's quality standards.
Please discuss this issue on the talk page or replace this tag with a more specific message.
This article has been tagged since December 2006.
Bot pages: Bot policy | Currently running bots | Requests for bot approval | How to create a bot | Bot Approvals Group
Requests for work to be done by a bot | Bot owners' noticeboard | Bot status page | BAG Admin Tools

Robots or bots are automatic processes which interact with Wikipedia as though they were human editors. This article attempts to explain how to carry out the development of a bot for use on wikipedia.

Contents

[edit] Why would I need to create a bot?

Bots can automate tasks and perform them much faster than humans. If you have a simple task which you need to perform lots of times (an example might be to add a template to all pages in a category with 1000 pages) then this is a task better suited to a bot than a human.

[edit] Do I have to create it myself?

This help page is mainly geared towards those who have some prior programming experience but are unsure of how to apply this knowledge to creating a Wikipedia bot.

If you lack any previous programming experience at all, it might be better to have others develop a bot for you. If so, please make a post to the bot requests page instead. If you have no previous programming experience, you could become an additional operator for one of the existing semi-bots. If you want to have a stab at making your own bot anyway, feel free to read the rest of this article.

If you have no prior programming experience whatsoever, you should be warned that learning a programming language is a non-trivial task. However, it is not black magic and anyone can learn how to program with sufficient time and effort.

[edit] Overview of the process of developing a bot

Actually coding or writing a bot is only one part of developing a bot. You should generally follow the development cycle below. Failure to comply with this development cycle, particularly the sections on Wikipedia bot policy, may lead to your bot failing to be approved or being blocked from editing Wikipedia.

overview of Wikipedia bot development cycle
overview of Wikipedia bot development cycle

[edit] Software Elements Analysis:

  • The first task in creating a Wikipedia bot is extracting the requirements or coming up with an idea. If you don't have an idea of what to write a bot for, you could pick up ideas at requests for work to be done by a bot.
  • Make sure an existing bot isn't already doing what you think your bot should do. To see what tasks are already being performed by a bot, see the list of currently operating bots.

[edit] Specification:

  • Specification is the task of precisely describing the software to be written, possibly in a rigorous way. You should come up with a detailed proposal of what you want it to do. Try to discuss this proposal with some editors and refine it based on feedback. Even a great idea can be made better by incorporating ideas from other editors.
  • In the most basic form, your specified bot must meet the following criteria:
  • The bot is harmless (it must not make edits that could be considered vandalism)
  • The bot is useful (it provides a useful service more effectively than a human editor could), and
  • The bot does not waste server resources.

[edit] Software architecture:

  • Think about how you might create it and which programming language and tools you would use. Architecture is concerned with making sure the software system will meet the requirements of the product as well as ensuring that future requirements can be addressed. There are different types of bots and the main body of the article below will cover this technical side.

[edit] Implementation:

Implementation (or coding) involves reducing design to code. It may be the most obvious part of the software engineering job but it is not necessarily the largest portion. In the implementation stage you should:

  • Create a user page for your bot. Your bot's edits must not be made under your own account. Your bot will need its own account with its own username and password.
  • Add these details to your proposal and post it to requests for bot approval
  • Add the same information to the user page of the bot. You should also add a link to the approval page (whether approved or not) for each function. People will comment on your proposal and it will be either accepted or rejected.
  • Code your bot in your chosen programming language.

[edit] Testing:

If accepted, it would probably be put on a trial period during which it may be run to fine-tune it and iron out any bugs. You should test your bot widely and ensure that it works correctly. At the end of the trial period it would hopefully be accepted.

[edit] Documentation:

An important (and often overlooked) task is documenting the internal design of your bot for the purpose of future maintenance and enhancement. This is especially important if you are going to allow clones of your bot. Ideally, you should post up the source code of your bot on its userpage if you want others to be able to run clones of it. This code should be well documented for ease of use.

[edit] Software Training and Support:

You should be ready to field queries or objections to your bot on your user talk page.

[edit] Maintenance:

Maintaining and enhancing your bot to cope with newly discovered problems or new requirements can take far more time than the initial development of the software. Not only may it be necessary to add code that does not fit the original design but just determining how software works at some point after it is completed may require significant effort.

  • If you want to make a major functionality change to your bot in the future, you should request this as above using the requests for bot approval.

[edit] Considerations about type of bot to be developed

  • Is it to run server-side or client-side?
  • Is it to be manually assisted or fully automated?
  • Will its requests be logged?
  • Will it be reporting its actions to a human?

[edit] How does a wikipedia bot work?

[edit] Overview of operation

Just like a human editor, a wikipedia bot can read wikipedia content and then make calculated decisions and then edit wikipedia content. However, computers are better than humans at some tasks and worse at others. They are able to make multiple reads and writes to pages per second. They are not so good at making value judgements based on page contents as humans are.

Like a human editor, a typical bot would operate by making one or more GET requests to wikipedia via the HTTP protocol (just as a human editor does with their browser), making calculations on the data returned by the server, and making another request based on what they calculated at the previous step.

For example, if one bot would want to get the list of pages that link to a specific page, they would tell the server "I want Special:Whatlinkshere". The server would respond with generated html data. Then the bot would act depending on what the html contained.

[edit] APIs for bots

An API is an Application Programming Interface. It describes how a bot should interact with wikipedia.

The most common API is probably Screen scraping. Screenscraping involves requesting a wikipedia page, and then looking at the raw HTML (just like a human could do by clicking View->Source in most browsers) and then extracting certain values from the HTML. It is not recommended, as interface may change at any moment, and it's a huge server load. However currently this is the only way to get an HTML-formatted wiki page. Also currently scraping the edit page and making a http post submission based on the data from the edit form is the only way to perform edits.

You can pass action=render to index.php to reduce the amount of data to transfer and the dependence to the user interface. Other parameters of index.php may be useful: Manual:Parameters to index.php is a partial list of these parameters and their possible values.

Other APIs that may be of use:

  • MediaWiki API extension - this is the next generation API that may one day be expanded to include data posting, as well as various data requests. Currently a number of features have already been implemented, such as revision+content, log events, watchlist, and many other. Data is available in many different formats (JSON,XML,YAML,...). See it live. Features are being ported from the older Query API interface.
    Status: Partially complete engine feature, available on all Wikimedia projects.
  • Query API -- a multi-format API to query data directly from the Wiki-servers (Live)
    Status: Production. This is an extension enabled on all WikiMedia servers.
  • Special:Export feature (bulk export of xml formatted data); see Parameters_to_Special:Export for arguments;
    Status: Production. Built-in engine feature, available on all WikiMedia servers.
  • Raw page: passing action=raw or action=raw&templates=expand to index.php allows for a direct access to the source of pages

[edit] Common hurdles

[edit] Logging in as your bot when making edits

It will be necessary for an approved bot to be logged in under its user account before making edits. A bot is free to make read requests without logging in, but ideally it should log in for all activites. One can login by passing a login string via HTTP such as http://en.wikipedia.org/w/index.php?title=Special:Userlogin&action=submitlogin&type=login through the query string. Data that should be POSTed to log in is wpName=BOTUSERNAME&wpPassword=BOTPASSWORD&wpRemember=1&wpLoginattempt=Log+in

Once logged in, the bot will need to save Wikipedia cookies and make sure it passes these back when making edit requests.

[edit] Edit tokens

Wikipedia uses a system of edit tokens for making edits to wikipedia pages.

What this means is that it is not possible for your bot to pass a single POST request via HTTP to make a page edit. Just like a human editor, it is necessary for the bot to go through a number of stages:

  • Request the page using a string such as http://en.wikipedia.org/w/index.php?title=Wikipedia:Creating_a_bot&action=edit
  • The bot is returned an edit page with a text field. It is also returned (and this is important) an edit token as a hidden input form field in the html form tag. The edit token is of the approximate form b66655fjr7fd5drr3411ss23456s65eg\. Note that this edit token is unique to each edit request, may not be transferred for use on other pages, will only work for a single edit, and has a time limit after which it will not work.
  • The bot must then make a "write my edit to the page" request passing back the edit token it has been issued.

[edit] Why am I being returned an empty or near-empty edit token?

Your bot may be returned an empty edit token or the edit token "\". This indicates that the bot's login has failed and that its authentication has not been approved by the server. This may be due to it failing to store and return the correct cookies.

[edit] Edit conflicts

A high possibility is that your bot will get caught in an edit conflict, in which another user will have made an edit between your requesting an edit (and getting your edit token) and actually making the edit.

One reason for this is that it might not have the correct wpStarttime and wpEdittime variables in the data sent to the server.

To check for edit conflicts, sadly you will have to do it the deprecated way, checking for some kind of indication it was a conflict in the html.

[edit] General guidelines for running a bot

In addition to the official bot policy, which covers the main points to consider when developing your bot, there are a number of more general advisory points to consider when developing your bot.

[edit] Bot best practice

  • Try not to make more than 10 requests (read and write added together) per minute.
  • Try to run the bot only at low server load times, or throttle the read/write request rate down at busy server times.
  • Edit/write requests are more expensive in server time than read requests. Be edit-light.
  • Do not make multi-threaded requests. Wait for one server request to complete before beginning another
  • Back off on receiving errors from the server. Errors are an indication often of heavy server load. Try to respond by backing off and not repeatedly hammering the server with replacement requests.
  • Try to consolidate edits. One single large edit is better than 10 smaller ones. Try and write your code with this in mind.

[edit] Common bot features you should consider implementing

[edit] Timers

You don't want your bot to edit too fast. Timers are ways for bots to control how fast they edit. A common way to do this is to send your bot to "sleep" for a certain time after performing an action before doing anything else. In Perl, this is accomplished by the simple command sleep(10); where 10 is the number of seconds to sleep for.

[edit] Manual assistance

If your bot is doing anything that requires judgement or evaluation of context (e.g., correcting spelling) then you should consider making your bot manually-assisted. That is, not making edits without human confirmation.

[edit] Disabling the bot

It is good bot policy to have a feature to disable the bot's operation if it is requested. You should probably have the bot refuse to run if a message has been left on its talk page, on the assumption that the message may be a complaint against its activities. This can be checked by looking for the "You have new messages..." banner in the HTML for the edit form. Remember that if your bot goes bad, it is your responsibility to clean up after it!

[edit] Signature

Just like a human, if your bot makes edits to a talk page on wikipedia, it should sign its post with four tildes (~~~~). It should not sign any edits to text in the main namespace.

[edit] What technique and language should I use?

[edit] Semi-Bots or auxiliary software

In addition to true bots, there are a number of semi-bots known as auxiliary software available to anyone. Most of these take the form of enchanced web browsers with Wikipedia-specific functionality. The most popular one of these is AWB (Click here for a full list of these and other editing tools).

[edit] Bot clones

There are already a number of bots running on wikipedia. Many of these bots publish their source code and allow you to subscribe to them, download and operate a copy (clone) of them in order to perform useful tasks whilst browsing wikipedia.

[edit] Developing a new bot

Bots are essentially small computer applications or applets written in one or more programming languages. There are generally two types of bots: client-side and server-side. Client-side bots run on your local machine, and can only be run by you. Server-side bots are hosted online by your own machine or on a remote webserver, and (if you chose) can be initiated by others.

An overview is given below of various languages that can be used for writing a wikipedia bot. For each, a list is given of external articles advising how to get started programming in that language, as well as a list of existing libraries (sub-programs) you can use with your bot to prevent having to "reinvent the wheel" for basic functions.

[edit] Perl

Perl has a run-time compiler. This means that it is not necessary to compile builds of your code yourself as it is with other programming languages. Instead, you simply create your program using a text editor such as Microsoft Notepad. You then run the code by passing it to an interpreter. This can be located either on your own computer or on a remote computer (webserver). If located on a webserver, you can start your program running and interface with your program while it is running via the Common Gateway Interface from your browser. Perl is available for most operating systems, including Microsoft Windows (which most human editors use) and UNIX (which many webservers use). If your internet service provider provides you with webspace, the chances are good that you have access to a perl build on the webserver from which you can run your Perl programs.

An example of some Perl code:

open (INPUT, "< $filepageid") || &file_open_error("$filepageid");

while ($page_id=<INPUT>){
    chomp($page_id);
    push @page_ids, $page_id;

    $hits_upto_sth->execute($page_id, $start_date);
    $hits{upto}{$page_id} = $hits_upto_sth->fetchrow_array();

    $hits_daily_sth->execute($page_id, $today);
    $hits{today}{$page_id} = $hits_daily_sth->fetchrow_array();

    $hits_daily_sth->execute($page_id, $yesterday);
    $hits{yesterday}{$page_id} = $hits_daily_sth->fetchrow_array();

    $hits_range_sth->execute($page_id, $start_of_30_days,
                                       $end_of_30_days,);
    $hits{monthly}{$page_id} = $hits_range_sth->fetchrow_array();
}

Guides to getting started with Perl programming:

Libraries:

  • Anura -- Perl interface to MediaWiki using libwww-perl. Not recommended, as the current version does not check for edit conflicts.
  • WWW::Mediawiki::Client -- perl module and command line client
  • WWW::Wikipedia -- perl module for interfacing wikipedia
  • Perl Wikipedia ToolKit -- perl modules, parsing wikitext and extracting data
  • perlwikipedia � A fairly-complete Wikipedia bot framework written in Perl.

[edit] PHP

If you create a bot in the PHP programming language, you would probably want to merge it with a interactive web app.

You could also run it through a Command Line Interface, like the way you would do with perl

When you put it on a server, you will not need to run it as a CGI script, but just a regular old php page.

For example, to have someone sign up for a newsletter on Wikipedia, but don't force them to edit the page directly, you could make an interface on the web that goes and edits "User:Me/spam" and adds the user's name (this is not a good idea, just an example)

Example PHP code:

<?php
if(!isset($_POST['username'])) {
        echo "<form method=\"post\" action=\"{$_SERVER['PHP_SELF']}\">\n".
             "Please enter your username here: <input type=\"text\" name=\"username\" />\n".
             "<input type=\"submit\" value=\"submit\" />\n".
             "</form>\n";
} else {
        echo "Welcome {$_POST['username']}!";
}
?>

[edit] Python

Introduction to use

An example of some Python code:

def addt5(x):
    return x+5

def dotwrite(ast):
    nodename = getNodeName()
    label=symbol.sym_name.get(int(ast[0]),ast[0])
    print ' %s [label="%s' % (nodename, label) ,
    if isinstance(ast[1], str):
        if ast[1].strip():
            print '= %s "];' % ast[1]
        else:
            print '"]'
    else:
        print '"]'
        children = []
        for n, child in eunmerate(ast[1:]):
            children.append(dotwrite(child))
        print ' %s -> {' % nodename,
        for name in children:
            print '%s' % name,

Getting started with Python:

Libraries:

[edit] Microsoft .NET

Generally developed in Microsoft Visual Studio development environment, languages include C++/CLI, J#, JScript .NET, Microsoft Visual C#, Visual Basic .NET, IronPython, and Windows PowerShell

Example of code:

 // Managed extensions for C++
 #using <mscorlib.dll>
 using namespace System::Collections;
 __gc class referencetype
 {
 protected:
     String* stringVar;
     int intArr __gc[];
     ArrayList* doubleList;
 public:
     referencetype(String* str, int* pointer, int number) // Which one is managed?
     {
         doubleList = new ArrayList();
         System::Console::WriteLine(str->Trim() + number.ToString());
     }
 };

Getting started:

  • Add links here!

Libraries:

[edit] AppleScript

Introduction to use

Example of code:

if testPrime is in {2, -2} then
        set testPrime to testPrime as string
        display dialog testPrime & " is a prime number." ¬
                buttons {"OK"} default button "OK"

else if testPrime is in {1, 0, -1} then
        set testPrime to testPrime as string
        display dialog testPrime & " is not a prime number." ¬
                buttons {"OK"} default button "OK"
end if

Getting started with AppleScript

Libraries:

Retrieved from "http://en.wikipedia.org../../../c/r/e/Wikipedia%7ECreating_a_bot_7f22.html"