Wikipedia:Template limits

From Wikipedia, the free encyclopedia

MediaWiki has several parameters that limit the amount of data that can be included into a page. These limits concern data that is transcluded or substituted onto a page but not data directly in the source of the page itself. This page explains how and why these limits are applied, and how users can work within the limits.

The inclusion limits were put into effect on the English Wikipedia by Tim Starling on 14 August 2006. A new preprocessor was enabled in January, 2008, removing the "pre-expand include limit" and replacing it with a "preprocessor node count" limit.

Contents

[edit] Background

[edit] What is this about?

The MediaWiki software, which generates the HTML of a page from its wiki source, uses a type of parser (called a recursive descent parser) to deal with included data. This is done using a "preprocessor" which converts the wiki source into a data structure known as a tree, and then using this tree to produce HTML.

During the conversion process, the software uses several counters to track the complexity of the page that is being generated. When the parsing of a page begins, these counters are set to zero, but they are incremented at appropriate times during the parsing process, as described below. There are upper limits on these counters, and the parser does not allow these limits to be exceeded.

[edit] Why are there limits?

Very long or complicated pages are slow to parse. Not only is this an inconvenience for users, but it can also be used to mount a denial of service (DoS) attack on the servers, in which a page request forces the MediaWiki software to parse an unreasonably large quantity of data. The limits help to prevent this kind of attack, and ensure that pages are rendered in a reasonable time.

[edit] Working within the limits

When a page reaches the template limits, the most common solution is to make the templates shorter, using methods described below. If this isn't possible, it may be necessary to include more data directly in the page source, rather than transcluding it from templates.

[edit] When do problems arise?

The inclusion limits are most commonly reached on pages that use the same template many times, for example using one transclusion per row of a long table. Even though the amount of data that the template adds to the final page may be small, it is counted each time the template is used, and so the limit may be encountered sooner than expected. Pages that only include a few dozen templates are unlikely to exceed the inclusion limits, unless these templates themselves include a lot of data.

[edit] How can you find out?

Once the page body is processed, a HTML comment is added towards the end of the HTML code of the page with the final values of the various counters. For example, the page AIDS (on April 9, 2008) contains the following comment in its generated HTML source:

<!-- 
NewPP limit report
Preprocessor node count: 33368/1000000
Post-expand include size: 326336/2048000 bytes
Template argument size: 153666/2048000 bytes
Expensive parser function count: 0/500
-->

Because of the way the counters are increased, these sizes will always be less than the limits. If either of these sizes is close to the limit, then it is likely that some templates have not been expanded. Each occurrence of an unexpanded template is identified in the page body by an HTML comment containing an error message.

[edit] Using the new preprocessor

The new preprocessor (introduced January, 2008) produces the line "NewPP limit report" shown above. The old parsing system does not generate this line. On wikis where the new preprocessor is optional but not the default, for pages for which the old pre-expand include size limit is exceeded you can use a URL like http://en.wikipedia.org/w/index.php?title=Table_of_municipalities_of_the_Netherlands&timtest=newpp. Put {{npp}} at the top of the page, this shows a warning with a link if the wrong preprocessor is used.

To check which preprocessor is active, apart from checking the HTML source one can also use {{#if:{{#if:x|{{{2}}}|}}|new pp|old pp}}, here producing "new pp".

[edit] Inclusion limits

There are two limits on the volume of data on a page: the post-expand include size and the template argument size limits.

Whenever the parser is instructed by the source code of a page to expand a template (that is, to replace it by transclusion or substitution), the parser adds together the length of the HTML generated by the template and the post-expand counter of the page. If this sum is more than the post-expand limit, the initial template is not replaced and an error message is added as a comment in the output HTML. Otherwise the post-expand counter is increased to the new value, and parser continues. The post-expand counter is increased each time a template is expanded. Thus a template that occurs more than once in the page source can contribute more than once to the post-expand size of the page.

The new preprocessor does not expand template in non-executed branches of conditional parser functions. For example, in the code {{#if:yes|{{bar}}|{{foo}} }}, the template {{bar}} is expanded, but the template {{foo}} is not expanded. Nevertheless, it is possible for a template argument to contribute to the post-expand count even though it does not appear on the final output. For example, if the code {{#if:{{foo}}|yes|no}} is parsed, the length of the expanded version of template {{foo}} will be added to the post-expand counter, because that template must be expanded to decide which branch of the conditional should be selected.

A second counter, the "template argument size" counter, keeps track of the total length of template arguments that have been substituted.

[edit] Using comments, noinclude and onlyinclude

Only data that is actually included on the final HTML is counted towards the post-expand counter. The length of HTML comments in the wikitext (which are not reproduced in the HTML source produced) is not included in the post-expand counter. Code which is either inside a <noinclude> section or outside an <onlyinclude> section does not get expanded, so these sections do not contribute to the post-expand size.

Formerly (prior to January 2008), text inside <noinclude> tags, although not expanded for templates, did FORMERLY limit the total size of the page buffer. For that reason, any templates used many times per page, such as more than 200 per page, had used a separate doc subpage "/doc" linked within <noinclude> tags, such as by "{{template doc}}" to link the documentation but only expanded when viewed as a standalone, non-included page. Even a template's category-links and interwiki-links (for other-language Wikipedias) had formerly (pre-2008) caused pages to hit buffer limits for templates used over 200 times per page. For templates used just a few times per page, internal documentation or category-links and interwiki-links did not affect the page-buffer limit. In 2008, templates with noinclude-sections of 4kb, used over 200 times per page, were tested to verify (double check) that the larger noinclude-sections no longer exceeded the template-processing limits, as had occurred in November 2007 (using the prior MediaWiki language parser).

[edit] Nested transclusions

At present, due to the way that the MediaWiki software computes post-expand include sizes, nested transclusions are particularly expensive (see bugzilla 13260). If page A transcludes B and B does nothing but transclude C, then the size of C will be counted twice towards the post-expand include size on page A. This may cause some pages to run into template limits much sooner than expected.

This phenomenon applies also to parser functions: a transclusion inside a parser function will count twice towards the post-expand include size. This can sometimes be mitigated by moving transclusions outside of parser functions, e.g. by replacing

{{#if|test|{{template1}}|{{template2}} }}

with

{{ {{#if|test|template1|template2}} }}.

[edit] Preprocessor node count

A third limit measures the complexity the page rather than the volume of data. As the parser is expanding a page, it creates a data structure known as a tree that corresponds to the HTML structure of the page. Each node of this tree is counted towards the preproccessor node count. If this count is exceeded, the parser will abort parsing with the error "Node-count limit exceeded" visible in the generated HTML. The node count is not merely a function of the number of templates; each link, section header, or other HTML element will add to the node count.

[edit] Expensive parser function calls

There is a limit of 500 to the expensive parser function count, i.e. the number of calls of expensive parser functions, which are:

  • #ifexist - used to change the behavior of a template depending on whether a particular page exists. If the limit on this counter is exceeded, additional #ifexist calls will act as though the pages they query do not exist.
  • PAGESINCATEGORY

Pages that exceed this limit are automatically categorized into Category:Pages with too many expensive parser function calls.

[edit] Special:Expandtemplates

The extension ExpandTemplates enables a new special page at Special:ExpandTemplates. It is used to manually expand wikitext containing template calls. The calls are recursively expanded to their plain wiki-text. Special:ExpandTemplates also recursively expands all ParserFunction calls — something that cannot be achieved with substitution (see bug 2777). In some situations, it may be necessary to completely expand a template using the ExpandTemplates page and then manually copy the resulting source code into an article or template. This is most useful when expanding the template reduces its length, such as when a call of #switch containing a list of alternatives is "expanded" to a single alternative.

[edit] References

Languages