Wikipedia talk:Category intersection

From Wikipedia, the free encyclopedia

Contents

[edit] About this proposal

This proposal was started by User:Rick Block and User:SamuelWantman. The initial discussions leading to the proposal are in Archive 1. There is also discussion about the different options here.

Please leave comments about the proposal on this page. Thank you for your input.

[edit] Laurence Fishburne

I haven't read all of this, but shouldn't the Laurence Fishburne article in your example end up with categories such as "MASH", and not "MASH Actors", "Miami Vice", and not "Miami Vice Actors"? If someone wants to know the actors in Miami Vice, they should be able to check the intersection of those two. Just a thought. --Kbdank71 11:01, 30 August 2006 (UTC)

I think the counter-example here is an actor who later becomes associated with Miami Vice, but not in an acting role. For example, if a famous actor produced or directed Miami Vice. Looking at the "American people" example, you might similarly ask why you can't intersect "American" and "people". I think these sort of cases will make it difficult to pin down the concept of a primary category, but I do like this proposal. I just haven't had time to read it again and decide which bits I like best. Carcharoth 11:09, 30 August 2006 (UTC)
It is possible to have the category "Actors" and the category "MASH". People would just need to realize that the intersection is not necessarily "people who acted in MASH", but is instead "People involved with MASH who also are Actors". I suspect this will get to be a commonly understood distinction. CFD will probably have discussions about whether a category should be replaced with an intersection or vice versa. -- Samuel Wantman 19:54, 30 August 2006 (UTC)

[edit] Already existing extensions or tools

Some of these may have been mentioned already, but just in case....

Existing software that's good to be aware of:

  1. The intersection mediawiki extension - aka DynamicPageList2.
    This extension now (February 2007) has some new features. See the new homepage at mediawiki.org and the special DPL demo website.
    I (mw:User:Algorithmix) continued the work of Unendlich, Dangerman, Haykinson and Amgine. I am interested in making DynamicPageList even more powerful and eventually positioning it as an official part of Mediawiki software.
    -- mw:User:Algorithmix 14:03, 17 February 2007
    I don't have experience with this... you can try pinging the contributors (n:en:User:IlyaHaykinson, n:en:User:Amgine, w:de:Benutzer:Unendlich, m:User:Dangerman) and seeing if they have comments about real-world use (eg. is it slow? what sort of categorization schemes do they find work well?)
    Pong. Two comments:
    1. Real world, server load and speed. Logical operations on categories generate complex database requests. We tried to keep the SQL queries to a minimum in DPL2 and I am not aware of any complaint on speed or load. But it doesn't mean there may be issues with big wiki communities like Wikipedia. However, for comparison, the DPL (not yet DPL2...but they have the same load and speed concerns) seems to be used successfully on Wikinews (see n:Wikinews:DynamicPageList). For other real-world info, there is a stub list of DPL2-powered wikis on the extension page. I hope this helps a little bit.
    2. User interface. I don't think the user interface - or I'd say the way users would do category intersection as mentioned in the proposal - is very flexible, the separate Intersection space in particular. One day, users may want the same thing for Union (you mix up articles from 2 categories together - different from showing articles in category 1, and then in category 2) or Negation (articles not in a category) or all operations together. That's why, on the long term, offering the user syntax of Category_math_feature seems like a more logical and flexible solution. --(m:User:Dangerman) Dangerville 13:21, 3 September 2006 (UTC)
    There are two distinct use cases we're attempting to address and I think a DPL (or DPL2) technique only addresses one of them. The two are 1) where there is an "intersection category" a user creates and adds to the category "hierarchy" (which DPL would address) and 2) where a user effectively "browses" (by some mechanism) to a dynamically created "intersection" that no one has previously set up in any way. CatScan is more like the latter case. To make it more general, the new namespace could have some name other than "intersection" (perhaps "operation" or something). The real benefit of using a namespace is that it comes with a URL and wikilink syntax almost automatically. -- Rick Block (talk) 19:46, 4 September 2006 (UTC)
  2. Duesentrieb's CatScan on the toolserver
    Lets you experiment with category intersection using the current categorization scheme. Allows searching to some limited depth, but it can get pretty slow
  3. Semantic MediaWiki extension
    The ultimate point of where this proposal is going... lets you answer questions like "list all female mayors who were in office after 1950". If you're serious about wanting complicated UIs and such, you may want to look into when it might be feasible for it to be used on something like Wikipedia. (mp3 from wikimania 2006)
    They also mentioned that their queries can be pretty slow.
This proposal calls for reworking most of the categorization system. I wanted to note that at least some reworking is required to make categories useful for automated tools, because the current categorization system is set up to only support human use. Automated use of current categories (eg. CatScan) can work up to a point, but currently, if you try to search as far as possible in an effort to get complete results, you get increasingly unrelated results the deeper you go.
See also: Category relationships and CatLadder --Interiot 12:08, 30 August 2006 (UTC)
Thanks for the link to the Category Ladder tool. I'd been looking for something like that for ages! I was having to manually construct category trees, and this does it for you! Carcharoth 19:37, 30 August 2006 (UTC)
Yes, this proposal calls for reworking the categorization system. The German language Wikipedia has already repopulated primary categories and has removed intersection categories. I think it would be a difficult battle to get that to happen here before category intersection is implemented. Once implemented, more and people will climb on board, and categories can be quickly depopulated, repopulated and redefined. -- Samuel Wantman 20:07, 30 August 2006 (UTC)

[edit] This is very interesting

I need time to think about it, but it looks like a great extension to the category functionality (with possible caveats of "people could be resistant to change" and "I wonder what the processor load would be?"). Syrthiss 12:13, 30 August 2006 (UTC)

Categories are currently implemented with a database, and the "large category" performance issue was addressed by limiting the search result. The intersection search results would be similarly limited, so I think the question boils down to how efficiently the underlying database can do intersection queries and how much such a feature might be used. In a real time sense, I think an intersection query could be quite fast (google seems to do OK, for example). It might be interesting to know what percentage of database queries are currently category related. I don't know, but I'd guess not much (I'd guess there are way more history, watchlist, and previous version related queries than category queries). -- Rick Block (talk) 13:55, 30 August 2006 (UTC)
  • I agree this is interesting. I'll think a bit on the specifics, too. >Radiant< 21:05, 4 September 2006 (UTC)

[edit] Support

I will have to check the implementation, but I wholeheartedly support this concept. Subcategorization constitutes, I dare say, 50% of the controversy at WP:CFD ... and boolean categories constitute the other 50%. But more importantly than making CFD easier, this will actually make categories useful to researchers -- which must be the guiding purpose of any project on Wikipedia. Right now, our category system is barely an improvement over traditional text searches. --M@rēino 14:03, 30 August 2006 (UTC)

I add my full support! -- SatyrTN (talk | contribs) 15:28, 30 August 2006 (UTC)
I want this as soon as possible. please implement Chendy 21:38, 2 November 2007 (UTC)

[edit] Query on intersecting

You have in your Laurence Fishbourne example the categories "People from Georgia (US State)" and "People from Augusta, Georgia". Now forgive me if my US geography is wrong, but isn't it possible to extrapolate the former from the latter? If we put someone into "People from Augusta, Georgia", shouldn't it be possible to make the database populate Laurence Fishbourne into "People from Georgia (US State)", and even "American people"?

Other than that, yes, it looks like a good idea, and would solve a lot of issues with categorisation. Be interested in a developer's comments. Well done! Hiding Talk 21:55, 30 August 2006 (UTC)

The issue about these categories, "People from Georgia (US State)" and "People from Augusta, Georgia" is how they are defined, and how they will be used. We're assuming that people will want to find "Actors from Georgia" or "Politicians from Augusta" etc... So you need to have an intersection that arrives with the correct population of articles. The intersection will depend upon how the categories are defined. Right now, the subcategories of "American people" have people who are American citizens, but the "People from Georgia" are not necessarily citizens. Politicians from Augusta may be people who are neither citizens nor people originally from Georgia. The distinctions between these categories can and should be made clearer, and they might change when category intersection is implemented. It would be possible to categorize any person closely associated with a location with the location. So ex-mayor Willie Brown could be put in category:San Francisco not because he was born there, but because he is a person very much connected with the city. He might also be put in the categories "People born in Texas" and "American people". There are other ways to do this as well. These sorts of conversations need to happen with quite a few categories. The criteria for making decisions about this should be, "What will be people be looking for?" and "What are the primary distinctions?". -- Samuel Wantman 23:38, 30 August 2006 (UTC)
Hmm. How would you stop Willie Brown showing up as a mayor of Texas? And I'm still unclear as to why it wouldn't be possible for the database to extrapolate back and see someone categorised as a mayor of san fransisco as being categorised as a mayor. Hiding Talk 00:00, 31 August 2006 (UTC)
The extrapolation is much more involved. We have not put that into this proposal. I'm thinking Willie might be categorized as "Mayors of San Francisco", (perhaps duplicated in "Mayors"), "People born in Texas", "San Francisco", "California Assemblyman", "American people", "Mineola, Texas", "Members of the California State Assembly", "People of African descent", "Politicians", "San Francisco State University alumni", "Alpha Phi Alpha brothers", "Freemasons", "1934 births", "Living people". This is more categories than currently (my mistake using an undercategorized article as an example!) I don't think you can say that "San Francisco mayors" is the intersection of "San Francisco" and "Mayor", He is a San Franciscan and a Mineola, Texan who was a mayor. That doesn't make him mayor of both places. So "San Francisco Mayors" is not an intersection and would be a primary category. The value in this proposal is not that it is going to lower the number of categories of all articles. For many articles, like this example, there will be more categories. The value is that you will be able to find the intersection of these categories. "San Franciscan African-American members of the CA State Assembly", "Freemasons from Texas", "San Francisco St. Univ. alumni born in 1934", "Politician Alpha Phi Alpha brothers", etc... It might be possible to add "Category unions" as the next step in the process. Unions would remove the need for duplications. I don't think duplication between parent and children primary categories will be a big problem. Perhaps only 2 or 3 levels in the hierarchy if at all. To make this work, categories must be fully populated. The first option describes methods of keeping categories fully populated that are the union of other categories. This system could be used in the other options as well. -- Samuel Wantman 02:08, 31 August 2006 (UTC)
It might be possible to automatically "upward populate", but I think this would be a little tricky to get right. The focus of this proposal is intersections, and basically doesn't address the issue of primary categories that are subsets of other primary categories (e.g. mayors of San Francisco and mayors, or people from Augusta, Georgia and People from Georgia). It's fairly clear for intersections to work, these categories have to be fully populated. This proposal just doesn't include a solution. -- Rick Block (talk) 02:14, 31 August 2006 (UTC)
Okay, so you're proposing that only specific categories become union level categories, for example category:born in foo. It seems like this is mostly going to work for geographic categories and top level occupation categories. I can see a use in this, yes. It's going to be interesting working out the union level cats, it would certainly allow the cat system to develop a better structure. It's a shame we can't also work on getting the database to extrapolate, since this would reduce the number of categories that would need to be on a page at any given time. My hope for the structure was that we could see Willie in Mayors of San Fransisco and this would automatically place him and display in Mayors of U.S. places and Mayors. Hiding Talk 09:12, 31 August 2006 (UTC)
It is possible to have software or bots take care of this. This is from the first option, but something similar could be added to any of the options. I'm not sure it is as tricky as Rick thinks. -- Samuel Wantman 09:58, 31 August 2006 (UTC)
I'm not saying we can't do this, just that I'd prefer to consider it to be outside the scope of the current proposal. If we decide on an approach that results in some namespace having permanently protected "metadata" content about categories, it likely makes sense to keep this in the back of our minds as another use of such metadata. And, BTW, I think it's tricky because any time the software automatically does something in addition to what you've specifically asked there's an issue of whether the automatic thing should be undone if you undo what you originally did. For example, if (through metadata or any other mechanism) adding an article to "Mayors of San Francisco" causes the article to automatically be added to "Mayors" (and, probably, "Mayors in the U.S." and "Mayors of California cities"), what happens if the article is removed from "Mayors". Do we automatically populate but not automatically remove? How many upward lines do we allow, only one or multiple? Isn't anyone who's a mayor also a politician? What happens if we introduce a new level of hierarchy affecting existing articles? Again, I'm not saying this can't be done but there are a fair number of issues that would need to be resolved and I'd rather not have this discussion sidetracked. -- Rick Block (talk) 14:33, 31 August 2006 (UTC)
Sorry, I don't mean to sidetrack the discussion. I'm happy to support the proposal. Hiding Talk 15:05, 31 August 2006 (UTC)

[edit] Wikipedia:Category math feature

Have you seen Wikipedia:Category math feature? Not sure how similar this is, but there's a good consensus on the talk page Wikipedia talk:Category math feature#Straw poll, although drawn from a small pool which has grown over time. Hiding Talk 22:38, 30 August 2006 (UTC)

[edit] Prima facie support with question

Having just scanned the (seemingly very well-prepared) proposal, I think I'd support its development; the Laurence Fishburne examples suggest it might be very useful. I've printed it out and will read more carefully anon. One question for now, with apologies if I've scanned past the answer: Is there an intention to provide folk with (say) an option on their preference page to use or ignore the system...?
Looks like some sterling work!  Best wishes, David Kernow 01:12, 31 August 2006 (UTC)

We're not thinking of a user preference setting to use or ignore the system. The basic point is to replace manually updated "intersection" categories with automatically dynamically generated ones. -- Rick Block (talk) 01:58, 31 August 2006 (UTC)
Ah, okay; thanks for clarification. Regards, David 02:45, 31 August 2006 (UTC)

[edit] automatic generation vs the old hype for xml

In the dark ages when xml was first coming out, it was proposed by some as a way to search and get good hits. So you could search for say duck as recipe=duck to only get hits on duck when it was in a recipe. I always thought that was a needed improvement. So I see this proposal as a step in the right direction. It would be nice if it was dynamic rather then something generated. But I guess that will depend on performance and something is better then nothing. Vegaswikian 03:02, 31 August 2006 (UTC)

I'm not sure I understand the distinction you're making between dynamic vs. generated. The intent is that these intersections reflect a database query executed at the time the page is requested to be viewed, i.e. the list of matching articles is generated dynamically from the current contents of the intersected categories. -- Rick Block (talk) 04:14, 31 August 2006 (UTC)
We are both looking for the same result then. I was a little worried by the use of automatically generated above. As far as I'm concerned, the result is dynamic. Vegaswikian 04:17, 31 August 2006 (UTC)

[edit] Can't wait...

Good work. I hope people won't bicker too long about the details of this proposal because such a feature is way overdue. I think it makes sense to fast-track this as soon as possible and then reflect in a few months on how well the proposed interface works so that we can make improvements on it. In that spirit, I support the proposal as is. Pascal.Tesson 07:53, 31 August 2006 (UTC)

[edit] Locations

How will locations work? I'm thinking that you would need to define locations with a great deal of precision for this to work. You could not simply list someone in a category of Valencia but you would need to provide exact details. Valencia, Spain or city=Valencia and country=Spain for the Spanish city and Valencia, California or city=valencia, state=California, county=United States to be able to extract the intersections correctly. I'm assuming that United States would map to North America and Western Hemisphere somehow. You would also need to have multiple entries for birth, death and other significant events. Being from foo, as pointed out above, is not of much use in many cases. This would require more work and a lot of thought, but would make it easier to build intersections that are virtually 100% accurate. As pointed out above, choosing the primary categories would be critical here. Vegaswikian 00:43, 1 September 2006 (UTC)

I've been mulling something over that directly relates to the Location question. There's a slight mod that might make things a whole lot easier. It may have already been covered here, so forgive me if I'm stating the obvious.
What I'm thinking is if taxonomies are more rigorously applied, the concern over searching through sub-categories is alleviated. From the example above, it's a fact that Valencia,_California is a subcat of California, and that Valencia,_Spain is a subcat of Spain. If the taxonomy is "hardwired" or something, then when the article on Tom Smith is put in the Valencia,_Spain category, the "super-categories" (those that his cat are a member of, not relative worth) are pre-determined. Thus a search for "people from Spain" knows what cats/sub-cats/sub-sub-cats... to look in.
As an added bonus, a "stricter" taxonomy would make the queries from the database much easier and more efficient.
As was just stated, though, determining primary cats is key. -- SatyrTN (talk | contribs) 01:11, 1 September 2006 (UTC)
The problem here is the semantic meaning of category membership is not defined. "X is in category Y" might mean "X is a subset of Y", or "X is a type of Y", or (really) nearly anything. The software has no clue what the universe of meanings is or which meaning any particular categorization carries. -- Rick Block (talk) 02:33, 1 September 2006 (UTC)

[edit] What's up with...

So I visit a category this afternoon and there's a new toggle for "Show as tree | Show as list". Is this new? Any info, anyone? -- SatyrTN (talk | contribs) 00:29, 2 September 2006 (UTC)

See Category Tree !   Regards, David Kernow 03:01, 2 September 2006 (UTC)
Or maybe the next link along: Meta:CategoryTree extension. Carcharoth 02:17, 3 September 2006 (UTC)
See also Special:CategoryTree! Carcharoth 02:20, 3 September 2006 (UTC)

It looks different now. I guess when they are done deciding how it will work, I'll update the mockups for this proposal to include the feature with intersections. -- Samuel Wantman 09:52, 3 September 2006 (UTC)

[edit] So what's next?

Where do we go from here? I've put requests on the talk pages of several developers, including the author of meta:DynamicPageList2 and Brion Vibber. I put a copy of the proposal in Jimmy Wales hand when I saw him at a meet up two weeks ago. His response (via e-mail) was:

I read it. I support it, it seems really powerful. It brings up a lot of interesting questions, but one of the things I realized while reading it is that too much "a priori" thinking is not needed. The value of it is clear, and the actual day to day epistemic questions that I start having really will have to be sorted out in the old fashioned wiki style of negotiation and compromise.

So what is the next step to make this happen? Rick and I have not publicized this widely. It seems premature to do so before we can get confirmation that this is not going to cause problems with the server load. Any suggestions as to how to proceed? -- Samuel Wantman 08:35, 2 September 2006 (UTC)

I think the next logical step is to make more people aware of it. It is going to need to have many more questions asked/answered before it is adopted. Having Jimbo's support is certainly a good start. The technical feasibility questions need to be evaluated by the MetaWiki Developers before getting too far along. Cacophony 09:43, 2 September 2006 (UTC)

One thing that needs to happen is that categories need to be changed from "multiple overlapping hierarchies" to "non-overlapping hierarchies". The current data isn't in a usable form. For instance, just try to simply "list all company names" or "list the names of all actors", nevermind the intersection part... it can't be done right now. The data is a mess and needs a lot of fixing, which isn't necessarliy surprising, but before you can start fixing data, you have to get agreement that policy can be changed to allow us to start changing the data in that direction. I brought it up before at Wikipedia talk:Categorization#Category relationships, and current consensus seems to be "categories aren't meant to be used like that". --Interiot 10:10, 5 September 2006 (UTC)

That might be contentious to do (as it is quite philosophical) unless the implementation is about to be done, and done for sure. I think intersecting categories would be very cool and would really like to see it, though... I forsee a lot of the category reshuffling as amenable to bots (put everything in a sub cat in both supers seems automatable to me), but probably not all. The bots should take notes on what they did, I suspect (more than just their contribs log, notes on why they moved something from X to Y) so humans can check their work easily. ++Lar: t/c 14:33, 5 September 2006 (UTC)
This is exactly the reason why Rich and I have not been posting links to this page from large community forums. We first wanted to see if the people who spend a bit of time working with categories would approve of the proposal, then we need to get developers on board. At that point we can present this as the new software feature that is on the way. Does anyone read German? It might be useful to know the history of categorization at German Wikipedia. Jimmy mentioned that they had already made the decision to move articles into primary categories and deleted intersection categories. I'm wondering how the decision making process went. --Samuel Wantman 17:38, 5 September 2006 (UTC)

[edit] Simplification

Many ideas in the proposal sound great to me, but much of it sounds too complicated. I think that adding new pages to the wiki namespace seems unnecessary. "User created category intersection" (the Flickr-like checkboxes) hits the nail on the head. This could be a very powerful tool for browsing categories. The user would just select the desired categories, then click a "go" button to open a new search that displays all articles that are contained within all the selected categories. This would require less work to implement, lead to less categories being required, and it wouldn't have "holes" (any category could be cross referenced with any other category (or multiple categories) without editors having to create intersection pages for every possible combination).

The other big issue here is fully populated "Primary" categories. The real trick is to make (some/all?) subcategories appear automatically in supercategories. For example, Lawerence Fishburn should not need to be included in both Cat:People from Agusta, Georgia and Cat:American people. Being a member of Cat:People from Georgia should automatically include him in Cat:American people. Ok, I guess that is where it starts getting complicated. What would be really cool would be to have a simplified category tree appear when scrolling over the category name and then be able to broaden/narrow the intersection "search" that way. Cacophony 09:29, 2 September 2006 (UTC)

It's not quite as easy as that because as I understand it, the People from Agusta is loosely based residency, and American people is more based on nationality. I think there may need to be some sort of tag on the parent that says what kind of relationship is implied. -- ProveIt (talk) 02:08, 8 September 2006 (UTC)
I came across the User:Duesentrieb/CatScan tool which does basically what I was talking about. It looks like this has been around since the beginning of this year. I've had limited sucess with it, but it is there. Cacophony 05:32, 14 September 2006 (UTC)

[edit] Too long

Page is too long and has too much extra prose and too many options. Trim it down a bit? --Brion 14:01, 4 September 2006 (UTC)

I think it's important for it to be long enough to expose the various possible options. The idea is not for this to become a policy (in which case conciseness would be an issue) but rather a proposal for major changes in the way categories are handled. Once the talk page has generated a consensus on what options are prefered by users and developers it will be possible to cut it down. Pascal.Tesson 17:00, 4 September 2006 (UTC)
It's nine pages printed. That's really too long for a feature request. :) --Brion 10:06, 6 September 2006 (UTC)
  • Let me paraphrase and/or summarize into something short...
    1. The intent is to allow users to view the intersection of two (or more) categories, similar to Dynamic Page Lists, only usable as a query. A plausible way of implementing this is through a Specialpage that invokes a SQL query. Syntax might be something like Special:Intersect/German people/Actors which would yield the intersection of Category:German people and Category:Actors (or the first 100 entries thereof). It's probably okay to limit this to three or four categories at the time.
    2. It would be nice if there was an interface to reach this from the relevant articles. For instance, one could add a checkbox to each of the categories mentioned at the bottom of an article, and one 'category' button. E.g. by selecting the checkboxes for the categories "1860 births" and "Spanish people", this would send the user to their intersection, Special:Intersect/1860 births/Spanish people.
    3. The community will likely setup redirects, e.g. Category:German actors could be made into a redirect to the query mentioned above. To facilitate this process, it would be useful if the software checked categorization whenever a page is saved (I believe it already does that) and if one of the categories the page is in is actually a redirect, it is replaced by whatever the redirect points to (i.e. if I were to save a page in Category:German actors, it would end up in Category:German people and Category:Actors instead). This could also be done by a bot, of course.
  • HTH. >Radiant< 16:41, 5 September 2006 (UTC)
    1. (The proposal actually calls for the creation of a new namespace for intersections. -- Samuel Wantman 20:12, 5 September 2006 (UTC))
      (Intersection:1860 births::Spanish people for the namespace.-- Samuel Wantman 20:12, 5 September 2006 (UTC))
The interface could certainly use special pages for intersections, it is not currently in the proposal. Until we know what is possible from the developers, we don't need to make decisions about specifics. -- Samuel Wantman 20:12, 5 September 2006 (UTC)
  • (unthreading) I saw that, but a namespace implies (1) being able to write things in it, and (2) having to create pages for all conceivable combination. Neither, imho, is true. The intent seems closer to a query like Special:Shortpages. But yeah, technical details are less important than the idea. >Radiant< 20:43, 5 September 2006 (UTC)
If the intent is to replace a largish number of existing categories, using a namespace is convenient since it allows things to be written into it but does not require creating pages for all conceivable combinations. You can't edit a special: to annotate it or put it in a category. Per my response to Dangerman (above), we could make it more general by using a namespace that does not imply the operation (perhaps "operation" or "catquery"), but I do think it would be useful to use a (writable) namespace. -- Rick Block (talk) 14:14, 6 September 2006 (UTC)
That's all very nice, but doesn't this just not say anything new? Everybody loves the idea of category intersections; implementation details are what need to be proposed if you want to get anything done. --Brion 10:07, 6 September 2006 (UTC)
I'm confused. Is it too long or not detailed enough? Or is it just too many words without enough code? -- Rick Block (talk) 14:14, 6 September 2006 (UTC)
Technical advice about the "implementation details" was what I was hoping you (Brion) would bring to the discussion. From what I'm reading on the wikitech list, it seems that the technical issue is speed. Is there any other technical considerations we should be concerned about? I'm assuming that the technical issue can be solved, either with cleaver code, or advancing technologies. Flickr seems to be able to do what we are talking about, so it seems just a matter of time before this is doable. As we are a group of people who spend quite a bit of time dealing with categorization issues, it seemed reasonable to start envisioning the type of interface we would find useful and helpful. If we have a sense of where we are going, we can start moving the masses and categorization policy in that direction. I began work on this proposal out of the frustration of trying to facilitate categorization policies that would make sense, be useful and reduce conflicts. As the project grows this becomes more and more difficult if not impossible. What do you suggest we do to help fix things? -- Samuel Wantman 18:15, 6 September 2006 (UTC)
  • Okay, I'll think a bit on technical details, I'd say I have enough of a technical background to qualify. >Radiant< 19:55, 6 September 2006 (UTC)
The question I'd like to ask Brion, is what are the cost in terms of database query, for an example how expensive would it be to find the intersection of all mathematicians (aprox 2300 entries, Category:Mathematicians and its sub cat) with category:living people? How would the cost change if the intersection of three categories was used? Is it feasiable to have live queries (uncached, allowing a general search) or will they need to be cached beforehand? Or basically would wikipedia grind to a halt if this was implemented? --Salix alba (talk) 20:02, 6 September 2006 (UTC)

[edit] Question

Hello. I already posed this question on the stub project page. The question was, in order to see which philosophy articles need expert attention, i.e. to search articles BOTH in the category 'Philosophy' and 'Needs Expert Attention'. I'm gathering from this page that it may not be possible. If not, add my name to the list of people who would like it. You can see the enormous power of this for those of us trying to make this place better. Dbuckner 15:49, 4 September 2006 (UTC)

Amen to that. I often try to cleanup the backlog of articles with unclear importance and I often would love to be able to group articles together. This would help for instance when asking help, say, to the Wikiproject chemistry for the numerous chemical compound pages tagged as unimportant. Pascal.Tesson 17:03, 4 September 2006 (UTC)
CatScan may be sufficient for purposes like this (and I believe it is well used by a variety of folks at the stub sorting project). My real intent for this proposal is to replace many (all?) of the existing manually maintained "intersection categories" with dynamically generated content. I'd prefer if a side effect were that many of these categories completely ceased to exist as well. -- Rick Block (talk) 19:35, 4 September 2006 (UTC)

[edit] Simetrical's response from Wikitech-1 mailing list

From: Simetrical

On 9/4/06, Samuel Wantman wrote:
I'm writing to ask a developer to take a look at a proposal being developed at English Wikipedia at:
http://en.wikipedia.org/wiki/Wikipedia:Category_intersection...

See http://bugs.wikimedia.org/show_bug.cgi?id=5244 and the various things duped to it. I'm pretty sure performance would be a major issue here; for instance, finding the first 200 pages in a category is limited to iterating over 200 members of the category, and likewise for all other operations currently supported by categories (as well as unions), but finding the first 200 pages in the intersection of two categories has no upper bound on the number of iterations required: you have to go through every page in each category in the event that they have fewer than 200 shared pages and neither is a subset of the other.

Has anyone written code that can handle this efficiently? Is such code even possible? Storing and updating expensive and often-used intersections as sort of "virtual" categories would probably be a good idea to begin with, but I'm not exactly knowledgeable on either databases or caching. When I mentioned it on IRC, Domas (database person, works for MySQL) was pessimistic. In addition to what I noted above about an unbounded number of checks, he also pointed out that intersection tends to make categories larger, which also affects performance.

So, as someone who has little personal knowledge of the issue, I'd hazard a guess that if one of the few devs who are knowledgeable enough about efficiency and databases and the MediaWiki schema (most likely Tim, I'd imagine) were willing to write the code, it could maybe be good enough to be acceptable, but otherwise I doubt this will be implemented.

(copied from e-mail -- Samuel Wantman 01:37, 5 September 2006 (UTC))
I think this all depends on how the database works. SQL syntax certainly supports "AND" queries, but MySQL might not have a tremendously efficient intersection algorithm. I think a "perfect" algorithm for a x2 intersection results in at most 2N database fetches where N is the number of elements to be displayed in the intersection. The database fetches are likely of blocks of some number of consecutive elements, so fetching N consecutive elements results in N/blocksize fetches. I think this means an intersection might be considerably more expensive, potentially involving a factor of (blocksize*2) more fetches than a simple list of N members of a category. Assuming a reasonable blocksize (maybe 4K bytes), this factor might total anywere from 20 to something like 1000. Note that this is worst case, but assuming a perfect intersection algorithm. Less than perfect and it gets worse. Worst case is no overlap at all, and all database records for each category being intersected must be fetched. Consider intersection:even numbers::odd numbers. On the other hand, my guess is that "intersection categories" are accessed infrequently enough that this factor really doesn't matter unless it makes it perceptably slow. -- Rick Block (talk) 04:05, 5 September 2006 (UTC) (update -- Rick Block (talk) 17:36, 5 September 2006 (UTC))
Of course efficient implementation is a big problem but isn't there some hope that the structure of the categories and the fact that the same intersection queries will likely be asked over and over can be exploited here? Pascal.Tesson 13:58, 5 September 2006 (UTC)
It might be possible to cache the results of intersection queries and before executing the query again check if any of the categories being intersected have changed. In general, this doesn't help the worst case analysis because there must always be a first time any given query is executed. Caching the database records (in memory) in the Apache servers would help offload the database and, in practice, might be sufficient. I think there's some caching of database records that is already done, but I don't know the details of this. -- Rick Block (talk) 17:36, 5 September 2006 (UTC)

[edit] Looks nice

I've given it a cursory glance, and I like what I see. I'll write a detailed opinion later. Dread Lord CyberSkull ✎☠ 03:38, 5 September 2006 (UTC)

[edit] Interim step?

I'm wondering about the possibility of creating an interim step on the way to category intersection. According to meta, DynamicPageList can be turned on for any wiki. Wikinews is already using it (see n:Wikinews:DynamicPageList). If we turned it on for English Wikipedia, we could replace intersection categories with dynamic lists. This would keep things looking pretty much the way they are now, as primary categories get repopulated. Many of these categories already use templates (like Template:Fooian fooers) which could be modified to include a dynamic page list. The page list would be in the top section of the category, and miscategorized articles would still appear at the bottom. The next step would be to create bots to recategorize the articles. I think this is all doable now. How does DynamicPageList get turned on? How does a decision like that get made? -- Samuel Wantman 17:51, 5 September 2006 (UTC)

DynamicPageList has a lot of advantages. First I'll describe some problems with a general intersetion as proposed here.
A general intersection across the whole of en-wiki involves a lot refactoring of already existing category trees, take category:Mathematicians, with three main sub trees, by nationality, by centuary and by sub-field. To make this work all 2000 mathematicians would have to be placed in an category:all mathematicians category, as I suspect it would be very inefficiant to try to query all the 100+ sub categories, which go about 4 levels deep. Then there are cases where sub categories are not strict subsets of the parent, say Category:Donald Knuth part of Mathematicians has as sub cat Category:TeX, whose articles are no longer mathematicians. We will find a lot of boundary cases like this. Moreover, a general scheme does not follow the incremental approach which wikipedia tends to follow. It becomes one large change, all or nothing to make it work.
DynamicPageList on the other hand could be introduced incrementally, if someone creates a page using DPL then its their responsibility to check the results are accurate. There is a lot of useful things which could be done with DPL today, for instance it would be quite easy to implement a project specific AfD watching page, similar to what the mathematics wikiproject does using a bot at Wikipedia:WikiProject Mathematics/Current activity. This would allow us to monitor performace, as the introduction would be gradual, it would not be a problem to switch if off if things grind to a halt. --Salix alba (talk) 20:47, 6 September 2006 (UTC)
  • The problem with DPL is that if it's an interim step, it creates a lot of work that we'd have to undo later. Recatting 2000 mathematicians is also a lot of work, of course, but it could be automated by bot or by suggestion #3 above. Imho. >Radiant< 21:40, 6 September 2006 (UTC)
    • A DPL to Category intersection transition might not be so difficult. The labour intensive part is reorganising all the categories. doing DPL first, might actually make the heavy lifting easier. Once there is a sufficent number of DPL'd categories, it should not be too hard to trsform a DPL enable category to whatever namespace is chosen. DPL might also help in ironing out the ontologial problems, which I don't think have been fully enumurated yet. --Salix alba (talk) 22:30, 6 September 2006 (UTC)

[edit] Category TOCs

The statement was made that since catagory TOCs were created that there is no longer any problem with large categories. I strongly disagree. There is still the problem that the subcategories and the pages are forced to use the same TOC. For example, if cat:American people were to be fully occupied then the first page might have articles for people with names going from Aa to Ab. Chances are there will be no subcategories with names within that range. This makes it look to the casual user like the category has no subcategories and even if the user does realize what is going on, it would be impossible to find all the subcategories (as visiting each page just wouldn't be feasible).

This problem must be fixed before really large categories are made common. --JeffW 20:55, 5 September 2006 (UTC)

A workaround (not a permanent solution) is to pipe-sort the subcategories so they appear before "A" (ie. sort by * or " " or something), and hence appear on the first page of the many pages for the articles (in large categories). Carcharoth 22:57, 5 September 2006 (UTC)
That's ok for a work-around. But I'd like to see a real solution in place before we start to fully populate large numbers of primary categories with articles. --JeffW 15:44, 6 September 2006 (UTC)
Right. I agree. Who would be best to ask? Carcharoth 18:36, 7 September 2006 (UTC)
This problem is already specifically addressed in the proposal. If a category like American People, gets really big, it would make sense to put all the subcategories in a different category (for example "American people by type", or rename "American people" to "Americans" or something similar). This already happens in many categories, ususally by having one categorie singular and the other plural, like "Opera" and "Operas". I don't know the best way to name these categories, but I'm sure we'll work out a good solution. -- Samuel Wantman 05:56, 6 September 2006 (UTC)
I just don't see how this addresses what I wrote at all. --JeffW 15:44, 6 September 2006 (UTC)
Try and find an example to show what you mean. Find a large category and ask people to find all the subcategories in it. Hmm. Actually, the recently implemented "unified tree view" of categories partially solves this problem. But some subcategories can still be "hidden" further down the alphabet. Carcharoth 18:43, 7 September 2006 (UTC)
I guess the proposal is not clear enough about this. Here is what it says in this section:
Some categories may need to be split because they are both primary categories and navigational categories. A navigational category is a category which contains subdirectories. An example of this is Category:American people by occupation. Navigational categories should not contain any articles. Currently, Category:American people functions as both a primary category and a navigational category. It probably should be split into Category:American people (which would be fully populated with articles about Americans,) and Category:American people by type or something similar (which would have all or most of the subcategories. Category:American people by type would then be a subcategory of Category:American people. This will make it easier to navigate through the subcategories, especially when primary categories are very large and have many subcategories.
There are other ways this could be structured, and I'm not sure what the best way is. It would also possible to have a fully populated primary category called "American". "American" would contain all things American -- people, things, etc... "Americans" would be the intersection of "American" and "People". "American People" could remain as a navigational category that only contains subcategories and links to intersections. The intersection to"Americans" would be one such link. --Samuel Wantman 19:42, 7 September 2006 (UTC)
OK, I can see how that helps alleviate the problem. However, if you have Category:People that contains all of the biography articles, I think under your plan it would also contain Category:People by nationality, Category:People by occupation and all the rest of the navigational categories. The navigational categories are still going to get lost, spread out among the thousands of pages needed to hold all the articles. Perhaps navigational and primary categories should be split off into a separate hierarchies? You could then have cat:People (navigation) contain cat:People (primary) and a link could be added in the other direction that would be guaranteed to show up on the first page. --JeffW 20:03, 7 September 2006 (UTC)
All the categories you mention are intersection categories (Rick likes to call these index categories) If you intersect "People" with "Nationalities" you end up with a set of intersections which would be what we now call "People by Nationality". In the first option this intersection gets transcluded into a category. In the second and third options this intersection page is either linked with a "See also" comment or is categorized. If it is a "See also" it remains prominently displayed on the top of the page. If it is categorized there would be a separate section for intersections (I don't think this is in any of the mockups). The third option also has a way to navigate directly to this intersection. Important links can always be edited into a page and they will always appear. Things that are categorized can be piped to appear at the top of the heap.
There's another possible way to handle this problem. Since there are now the plus signs that expand subcategories, pehaps what is needed is a plus sign to expand or collapse the entire subcategory section and article section. If you collapse the articles, you'd see all the subcategories. -- Samuel Wantman 06:47, 8 September 2006 (UTC)

[edit] American people a topic level category

Another thing that struck me as I read the proposal is why would the American people category be a primary or topic level category? Surely cat:People would be the primary category with cat:People by nationality as subset of that? Under your proposal should both American people and People be fully populated and is there a mechanism to ensure that is done? --JeffW 21:01, 5 September 2006 (UTC)

There are a couple of issues around this question:
First issue, what makes "American people" a primary category. I think there are two reasons why it is primary. The first is that it does not fit the general rule for an intersection: "If a category can be completely and totally expressed as the intersection of other categories, it is not a primary category and should be defined only as this intersection." If you intersect People with United States, you might not end up with Americans. There may be many people who's biographies are significant to the US without them being American. This is similar to how the intersection of "Mayor" and "San Francisco" is not necessarily a mayor of San Francisco. What seems to be implied in the "nationality" categories is citizenship. This is the primary distinction being made about "People by nationality". The second reason has to do with notability. If people are noted for being "American people" it is worth having a category for them. For some categories, like "film directors", nationality seems somewhat irrelevant. For others, like "Politicians" it is very relevant. Since there are so many relevant intersection categories that use intersections of people by nationality, having this category is essential. Intersecting "United States" with "Politics" and "People" is ambiguous. It could mean "American people" intersected with "Politics" which would include Americans who write about the politics of other countries. It could mean "Politicians" intersected with "United States", which wouldn't necessarily be Americans. It could also mean "US Politics" intersected with "People" which would include all non-Americans who write about US Politics. So "Politicians" and "American people" are both needed.
I think what this implies is that a person should have categories for the city, state or province, and country they were born in. Perhaps we also need categories for the cities, states, and countries that they are strongly associated with. I suppose they should also be placed in categories for their occupation and their class of occupation (news anchor and media occupations) as both of those would be useful to intersect on. This is beginning to look like each article will be in a lot of categories. I think that for this proposal to be useful you will also need to propose a mechanism to allow a person to be placed only in the most restrictive category and automatically be propogated up the chain of super-categories (but without the super-categories actually being added to the article). I'm thinking of some syntax put on the category page that means to propogate all the articles in the category to a specific super-category). --JeffW 15:36, 6 September 2006 (UTC)
Careful! :-) You are sliding down the slippery slope towards categorising an article based on nearly everything that could be linked from it instead. ie. "Y is a person from the USA and from the state of Georgia and the city of Augusta. They have worked as a film actor and as a roadsweeper..." Carcharoth 01:29, 7 September 2006 (UTC)
I think that just illustrates my point since if you thought it important to catagorize a person into People from Augusta then you will be required (or a bot will do it) to also place the biography into the People from Georgia, American people, and People. Otherwise it wouldn't appear in the intersection of Plumbers and People from Georgia, for example. --JeffW 16:06, 7 September 2006 (UTC)
What this illustrates is that all categories need to be well defined. They all have to have clear distinctions about what belongs and what does not belong. Once we have made those distinctions, categories should be fully populated with every article that would be helpful for people looking in that category or the intersections that use the category. One of the reasons the categorization system is a mess is because many categories get populated before they are defined. -- Samuel Wantman 07:09, 8 September 2006 (UTC)
Second issue, should "People" be fully populated. Rick and I actually discussed having some basic category tags, such as PEOPLE, PLACE, THING, TOPIC, LIST, EVENT. We could perhaps even require that at least one be assigned for every article. (This discussion is in Archive 1) This is sort of like how libraries had card catalogs for Subject/Title/Author. In the best of all worlds, I think these categories would be useful, but for our purposes they also have some problems. The biggest problem is they would be quite huge, bigger than the biggest categories we now have (Living people?) In print, a master index is needed to find something, but we have the search box for that. If you know a name you look it up. These categories would not be very useful for browsing. If you are browsing, you probably want to browse through similar things, not similarly spelled things. However, there's great utility to having these categories available to create intersections. By having "List" as a category, we can get rid of ALL the list categories because they will be intersections. For some topics (like World War II) it would be very useful to have intersections using these categories such as "World War II events", "World War II people", etc... So there is a utility to having these categories fully populated, and there is a down-side. If possible, I'd say have them fully populated.
Third issue: Keeping categories fully populated. I think this proposal will require that bots or software help maintain things. Option one describes how this might be done. Rick mentioned the problem of undoing categorization if larger categories are automatically populated from subcategories. I think this could also be handled somewhat automatically. For example, when an article page is saved, the software already looks at the categories and adds them or removes them from category pages. It could also look at the flags or bots set to see if any of the other categories in the page will end up duplicating the category being removed. The user could then be warned of this duplication and have the option of keeping the removed category or removing the child categories as well. Even without the technology, people can be warned to remove parents and children together. --Samuel Wantman 07:09, 6 September 2006 (UTC)
Bots can solve the problem of keeping parent categories fully populated, but the don't solve the problem of massive numbers of categories on each article once you have to put it in every category up and down the hierarchy.--JeffW 15:39, 6 September 2006 (UTC)
It will add some categories, but many will also go away since they are intersections. Many categories will only need one or two levles in a hierarchy. Perhaps articles with 10 categories now will have 15 in this new system. But from those 15 you will be able to generate hundreds of intersections. So even if we double the clutter there will be much larger increase of utility by several orders of magnitude. Ultimately this is a huge reduction in clutter. Many categories are now routinely deleted at CfD because they contribute to clutter. This is a reality of the current system because if left unabated, articles would accumulate scores of intersection categories. With the proposed system adding a single category yeilds multiple taxonomies with minimal additional clutter. -- Samuel Wantman 07:09, 8 September 2006 (UTC)

[edit] Namespace vs Specialpage

As to the namespace/specialpage debate, let me just point out that

    1. There really isn't anything you could state on a Intersect:Foo:Bar page that isn't already on Category:Foo and Category:Bar
    2. Most category pages (and the corresponding talk pages) are relatively unused; that's what {{mainarticle}} is for
    3. If we were to put text on Intersect:Foo:Bar, the same text should be on Intersect:Bar:Foo. This gets worse if people intersect three or four categories
    4. And also, we shouldn't have to manually create pages for each plausible intersection - and if a bot were to do it, it might just as easily done on a specialpage.
  • In other words, as I see it, it's a lot more work for virtually no extra gain. The intersection is kind of like Special:Shortpages in that it dynamically lists a number of pages with a certain characteristic. Rebuttals are, of course, welcome. >Radiant< 19:55, 6 September 2006 (UTC)
Multipoint rebuttals get awkward, but:
  1. Consider an article List of American film directors. Probably not in both category:Film directors and category:American people, but likely to be a useful link from Intersect:Film directors:American people.
  2. I completely agree (except I think you mean template:catmore).
  3. Yes, and the proposal addresses this. In the intersect namespace the "article" content has to be accessed indirectly based on the name. The "real" name (name in the database) is in canonical (sorted) order. So, before fetching intersect:D::E::C::F::A::B the pagename is manipulated into (canonical) A::B::C::D::E::F format, so any order matches the same "page".
  4. I completely agree. The proposal includes three versions of how this might work. All include the ability to traverse to an arbitrary intersection, with no manually created anything.
In other words, other than point #1, I don't know that I care whether it's a page in a new namespace or a special: page. I can see some value in allowing annotations, but it does indeed boil down to generating a list of pages that meet a certain characteristic. The "intersection selection table" concept (from the 3rd version of the UI) is something that could be added to the output of a special: page. -- Rick Block (talk) 02:46, 7 September 2006 (UTC)
To add to what Rick said, I think there are some potentially important uses for the intersection pages. In option one, they are used for category maintenance parameters. Things like how categories should be linked, duplicated, etc... (This could be done in other ways.) In option two and three, the pages are used to create links to other intersection pages. Option three has the ability to go pretty much anywhere in "intersection space" but that doesn't mean it would be easy or obvious for the user to know where to go and how to get there. By adding links to intersections like "Nationalies by occupation", you could help direct people through the space. There will also be cases where you are at the intersection of two categories and there are useful intersection with a third. This requires links in all three options. Links are more important in Option two where full navigation is more constrained. This option relies on links to make useful connections between intersection pages. There are examples of this in the mockups. -- Samuel Wantman 06:13, 7 September 2006 (UTC)
  • I see... however, if the intersects were a specialpage, they would not need maintenance parameters. I'm somewhat worried about having the software automatically convert C:A:D:B into A:B:C:D because no other namespace works like that (KISS principle). If you're at the intersection of two categories, any intersection with a third would be a subset thereof... I think an easier way to cover this is to create a few menu screens with pulldowns where you can select e.g. country, profession and birth year. It might also help to create redirects from (current) categories to the specialpage (note that contrary to popular belief, redirects to specialpages work fine). Anyway I'm not yet convinced, and I think neither are you, so let's think and talk on this some more. >Radiant< 21:32, 7 September 2006 (UTC)

[edit] Another example (Disasters)

Is it possible to have another set of examples to see how this would work? For no other reason than I am familiar with it, I propose Category:Disasters. It is not fully fleshed out, but there are categories by location (mostly country), by date (mostly year) and by type (many of these). How would this change with this proposed system? Carcharoth 10:06, 7 September 2006 (UTC)

Here's my take on what would change:
  • category:Disasters would be fully populated as a primary category.
  • The "by-type" subcategories would be fully populated (as primary categories).
  • The "by-country" and "by-year" "index" subcategories could be turned into intersections, by adding individual disasters to an "events in <country>" category and an "events in <year>" category (this could be done by a bot).
  • Then, for example, Category:1971 disasters becomes Intersection:Events in 1971::Disasters, and the category either goes away or is left as a category linked to the intersection (depending on which version of the UI we're talking about).
So, how is this better than what we have today? For disasters, at least superficially, I don't think it makes much difference. An individual disaster would be categorized into category:Disasters (which would be an addition), [[:category:<some type of disaster>]] (just like it is today), [[:category:events in <some country>]] (rather than [[:category:disasters in <country>]]), and [[:category:events in <some year>]] (rather than [[:category:disasters in <some year>]]).
This immediately makes some things available that currently aren't like "fires by year" or "fires by country" (and please don't go create these!). I don't think this is an overly huge deal, but changing to this sort of structure would also include disasters in the event-by-year-by-country categories/intersections (which they aren't today). We could fix this by fully populating the Category:Years by country category hierarchy (don't do this either). As it stands today, I think we have a lot of "x by y by z" index categories that aren't complete and would take a huge amount of manual effort to create and maintain. -- Rick Block (talk) 15:13, 7 September 2006 (UTC)
Thanks for explaining all this. It sounds great, and if this can all be implemented fairly simply and painlessly, then I think it is a great idea. I started to create lots of "<year> disasters" categories, but it was taking a long time. The problem is actually that not all the disasters are tagged with their year yet. That would still need to be done manually, right? Also, when the bot fully populates the primary disasters category, it will show some miscategorisation. An example is the eponymous categories like Category:Columbine_High_School_massacre and Category:Beslan_school_hostage_crisis and Category:2004 Indian Ocean earthquake and Category:September 11, 2001 attacks. These all contain articles that appear under the "Disasters" hierarchy, but are not disasters - rather, they are related to a specific disaster. Also, the term disaster is used in a rather broad and vague sense in this category, so I'm not entirely sure how that would work. Articles that happily sit together under "Disasters" as examples of "damage to life and property" are currently labelled accurately by their immediate designation (eg. earthquake or terrorist bombing), but applying the broad term "disaster" to all of them would not work so well. I think that is more a problem with the semantics of this particular area though, rather than the system you are proposing. But it is something to think about. I've also noticed things like Category:Giant monster films creeping in! :-) Carcharoth 18:28, 7 September 2006 (UTC)

[edit] Tossing some thoughts out

So what we could well be looking at is a complete overhaul of categories, for example having categories that are defined as born in fooand naturalised citizen of foo, which sit inside nationality people and would also be top level categories? I'm not sure how to catch the overlap though. What sort of definition doe we place on our nationality categories? Are we going to have to completely overhaul the category structure? Are we going to have to be more rigid in our categorisation policy? Hiding Talk 09:11, 8 September 2006 (UTC)

I'm not sure we need to plan every detail about categorization policy before implementing this. There will be a natural desire to make categories that create useful and meaningful intersections. Perhaps "useful and meaningful" is all we need to have as a policy. Things can evolve. I do wonder about the issue about the "x is a y" and "x is related to y" duality to our categorization structure. Clearly, "x is a y" is not a problem when creating intersections. I don't think "x is related to y" would be a problem either. For example, if you take Category:George W. Bush which is a "related to" category and intersect it with Category:Children's books you end up with an intersection that contains The Pet Goat. If the categories are "related to" the intersections will be as well. But as this is a source of confusion and conflict with our current system, it will remain so with an intersection system. -- Samuel Wantman 09:56, 9 September 2006 (UTC)

[edit] Some technical bits

Ok, let's forestall the namespace vs specialpage bit and see what we do agree on...

  • What we want is a way of getting the intersection of two or more categories.
  • Obviously, intersection of A and B equals intersection of B and A.
  • Because of performance reasons, it might be useful to restrict this to the intersection of, say, five or six categories maximum.
  • It would be nice to have some browser page, that makes it easy to get from "actors" to "italian actors" or "actors born in 1950".
  • It would be nice if this functionality was available from the category list at the bottom of each page; for example, we could add a checkbox to each cat, and an intersect button that, if clicked, shows the intersection of all 'checked' cats.
  • Thoughts please? >Radiant< 10:20, 9 September 2006 (UTC)
  • Also, I've written this script as a possible addition to MediaWiki. It currently uses Specialpages but it would not be problematic to rewrite it using the Category routines, to create a namespace. >Radiant< 16:08, 9 September 2006 (UTC)
    • There may be a difference in performance of intersection of A and B and the intersection of B and A, especially if one is much larger than the other. --Salix alba (talk) 16:13, 9 September 2006 (UTC)
    • And it might interesting to look at the current query code for categories, which limit the search to 200 hits at a time. I assume running a "null" result intersection query against a very large category (like Category:Living people intersected with Category:Ancient Greeks) would have roughly the same database impact as the unlimited category search used to have. The unlimited category searches presumably had other problems as well, but I suspect Tim might have some insight here. -- Rick Block (talk) 20:13, 9 September 2006 (UTC)
Looks like Tim's insight might have to wait a bit. See User:Tim Starling/Get Well Soon. :-( Carcharoth 22:14, 19 September 2006 (UTC)
  • I spoke to Brion yesterday and he likes the idea. The main problem seems to be server load: if two huge (>10000) categories are intersected but have no actual matches, that would be problematic. Suggestions on how to alleviate this would be appreciated. >Radiant< 22:27, 19 September 2006 (UTC)
Without prebuilding indices in the database for any conceivable intersection search, I don't think there is a simple solution. A ∩ B for large sets A and B that have no intersection requires reading all elements of A and all elements of B (even if A and B are just indices). The only alternatives I can think of are:
  1. run such queries in a non-blocking manner at a lower priority than other work going on in the database (but it looks like MySQL doesn't support this, see [1])
  2. slough them off to a different physical server (which would likely need to have its own copy of the category database)
  3. expand the query into multiple queries that could be run in series to obtain the result (for example run 26 queries restricting the intersection by first character of the category name).
None of these are particularly easy, and #2 even requires throwing hardware at the problem. -- Rick Block (talk) 01:43, 20 September 2006 (UTC)
Is it possible to prebuild indicies for intersection categories on a daily basis, or the first time an intersection is requested each 24 hours, or whenever the servers have low loads? This would mean that the information would not be totally current, but that is better than non-existant. -- Samuel Wantman 08:57, 21 September 2006 (UTC)
You're basically asking for (number of categories) squared tables to be built every 24 hours (and that's only for two way intersections) which doesn't seem feasible. Also, the vast majority of those category intersections will never be requested. Perhaps, since real-time intersections of categories with a low number of entries (less than 200?) wouldn't be a load on the server, you could limit the tables built to just the intersections of large categories. Or maybe something like creating a daily list of categories that don't intersect would be useful as that information could be stored in one simple list for all such intersections. --JeffW 15:57, 21 September 2006 (UTC)
  • I figured it might be possible to cut off the query after a second or so, or to cache a list of queries that are known not to yield results so that they need not be recalculated (of course this cache should be timestamped). >Radiant< 16:54, 21 September 2006 (UTC)

From the discussions on wikitech-l and some personal experimentation, I think the issues are exactly 1) almost entirely technical - how to actually get the pages at the intersection at one or more categories and 2) demonstrated in the worst case by those very large categories mentioned (Category:Living_people for example). There are a few differnent ways to implement using existing tables, but so far it looks like, in the worst cases, these all take at least several seconds to complete. So, that means either new tables (likely with millions and millions of records - it's not exponential, but it is big) or perhaps a way to identify unreasonable queries or cut them off as Radiant mentioned. I'm going to try a few more experiments... --Aerik 00:06, 12 October 2006 (UTC)

[edit] Anime and Manga case

I find this proposal the right approach to solve supercategorization and long discussions about where and how put an article. On italian Wikipedia, I have made a simple SVG that can be useful to explain the differences in this two categorization approaches in Anime and Manga case ((Italian) read the discussion), where genres are the same for both and things like shounen anime and shounen manga are redundant, confusing and create an huge amount of subcategories. Since I've put that SVG in Public Domain, you can copy and modify it without restriction (using an SVG editor like Inkscape) to represent other supercategorization cases. --Dzag 10:03, 4 November 2006 (UTC)

[edit] Also check out Faceted classification

Much of what you (and I) are looking for is described as Faceted classification, which is fundamentally about intersections of semantically orthogonal facets.

Of course, we could have lots of arguments about what constitutes a "proper" facet as well as whether a "faceted approach" should include multiple picks from the same facet. -- PhilMurray46 16:44, 14 November 2006 (UTC)

[edit] DynamicPageList

I see it mentioned a few times on this page. Is there a reason why meta:DynamicPageList or meta:DynamicPageList2 are not turned on in en-wikipedia? It seems like it would be very useful. Carcharoth 10:41, 12 January 2007 (UTC)

Maybe DynamicPageList could be of some help. I put a lot of effort in improving it over the last weeks. Some of the features discussed here are already available, others could be added I think. You may want to have a look at the DPL demo website. -- Algorithmix 13:49, 18 February 2007

[edit] Temporary flat category solution?

I like the proposal at Wikipedia:Category_intersection#Category_viewed_as_an_outline. I was vaguely aware of this, and there is currently a way to rather kludgingly do this in a manual and non-updateable way. See what I did at Category:Earthquakes, Category:School massacres and Category:Anti-communists. What I want is category union. Intersection seems to talk about category A AND category B, when I am also interested in category A OR category B, with tweaks to allow simple union to a specified depth (ie. specify a depth of 3 go down three levels into the subcats, and unify and list all the articles in the subcats and the original parent category), as opposed to, say, unifying two unrelated categories. What I really want is feedback on whether the linking to page history that I did is acceptable, and whether such lists of all articles in a set of categories are useful. Carcharoth 10:53, 12 January 2007 (UTC)

[edit] Some test code for intersections implmented in a fulltext index

I have a page up for testing some code which does category intersections using a boolean search on a MySQL fulltext index of a table containing all the categories for each page. My database is copied from November and only has the tables I needed to test this code.

http://aerik.com/wikintersections.php

I know the interface is no good, but I am trying to collect statistics about query performance. Please take a look and run some intersection queries. Thanks, --Aerik 18:37, 16 January 2007 (UTC)

Looks good. Would you be able to comment on the above two sections as well? I would dearly love to unify any number of categories I selected and create alphabetical indices for large numbers of fragmented categories. Would your system do that as well as intersection (ie. would it do OR as well as AND)? Carcharoth 11:48, 17 January 2007 (UTC)

I've put some links at the top of Category:Unassessed Tolkien articles. Unfortunately, those intersections are only helpful if you are using current data, but that is an example of the reasons why category intersection would be very helpful. Carcharoth 16:20, 21 January 2007 (UTC)

Just to try some worst case queries, I intersected category:Living people with category:album covers (no intersection, query took 4.8575 seconds) and category:Living people with category:Disambiguation (one intersection, query took 7.2953 seconds). I don't know if these are elapsed or CPU times (seem to be elapsed). My understanding is that in at least some cases MySQL processes queries serially, so while one of these is running other queries would wait. Running some more tests, specifically intersecting category:Unassessed_Tolkien_articles with category:Top-importance_Tolkien_articles at the same time as one of the long running queries, it appears reads involving different specific database records are not serialized (and, with this interface there's no way to try simultaneous writes, so I have no idea how writes would interact with these long running reads). The smaller intersection finished quite quickly when starting it during a long query, but repeating the same long running query while one is already active seems to slow down both (implying multiple reads using the same index record are processed concurrently). Intersecting category:Living people with category:Living people (!) takes 15 seconds or so of elapsed time, but the query result says 0.0045 seconds. I think the bottom line is that with read-only access it seems to not be too bad but we should have considerably more benchmarking involving simultaneous read and write operations before imagining we might be able to deploy anything like this to the live en database server(s). It seems like it might be worth getting a MySQL performance expert to comment about this. -- Rick Block (talk) 18:02, 21 January 2007 (UTC)
Thanks for the comments. Over here someone asked when Category Intersection is likely to happen. I said it was anyone's guess, really, but if anyone here can say something more informative, that would be great. Carcharoth 12:51, 23 January 2007 (UTC)

Hi, as an update, I'd call the result of the tests so far, "mixed". Yes, the "Query completed " time is elapsed time that I calculate using a microtime right before the query is passed to MySQL and getting it again right the result set is returned. This is always for a statement limited to 30 records. Rick, thanks for your in-depth observations and tests. I'm going to go ahead and create a lucene index (I need to go educate myself about lucene) of the same data so we can play with that and make some comparisons. Carcharoth, I think we're maybe a couple of months away from having a working solution, but I will likely need a more senior developer's help to integrate the agreed upon solution into the main body of code as efficiently as possible... so there is always the question of politics/bureaucracy... --Aerik 18:20, 23 January 2007 (UTC)

Great. Thanks for the update. Let us know what is happening and if we can do anything to help, like run more tests. I'll try and keep an eye on this page. Carcharoth 23:29, 23 January 2007 (UTC)
Carcharoth, thank you! More tests would be great! I'll post more as soon as I have any data/updates. --Aerik 19:57, 24 January 2007 (UTC)

I had to change hosts and upgraded to a VPS (too cheap for a dedicated server) an in the process deleted some stuff I shouldn't have. To make a long story short, I'm working on a new download (from Apr 7, I believe) and I am also playing with Zend_Search_Lucene and *trying* to create a Lucene index. I hope to put up a new MySQL fulltext index in a day or two, and the Lucene index in maybe a couple of weeks (still working on technical problems and even after that it's going to take days to index everything - I'm planning to go ahead and index through all the namespaces to really see what the performance is like).--Aerik 16:28, 19 April 2007 (UTC)

[edit] If this is implemented it would make it a lot easier to

look for octogenarians, for example. Just search for an intersection between category:living people and category:1918 births until category:1927 births. --Wirosableng 20:33, 24 February 2007 (UTC)

Yes I agree, I think we should start a voting process for this proposal and if the majority agrees, the developers could begin the work. --Drgerard 17:06, 25 February 2007 (UTC)

[edit] This proposal would wreck the utility of the categories for browsing articles

This system misunderstands what categories primarily for, which is browsing, not searching. If implemented people might now and again find it useful as a search function, but they would miss out on all the detailed connections that one comes across serendipitiously with the current detailed system. Therefore it should be abandoned. Casperonline 15:26, 27 March 2007 (UTC)

I think it would make it easier to browse, which was part of the inspiration for writing the proposal. All the current categories would still exist in some form, and many more would be possible using intersections. Every topic would have fully populated categories and you could navigate to fully populated sub-topics or intersections. Of the three ways we mocked this up, the first would look almost identical to what we already have, so why is the serendipity lost? -- Samuel Wantman 09:03, 28 March 2007 (UTC)
After a lot of problems in WP Films categorization and taking in consideration all views presented, I also think this system would help rather than wreck browsing. Actually it doesn't wreck at all the browsing of articles, it just involves having the primary category additionally populated, offering multiple possibilities, whereas our present conventions limit them. I was also scarred when I first learned about it, but as problems continue to accumulate unanswered, I start seeing its huge merits. Hoverfish Talk 14:22, 10 April 2007 (UTC)
I agree as well it would help browsing. For example, recently I was browsing Category:FA-Class biography (arts and entertainment) articles. But I didn't just want them all, I specifically wanted the subset that were also BLPs. Many subcategories are so large that browsing them may not be of interest. However browsing a subset may very well be. Nil Einne (talk) 19:13, 31 March 2008 (UTC)

[edit] So what's going on?

I am interested in doing what I can (as a non-dev) to help push this along. Please tell me that at least one person is still tinkering around in the background... :) Girolamo Savonarola 04:50, 21 August 2007 (UTC)

I'm still watching the talk page. Well, sort of. If you want to get people interested in this again, the best thing to do is get a developer interested, and then get a group of people developing the proposals and putting them into workable formats for wider discussion and, well, in a format that developers can work on. Carcharoth 15:59, 3 October 2007 (UTC)
Okay, fair enough. Where can I hunt for a dev? :) Girolamo Savonarola 17:42, 3 October 2007 (UTC)
I too, keep an eye on this page, and also scan the Wikitech mailing list. I put a copy of this proposal in Jimmy Wales hands last year, I e-mailed the tech list, e-mailed developers that seemed interested. The problem, it turns out, is not getting people interested. Pretty much everyone I've ever talked to about this thinks it should happen. The problem is technical. It involves creating a system that will be able to handle category intersections without bringing the entire site to its knees. Devoloper Gmaxwell, is working on this using Lucene, and in an e-mail to wikitech on September 9th said:
I have a new search system running externally right now which is much more powerful, for example it is able to apply category intersections, fuzzy title matches, text regular expressions, page link constraints and geographic filtering. Most of the functionality is fast enough to make available to everyone.
If you'd like to be a beta-tester please let me know.
I e-mailed him and offered my help in trying to coordinate efforts between the developers and the people who had been discussing the user interface here. I got no response. -- ☑ SamuelWantman 20:26, 3 October 2007 (UTC)
That sounds like an external system. Did he give any indication of how it would be integrated to work in the system? I'd try e-mailing him again, or dropping a note off on his talk page. It sounds promising. Did he try it out on the very largest categories? Carcharoth 23:04, 3 October 2007 (UTC)
For what it's worth, I watch this page as well. I think the most promising approach involves using a database for category searches that would allow these searches to be run at a lower priority than other work, although AFAICT MySQL does not currently support this (browser timeouts are another problem, but I think there are potential solutions for this issue). I would think the MySQL folks would be interested in working with the MediaWiki development team to come up with a solution - it would make a great entry on their case studies promo page. Has anyone approached the MySQL folks? -- Rick Block (talk) 00:40, 4 October 2007 (UTC)
How about a well-advertised, open competition to see which system or approach can come up with the goods? Or does that sort of thing not really work any more? Carcharoth 00:44, 4 October 2007 (UTC)
I don't know if it's official, but it's at least an unofficial policy of the board to use free software. An open competition restricted to open source databases might work, but I suspect winning a competition of this nature might be more valuable as advertising to a commercial vendor. MySQL exists in both commercial and free forms (same code). I'm not sure how many others are like this. Asking MySQL if they'd work with the development team on this doesn't seem like it could hurt, and since we use MySQL now seems like the easiest approach. For them to list Wikipedia as a case study would probably require the board's approval. -- Rick Block (talk) 04:26, 4 October 2007 (UTC)

[edit] Is "American film directors" a good example?

"American film directors" can be interpreted as "Film directors who are American" or as "Directors of films first published in America". Is this important? --Damian Yerrick (talk | stalk) 22:33, 26 November 2007 (UTC)

The former is the only correct interpretation. UnitedStatesian (talk) 22:35, 26 November 2007 (UTC)

[edit] Semantic MediaWiki extension

I am excited by the potential of this extension. I understand this fancy features load servers, complicate wikipedia editing, but these will take wikipedia to the next level. This feature is very scalable. We could initially only give it a limited scope, say dates or birth dates; why have category: 1976 births when just as easily we could have semantic data: "birth::27 may 1976" meaning the semantic tag which goes beyond generic category info - it understands the birthday in question is 27 may 1976. This could be applied to lots of things, and i feel it will (after an adjustment period) enhance and even simplify the categorization element of wikipedia. Chendy (talk) 14:38, 1 February 2008 (UTC)

Indeed, SMW does category intersection, and it lets you do much more precise querying than nebulous things like "Child actors".

{{#ask:
 [[Category:Actors]]
 [[Category:Directors]]
}}

with no changes to wiki markup will find the intersection of those two categories and their subcategories (to a configurable subcategory depth). Then if you've gone on to annotate with properties like [[Date of birth::2004-05-27]], you can ask for

{{#ask:
 [[Category:Actors]]
 [[date of birth::>1992-01-01]]
 |?Date of birth
 |?Category
 |sort=date of birth
 |?
}}

and that will display all actors 16 years old or less, ordered by date of birth, together with any other categories they've been assigned. -- Skierpage (talk) 05:14, 16 April 2008 (UTC)

I certainly think we should be working towards implementing SMW somehow on Wikipedia. It would be useful for categorization and much more besides. In fact I would like to see an SMW database developed somehow outside any specific language version of Wikipedia (something like commons), to enable the sharing of key information between Wikipedias. (People might like to check out my partially-formed idea at User:Kotniski/wdb, which I wrote before I discovered SMW - I believe SMW would be a perfect way to implement something like this.)--Kotniski (talk) 07:45, 16 April 2008 (UTC)

[edit] When can we get ?

I would like to use it now. People are creating manual intersection categories all over the place. Perhaps the overhead of the dynamic processing will be offset by dramatically reducing the number of manual intersction categories . . . ? Pee Tern (talk) 00:23, 22 February 2008 (UTC)

PS: They are also creating (non encylcopedic) intersection list articles all over the place. Pee Tern (talk) 00:27, 22 February 2008 (UTC)

I would also be curious to know what exactly is going on. It seems that every few months there's a clamor, we're given a quick update and told it should be around shortly, and then suddenly...nothing for months. At the very least, it would be nice to have some regular biweekly or monthly updates as to the progress being made. This would also increase involvement with this page and (hopefully) editorial advocacy for prioritizing this development work. Thanks, Girolamo Savonarola (talk) 06:06, 22 February 2008 (UTC)
I am thinking of just starting to create semantically primitive categories anyway, and try to get some momentum up. For example, instead of (actually as well as for the being) having a category for law enforcement agency the article would have three semantic primitive categories, law, enforcement, and organisation . . . Pee Tern (talk) 21:22, 24 February 2008 (UTC)

[edit] Initial implementation option

Rather than trying to go for the whole solution immediately, perhaps we go for in stage 1 only binary intersections, with a not and groupby clause. In stage 2 we go for nested binaries . . . Pee Tern (talk) 21:53, 24 February 2008 (UTC)

[edit] Alternative syntax

The syntax proposed to date seems to introduce another way of "linking" to information in Wikipedia. I would like to propose that we use the existing syntax as follows:

Expression Effect
{{Category:Intersection |category1 |category2}} To produce a category page of those pages in both category1 and category2.
{{Category:Intersection |category1 |category2 |not=category1}} To produce a page of those pages in category2 but not in category1.
{{Category:Intersection |category1 |category2 |not=category2}} To produce a page of those pages in category1 but not in category2.
{{Category:Intersection |category1 |category2 |groupby=category2}} To produce a page of articles in both category1 and category2 where category2 is a parent category. For example {{Category:Intersection |countries |organisation |groupby=countries}} would produce a page of organisations grouped by country (there being a category for each country which is in a parent categroy of countries).
[[:Category:Intersection |category1 |category2]] To produce a link on a page to the corresponding intersection category.
[[Category:Intersection |category1 |category2 ]] Is by definition not defined, or, it could put the page into both categories at once and be equivalent to [[Category:category1]] [[Category:category2]]?
{{Category:Intersection |{{Category:Intersection |category1 |category2 }} |category2 }} For nesting?

Pee Tern (talk) 21:53, 24 February 2008 (UTC)

This is basically creating a 2-level namespace where "Intersection" is a qualifier within the category namespace. I don't actually know the code, but I'd expect that creating "intersection" as a namespace would be a whole lot easier and cleaner. Note that the main problem is (and has always been) the potential performance hit of a sparse intersection between two very large categories. It can take multiple CPU seconds (!) to figure out which members of category A are also in category B (worst case is two very large categories with very little intersection). Allowing individual users to invoke operations that make the DB server(s) consume multiple CPU seconds at wikipedia.org is simply not acceptable (invites a denial of service attack). -- Rick Block (talk) 05:20, 27 February 2008 (UTC)
I guess I am not to hung up on the exact syntax. The currently suggested one seems to be going too far away from what a category looks like and I think if set logic on categories is going to be successful then it needs to look like and feel like a cagegory all the time.
Yep. Performance could be an issue. Perhaps category set logic is only routinely processed by bots at certain times and they can check for malsetups and the only people who can dynamically reevaluate a specific category set logic expression are fully fledged administrators . . . ?

Pee Tern (talk) 10:01, 27 February 2008 (UTC)

[edit] A non-wikipedia example

It seems important to note that this idea is important beyond the particular wiki implementation that we call wikipedia, and would be incredibly useful to anyone using the underlying software. A case in point is a project that I'm working on now. We are using MediaWiki to organize and categorize all of the projects that my organization is working on. We want to browse by multiple categories, and also by the interersection of those categories. Example: show me all the present projects. show me all the future projects. Show me all the projects in the IT department. Show me all the past IT department projects. We chose a wiki over a formal database since it was so easy to use and flexible; it would be really wonderful if it could also provide more powerful reporting functions. Consider this a vote in favor of devising a scheme for allowing for union and intersection functions that will make browsing much more powerful. -- mike roy, kenyon college, gambier, OH, USA —Preceding unsigned comment added by 76.181.107.71 (talk) 04:54, 27 February 2008 (UTC)

The main branch of MediaWiki development is typically focused on the needs of wikipedia.org, however (as you note) MediaWiki is actually separate from Wikipedia and there are extensions that can be deployed on sites other than wikipedia.org. I think category intersection isn't being pursued by the developers basically because of potential performance impacts at wikipedia.org (I think several of the suggestions here could clearly be implemented). If I ever end up with a significant amount of time to devote to this (not likely in the short term), I may actually implement one of these. I'd invite anyone with development skills to consider implementing one of these suggestions. -- Rick Block (talk) 05:37, 27 February 2008 (UTC)
I know zip about the "backend"s of wiki*, either from technical perspective or a protocol perspective. If no one gets to this in the next few months, I might have some time after midyear to do a bit more than just muse about it. Where would I start? Pee Tern (talk) 10:08, 27 February 2008 (UTC)
Start at the MediaWiki homepage, http://www.mediawiki.org/wiki/MediaWiki (it's written in PHP, with an SQL database backend). -- Rick Block (talk) 14:48, 27 February 2008 (UTC)

[edit] Category Union

I posted this to the Wikitech list...

With the advent of __HIDDENCAT__, I've been wondering about using hidden categories to create indexes. My initial hope with Wikipedia was that we could reorganize categories so that categories could function as broad indexes of single attributes such as "People", "Films", "Bridges", etc... and hide all the intersection categories of parents. Later, if and when category intersection was implemented, all the hidden categories would no longer be needed. However, implementing major changes seems to be near impossible in a project as large and set in its ways as Wikipedia. There is just too much resistance to change. If category intersection was implemented there would be an technical compelling reason to make the change, but short of that upgrade, it seems like a very difficult -- if not impossible -- sell.

It really bothers me (and others, especially librarians), that Wikipedia is not indexed. You cannot find a master index of People, places, books, films, etc... To find anything you have to know in advance, where it is subcategorized. This only works if you know where to browse, and it is your desire to only browse in a small well-defined place. One of the big joys of libraries is the ability of finding things you didn't know about in broad swaths of knowledge. This ability is often lacking in Wikipedia because of categories being constantly broken into smaller pieces. For example, If I want to browse through the bridges in Europe, I have to look at a category for each country separately, and in some countries (like the UK) I have look at one for each county. It is just too difficult and time consuming a task to be a pleasurable leisurely browse.

So I've been thinking of alternative approaches. One possibility is to use hidden categories to create index categories. For instance, Category:Index-Films could contain all films, Category:Index-People could contain all people, etc... However, this would be difficult to maintain because the categories would be hidden, and it would take a tremendous amount of work to populate these categories. It seems crazy to have people doing all the mindless busywork necessary to create categories like these. That is why we have computers.

This is where developers come in...

I'm wondering about creating a new namespace, called (you guessed it) INDEX. Any category of people could be put in an index by adding Index:People on the category page. The "People" INDEX page, into which the category get put, would have links to all the articles and subcategories from the categories in the INDEX. The contents of the subcategories of those categories would NOT be added automatically. Each would have to be manually added to the index if appropriate. Just like a category there would be text that could be edited for each INDEX page. So in essence, an INDEX is a way to do category unions. This would be much, much easier than trying to create and maintain these indexes manually using categories.

It would be great if an INDEX page could be viewed two different ways (and easily switched). The first way would look similar to current categories, showing a category tree at the top, and all the articles below arranged alphabetically. It would also be great to see categories viewed hierarchically, like an index in a book. So the categories would be listed alphabetically and then all the subcategories and articles in the categories would be listed together alphabetically and indented. The categories could be differentiated by either making them bold, italic, or by labeling them as categories. If the subcategories have also been included in the index, their contents would also appear indented in one more level (this could be closed at first and opened using a "+, the same way category trees look. Users might also be able to set the default number of levels that appear -- perhaps two?).

I don't think there is any need to be able to add anything but categories to an INDEX. Adding anything else would probably make it harder to maintain the INDEX, and would probably confuse newbies. Of course, you should be able to create a link to an index page by typing [[:Index:People|Index of people]].

If you think this idea has merit and is a possibility, would it be difficult to implement? It has long been my understanding that category unions would be much less server intensive than category intersections. Perhaps each INDEX display process could be done dynamically?

Thanks, -- ☑ SamuelWantman 08:38, 29 February 2008 (UTC)

From the initial response from Symetrical, it looks like this has mySQL limitation problems as well... -- ☑ SamuelWantman 08:38, 29 February 2008 (UTC)

[edit] Property categories and other thoughts

I've noticed an occasional convention on English wikipedia that Category:Cities (plural) is for things that are cities, while Category:City (singular) is for stuff about the topic "City". -- Skierpage (talk) 05:18, 16 April 2008 (UTC)

[edit] Database memory X Server performance/ Wait time Trade-off

I've read most of the project page about the category intersection project; but I see it lacks a mid-term for the project. It was listed an example that included 'American People' then 'Film Directors'. I've noticed this could be a bit overwhelming to do 'on the fly' as the page mentioned; as both categories are relatively large, being 'American People' especially huge. This kind of approach maximizes memory use for the category pages, as it would spare 1 category as you would just 'request' a fusion of categories while navigating. But maybe wouldnt this case be a little extremist? A more convienient solution, imo, would be to keep the biggest intersective categories (saving performance), while having small categories only to be possible to be subject of intersection. Like this we would, over a transition time, keep categories as they are. No changes; no massive relocating in Wikipedia's core structure and lots of work for users; instead, just one possibility: intersect (possibly like a system mentioned on proj page) two or more categories from request. BUT (and heres the change), you may only intersect categories with under X articles. Like this we would keep major intersections (they could work as normal intersections, only they would be alredy stored in the DB), while removing smaller unnecessary ones (many were removed lacking notability, which wouldnt happen anymore). Let me list one example. Someone is looking for some real-time strategy game. But he is looking for platform ones, especifically PS2. Also, he wants the ones released in 2006.

So, acording to what was proposed at project page, we should have this:

 Real-time Strategy games - PS2 games - Games realeased in 2006

You may notice that all categories are huge, so for the server to make such an intersection would take time; server load. Now look at my proposal:

 Real-time strategy games - Ps2 games released in 2006 (1) OR

 Real-time strategy games - Ps2 games - US Platform Games realeased in 2006(2)

Notice that categories mentioned in suggestion 1 alredy exist, and they have both medium sizes. So here intersection comes as a very useful and not much server-expensive feature, excluding the necessity of categories of uncertain notability (ps2 real-time strat games of 2006, in the example), while keeping most categories we have now with a gradual detetion of the alredy existant lesser intersections. (And by doing so decreasing a bit database memory use, while having ocasionally-used intersection tool.) Frequent intersections could be saved as a category by the server, but they would still be seen as a regular intersection.

Feel free to add anything as you please; or to disagree. --Almighty11 (talk) 02:20, 4 March 2008 (UTC)

The main benefit of category intersection is that it allow users to intersect categories in ways that just are not possible in the current system. For example, intersecting Disasters, Events in China, and Events in 1935, or Films from 1963 and Comedy films. -- ☑ SamuelWantman 19:51, 8 March 2008 (UTC)

[edit] Redundancy and hidden categories

First, I'd like to know how this proposal relates to Wikipedia talk:Categorization#Hidden categories. I think the hidden categories is a tremendously hideous idea, though the intersection is very interesting.

Second, the example listed is redundant. The example shown is that Lawrence Fischburne's categories would be

Categories: 1961 births | American people | People of African descent | Actors |
 Child actors | Film actors | Television actors | Living people |
 People from Georgia (US State) | People from Augusta, Georgia | Tony Award winners

However, this is redundant. All People from Augusta, Georgia are consecquetially People from Georgia (US State), and all Child actors are consequentially Actors. Surely you can easily see the redundancy. Often I will remove redundant categories such as these from articles, and I refuse to include a main category when an article is already in its subcategory. I like the checkbox format to intersect cateogries, but there must be a way to do it wothout putting an article in both a subcategory and the main category. Reywas92Talk 19:05, 8 March 2008 (UTC)

First: This proposal predates the hidden category proposal. There is widespread support for implementing some sort of category intersection by both developers and long-time editors. The reason it hasn't happened yet is technical, involving managing the server load that will result. Once implemented, intersection categories will most likely be deleted because they will be able to be created dynamically. If the hidden category proposal were implemented, the result would look very similar to this category intersection proposal.
Second: It is the removal of "redundant" categories that is one of the biggest problems with the current categorization system. I think you are thinking of categories as a way of classifying articles, so any additional categorization in the same taxonomy would be redundant. However, classifying articles is just one of the functions of categories, and in my view, not the most important one. Categories help users browse through similar articles. When you remove a redundant categorization, you make it much harder to browse through similar articles. To use your example of child actors, when you remove the redundancy of including them in the parent category, you make the parent much less useful because a large portion of the members will be missing. This case is the reason why we have WP:SUBCAT -- a guideline NOT to remove the redundancy which I hope you will read. Just because smaller subunits are created does not mean that the parent category is no longer useful for browsing, or that the children will be more useful than the parents.
Category intersection requires some of these redundancies, and their necessity is described in the proposal. To be redundant, a category must be fully described by the intersection of two other categories. In the case of "Child actors", what two categories would you intersect to have a result that only includes children actors? You could not intersect "Children" and "Actors" (this would result in people who are currently children and actors, but not former children actors), you could not intersect "Children performers" and "Actors" (this would mistakenly include children that were other types of performers who became actors later in life). See the problem? -- ☑ SamuelWantman 19:30, 8 March 2008 (UTC)
In most instances, I still don't think both the parent and child categories should be listed, but fine. About hidden categories again, if this is implemented, then then Hidden cats for normal categories won't need to be, right? I like how it hides maintenance categories, but the rest must be shown. I didn't read everything, but Interesection seems simpler than hidden categories. Reywas92Talk 20:13, 8 March 2008 (UTC)

[edit] Problem with redundancy and better proposal

The more I think about this, the more I don't like it. Above, you say that in many cases redundancy of categories would be necessary (Child actors and Actors) to allow for better intersection selections. Although in a few cases (which I disagree with) it may be preferred to have both the child and parent categories, it can really get out of hand.

For example, Warren Buffett. If Laurence Fishburne should be in both Child actors and Actors and in both People from Augusta, Georgia and People from Georgia (US State), then what about a little further. For the most possible intersections available on one article, shouldn't Warren Buffett be categorized in People from Omaha, Nebraska, People from Harrison County, Iowa, People from Iowa, American people, North American people, AND People? If the redundancy of two lower-level cats is advocated, why not all of them? North American people DOES apply to Warren Buffet, and someone may want to intersect that. What about listing him in 1930 births, 1930s births, 20th century births, AND Births of the last 123 years? They all apply, and someone may want to intersect them, but that's just too much redundancy on the article.

Here's a better idea: Rather than having checkboxes on every pages next to every category with an intersection button, why not have a special page with the tool? It could be Special:Category intersection, and it would simply have spaces for the categories to be intersected. It is much simpler than effecting every page, and is is surely no harder to implement. Reywas92Talk 20:23, 10 March 2008 (UTC)

What I'd really like to see is some well defined hierarchies that are integrated into how we do category intersection. So for example, if as you say -- Warren Buffett can be categorized in People from Omaha, Nebraska, People from Harrison County, Iowa, People from Iowa, American people, North American people, and People -- Perhaps only one of these categories is displayed on his page (probably People from Omaha) but there'd be a pull down list that would have all the other layers of the hierarchy. Currently, hierarchy, intersection, and related attributes are all combined into the categorization system and all look the same -- they are all subcategories. -- ☑ SamuelWantman 23:40, 11 March 2008 (UTC)
I actually like the pull-down idea, though only if it was small. My biggest problem is big categories. Some say that large categories are okay and make searchability easier, but when there are more than 200 members, it gets rediculous. Also, as some categories are sub-cats of many others, which one would go in the box? Many categories are not for articles, anyway. Reywas92Talk 23:47, 11 March 2008 (UTC)