User talk:William Pietri/Archive 01

From Wikipedia, the free encyclopedia

This is an archive of past discussions. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.

Downloading log files?

Hello William, welcome to Wikipedia. I've moved your question about downloading log files here as it was time to clean the Village pump. Angela. 14:42, Sep 1, 2004 (UTC)

Does anybody know where I can download a few days worth of Wikipedia hit log files? I'd like to do some analysis on them, but I can't figure out where they're kept.

Thanks, --William Pietri 00:30, 28 Aug 2004 (UTC)

We don't have daily log files. We have log files for the current month, updated nightly. If you take the difference from one nightly log file to the next, you can calculate the day's hits. →Raul654 00:37, Aug 28, 2004 (UTC)

Ah, I was looking for the raw log files, with Referer information. I was curious do some analysis on Wikipedia usage patterns, including how people come in via search engines and what they do after that point. But I gather you're saying that raw hit logs just aren't collected on a regular basis? In that case, are there older ones around, perhaps collected during an optimization run? --William Pietri 00:44, 28 Aug 2004 (UTC)

The raw logs are available only to wikimedia's developers. Publishing them would be a major issue, as in general we view people's IPs, the sites and pages they visit, and what they do here to be private information. A lot of people would be very pissed off if the raw logs were available to the casual visitor. I suppose if there were such a thing as a decent unified logfile sanitiser (which removed/abstracted IP information) then publishing that would be an option. But if I were running such a sanitiser, I'd surely have it purge the referrer information too, as (occasionally) there's juicy info in the referring URL (particularly when the referrer is an email in a webmail service, and sometimes a search query). That said, there doesn't seem to be such a program anyway (I guess I don't know what it would be called, so it's rather hard to google for it). The problem of examining the log files is compounded by wikimedia's server setup - most pages are served ex-cache by one of the frontline squid servers; so we'd need to publish a number of the (huge) squid logs too. It would indeed be a very interesting exercise to run some analyses of the logs, as there's all kinds of things we don't know about how visitors and search engines enter, move around, and leave the site. If we could figure out an acceptable way to do things then there is much value to be mined. -- Finlay McWalter | Talk 00:58, 28 Aug 2004 (UTC)

Right, logs are private. You can find some statistics extracted from log files, including a list of referrers, on the Webalizer stats page. -- Tim Starling 01:28, Aug 28, 2004 (UTC)

Interesting. I hadn't thought about the privacy issues, but they make complete sense to me. If I get enough time to do the research properly, I'll contact y'all with a proposal to either A) write a log sanitizer that meets your standards, or B) write an analysis program that you guys run, so I never see the log data. Also, thanks for the link to the Webalizer pages; I had missed those the first time through. Thanks, --William Pietri 20:54, 30 Aug 2004 (UTC)

When you write your log file analysis program, keep in mind that the squid logs for each day come to about five gigabytes. Also, depending on how much spare CPU time there is, running the analyzer might be done only intermittently, as has been the case with the Webalizer statistics. It will be interesting to see the results of this. Until now, everybody wanting to do this has been all talk and no code. -- Jeronim 05:26, 1 Sep 2004 (UTC)

Gilian

The answer is out there somewhere! Seriously I doubt it cause i didn't babysit anyone with that name. Bye, Gillian