Talk:Performance analysis

From Wikipedia, the free encyclopedia

Contents

[edit] Not synonym

The page used to give "program analysis" as a synonym, but that is misleading to my ears. I have clarified it to "dynamic program analysis". --Lexspoon 14:59, 9 April 2006 (UTC)

[edit] Wording change

Changed "In computer science, ..." to "In software engineering, ...". I think realistically, performance analysis is something engineers do--and only when they have to. --Jorend 12:54, 14 April 2006 (UTC)


[edit] Disam?

Shouldn't a profiling disambig be made? 134.193.168.234 23:18, 15 June 2006 (UTC) site:en.wikipedia.org profiling

[edit] Extern links just advertising

Needs to be trimmed. All the tools listed could be made into a list in the article (without the extern links), but as it is, the links offer no further info on the topic, just ads for various tools. — Frecklefoot | Talk 14:58, 15 August 2006 (UTC)

Better than none. :-) --Leo 19:44, 27 October 2006 (UTC)
Please don't delete that collection of links, maybe create a list page sorted by language and profiler features though, like list of unit testing frameworks. --Chris Pickett 01:48, 4 December 2006 (UTC)

[edit] Systems?

What about the performance analysis that Systems Administrators(/Programmers/Whatever) do?

[edit] How about static analysis?

Shouldn't this page also say something about static analysis? Maybe also a link to Algorithmic efficiency could be added. --Bernard François 21:08, 2 January 2007 (UTC)

[edit] Whither profiling?

Forgive me, but I've never seen any profiler that is nearly as effective as the simple technique of getting the program in question running under a debugger, hitting the "pause" button, and then examining the call stack. If this is done several times, if there is a slowness bug in the program, it becomes very obvious. I hate to sound like I'm voicing just a personal opinion, but this is easily put to an objective test. Any comments? 65.96.64.14 21:41, 12 March 2007 (UTC)

Low-impact tracing profilers allow you to cover far more ground quickly, and on applications that cause debuggers to choke. Example for the former is a GUI application which is sluggish, but not slow enough to allow you to switch applications to the debugger and pause. I ran into this when tuning the performance of a code editor product - it was just slow enough to be annoying, without being so slow you could apply simple tricks to it.

For the latter, consider a D3D game. Trying to pause a fullscreen game in a debugger is a nightmare, as you lose your video resources when you switch apps. If you tried to sample it with the debugger technique above, you'd be spending all your time in the driver, loading resources.

--Jim —Preceding unsigned comment added by 203.173.169.118 (talk) 03:17, 3 September 2007 (UTC)

Pardon me, but you don't need to "cover ground". Performance problems are not needles in haystacks. The worse they are, the more obvious they are. But you are right, dropping in on a program at random is not always easy to do. For the first problem, if something takes less than a second, you can put a loop around it - do it 100 times, find/fix the problem, then take away the loop. For the second problem, I've done extreme things to get call stack samples. On one project, I used an in-circuit emulator. On another, I made an "alarm-clock" signal dump the program. On another the CPU had a "halt" button on the front panel - then I could toggle through memory and read the lights. It wouldn't be worth the trouble if there were a better way. In head-to-head comparisons, "stackshots" show the problem immediately. Profilers only give statistics, graphs, and colorful displays to puzzle over. MikeDunlavey 20:30, 18 October 2007 (UTC)

I think all we've really managed to say here is that the current breed of profilers aren't doing their job properly. I'm personally quite fond of a little .NET tool that takes you straight to your performance problem. Run it up, do the slow thing, let it crunch the numbers, and `hey, here's your perf issue` a couple seconds later. Sure, you could do it by hand, but I'd rather automate the bits I can. However, I'm a profiler vendor, so I'm biased ;) --Jim —Preceding unsigned comment added by 203.97.223.50 (talk) 21:28, 23 October 2007 (UTC)

Understood, but I bet the programs you profile are either small or have lots of small functions. I'm accustomed to ugly million-liners, and profilers I've seen tell you piles of nothing. I built a tool once to automate the stack-sampling approach. It had a graph-browser style UI. It worked pretty well, but still took longer than the manual method which lets you see all the context and really understand what it's doing. Just last week I pinpointed three major problems in less time than my teammate could even think about setting up the profiler. I could say this is what it's spending X% of its time doing and maybe it doesn't need to. Result - massive speedup.

I am just trying to awaken the gut understanding that if the software is doing something unnecessary and making you wait, then it's doing it now, while you're waiting, and if you interrupt it chances are you'll catch it in the act, like an employee trying to "look busy". You don't need fine-tooth-comb detective work, intuition, cleverness, precise timing, execution counts, statistics, or graphs. What has me completely baffled is that this has not been totally obvious since day 1 of computers.

In my opinion, what the next breed of profilers should do is automate the manual technique that works, namely:

  • sampling the entire call stack, not just the program counter or a limited number of levels.
  • sampling only during during the interval of interest.
  • taking only a moderate number of samples. 20 is enough. 100 is more than enough.
  • computing for each address that appears on call stacks the percent of call stack samples it appears on. Note that this eliminates concerns about recursion. If a statement appears on a call stack sample, that counts as 1, even if it appears 10 times on that one sample.
  • displays the statements in decreasing order by that percentage. Then the user looks down that list for something he/she can optimize. If a statement can be eliminated or not called, that percentage is roughly how much time will be saved.
  • presentation: some sort of statement-call-graph would be OK, but not if it is only at the function level. It needs to retain call-site information. It should also let you examine individual call stack samples and the source code at each level so you can see why that particular nanosecond was being spent.

All of this is public-domain information. Anyone who wants to can build and sell such a profiler.

People will ask "Isn't this what current profilers do?" The answer is NO, they don't. They lose information right and left. For sampling, the PC alone is such a small part of the program's state that it tells you almost nothing. For instrumentation (function timing and call graph capture) it loses crucial call-site information, not to mention calls to functions that are not instrumented, like library functions. Then, to further fog the issue, they tell you lots of other stuff that you might care about but have only indirect effect on performance, such as cache misses, memory usage, and code coverage. MikeDunlavey 13:39, 11 November 2007 (UTC)

[edit] Monte Carlo Profiler?

To my knowledge "Monte Carlo Profilers" work by sampling the program counter on a timer interrupt.

Yes, and some of them also sample the entire call stack. I believe the Sun profiler is the only one (currently) that allows you to figure out, for individual call statements (as opposed to entire functions), what fraction of the time they are active on the stack.. In my experience, this is the number that matters. What's more, most of the information comes in the first few samples, the rest only giving needless precision. For example, if a program takes 20 seconds to run, and in 5 out of 10 samples the instruction "file foo.c, line 101: call bar()" appears, removing that call, if possible, would save 10 seconds (more or less). If bar() takes 10 seconds and is called once, or takes 10 microseconds and is called a million times makes no difference. Of course, removing the entire function bar() (or speeding it way up) would get the same effect, but the call is more likely to be removeable than the entire function.

To put it another way, different instructions take different amounts of time. "fadd" (floating point add) takes a lot more cycles than "mov eax,ebx" (move register). This shows up in the program counter histogram, because the timer interrupt is more likely to occur in the longer instruction. It makes sense, then, to try to use fewer of such instructions, so the program will take fewer cycles. However, some instructions take millions of times longer, and would help enormously if they could be removed, but they are essentially ignored by the program counter histogram. An example is "call _fprintf". If when a timer interrupt occurred, not only the PC were histogrammed, but every unique return address on the call stack, this deficiency would be remedied. If that slows down the program being profiled, so what? The object is to find out what needs to be optimized, not to measure how fast it isn't. MikeDunlavey 20:41, 24 May 2007 (UTC)

[edit] Time to re-org this page?

As it is, this page sort of assumes these things are the same:

  • performance analysis = profiling = measuring performance = diagnosing performance problems = optimizing performance

I wonder if it makes sense to re-arrange it to split these ideas apart.

Any comments? Thanks, MikeDunlavey 14:29, 4 June 2007 (UTC)

[edit] Linkfarm

I think all the external links in the list of profilers should be removed per WP:SPAM, WP:EL, and WP:NOT#LINK. --Ronz 23:38, 25 July 2007 (UTC)

But since the tools are external, creating a page for each one would be creating an article about a product, which isn't allowed WP:SPAM. Surely an external link is more appropriate than a dead link or no link at all. Ian Broster 10:25, 23 August 2007 (UTC)

I think that there should be some kind of ordering on the links, or at least some sense of what is 'acceptable' when adding a new link:

- Alphabetical order would work.
- If someone wanted to actually evaluate all these tools, and rank them by merit, that would work too, but likely take a significant amount of effort.
- First-in, first-served also works.
- Any other well-defined ordering.

What I'm taking issue with here, is when some user (often a tool vendor) comes along and places a link to their product at the very top of the list. Frankly, its rude, and very much WP:SPAM. Instead, everyone should be considerate, and place their link either:

- In the correct place, if the list has an ordering.
- Discreetly in the middle of the list, or better, toward the bottom. Wiki isn't here to advertize your product.

-- Jim (203.173.169.118)

You can't rank the tools by merit...we would never agree on criteria :-)

Can we just go back to Alphabetical ordered links? Or follow the suggestion above from Chris Pickett to be like list of unit testing frameworks. Ian Broster 08:11, 3 September 2007 (UTC)

See WP:LIST. We can try to come up with an alternative criteria for inclusion in the list other than what I'm proposing, which is to only include tools that have their own article (or have appropriate mention in another article such as a section about a tool in an article about the company that makes the tool). --Ronz 15:29, 3 September 2007 (UTC)

Most tools cannot have their own article. Most companies cannot have their own article. We would end up with a very short list of the few tools that are famous enough (or free enough that it's not seen as advertising!) ... this would not make an informative or neutral list. The criteria for entry to the "List of profilers" is surely as simple as "Is a profiler". Nothing to do with whether the tool is GPL, or how large the company is. Ian Broster 16:19, 3 September 2007 (UTC)

Yep. Either that or we come up with another criteria. Can we find a list of these tools to refer too, preferably as part of a WP:RS? --Ronz 17:17, 3 September 2007 (UTC)

There's a couple links in the C/C++ section that don't make sense. In particular, Shark and Ants are probably not linking to the correct pages. --Jim —Preceding unsigned comment added by 203.173.169.118 (talk) 04:42, 13 September 2007 (UTC)