|
pmcannotate is a tool that prints out sources of a tool (in C or
assembly) with inlined profiling informations retrieved by a prior pmcstat analysis. If compared with things like callgraph generation, it prints out profiling on a per-instance basis and this can be useful to find, for example, badly handled caches, too high latency instructions, etc. The tool usage is pretty simple: pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj where samples.out is a pmcstat raw output and binaryobj is the binary object that has been profiled and is accessible for (ELF) symbols retrieving. The options are better described in manpages but briefly: - a: performs analysis on the assembly rather than the C source - h: usage and informations - k: specify a path for the kernel in order to locate correct objects for it - l: specify a lower boundary (in total percentage time) after which functions will be displayed nomore. A typical usage of pmcannotate can be some way of kernel annotation. For example, you can follow the steps below: 1) Generate a pmc raw output of system samples: # pmcstat -S ipm-unhalted-core-cycles -O samples.out 2) Copy the samples in the kernel building dir and cd there # cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd /usr/src/sys/i386/compile/GENERIC/ 3) Run pmcannotate # pmcannotate -k . samples.out kernel.debug > kernel.ann In the example above please note that kernel.debug has to be used in order to produce a C annotated source. This happens because in order to get the binary sources we rely on the "objdump -S" command which wants binary compiled with debugging options. If not debugging options are present assembly analynsis is still possible, but no C-backed one will be available. objdump is not the only one tool on which pmcannotare rely. Infact, in order to have it working, pmcstat needs to be present too because we need to retrieve, from the pmcstat raw output, informations about the sampled PCs (in particular the name of the function they live within, its start and ending addresses). As long as currently pmcstat doesn't return those informations, a new option has been added to the tool (-m) which can extract (from a raw pmcstat output) all pc sampled, name of the functions and symbol bundaries they live within. Also please note that pmcannotate suffers of 2 limitations. Firstly, relying on objdump to dump the C source, with heavy optimization levels and lots of inlines the code gets difficult to read. Secondly, in particular on x86 but I guess it is not the only one case, the sample is always attributed to the instruction directly following the one that was interrupted. So in a C source view some samples may be attributed to the line below the one you're interested in. It's also important to keep in mind that if a line is a jump target or the start of a function the sample really belongs elsewhere. The patch can be found here: http://www.freebsd.org/~attilio/pmcannotate.diff/ where pmcannotate/ dir contains the code and needs to go under /usr/src/usr.sbin/ and the patch has diffs against pmcstat and Makefile. This work has been developed on the behalf of Nokia with important feedbacks and directions from Jeff Roberson. Testing and feedbacks (before it hits the tree) are welcome. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
> pmcannotate is a tool that prints out sources of a tool (in C or
> assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. [snip] > This work has been developed on the behalf of Nokia with important > feedbacks and directions from Jeff Roberson. Are you "rookie" on IRC, who used to catch me with tons of questions whenever I logged on? > Testing and feedbacks (before it hits the tree) are welcome. The pmcstat changes seem fine. Koshy _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008
14:02:22 +0100): > pmcannotate is a tool that prints out sources of a tool (in C or > assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. > If compared with things like callgraph generation, it prints out > profiling on a per-instance basis and this can be useful to find, for > example, badly handled caches, too high latency instructions, etc. Can this also be used to do some code coverage analysis? What I'm interested in is to enable something, run some tests in userland, disable this something, and then run a tool which tells me which parts of specific functions where run or not. At first I hoped I can use dtrace for this... I had a dtrace training and seen the userland probes in action, where you can trace every ASM instruction, but unfortunately you can not do this with kernel probes. I tried with fbt and syscall on a Solaris 10 machine. I haven't tested with FreeBSD-dtrace yet, but I doubt it is more advanced in this regard than the Solaris dtrace. So I'm still searching. Bye, Alexander. -- We should keep the Panama Canal. After all, we stole it fair and square. -- S. I. Hayakawa http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
I know I am going to really show my FreeBSD ignorance here, but this is a
patch of FreeBSD 8.0 Current isn't it ? Thanks Ray Kinsella On Sun, Nov 23, 2008 at 1:02 PM, Attilio Rao <[hidden email]> wrote: > pmcannotate is a tool that prints out sources of a tool (in C or > assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. > If compared with things like callgraph generation, it prints out > profiling on a per-instance basis and this can be useful to find, for > example, badly handled caches, too high latency instructions, etc. > > The tool usage is pretty simple: > pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj > > where samples.out is a pmcstat raw output and binaryobj is the binary > object that has been profiled and is accessible for (ELF) symbols > retrieving. > The options are better described in manpages but briefly: > - a: performs analysis on the assembly rather than the C source > - h: usage and informations > - k: specify a path for the kernel in order to locate correct objects for > it > - l: specify a lower boundary (in total percentage time) after which > functions will be displayed nomore. > > A typical usage of pmcannotate can be some way of kernel annotation. > For example, you can follow the steps below: > 1) Generate a pmc raw output of system samples: > # pmcstat -S ipm-unhalted-core-cycles -O samples.out > 2) Copy the samples in the kernel building dir and cd there > # cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd > /usr/src/sys/i386/compile/GENERIC/ > 3) Run pmcannotate > # pmcannotate -k . samples.out kernel.debug > kernel.ann > > In the example above please note that kernel.debug has to be used in > order to produce a C annotated source. This happens because in order > to get the binary sources we rely on the "objdump -S" command which > wants binary compiled with debugging options. > If not debugging options are present assembly analynsis is still > possible, but no C-backed one will be available. > objdump is not the only one tool on which pmcannotare rely. Infact, in > order to have it working, pmcstat needs to be present too because we > need to retrieve, from the pmcstat raw output, informations about the > sampled PCs (in particular the name of the function they live within, > its start and ending addresses). As long as currently pmcstat doesn't > return those informations, a new option has been added to the tool > (-m) which can extract (from a raw pmcstat output) all pc sampled, > name of the functions and symbol bundaries they live within. > > Also please note that pmcannotate suffers of 2 limitations. > Firstly, relying on objdump to dump the C source, with heavy > optimization levels and lots of inlines the code gets difficult to > read. Secondly, in particular on x86 but I guess it is not the only > one case, the sample is always attributed to the instruction directly > following the one that was interrupted. So in a C source view some > samples may be attributed to the line below the one you're interested > in. It's also important to keep in mind that if a line is a jump > target or the start of a function the sample really belongs elsewhere. > > The patch can be found here: > http://www.freebsd.org/~attilio/pmcannotate.diff/<http://www.freebsd.org/%7Eattilio/pmcannotate.diff/> > > where pmcannotate/ dir contains the code and needs to go under > /usr/src/usr.sbin/ and the patch has diffs against pmcstat and > Makefile. > > This work has been developed on the behalf of Nokia with important > feedbacks and directions from Jeff Roberson. > > Testing and feedbacks (before it hits the tree) are welcome. > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein > _______________________________________________ > [hidden email] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > [hidden email]" > [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Alexander Leidinger
2008/11/23, Alexander Leidinger <[hidden email]>:
> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22 > +0100): > > > > pmcannotate is a tool that prints out sources of a tool (in C or > > assembly) with inlined profiling informations retrieved by a prior > > pmcstat analysis. > > If compared with things like callgraph generation, it prints out > > profiling on a per-instance basis and this can be useful to find, for > > example, badly handled caches, too high latency instructions, etc. > > > > Can this also be used to do some code coverage analysis? What I'm > interested in is to enable something, run some tests in userland, disable > this something, and then run a tool which tells me which parts of specific > functions where run or not. Yes, this is exactly what it does. You can see traces for any sampled PC and so get a profiling anslysis on a per-instance basis. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Ray Kinsella
2008/11/23, Ray Kinsella <[hidden email]>:
> I know I am going to really show my FreeBSD ignorance here, but this is a > patch of FreeBSD 8.0 Current isn't it ? Yes, it is for 8.0. Is it giving you problems? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
On Mon, 24 Nov 2008, Attilio Rao wrote: > 2008/11/23, Alexander Leidinger <[hidden email]>: >> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22 >> +0100): >> >> >>> pmcannotate is a tool that prints out sources of a tool (in C or >>> assembly) with inlined profiling informations retrieved by a prior >>> pmcstat analysis. >>> If compared with things like callgraph generation, it prints out >>> profiling on a per-instance basis and this can be useful to find, for >>> example, badly handled caches, too high latency instructions, etc. >>> >> >> Can this also be used to do some code coverage analysis? What I'm >> interested in is to enable something, run some tests in userland, disable >> this something, and then run a tool which tells me which parts of specific >> functions where run or not. > > Yes, this is exactly what it does. > You can see traces for any sampled PC and so get a profiling anslysis > on a per-instance basis. I would add that it is only sampled so you don't see every instruction executed. You can use gcov for that however. That's precisely what it's for. Thanks, Jeff > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein > _______________________________________________ > [hidden email] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "[hidden email]" > [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
Quoting Attilio Rao <[hidden email]> (from Mon, 24 Nov 2008
00:46:29 +0100): > 2008/11/23, Alexander Leidinger <[hidden email]>: >> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22 >> +0100): >> >> >> > pmcannotate is a tool that prints out sources of a tool (in C or >> > assembly) with inlined profiling informations retrieved by a prior >> > pmcstat analysis. >> > If compared with things like callgraph generation, it prints out >> > profiling on a per-instance basis and this can be useful to find, for >> > example, badly handled caches, too high latency instructions, etc. >> > >> >> Can this also be used to do some code coverage analysis? What I'm >> interested in is to enable something, run some tests in userland, disable >> this something, and then run a tool which tells me which parts of specific >> functions where run or not. > > Yes, this is exactly what it does. > You can see traces for any sampled PC and so get a profiling anslysis > on a per-instance basis. Cool. Would be great if you could provide some example in the man page or as a script which shows how to do this for kernel code. This would immediately show us how good our regression tests are for their specific areas of test. Bye, Alexander. -- In a family recipe you just discovered in an old book, the most vital measurement will be illegible. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Jeff Roberson-2
Quoting Jeff Roberson <[hidden email]> (from Sun, 23 Nov 2008
13:50:35 -1000 (HST)): > > On Mon, 24 Nov 2008, Attilio Rao wrote: > >> 2008/11/23, Alexander Leidinger <[hidden email]>: >>> Quoting Attilio Rao <[hidden email]> (from Sun, 23 Nov 2008 14:02:22 >>> +0100): >>> >>> >>>> pmcannotate is a tool that prints out sources of a tool (in C or >>>> assembly) with inlined profiling informations retrieved by a prior >>>> pmcstat analysis. >>>> If compared with things like callgraph generation, it prints out >>>> profiling on a per-instance basis and this can be useful to find, for >>>> example, badly handled caches, too high latency instructions, etc. >>>> >>> >>> Can this also be used to do some code coverage analysis? What I'm >>> interested in is to enable something, run some tests in userland, disable >>> this something, and then run a tool which tells me which parts of specific >>> functions where run or not. >> >> Yes, this is exactly what it does. >> You can see traces for any sampled PC and so get a profiling anslysis >> on a per-instance basis. > > I would add that it is only sampled so you don't see every > instruction executed. You can use gcov for that however. That's > precisely what it's for. How to use gcov for the kernel? Bye, Alexander. -- If only you knew she loved you, you could face the uncertainty of whether you love her. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
No, not at all.
I am using 6.2 and 7.0 at the moment, I will build another disk with FreeBSD 8.0-CURRENT. Thanks Ray Kinsella On Sun, Nov 23, 2008 at 11:47 PM, Attilio Rao <[hidden email]> wrote: > 2008/11/23, Ray Kinsella <[hidden email]>: > > I know I am going to really show my FreeBSD ignorance here, but this is a > > patch of FreeBSD 8.0 Current isn't it ? > > Yes, it is for 8.0. > Is it giving you problems? > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein > [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
At Sun, 23 Nov 2008 14:02:22 +0100,
Attilio Rao wrote: > > pmcannotate is a tool that prints out sources of a tool (in C or > assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. > If compared with things like callgraph generation, it prints out > profiling on a per-instance basis and this can be useful to find, for > example, badly handled caches, too high latency instructions, etc. > > The tool usage is pretty simple: > pmcannotate [-a] [-h] [-k path] [-l level] samples.out binaryobj > > where samples.out is a pmcstat raw output and binaryobj is the binary > object that has been profiled and is accessible for (ELF) symbols > retrieving. > The options are better described in manpages but briefly: > - a: performs analysis on the assembly rather than the C source > - h: usage and informations > - k: specify a path for the kernel in order to locate correct objects for it > - l: specify a lower boundary (in total percentage time) after which > functions will be displayed nomore. > > A typical usage of pmcannotate can be some way of kernel annotation. > For example, you can follow the steps below: > 1) Generate a pmc raw output of system samples: > # pmcstat -S ipm-unhalted-core-cycles -O samples.out > 2) Copy the samples in the kernel building dir and cd there > # cp samples.out /usr/src/sys/i386/compile/GENERIC/ ; cd > /usr/src/sys/i386/compile/GENERIC/ > 3) Run pmcannotate > # pmcannotate -k . samples.out kernel.debug > kernel.ann > > In the example above please note that kernel.debug has to be used in > order to produce a C annotated source. This happens because in order > to get the binary sources we rely on the "objdump -S" command which > wants binary compiled with debugging options. > If not debugging options are present assembly analynsis is still > possible, but no C-backed one will be available. > objdump is not the only one tool on which pmcannotare rely. Infact, in > order to have it working, pmcstat needs to be present too because we > need to retrieve, from the pmcstat raw output, informations about the > sampled PCs (in particular the name of the function they live within, > its start and ending addresses). As long as currently pmcstat doesn't > return those informations, a new option has been added to the tool > (-m) which can extract (from a raw pmcstat output) all pc sampled, > name of the functions and symbol bundaries they live within. > > Also please note that pmcannotate suffers of 2 limitations. > Firstly, relying on objdump to dump the C source, with heavy > optimization levels and lots of inlines the code gets difficult to > read. Secondly, in particular on x86 but I guess it is not the only > one case, the sample is always attributed to the instruction directly > following the one that was interrupted. So in a C source view some > samples may be attributed to the line below the one you're interested > in. It's also important to keep in mind that if a line is a jump > target or the start of a function the sample really belongs elsewhere. > > The patch can be found here: > http://www.freebsd.org/~attilio/pmcannotate.diff/ > > where pmcannotate/ dir contains the code and needs to go under > /usr/src/usr.sbin/ and the patch has diffs against pmcstat and > Makefile. > > This work has been developed on the behalf of Nokia with important > feedbacks and directions from Jeff Roberson. > > Testing and feedbacks (before it hits the tree) are welcome. > First of all, this is excellent work. As soon as this and some other changes in PMC hit 7.x I'll be rolling this out to all the developers I work with. I've tested this on amd64 on HEAD, and with the changes we have talked about privately (%jx vs. %x) it works quite well. Secondly, I would like to request a feature. I would like to be able to get output in a more easily parsable format so I can write some Emacs code to highlight C code with the output. I'd like something along the lines of: path:function:line:percentage Keep up the good work! Later, George _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
|
In reply to this post by Attilio Rao-2
Hi Attilio Rao!
On Sun, 23 Nov 2008 14:02:22 +0100; Attilio Rao wrote about '[PATCH] pmcannotate tool': > pmcannotate is a tool that prints out sources of a tool (in C or > assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. > If compared with things like callgraph generation, it prints out > profiling on a per-instance basis and this can be useful to find, for > example, badly handled caches, too high latency instructions, etc. [...] > objdump is not the only one tool on which pmcannotare rely. Infact, in > order to have it working, pmcstat needs to be present too because we > need to retrieve, from the pmcstat raw output, informations about the > sampled PCs (in particular the name of the function they live within, > its start and ending addresses). As long as currently pmcstat doesn't > return those informations, a new option has been added to the tool > (-m) which can extract (from a raw pmcstat output) all pc sampled, > name of the functions and symbol bundaries they live within. [...] > The patch can be found here: > http://www.freebsd.org/~attilio/pmcannotate.diff/ > where pmcannotate/ dir contains the code and needs to go under > /usr/src/usr.sbin/ and the patch has diffs against pmcstat and > Makefile. It looks like this the thing I need, please see parallel thread "hwpmc granularity and 6.4 network performance", I have CPU hog ipfw_chk() which is 1200+ lines long and want to know which parts are most hogs. But that part of your patch which is for pmcstat isn't applying to 6.4 :( Could you please provide another for this ver? -- WBR, Vadim Goncharov. ICQ#166852181 mailto:[hidden email] [Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight] _______________________________________________ [hidden email] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[hidden email]" |
| Powered by Nabble | Edit this page |
