ULE Scheduler

[Momchil Ivanov-3]

Hi,

today I had a really strange experience with my laptop. I ran 2 lisp
processes each consuming 100% CPU (i.e. in top, 100% CPU means one
full core), since I have 2 cores, it means full processor load. I was
moreover running Emacs and Opera, but that is more or less irrelevant
since they consume negligible amount of CPU time. At some point my
Xorg died and I was dropped in the terminal and the first thought
through my mind was that my laptop just said "goodbye, it was nice
meeting you after 4+ years". A second later I saw:

Jun  7 01:00:06 t61 kernel: acpi_tz1: WARNING - current temperature (100.1C) exceeds safe limits

which was a sign of relief: "oh, maybe the fan got busted...". So I
took a screw driver and disassembled my Lenovo T61. Cleaning all the
dust and putting a fresh amount of thermal fluid (did it 1 year ago),
I booted again and started both processes again and took a look at the
temperature. It was constantly increasing from about 33 C. I took a
look at top and saw that both processes were wildly jumping accross
the cores, i.e. CPU0 and CPU1.

So before reading all the papers about the ULE scheduler and the
source code, I would like to as a simple question: is it that stupid?

I mean, there are just 2 processes running (except of top, X and
... which should be scheduled occasionally) on 2 cores of one physical
processor. Why sould each be scheduled on a different core each time?

I did cpuset to pin each to a specific core and got to about a
constant temperature of 72 C. I am affraid to "cpuset -l 0,1 -p <...>"
both of them since I might again get at 100 C.

Is there some remedy?

Please CC me, since I am not subscribed to the list.

Regards,
Momchil
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[dougb]

On 06/06/2012 18:01, Момчил Иванов wrote:
> Is there some remedy?

Try the 4BSD scheduler.

--

    This .signature sanitized for your protection
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Erich Dollansky-2]

Hi,

On 07 June 2012 3:01:07 Момчил Иванов wrote:

> temperature. It was constantly increasing from about 33 C. I took a
> look at top and saw that both processes were wildly jumping accross
> the cores, i.e. CPU0 and CPU1.
>
> So before reading all the papers about the ULE scheduler and the
> source code, I would like to as a simple question: is it that stupid?

maybe, maybe not. It could be that the difference is minor as the cache for both kernels is in the same chip.
>
> I mean, there are just 2 processes running (except of top, X and
> ... which should be scheduled occasionally) on 2 cores of one physical
> processor. Why sould each be scheduled on a different core each time?
>
> I did cpuset to pin each to a specific core and got to about a
> constant temperature of 72 C. I am affraid to "cpuset -l 0,1 -p <...>"
> both of them since I might again get at 100 C.

This would be the interesting point? Did it happen because of the dirt or because or the scheduler.
>
> Is there some remedy?

I think that the only remedy available is the one you applied.

Erich
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Momchil Ivanov-3]

At Thu, 07 Jun 2012 09:12:55 +0700,
Erich wrote:

>
> Hi,
>
> On 07 June 2012 3:01:07 Момчил Иванов wrote:
>
> > temperature. It was constantly increasing from about 33 C. I took a
> > look at top and saw that both processes were wildly jumping accross
> > the cores, i.e. CPU0 and CPU1.
> >
> > So before reading all the papers about the ULE scheduler and the
> > source code, I would like to as a simple question: is it that stupid?
>
> maybe, maybe not. It could be that the difference is minor as the cache for both kernels is in the same chip.
> >
> > I mean, there are just 2 processes running (except of top, X and
> > ... which should be scheduled occasionally) on 2 cores of one physical
> > processor. Why sould each be scheduled on a different core each time?
> >
> > I did cpuset to pin each to a specific core and got to about a
> > constant temperature of 72 C. I am affraid to "cpuset -l 0,1 -p <...>"
> > both of them since I might again get at 100 C.
>
> This would be the interesting point? Did it happen because of the dirt or because or the scheduler.
> >
> > Is there some remedy?
>
> I think that the only remedy available is the one you applied.
>
> Erich
>

I've repeated the same experiment just now, setting both processes on
both cores with cpuset. The temperature got to about 72-74 C, so the
two small pieces of dirt that came out, the fresh thermal liquid and
tightening the screws probably gave me 10 about C on idle (from 53 C
down to 45 C) and 30 C on full load. I didn't expect that much...

Though, it was strange seeing both processes hopping around... I will
probably go back to the 4BSD scheduler if my laptop does another
self-shutdown in the next few days as Doug suggested.

Regards,
Momchil
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Erich Dollansky-2]

Hi,

On 07 June 2012 10:16:07 Momchil Ivanov wrote:
> At Thu, 07 Jun 2012 09:12:55 +0700,
> Erich wrote:
> >
> I've repeated the same experiment just now, setting both processes on
> both cores with cpuset. The temperature got to about 72-74 C, so the
> two small pieces of dirt that came out, the fresh thermal liquid and
> tightening the screws probably gave me 10 about C on idle (from 53 C
> down to 45 C) and 30 C on full load. I didn't expect that much...
>
this sounds very normal to me. I clean my desktop every two months of operation.

A drop of 10K is normal just for blowing the heat sink.

The best design of a notebook I every have seen was my old Fujitsu. I used it from 2004 until the beginning of the year without any cleaning and any problems with dirt. After it was dead I disassembled it and did not find any dust blocking anything.

> Though, it was strange seeing both processes hopping around... I will
> probably go back to the 4BSD scheduler if my laptop does another
> self-shutdown in the next few days as Doug suggested.

This switch might always help. I did some tests more than a year ago. I left the BSD Scheduler then on the single core Fujitsu notebook mentioned above but used the new one my desktop machine.

It will also depend on the software you are running.

Erich
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Daniel Kalchev]

On 07.06.12 11:16, Momchil Ivanov wrote:
> Though, it was strange seeing both processes hopping around... I will
> probably go back to the 4BSD scheduler if my laptop does another
> self-shutdown in the next few days as Doug suggested.

You never run just two processes on FreeBSD, ever. The kernel too runs
multiple threads.

However small the CPU usage of the other processes is, they must run
from time to time, kicking out at least one of your CPU intensive
processes, possibly kicking them out both, as well. When that happens,
and they are queued to run again it does not matter much on which core
they ran before, because chances are it's cache will be invalidated
anyway. Also, different CPUs have different cache affinity. ULE is
supposed to be aware of this, while the 4BSD scheduler is not.

In any case, on an older single/dual core CPU there is rarely any
difference between both schedulers. Differences might appear in modern
multi-core CPUs..

Daniel
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Martin Sugioarto-3]

Am Thu, 07 Jun 2012 03:01:07 +0200
schrieb Момчил Иванов <[hidden email]>:

> Is there some remedy?

Hi,

I remember this series, I've had a T60p and when I compiled world, I
placed a fan in front of it to cool it down from 100°C. The difference
with T60p was that it simply shut off reaching 101°C.

The problem is the hardware, not FreeBSD. T60p and obviously T60, too,
was made by some crazy people who had the idea to cool the CPU und the
GPU under the same heat sink. The funny thing is that the GPU is
running at 70°C all the time, because FreeBSD does not implement
voltage regulation for the VGA chipset. The result is that the GPU
warms up the CPU to at least 55°C while idle.

If you want to have a cooler CPU implement power saving for the Radeon
chipset there.

Martin

[Momchil Ivanov-3]

At Fri, 8 Jun 2012 00:54:15 +0200,
Martin Sugioarto wrote:

>
> [1  <text/plain; UTF-8 (quoted-printable)>]
> Am Thu, 07 Jun 2012 03:01:07 +0200
> schrieb Момчил Иванов <[hidden email]>:
>
> > Is there some remedy?
>
> Hi,
>
> I remember this series, I've had a T60p and when I compiled world, I
> placed a fan in front of it to cool it down from 100°C. The difference
> with T60p was that it simply shut off reaching 101°C.
>
> The problem is the hardware, not FreeBSD. T60p and obviously T60, too,
> was made by some crazy people who had the idea to cool the CPU und the
> GPU under the same heat sink. The funny thing is that the GPU is
> running at 70°C all the time, because FreeBSD does not implement
> voltage regulation for the VGA chipset. The result is that the GPU
> warms up the CPU to at least 55°C while idle.
>
> If you want to have a cooler CPU implement power saving for the Radeon
> chipset there.
>
> Martin
> [2 signature.asc <application/pgp-signature (7bit)>]
>

Hi,

well, that is not true, I have been using the laptop since more than 4
years without any problems. The thing is that yesterday I had it
docked and that seems to raise the idle temperature by about 10°C, so
I get docked somewhere about 42°C when doing nothing computationally
intensive:

hw.acpi.thermal.tz0.temperature: 50.0C
hw.acpi.thermal.tz1.temperature: 42.0C
dev.cpu.0.temperature: 42.0C
dev.cpu.1.temperature: 42.0C

So I probably have to shift things around to give the dock a bit more
room. However, the dust, the thermal liquid and the screws seem to
have contributed to the temperature increase too. The GPU (Nvidia
Quadro NVS 140M) might be an issue, nvidia-settings says 58°C (I am
not running any fancy graphics) and I've seen it getting over
100-120°C before, when I am doing some opengl stuff. With the latter I
mean, that I know how to intentionally kill it.

Anyway, I have solved my problem and that seems to not be related to
ULE at all. However, it was still surprising for me to find out how
ULE schedules computationally intensive tasks.

Regards,
Momchil
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Andreas Nilsson-8]

On Fri, Jun 8, 2012 at 2:29 AM, Momchil Ivanov <[hidden email]> wrote:

> At Fri, 8 Jun 2012 00:54:15 +0200,
> Martin Sugioarto wrote:
> >
> > [1  <text/plain; UTF-8 (quoted-printable)>]
> > Am Thu, 07 Jun 2012 03:01:07 +0200
> > schrieb Момчил Иванов <[hidden email]>:
> >
> > > Is there some remedy?
> >
> > Hi,
> >
> > I remember this series, I've had a T60p and when I compiled world, I
> > placed a fan in front of it to cool it down from 100°C. The difference
> > with T60p was that it simply shut off reaching 101°C.
> >
> > The problem is the hardware, not FreeBSD. T60p and obviously T60, too,
> > was made by some crazy people who had the idea to cool the CPU und the
> > GPU under the same heat sink. The funny thing is that the GPU is
> > running at 70°C all the time, because FreeBSD does not implement
> > voltage regulation for the VGA chipset. The result is that the GPU
> > warms up the CPU to at least 55°C while idle.
> >
> > If you want to have a cooler CPU implement power saving for the Radeon
> > chipset there.
> >
> > Martin
> > [2 signature.asc <application/pgp-signature (7bit)>]
> >
> Hi,
>
> well, that is not true, I have been using the laptop since more than 4
> years without any problems. The thing is that yesterday I had it
> docked and that seems to raise the idle temperature by about 10°C, so
> I get docked somewhere about 42°C when doing nothing computationally
> intensive:
>
> hw.acpi.thermal.tz0.temperature: 50.0C
> hw.acpi.thermal.tz1.temperature: 42.0C
> dev.cpu.0.temperature: 42.0C
> dev.cpu.1.temperature: 42.0C
>
> So I probably have to shift things around to give the dock a bit more
> room. However, the dust, the thermal liquid and the screws seem to
> have contributed to the temperature increase too. The GPU (Nvidia
> Quadro NVS 140M) might be an issue, nvidia-settings says 58°C (I am
> not running any fancy graphics) and I've seen it getting over
> 100-120°C before, when I am doing some opengl stuff. With the latter I
> mean, that I know how to intentionally kill it.
>
> Anyway, I have solved my problem and that seems to not be related to
> ULE at all. However, it was still surprising for me to find out how
> ULE schedules computationally intensive tasks.
>
> Regards,
> Momchil
>

My t61p also had overheating problems with fbsd, but never in linux. For me
the fan control was somewhat broken: I had to turn off auto-mode and set
max myself to get any heavy usage out of it.

You might want to check that as well.

Regards
Andreas
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Martin Sugioarto-3]

Am Fri, 8 Jun 2012 08:04:12 +0200
schrieb Andreas Nilsson <[hidden email]>:

> My t61p also had overheating problems with fbsd, but never in linux.
> For me the fan control was somewhat broken: I had to turn off
> auto-mode and set max myself to get any heavy usage out of it.
>
> You might want to check that as well.
>
> Regards
> Andreas

I checked this and my fan was already at full speed (all the time).
Linux has got voltage regulation in fglrx as far as I know. As soon as
the GPU was using power saving functions on MS-Windows the GPU
temperature was far below 50°C. With FreeBSD it was always at 70°C!

I don't have the T60p anymore (thanks god). It was an awful laptop. So
I cannot give more precise information about this anymore.

Martin

[Adrian Chadd-2]

Did anyone ever file a PR for this kind of thing?



Adrian


On 8 June 2012 10:07, Martin Sugioarto <[hidden email]> wrote:

> Am Fri, 8 Jun 2012 08:04:12 +0200
> schrieb Andreas Nilsson <[hidden email]>:
>
>> My t61p also had overheating problems with fbsd, but never in linux.
>> For me the fan control was somewhat broken: I had to turn off
>> auto-mode and set max myself to get any heavy usage out of it.
>>
>> You might want to check that as well.
>>
>> Regards
>> Andreas
>
> I checked this and my fan was already at full speed (all the time).
> Linux has got voltage regulation in fglrx as far as I know. As soon as
> the GPU was using power saving functions on MS-Windows the GPU
> temperature was far below 50°C. With FreeBSD it was always at 70°C!
>
> I don't have the T60p anymore (thanks god). It was an awful laptop. So
> I cannot give more precise information about this anymore.
>
> Martin

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[dougb]

On 06/06/2012 18:16, Doug Barton wrote:
> On 06/06/2012 18:01, Момчил Иванов wrote:
>> Is there some remedy?
>
> Try the 4BSD scheduler.

Did you ever try this? Did it help?


--

    This .signature sanitized for your protection


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Momchil Ivanov-3]

At Sat, 09 Jun 2012 20:23:44 -0700,
Doug Barton wrote:
>
> On 06/06/2012 18:16, Doug Barton wrote:
> > On 06/06/2012 18:01, Момчил Иванов wrote:
> >> Is there some remedy?
> >
> > Try the 4BSD scheduler.
>
> Did you ever try this? Did it help?

I compiled the same kernel with the 4BSD scheduler today and it seems
that the processes jump accross cores too. My "eye" measure with "top"
fells like it's more stable and probably converges faster to a stable
state after "top" jumps accross cores. But in order to talk with
numbers, I need to replace "top" with somethings that dumps the
process number and the cpu id continuously in order to get some
statistics out of it, otherwise you can just forget all the things
that I have written. Is there an easy way to do that and are you
interested?

Regards,
Momchil
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Lars Engels-2]

On Mon, Jun 11, 2012 at 09:06:26PM +0200, ???????????? ???????????? wrote:

> At Sat, 09 Jun 2012 20:23:44 -0700,
> Doug Barton wrote:
> >
> > On 06/06/2012 18:16, Doug Barton wrote:
> > > On 06/06/2012 18:01, ???????????? ???????????? wrote:
> > >> Is there some remedy?
> > >
> > > Try the 4BSD scheduler.
> >
> > Did you ever try this? Did it help?
>
> I compiled the same kernel with the 4BSD scheduler today and it seems
> that the processes jump accross cores too. My "eye" measure with "top"
> fells like it's more stable and probably converges faster to a stable
> state after "top" jumps accross cores. But in order to talk with
> numbers, I need to replace "top" with somethings that dumps the
> process number and the cpu id continuously in order to get some
> statistics out of it, otherwise you can just forget all the things
> that I have written. Is there an easy way to do that and are you
> interested?

Maybe sysutils/atop is useful here?

[Oliver Fromme]

?????? ?????? <[hidden email]> wrote:
 > I compiled the same kernel with the 4BSD scheduler today and it seems
 > that the processes jump accross cores too.

What exactly is the problem that you're seeing?  Do you have
performance problems?  If so, then they're probably *not*
caused by processes "jumping across cores".

Have you read Daniel Kalchev's reply in this thread?
He explained very well why that's not a problem usually.

Also note that top(1) only shows one snapshot every second
or two.  It does not show you the hundreds (or thousands)
of thread switches that happen every second.  In fact,
top(1) shows a very misleading picture because it creates
the wrong impression that your CPU-bound processes are
almost the only ones being scheduled on your cores.

Most of the time, people looking at top(1) see problems that
don't exist.  Another example is the amount of "free" memory
displayed by top that is often misinterpreted.

I suggest you just keep the standard scheduler (i.e. ULE).
If you don't have a performance problem (i.e. a problem that
you can measure by other means than top), then don't try to
fix it.  Chances are you make things worse.

Best regards
   Oliver


--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"The scanf() function is a large and complex beast that often does
something almost but not quite entirely unlike what you desired."
        -- Chris Torek
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Momchil Ivanov-3]

At Tue, 12 Jun 2012 11:11:36 +0200 (CEST),
Oliver Fromme wrote:
>
> ?????? ?????? <[hidden email]> wrote:
>  > I compiled the same kernel with the 4BSD scheduler today and it seems
>  > that the processes jump accross cores too.
>
> What exactly is the problem that you're seeing?  Do you have
> performance problems?  If so, then they're probably *not*
> caused by processes "jumping across cores".

When my laptop said bye bye because of the heat I thougt it might be
the scheduler, which was a mistake since I got 30 C reduction on full
load after cleaning. Therefore, I don't think that there is any
problem. I was just curious why both processes are hopping around,
because I would naively think that should not happen. Note that I am
neither software nor hardware expert, but a mere user of both. I tried
the 4BSD scheduler just because Dough asked if I did. For me the
question is resolved and I thank all of you for the replies.

> Have you read Daniel Kalchev's reply in this thread?
> He explained very well why that's not a problem usually.

No, I haven't received that one.

Regards,
Momchil
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Oliver Fromme]

Momchil Ivanov <[hidden email]> wrote:
 > I was just curious why both processes are hopping around,
 > because I would naively think that should not happen.

I'll try to explain ...

There are always many more processes and threads being executed
beside the two CPU-bound ones that you see at the top of the
top(1) display.  For example, there are at least 15 threads
inside the kernel (see "ps -auxww") that are scheduled every
now and then.  Or look at "vmstat -i" output to see the
interrupt statistics:  Several hundred times per second,
the interrupt handlers have to be executed.

So, what happens is that an interrupt occurs (from a hardware
clock, from a hard disk controller, from an input device, or
anything else).  Then your process is _removed_ from the core
on which it is currently executing, and the interrupt handler
starts executing.  The same can happen on the second core at
the same time, because different kernel threads can execute
simultaneously (if they don't share resources).  Now, if the
interrupt handler on one of the two cores is done, your own
process has a chance to be scheduled again.  In this moment,
it does not matter at all on which core it is going to be
executed.  The only difference is cache contents, but the
first-level-caches are usually too small anyway.  The ULE
scheduler takes a lot of information into account when making
the decision, including cache affinity (the 4BSD scheduler
doesn't know about that at all).

All of that happens several hundreds times per second.  You
don't have a chance to see it with a tool like top(1).

 > > Have you read Daniel Kalchev's reply in this thread?
 > > He explained very well why that's not a problem usually.
 >
 > No, I haven't received that one.

Ok, I've appended it below.

Best regards
   Oliver

-------- begin of quoted text --------

From: Daniel Kalchev <[hidden email]>
Subject: Re: ULE Scheduler
Date: Thu, 7 Jun 2012 12:19:43 +0200 (CEST)
Message-ID: <[hidden email]>

On 07.06.12 11:16, Momchil Ivanov wrote:
 > Though, it was strange seeing both processes hopping around... I will
 > probably go back to the 4BSD scheduler if my laptop does another
 > self-shutdown in the next few days as Doug suggested.

You never run just two processes on FreeBSD, ever. The kernel too runs
multiple threads.

However small the CPU usage of the other processes is, they must run
from time to time, kicking out at least one of your CPU intensive
processes, possibly kicking them out both, as well. When that happens,
and they are queued to run again it does not matter much on which core
they ran before, because chances are it's cache will be invalidated
anyway. Also, different CPUs have different cache affinity. ULE is
supposed to be aware of this, while the 4BSD scheduler is not.

In any case, on an older single/dual core CPU there is rarely any
difference between both schedulers. Differences might appear in modern
multi-core CPUs..

Daniel

-------- end of quoted text --------


--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"What is this talk of 'release'?  We do not make software 'releases'.
Our software 'escapes', leaving a bloody trail of designers and quality
assurance people in its wake."
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Momchil Ivanov-3]

At Tue, 12 Jun 2012 14:03:10 +0200 (CEST),
Oliver Fromme wrote:

>
> Momchil Ivanov <[hidden email]> wrote:
>  > I was just curious why both processes are hopping around,
>  > because I would naively think that should not happen.
>
> I'll try to explain ...
>
> There are always many more processes and threads being executed
> beside the two CPU-bound ones that you see at the top of the
> top(1) display.  For example, there are at least 15 threads
> inside the kernel (see "ps -auxww") that are scheduled every
> now and then.  Or look at "vmstat -i" output to see the
> interrupt statistics:  Several hundred times per second,
> the interrupt handlers have to be executed.
>
> So, what happens is that an interrupt occurs (from a hardware
> clock, from a hard disk controller, from an input device, or
> anything else).  Then your process is _removed_ from the core
> on which it is currently executing, and the interrupt handler
> starts executing.  The same can happen on the second core at
> the same time, because different kernel threads can execute
> simultaneously (if they don't share resources).  Now, if the
> interrupt handler on one of the two cores is done, your own
> process has a chance to be scheduled again.  In this moment,
> it does not matter at all on which core it is going to be
> executed.  The only difference is cache contents, but the
> first-level-caches are usually too small anyway.  The ULE
> scheduler takes a lot of information into account when making
> the decision, including cache affinity (the 4BSD scheduler
> doesn't know about that at all).

So the L2 cache is shared between both cores and hence it's size does
not matter at all?
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Daniel Kalchev]

On 12.06.12 16:08, Momchil Ivanov wrote:
> So the L2 cache is shared between both cores and hence it's size does
> not matter at all?

If the cache is shared between both cores then it does not matter on
which core the process runs, as long as data is in teh case. The cache
size is irrelevant.

Some CPUs have shared cache between cores, some don't. The ULE scheduler
takes this into account, the 4BSD does not. Even if the ULE scheduler
takes the CPU topology into consideration, if you only have two cores,
it is almost guaranteed that processes will be switched between both,
because the OS is running way more than two processes "at the same time".

Even with more cores... it is not guaranteed an computational process
won't be 'bouncing'. Here is an example.
Suppose you have an 8 core (or threads) CPU. If you happen to have an
modern Ethernet controller, like the Intel 82576 (the igb driver in
FreeBSD), then it will use up to 8 interrupt lines, by default routing
them each to a different core. Then, if you have heavier network
traffic, chances are that at any given moment all 8 interrupts might be
fired and all 8 cores switched to service network traffic -- removing
your computational process from the running queue. The next time it
runs, it might run on any other core, especially if the cache is not shared.

Of course, if you have sufficiently large number of CPUs, you can
configure your system so that such things do not happen, like by
limiting the number of cores the igb driver attaches to, and have some
of the cores dedicated to 'only' running an computational task.

There is however, very little sense doing so.

Daniel
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"

[Momchil Ivanov-3]

At Wed, 13 Jun 2012 17:49:15 +0300,
Daniel Kalchev wrote:

>
>
>
> On 12.06.12 16:08, Momchil Ivanov wrote:
> > So the L2 cache is shared between both cores and hence it's size does
> > not matter at all?
>
> If the cache is shared between both cores then it does not matter on
> which core the process runs, as long as data is in teh case. The cache
> size is irrelevant.
>
> Some CPUs have shared cache between cores, some don't. The ULE scheduler
> takes this into account, the 4BSD does not. Even if the ULE scheduler
> takes the CPU topology into consideration, if you only have two cores,
> it is almost guaranteed that processes will be switched between both,
> because the OS is running way more than two processes "at the same time".
>
> Even with more cores... it is not guaranteed an computational process
> won't be 'bouncing'. Here is an example.
> Suppose you have an 8 core (or threads) CPU. If you happen to have an
> modern Ethernet controller, like the Intel 82576 (the igb driver in
> FreeBSD), then it will use up to 8 interrupt lines, by default routing
> them each to a different core. Then, if you have heavier network
> traffic, chances are that at any given moment all 8 interrupts might be
> fired and all 8 cores switched to service network traffic -- removing
> your computational process from the running queue. The next time it
> runs, it might run on any other core, especially if the cache is not shared.
>
> Of course, if you have sufficiently large number of CPUs, you can
> configure your system so that such things do not happen, like by
> limiting the number of cores the igb driver attaches to, and have some
> of the cores dedicated to 'only' running an computational task.
>
> There is however, very little sense doing so.

OK, thank you for the explanation.

Regards,
Momchil
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"