SCHED_ULE on sparc64

[Peter Jeremy-6]

[Cc'ing jeff@ as the "owner" of ULE]

I notice sparc64 is now the only architecture that doesn't support
ULE.  According to the sched_ule.c commit log, this is due to lack of
support for the 3rd argument to cpu_switch().  Followup to r202889,
committed as r203185 (MFC'd as r203554 and r203573) suggests that
cpu_switch() has now been fixed.  Are there still outstanding issues
preventing the use of ULE on sparc64 or did removing the #error just
fall thru the cracks?

--
Peter Jeremy

[Marius Strobl]

On Fri, May 20, 2011 at 05:52:45AM +1000, Peter Jeremy wrote:

> [Cc'ing jeff@ as the "owner" of ULE]
>
> I notice sparc64 is now the only architecture that doesn't support
> ULE.  According to the sched_ule.c commit log, this is due to lack of
> support for the 3rd argument to cpu_switch().  Followup to r202889,
> committed as r203185 (MFC'd as r203554 and r203573) suggests that
> cpu_switch() has now been fixed.  Are there still outstanding issues
> preventing the use of ULE on sparc64 or did removing the #error just
> fall thru the cracks?
>

The main problem with SCHED_ULE on sparc64 is that the MD code
(ab)uses the global sched_lock of SCHED_4BSD to protect pm_context,
pm_active and pc_pmap, partially of all CPUs, and SCHED_ULE doesn't
use/provide such a lock. One could replace the use of sched_lock
for that with a global MD spin lock but this has the issue that it
would have to be acquired and released in cpu_switch(), which is next
to impossible to do properly in assembler. The best thing one can do
there is to call C-wrappers which then call mtx_{,un}lock_spin(),
which isn't very elegant however (I think I've a patch which actually
implements just that though). There might be an alternate approach to
achieve the required level of protection not involving using a spin
lock but that needs thorough thinking and testing. The bottom line
is that watching the various mailing lists so far didn't provide the
necessary motivation to work on that to me though (even today you still
find reports about performance problems with SCHED_ULE and suggestions
to use SCHED_4BSD instead, just see [hidden email] as current
example).

Marius

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Peter Jeremy-6]

On 2011-May-20 12:38:41 +0200, Marius Strobl <[hidden email]> wrote:
>The main problem with SCHED_ULE on sparc64 is that the MD code
>(ab)uses the global sched_lock of SCHED_4BSD to protect pm_context,
>pm_active and pc_pmap, partially of all CPUs, and SCHED_ULE doesn't
>use/provide such a lock. One could replace the use of sched_lock
>for that with a global MD spin lock but this has the issue that it
>would have to be acquired and released in cpu_switch(), which is next
>to impossible to do properly in assembler.

Definitely messy but MIPS and PPC do it (at least the acquire - I
don't see how the lock is released in either case).

> There might be an alternate approach to
>achieve the required level of protection not involving using a spin
>lock but that needs thorough thinking and testing.

The lack of atomic RMW operations definitely makes sparc messier than
(eg) i386/amd64 here.  It has been a long while since I studied SPARC
assembler in any detail (and that was pre sun4u) so I can't really
help here.

> The bottom line
>is that watching the various mailing lists so far didn't provide the
>necessary motivation to work on that to me though (even today you still
>find reports about performance problems with SCHED_ULE and suggestions
>to use SCHED_4BSD instead, just see [hidden email] as current
>example).

OTOH, not using it won't get the bugs fixed.  My rationale for firing
up the spare V890 at $work was to try and stress some of the big
systems code and SCHED_ULE is supposed to be better at handling lots of
CPUs than SCHED_4BSD.

--
Peter Jeremy

[Marius Strobl]

On Fri, May 20, 2011 at 10:41:02PM +1000, Peter Jeremy wrote:

> On 2011-May-20 12:38:41 +0200, Marius Strobl <[hidden email]> wrote:
> >The main problem with SCHED_ULE on sparc64 is that the MD code
> >(ab)uses the global sched_lock of SCHED_4BSD to protect pm_context,
> >pm_active and pc_pmap, partially of all CPUs, and SCHED_ULE doesn't
> >use/provide such a lock. One could replace the use of sched_lock
> >for that with a global MD spin lock but this has the issue that it
> >would have to be acquired and released in cpu_switch(), which is next
> >to impossible to do properly in assembler.
>
> Definitely messy but MIPS and PPC do it (at least the acquire - I
> don't see how the lock is released in either case).

I don't think these actually acquire a lock, all lock-related I can
identify there are the equivalents of the following:
        atomic_store_rel_ptr(&old->td_lock, mtx);
and:
#if defined(SCHED_ULE) && defined(SMP)
                while (atomic_load_acq_ptr(&new->td_lock) == &blocked_lock)
                        cpu_spinwait();
#endif

> > The bottom line
> >is that watching the various mailing lists so far didn't provide the
> >necessary motivation to work on that to me though (even today you still
> >find reports about performance problems with SCHED_ULE and suggestions
> >to use SCHED_4BSD instead, just see [hidden email] as current
> >example).
>
> OTOH, not using it won't get the bugs fixed.

They certainly won't but typically I hit enough problems when trying to
get code developed on x86 or actually written with only x86 in mind to
work on sparc64 that I don't really feel the desire to go out hunting for
generic bugs in that code. In any case my motivation for getting SCHED_ULE
to work on sparc64 suddenly vanished with r171488 for some strange reason.

> My rationale for firing
> up the spare V890 at $work was to try and stress some of the big
> systems code and SCHED_ULE is supposed to be better at handling lots of
> CPUs than SCHED_4BSD.
>

I don't think 16 cores counts as a lot these days :)

Marius

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Jeff Roberson-2]

On Fri, 20 May 2011, Marius Strobl wrote:

> On Fri, May 20, 2011 at 10:41:02PM +1000, Peter Jeremy wrote:
>> On 2011-May-20 12:38:41 +0200, Marius Strobl <[hidden email]> wrote:
>>> The main problem with SCHED_ULE on sparc64 is that the MD code
>>> (ab)uses the global sched_lock of SCHED_4BSD to protect pm_context,
>>> pm_active and pc_pmap, partially of all CPUs, and SCHED_ULE doesn't
>>> use/provide such a lock. One could replace the use of sched_lock
>>> for that with a global MD spin lock but this has the issue that it
>>> would have to be acquired and released in cpu_switch(), which is next
>>> to impossible to do properly in assembler.
>>
>> Definitely messy but MIPS and PPC do it (at least the acquire - I
>> don't see how the lock is released in either case).
>
> I don't think these actually acquire a lock, all lock-related I can
> identify there are the equivalents of the following:
> atomic_store_rel_ptr(&old->td_lock, mtx);
> and:
> #if defined(SCHED_ULE) && defined(SMP)
> while (atomic_load_acq_ptr(&new->td_lock) == &blocked_lock)
> cpu_spinwait();
> #endif

Yes the goal of passing the lock pointer into the switch function is so
that the outgoing thread's lock is not released until we are off of its
stack.  Otherwise another cpu could start switching into it as we are on
the way out.

>
>>> The bottom line
>>> is that watching the various mailing lists so far didn't provide the
>>> necessary motivation to work on that to me though (even today you still
>>> find reports about performance problems with SCHED_ULE and suggestions
>>> to use SCHED_4BSD instead, just see [hidden email] as current
>>> example).

Can you give me another reference to this?  You have to realize that no
scheduling policy will be faster for everything.  The goal is to be faster
for most things and eliminate worst case scenarios.  I can look at this
soon if there is something to be done.

>>
>> OTOH, not using it won't get the bugs fixed.
>
> They certainly won't but typically I hit enough problems when trying to
> get code developed on x86 or actually written with only x86 in mind to
> work on sparc64 that I don't really feel the desire to go out hunting for
> generic bugs in that code. In any case my motivation for getting SCHED_ULE
> to work on sparc64 suddenly vanished with r171488 for some strange reason.

I really don't know what the status of the sparc64 port is.  If it is
intended to be first tier it should support ULE.  Features like cpusets
and topology aware scheduling are better supported on ULE.  It is
generally considered the path forward for SMP.  4BSD with its global run
queue and global lock is a dead end unless someone wants to salvage its
priority computation mechanism and add the cpu load balancing features
that end up making ULE slower in some cases.

>
>> My rationale for firing
>> up the spare V890 at $work was to try and stress some of the big
>> systems code and SCHED_ULE is supposed to be better at handling lots of
>> CPUs than SCHED_4BSD.
>>
>
> I don't think 16 cores counts as a lot these days :)

The per-cpu scheduler locks showed massive improvements on some workloads
with only 4 cores.  The global scheduler lock is a significant point of
contention probably for any workload at 16 cores.  16 cores is not a big
machine anymore but it's plenty to have heavy contention soak up too many
cycles.

Thanks,
Jeff

>
> Marius
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Marius Strobl]

On Fri, May 20, 2011 at 10:37:31AM -1000, Jeff Roberson wrote:

> On Fri, 20 May 2011, Marius Strobl wrote:
>
> >>>The bottom line
> >>>is that watching the various mailing lists so far didn't provide the
> >>>necessary motivation to work on that to me though (even today you still
> >>>find reports about performance problems with SCHED_ULE and suggestions
> >>>to use SCHED_4BSD instead, just see [hidden email] as current
> >>>example).
>
> Can you give me another reference to this?

No, sorry, this was just one complaint about SCHED_ULE I noticed that
day.

Marius

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Mark Linimon-2]

On Fri, May 20, 2011 at 05:04:39PM +0200, Marius Strobl wrote:
> I don't think 16 cores counts as a lot these days :)

It's more than we are currently running any sparc64 machines on IIRC.

mcl
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Marius Strobl]

On Tue, May 24, 2011 at 02:17:45PM -0500, Mark Linimon wrote:
> On Fri, May 20, 2011 at 05:04:39PM +0200, Marius Strobl wrote:
> > I don't think 16 cores counts as a lot these days :)
>
> It's more than we are currently running any sparc64 machines on IIRC.
>

Peter's V890 actually is a 16-core machine.

Marius

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Rob Farmer-2]

On Tue, May 24, 2011 at 7:05 AM, Marius Strobl
<[hidden email]> wrote:
> On Fri, May 20, 2011 at 10:37:31AM -1000, Jeff Roberson wrote:
>> Can you give me another reference to this?
>
> No, sorry, this was just one complaint about SCHED_ULE I noticed that
> day.
>

I think he means like a http link to the mail on lists.freebsd.org or
whatever. Message ID cannot be googled and most archive sites don't
show them, so unless there is a secret I don't know, they are a PITA
for looking things up.

--
Rob Farmer
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"

[Marius Strobl]

On Tue, May 24, 2011 at 12:51:26PM -0700, Rob Farmer wrote:

> On Tue, May 24, 2011 at 7:05 AM, Marius Strobl
> <[hidden email]> wrote:
> > On Fri, May 20, 2011 at 10:37:31AM -1000, Jeff Roberson wrote:
> >> Can you give me another reference to this?
> >
> > No, sorry, this was just one complaint about SCHED_ULE I noticed that
> > day.
> >
>
> I think he means like a http link to the mail on lists.freebsd.org or
> whatever. Message ID cannot be googled and most archive sites don't
> show them, so unless there is a secret I don't know, they are a PITA
> for looking things up.
>

Actually the FreeBSD website provides an search engine for Message IDs
on the hosted lists:
http://www.freebsd.org/search/search-mid.html
The mail in question is:
http://www.freebsd.org/cgi/getmsg.cgi?fetch=249845+0+/usr/local/www/db/text/2011/freebsd-stable/20110522.freebsd-stable

Marius

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
To unsubscribe, send any mail to "[hidden email]"