Why I don't like the 4.4x scheduler

Message boards : Number crunching : Why I don't like the 4.4x scheduler
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
Profile Bruno G. Olsen & ESEA @ greenholt
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 875
Credit: 4,386,984
RAC: 0
Denmark
Message 118060 - Posted: 3 Jun 2005, 13:21:05 UTC

In this thread you can tell why you don't like the scheduler. I strongly suggest you not use this thread for reasons why you like it.


ID: 118060 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 32786
Credit: 79,922,639
RAC: 80
Germany
Message 118064 - Posted: 3 Jun 2005, 13:38:11 UTC

Hi

Same as in your other thread.

greetz Mike

With each crime and every kindness we birth our future.
ID: 118064 · Report as offensive
Vid Vidmar*
Volunteer tester
Avatar

Send message
Joined: 19 Aug 99
Posts: 136
Credit: 1,830,317
RAC: 0
Slovenia
Message 118079 - Posted: 3 Jun 2005, 14:11:11 UTC

I cannot say, that this bug is still in the code, but: when manualy updating, the scheduler sometimes sends 2 consecutive requests to the project scheduler, which IMO causes the "ghost WUs" problem. Since I didn't see this happen in v4.44 it might have been fixed. But 4.44 IS development version...

ID: 118079 · Report as offensive
Profile Nightbird
Volunteer tester

Send message
Joined: 2 Feb 03
Posts: 73
Credit: 53,523
RAC: 0
France
Message 118146 - Posted: 3 Jun 2005, 17:04:57 UTC
Last modified: 3 Jun 2005, 17:07:24 UTC

I don't like it and even not sure that i like the 4.37 on my main machine.
I ran with the 4.19 3 or 4 projects on every machine that i own and rarely i lost wus because of a deadline.
Needed "only" some fine "tuning" with the ressource share and the cache size.
If the 4.19 had some options seen in the versions 4.3x or 4.4x, it would be perfect for me.

For now, i have the feeling that i can't control really Boinc and a project with the 4.43.





Do you want to get banned for 31 years and your account & credits deleted at a Boinc project ? Predictor@home is your best choice.
ID: 118146 · Report as offensive
keputnam
Volunteer tester

Send message
Joined: 2 Jul 99
Posts: 242
Credit: 2,736,564
RAC: 3
United States
Message 118201 - Posted: 3 Jun 2005, 19:53:27 UTC

I'm gradually liking it better and better, but I still have a problem with projects accumulating LongTermDebt when they don't have a WU on the client (for whatever reason). Seems to me that when this happens, when a flow of WUs does start again, those projects will get significantly more cycles (in order to reduce the debt) than the ones that maintain a more or less constant flow of WUs, thus penalizing the ones that do maintain a flow.

I know, I know, over weeks or months everything will balance out eventually, but if a project can't keep delivering WUs, why should it be rewarded when it finally does?

If I have mis-understood the way that the new scheduler works, I appologize.

And I will add my name to the request to create separate settings for "Connect Every" and "Cache Size" (Keep enough work for xx days of processing on my machine).

ID: 118201 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 118209 - Posted: 3 Jun 2005, 20:21:40 UTC - in response to Message 118201.  

I'm gradually liking it better and better, but I still have a problem with projects accumulating LongTermDebt when they don't have a WU on the client (for whatever reason). Seems to me that when this happens, when a flow of WUs does start again, those projects will get significantly more cycles (in order to reduce the debt) than the ones that maintain a more or less constant flow of WUs, thus penalizing the ones that do maintain a flow.

I know, I know, over weeks or months everything will balance out eventually, but if a project can't keep delivering WUs, why should it be rewarded when it finally does?

If I have mis-understood the way that the new scheduler works, I appologize.

And I will add my name to the request to create separate settings for "Connect Every" and "Cache Size" (Keep enough work for xx days of processing on my machine).

4.43 or 4.44. I believe that I have addressed this in 4.44.


BOINC WIKI
ID: 118209 · Report as offensive
csarsenault
Volunteer tester

Send message
Joined: 22 Jan 01
Posts: 51
Credit: 4,164
RAC: 0
United States
Message 118229 - Posted: 3 Jun 2005, 21:18:09 UTC

I have 4.43 and I can't get it to process anything but protein predictor unless I suspend it or not allow new work. The predictor wu deadlines are a lot less than seti, einstein and climate, and it goes to earliest deadline first mode. Problem is, after it crunches a wu, it downloads another, with a deadline still too close, so it stays on earliest deadline first. So if I left predictor going, that is all it would crunch until it got to the deadlines of the other projects.
I also saw this problem on a 4.3x version I tried and subsequently reinstalled 4.25.
Now what should I do, go back to 4.25 and let all my projects run, or stay current and only let protein predictor run, or stay current and suspend predictor.
This scheduler is lame! Stupid debts too. There should be a option to set or reset these debts so when a project goes down for a month, it shouldn't crunch that projects for a month because of debt.

ID: 118229 · Report as offensive
Profile Steve Cressman
Volunteer tester
Avatar

Send message
Joined: 6 Jun 02
Posts: 583
Credit: 65,644
RAC: 0
Canada
Message 118258 - Posted: 3 Jun 2005, 22:31:04 UTC

John I have a sugestion that might help the debt problem when upgrading to the client with the new scheduler. Have all projects start with a clean slate or in other words have all the debt values set to zero. This way it should work as intended right from the begining instead of having to juggle things for a week or two. Also a debt reset button that zeros all debt for all projects might be useful but not needed if you implement my first idea.

98SE XP2500+ @ 2.1 GHz Boinc v5.8.8

And God said"Let there be light."But then the program crashed because he was trying to access the 'light' property of a NULL universe pointer.
ID: 118258 · Report as offensive
Profile Nightbird
Volunteer tester

Send message
Joined: 2 Feb 03
Posts: 73
Credit: 53,523
RAC: 0
France
Message 118336 - Posted: 4 Jun 2005, 0:57:04 UTC

I want to be able to control Boinc and the projects like i want and not like Boinc wants it...
Since these debts seem to be very important for this version 4.4x, is it possible to "deconnect" them (an option ?) and get Boinc working like an "old but classical" version ?
Managing the deadline like i want (and in short the projects) is my wish.

Do you want to get banned for 31 years and your account & credits deleted at a Boinc project ? Predictor@home is your best choice.
ID: 118336 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 118382 - Posted: 4 Jun 2005, 2:00:26 UTC - in response to Message 118229.  

I have 4.43 and I can't get it to process anything but protein predictor unless I suspend it or not allow new work. The predictor wu deadlines are a lot less than seti, einstein and climate, and it goes to earliest deadline first mode. Problem is, after it crunches a wu, it downloads another, with a deadline still too close, so it stays on earliest deadline first. So if I left predictor going, that is all it would crunch until it got to the deadlines of the other projects.
I also saw this problem on a 4.3x version I tried and subsequently reinstalled 4.25.
Now what should I do, go back to 4.25 and let all my projects run, or stay current and only let protein predictor run, or stay current and suspend predictor.
This scheduler is lame! Stupid debts too. There should be a option to set or reset these debts so when a project goes down for a month, it shouldn't crunch that projects for a month because of debt.


I bet you have a queue of 6 to 10 days. With multiple projects this is not really needed unless you are a modem user. It is processing results in Earliest Deadline First mode so that a modem user that really only can connect with the frequency that you have indicated will be able to get work in on time.


BOINC WIKI
ID: 118382 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 118389 - Posted: 4 Jun 2005, 2:04:39 UTC - in response to Message 118258.  

John I have a sugestion that might help the debt problem when upgrading to the client with the new scheduler. Have all projects start with a clean slate or in other words have all the debt values set to zero. This way it should work as intended right from the begining instead of having to juggle things for a week or two. Also a debt reset button that zeros all debt for all projects might be useful but not needed if you implement my first idea.

All of the clients start with a clean slate for LT debt when first upgraded from versions before 4.35. The couple of weeks adjustment is because many computers (not all) are somewhat overcommitted, and the scheduler has to sort that out first.

Resetting on a version change is not a great idea. Suppose that you were just finished with a CPDN WU that had been crunching solo for the last 6 months to make the deadline. Woould you really want another CPDN WU so soon?


BOINC WIKI
ID: 118389 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 118391 - Posted: 4 Jun 2005, 2:06:38 UTC - in response to Message 118336.  

I want to be able to control Boinc and the projects like i want and not like Boinc wants it...
Since these debts seem to be very important for this version 4.4x, is it possible to "deconnect" them (an option ?) and get Boinc working like an "old but classical" version ?
Managing the deadline like i want (and in short the projects) is my wish.

You control which projects to run in what shares by attaching to the projects that you want and setting the resource shares. It is then BOINCs job to ensure that the work that is downloaded is done on time, and that over time your resource shares are honored. The larger the cache, the longer it will take to balance the resource shares.


BOINC WIKI
ID: 118391 · Report as offensive
ABT Chuck P
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 91
Credit: 316,669
RAC: 0
United States
Message 118435 - Posted: 4 Jun 2005, 2:38:09 UTC - in response to Message 118209.  

I'm gradually liking it better and better, but I still have a problem with projects accumulating LongTermDebt when they don't have a WU on the client (for whatever reason). Seems to me that when this happens, when a flow of WUs does start again, those projects will get significantly more cycles (in order to reduce the debt) than the ones that maintain a more or less constant flow of WUs, thus penalizing the ones that do maintain a flow.

I know, I know, over weeks or months everything will balance out eventually, but if a project can't keep delivering WUs, why should it be rewarded when it finally does?

If I have mis-understood the way that the new scheduler works, I appologize.

And I will add my name to the request to create separate settings for "Connect Every" and "Cache Size" (Keep enough work for xx days of processing on my machine).

4.43 or 4.44. I believe that I have addressed this in 4.44.

==============
Looks like we may find out sooner than you thought. Berkeley may be having another weekend outage now.

ID: 118435 · Report as offensive
Profile Angus
Volunteer tester

Send message
Joined: 26 May 99
Posts: 459
Credit: 91,013
RAC: 0
Pitcairn Islands
Message 118447 - Posted: 4 Jun 2005, 2:50:40 UTC - in response to Message 118336.  

I want to be able to control Boinc and the projects like i want and not like Boinc wants it...
Since these debts seem to be very important for this version 4.4x, is it possible to "deconnect" them (an option ?) and get Boinc working like an "old but classical" version ?
Managing the deadline like i want (and in short the projects) is my wish.


There are four criteria that a scheduler needs to honor to be called 'successful'.

1. Cache size
2. Resource Share
3. Task switching time
4. Connect schedule

It should download the amount of work you want (Cache Size), in the proportion you want (Resource Share), switch CPU between project on the schedule you want every time (Task Switching), and upload finished work and download new work when you want (Connect Schedule).

Right now it only honors 3 of those - since Cache size and Connect schedule are the same setting. Separating those two preferences will allow the scheduler to work as you want, not how the projects want and driven by deadlines.

If you have 4 projects, a 25/25/25/25 percent split, a 10 day cache, a 60 minute task switch setting, and an always on connect schedule, the scheduler should always keep 2.5 days of work for each project in your cache, switch the CPU from one project to the next every hour, report each WU when done, and MAYBE not download that next WU for the project if it won't get done inside the 25% of 10 days.

ID: 118447 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 118459 - Posted: 4 Jun 2005, 2:58:55 UTC - in response to Message 118447.  

I want to be able to control Boinc and the projects like i want and not like Boinc wants it...
Since these debts seem to be very important for this version 4.4x, is it possible to "deconnect" them (an option ?) and get Boinc working like an "old but classical" version ?
Managing the deadline like i want (and in short the projects) is my wish.


There are four criteria that a scheduler needs to honor to be called 'successful'.

1. Cache size
2. Resource Share
3. Task switching time
4. Connect schedule

It should download the amount of work you want (Cache Size), in the proportion you want (Resource Share), switch CPU between project on the schedule you want every time (Task Switching), and upload finished work and download new work when you want (Connect Schedule).

Right now it only honors 3 of those - since Cache size and Connect schedule are the same setting. Separating those two preferences will allow the scheduler to work as you want, not how the projects want and driven by deadlines.

If you have 4 projects, a 25/25/25/25 percent split, a 10 day cache, a 60 minute task switch setting, and an always on connect schedule, the scheduler should always keep 2.5 days of work for each project in your cache, switch the CPU from one project to the next every hour, report each WU when done, and MAYBE not download that next WU for the project if it won't get done inside the 25% of 10 days.

You did not include deadlines. By your definition, the scheduler can return work too late to get credit and still be considered "successful."
ID: 118459 · Report as offensive
Profile Angus
Volunteer tester

Send message
Joined: 26 May 99
Posts: 459
Credit: 91,013
RAC: 0
Pitcairn Islands
Message 118482 - Posted: 4 Jun 2005, 3:18:14 UTC - in response to Message 118459.  
Last modified: 4 Jun 2005, 3:18:45 UTC

I want to be able to control Boinc and the projects like i want and not like Boinc wants it...
Since these debts seem to be very important for this version 4.4x, is it possible to "deconnect" them (an option ?) and get Boinc working like an "old but classical" version ?
Managing the deadline like i want (and in short the projects) is my wish.


There are four criteria that a scheduler needs to honor to be called 'successful'.

1. Cache size
2. Resource Share
3. Task switching time
4. Connect schedule

It should download the amount of work you want (Cache Size), in the proportion you want (Resource Share), switch CPU between project on the schedule you want every time (Task Switching), and upload finished work and download new work when you want (Connect Schedule).

Right now it only honors 3 of those - since Cache size and Connect schedule are the same setting. Separating those two preferences will allow the scheduler to work as you want, not how the projects want and driven by deadlines.

If you have 4 projects, a 25/25/25/25 percent split, a 10 day cache, a 60 minute task switch setting, and an always on connect schedule, the scheduler should always keep 2.5 days of work for each project in your cache, switch the CPU from one project to the next every hour, report each WU when done, and MAYBE not download that next WU for the project if it won't get done inside the 25% of 10 days.

You did not include deadlines. By your definition, the scheduler can return work too late to get credit and still be considered "successful."


I DID include deadline.
and MAYBE not download that next WU for the project if it won't get done inside the 25% of 10 days.

by which I meant that it couldn't meet deadline if only crunched for 25% of 10 days.
ID: 118482 · Report as offensive
Profile ksnash

Send message
Joined: 28 Nov 99
Posts: 402
Credit: 528,725
RAC: 0
United States
Message 118495 - Posted: 4 Jun 2005, 3:30:19 UTC

You mind telling me how the scheduler is settling down?

CPDN 360000 LTD
Einstein -250000 LTD
Setiathome 240000 LTD

A week ago the LTD was around 30000.
Scheduler doesn't work.
ID: 118495 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 118501 - Posted: 4 Jun 2005, 3:36:38 UTC - in response to Message 118482.  
Last modified: 4 Jun 2005, 3:36:59 UTC

There are four criteria that a scheduler needs to honor to be called 'successful'.

1. Cache size
2. Resource Share
3. Task switching time
4. Connect schedule

It should download the amount of work you want (Cache Size), in the proportion you want (Resource Share), switch CPU between project on the schedule you want every time (Task Switching), and upload finished work and download new work when you want (Connect Schedule).

Right now it only honors 3 of those - since Cache size and Connect schedule are the same setting. Separating those two preferences will allow the scheduler to work as you want, not how the projects want and driven by deadlines.

If you have 4 projects, a 25/25/25/25 percent split, a 10 day cache, a 60 minute task switch setting, and an always on connect schedule, the scheduler should always keep 2.5 days of work for each project in your cache, switch the CPU from one project to the next every hour, report each WU when done, and MAYBE not download that next WU for the project if it won't get done inside the 25% of 10 days.

You did not include deadlines. By your definition, the scheduler can return work too late to get credit and still be considered "successful."


I DID include deadline.
and MAYBE not download that next WU for the project if it won't get done inside the 25% of 10 days.

by which I meant that it couldn't meet deadline if only crunched for 25% of 10 days.


You said "four criteria" and listed them:

1. Cache size
2. Resource Share
3. Task switching time
4. Connect schedule

"Deadline" is not one of the four.

Task switching time is, and that one really only works if your resource numbers are pretty much equal. If you had four projects with 85/5/5/5 shares, then you can't possibly switch at each switch time and be true to resource share.

You also have to allow for the fact that we have work units from various projects that range from something near an hour (maybe less) up to CPDN. JM7 says one of the Shhh projects has a four month WU on a fast machine.

If all of the projects had roughly the same size WUs and the same deadlines, the scheduler would be drop-dead-simple.
ID: 118501 · Report as offensive
keputnam
Volunteer tester

Send message
Joined: 2 Jul 99
Posts: 242
Credit: 2,736,564
RAC: 3
United States
Message 118502 - Posted: 4 Jun 2005, 3:37:01 UTC - in response to Message 118209.  

I'm gradually liking it better and better, but I still have a problem with projects accumulating LongTermDebt when they don't have a WU on the client (for whatever reason). Seems to me that when this happens, when a flow of WUs does start again, those projects will get significantly more cycles (in order to reduce the debt) than the ones that maintain a more or less constant flow of WUs, thus penalizing the ones that do maintain a flow.

I know, I know, over weeks or months everything will balance out eventually, but if a project can't keep delivering WUs, why should it be rewarded when it finally does?

If I have mis-understood the way that the new scheduler works, I appologize.

And I will add my name to the request to create separate settings for "Connect Every" and "Cache Size" (Keep enough work for xx days of processing on my machine).

4.43 or 4.44. I believe that I have addressed this in 4.44.


OK - I'm on 4.44 so we'll see how it goes

ID: 118502 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 118512 - Posted: 4 Jun 2005, 3:43:21 UTC - in response to Message 118201.  

I know, I know, over weeks or months everything will balance out eventually, but if a project can't keep delivering WUs, why should it be rewarded when it finally does?

The theory here is that you could crunch several projects, run with a short cache, and if someone had an outage (like we may be seeing at SETI right now) they'd get their time back later -- an outage generally isn't the project's fault.

Trouble is, long dry spells like we've seen with LHC (out of work since November) and outages are alot alike.

So, in practice granting debt for projects that are "out" is a bad thing.

ID: 118512 · Report as offensive
1 · 2 · 3 · 4 · Next

Message boards : Number crunching : Why I don't like the 4.4x scheduler


 
©2021 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.