Panic Mode On (109) Server Problems?

Message boards : Number crunching : Panic Mode On (109) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 31 · 32 · 33 · 34 · 35 · 36 · 37 . . . 38 · Next

AuthorMessage
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 3506
Credit: 74,863,190
RAC: 106,663
Australia
Message 1913390 - Posted: 17 Jan 2018, 2:19:42 UTC - in response to Message 1913315.  

Looks like we've reached that critical point of returned-last-hour, WU-waiting-deletion, in-progress and splitter load.
Waiting-deletion is going up, splitter output has gone down, and Ready-to-send buffer is rapidly emptying. And with the shorter WU run times, channels left to be split is rapidly diminishing also.


. . On that subject (channels to be split) I am left pondering why it is that all the "tapes" (Disks) show as 52.24 GB but while some show 80 to 120 channels others show only one or two. What causes that level of discrepancy?

Stephen

??
ID: 1913390 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 10088
Credit: 133,245,111
RAC: 83,822
Australia
Message 1913400 - Posted: 17 Jan 2018, 2:54:20 UTC - in response to Message 1913383.  
Last modified: 17 Jan 2018, 3:22:48 UTC

And the splitters are back online. But the SSP has not updated in 15 minutes. So who knows?

Splitters showing Green, and status page has updated. But still no work actually coming from the splitters.
Server code update broke our splitters as well as Betas?


Edit- over 2 hours since the project came back up, but still no splitter output.
Grant
Darwin NT
ID: 1913400 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 10088
Credit: 133,245,111
RAC: 83,822
Australia
Message 1913402 - Posted: 17 Jan 2018, 2:56:27 UTC - in response to Message 1913390.  

. . On that subject (channels to be split) I am left pondering why it is that all the "tapes" (Disks) show as 52.24 GB but while some show 80 to 120 channels others show only one or two. What causes that level of discrepancy?

Only those in progress (dark green), or have been processed (light green), show up.
Grant
Darwin NT
ID: 1913402 · Report as offensive
juan BFP Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 6939
Credit: 394,593,791
RAC: 140,518
Panama
Message 1913411 - Posted: 17 Jan 2018, 3:45:40 UTC
Last modified: 17 Jan 2018, 3:46:25 UTC

Apparently the splitters are splitting as showed by the Splitter status but where the splitted data is going?
ID: 1913411 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 7939
Credit: 18,939,559
RAC: 8,452
United States
Message 1913412 - Posted: 17 Jan 2018, 3:50:37 UTC

I made it thru todays outrage but it looks like I will go dry this evening unless something gets fixed.
ID: 1913412 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 10088
Credit: 133,245,111
RAC: 83,822
Australia
Message 1913414 - Posted: 17 Jan 2018, 4:09:37 UTC - in response to Message 1913411.  

Apparently the splitters are splitting as showed by the Splitter status but where the splitted data is going?

I think it's a case of the splitters are running, but they're not actually doing anything.

Apparently, the updated server code broke the splitters on Beta.
They got that fixed, but it would appear there's some differences between the splitters here & at Beta that didn't cope too well with the new code.
It could be a while before work is flowing again.
Grant
Darwin NT
ID: 1913414 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 4107
Credit: 247,614,641
RAC: 329,306
United States
Message 1913428 - Posted: 17 Jan 2018, 5:06:12 UTC

Nothing new on the graphs, the In Progress is still headed Down;


about a couple hours left, then my machines will be going cold...like the weather outside.
ID: 1913428 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 10088
Credit: 133,245,111
RAC: 83,822
Australia
Message 1913429 - Posted: 17 Jan 2018, 5:12:32 UTC - in response to Message 1913428.  

Nothing new on the graphs, the In Progress is still headed Down;

Even if they get the splitters fixed quickly, and they then split like never before, and maintain that level of output like never before, it's going to be a long and tough recovery.
Grant
Darwin NT
ID: 1913429 · Report as offensive
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 3506
Credit: 74,863,190
RAC: 106,663
Australia
Message 1913430 - Posted: 17 Jan 2018, 5:13:54 UTC - in response to Message 1913412.  

I made it thru todays outrage but it looks like I will go dry this evening unless something gets fixed.


. . I have about 3 to 5 hours for all machines to be empty. Considering it is after 9pm in Berkeley I guess the guys (computers) will get to sleep tonight...

Stephen

:(
ID: 1913430 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 2992
Credit: 51,350,106
RAC: 18,199
United States
Message 1913444 - Posted: 17 Jan 2018, 8:15:48 UTC
Last modified: 17 Jan 2018, 8:18:45 UTC

someone needs to give the splitters a (re-) boot in the bum... 0 WU ready to send and only .7584 WU per second created (with 0 WU available, that should be in the 49-50 range!)

It's time to give your alternate projects some love... (IF they aren't also down [hello, GPUgrid?])
.

Hello, from Bangkok, Thailand!...
ID: 1913444 · Report as offensive
Tutankhamon
Volunteer tester
Avatar

Send message
Joined: 1 Nov 08
Posts: 7217
Credit: 44,941,174
RAC: 9,175
Sweden
Message 1913449 - Posted: 17 Jan 2018, 8:44:03 UTC

I bet it is the patches for Meltdown and Spectre that's the reason for the server slowdowns.
ID: 1913449 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1097
Credit: 108,833,129
RAC: 31,671
United States
Message 1913450 - Posted: 17 Jan 2018, 8:46:12 UTC - in response to Message 1913444.  

someone needs to give the splitters a (re-) boot in the bum... 0 WU ready to send and only .7584 WU per second created (with 0 WU available, that should be in the 49-50 range!)
Apparently a change was made that didn't go well. Splitters crashed shortly after the outage.

It's time to give your alternate projects some love... (IF they aren't also down [hello, GPUgrid?])
Yeah, only good news here is I finally got a config I'm happy with that lets Einstein take over when SETI goes sideways without having to dump a bunch of work when SETI resurfaces.
ID: 1913450 · Report as offensive
Profile Stargate (S.A.)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1503
Credit: 473,208
RAC: 1,282
Australia
Message 1913451 - Posted: 17 Jan 2018, 8:53:04 UTC

I don't personally think so IMO maybe the project is in disarray after what happened at Arecibo, they would be still finding there feet one would imagine..
ID: 1913451 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 10088
Credit: 133,245,111
RAC: 83,822
Australia
Message 1913452 - Posted: 17 Jan 2018, 8:54:16 UTC - in response to Message 1913449.  
Last modified: 17 Jan 2018, 8:54:46 UTC

I bet it is the patches for Meltdown and Spectre that's the reason for the server slowdowns.

If we're lucky.
If not, it's the increased load (of all those short running GBT WUs) showing the present systems limits, and the patches will just make things even worse when they are applied.

Present lack of work- most likely the server update that worked on Beta (after it was fixed for breaking the splitters there), broke the splitters here.
Grant
Darwin NT
ID: 1913452 · Report as offensive
Cruncher-American Special Project $75 donor

Send message
Joined: 25 Mar 02
Posts: 1453
Credit: 273,575,455
RAC: 140,736
United States
Message 1913459 - Posted: 17 Jan 2018, 9:58:28 UTC

Currently, the server status page is in an inconsistent state vis a vis the splitters for BLC.

It shows 14 splitters running, but 17 tapes being processed, one of which appears to have > 1 splitter associated with it.

With all the things that have happened recently, I don't recall ever seeing ghost splitters before.

Did I miss something?
ID: 1913459 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 12055
Credit: 121,755,513
RAC: 63,048
United Kingdom
Message 1913460 - Posted: 17 Jan 2018, 10:08:55 UTC - in response to Message 1913459.  

Yes. "Being processed" on the SSP means that work has started on that tape. But when the splitters are paused (e.g. for maintenance yesterday), or when new tapes are loaded, the tape selection mechanism sometimes picks a different tape for processing. So then the old tape goes into a 'waiting to run' state, such as we sometimes see on our BOINC Managers - except you can't see it on the SSP. Those are your ghosts: they are very common, even if you haven't noticed them before. And they're nothing to worry about - eventually the tape selection process will pick them out again and give them another go.
ID: 1913460 · Report as offensive
Profile Piotr

Send message
Joined: 24 May 17
Posts: 9
Credit: 16,272,461
RAC: 17,509
Poland
Message 1913473 - Posted: 17 Jan 2018, 12:12:06 UTC

Look's like after each servicing outage the project getting worse maladies ...
ID: 1913473 · Report as offensive
juan BFP Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 6939
Credit: 394,593,791
RAC: 140,518
Panama
Message 1913476 - Posted: 17 Jan 2018, 12:41:50 UTC

Murphy`s Law. Nothing is so bad that can not get worse.
ID: 1913476 · Report as offensive
Profile Brent Norman Special Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2218
Credit: 230,434,794
RAC: 471,822
Canada
Message 1913477 - Posted: 17 Jan 2018, 12:47:23 UTC - in response to Message 1913476.  

Well our first found message could contain "Incoming Nuke, Take Cover!"
ID: 1913477 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 179
Credit: 14,969,476
RAC: 19,751
Norway
Message 1913482 - Posted: 17 Jan 2018, 13:30:57 UTC

Actually, Seti@Home is not much fun any more, with the outages, droughts, problems, speculations...and CreditScrew.
Sorry about being so pessimistic, folks.... ;-)

...Ghia...
Humans may rule the world...but bacteria run it...
ID: 1913482 · Report as offensive
Previous · 1 . . . 31 · 32 · 33 · 34 · 35 · 36 · 37 . . . 38 · Next

Message boards : Number crunching : Panic Mode On (109) Server Problems?


 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.