Jump to content

WxRouting. National Buoy Data Center down-No idea when it may return.


Recommended Posts

While helping out a friend who is delivering a new to him boat down from San Francisco I came upon a startling discovery. One of the resources I use to help people in near real time with deliveries is the array of buoys maintained by NOAA and the National Buoy Data Center. This morning at 4AM when I logged in I was stunned to discover that pretty much the entire array of buoys had no data available.  The message in bold red letters at the top of the data page said words to the effect of: "the NBDC primary processing servers were shit off due to a facilities issue. There is no ETR as to when we may get the system functioning again."

On the surface this may not seem like that big a deal...except guess where your data points come from for the various weather models? For those on the East Coast and middle of the country, less of a problem but for those on the west coast, you are now pretty much blind as even the ship data is gone. Rut Roh. 

https://www.ndbc.noaa.gov/

NBDC OFFLINE.jpg

  • Like 2
Link to post
Share on other sites

that's a big problem, I call buoy data when we are offshore and the models are not matching up with what we are observing.    

Link to post
Share on other sites
15 minutes ago, Maxx Baqustae said:

Thanks Mark. I saw that on your Wastebook feed. We have a similar application for Environment Canada Marine Weather. You might have to dig down on it but there actuals looking for Weather Conditions tab.   https://weather.gc.ca/marine/forecast_e.html?mapID=02&siteID=14305

 

Thank you for that information. 

Link to post
Share on other sites

On 03/09/2021 the NDBC primary processing servers were shut off due to a facilities issue. Station pages on the NDBC website are not updating and there is no ETR at this time. We will update this banner as we learn more.

Translation:  Hacked.

  • Like 2
Link to post
Share on other sites
9 minutes ago, Hitchhiker said:

On 03/09/2021 the NDBC primary processing servers were shut off due to a facilities issue. Station pages on the NDBC website are not updating and there is no ETR at this time. We will update this banner as we learn more.

Translation:  Hacked.

Yep, there was a widespread hack the other day too.

  • Like 1
Link to post
Share on other sites

The internet is one big hackfest. Infrastructure should NOT be connected to it. Air Gap the freak out of it. That is going to be the future--if we want to keep the fucking Chinese Communists out of our shorts.

  • Like 1
Link to post
Share on other sites

Saying the below in jest don't flame me.  I feel the pain for the people who us that data.  Nobody pays enough attention to security as they should until they get hacked.

 

The weather station I support is still spitting out data.  On a rocky outcrop about 300 meters from land.  Picture is taken at low tide.  Davis weather station that transmits  to shore using their own wireless tech.  No cellular connection needed.  

https://www.wunderground.com/dashboard/pws/KNY273

IMG_6627.thumb.JPG.f3e44ef1a9615a0620662dd5edca105a.JPG

A 75knot puff once knocked it down but not out.  It was still transmitting but data was a little off.  (Sorry for the image rotation it looked right in my computer.)  There is more ballast keeping it straight now.

IMG_4208.thumb.jpeg.758e7b68801325237d63add6a2a5eb27.jpeg

Link to post
Share on other sites
9 hours ago, fastyacht said:

The internet is one big hackfest. Infrastructure should NOT be connected to it. Air Gap the freak out of it. That is going to be the future--if we want to keep the fucking Chinese Communists out of our shorts.

#FiberOpticUSnot5Gus

 

Link to post
Share on other sites

From the National Data Bouy Center - Marine Forecast....It is now Saturday  morning...five days later. No updates. Station TPLM2 about 5 nm S of Greenbury Point Annapilis, MD USA

Screen Shot 2021-03-13 at 07.03.43.png

Edited by stanlutz
Inccorrect info / it's correct NDBC vs NWS
Link to post
Share on other sites
On 3/12/2021 at 6:38 PM, fastyacht said:

The internet is one big hackfest. Infrastructure should NOT be connected to it. Air Gap the freak out of it. That is going to be the future--if we want to keep the fucking Chinese Communists out of our shorts.

No kidding, my little 3ktsb website for teachers is being hit constantly with SQL injection attempts and probes for vulnerabilities.   It used to be I could ignore the logs for long periods of time.  Not anymore.  

I don't know what use it will be to them to find out little Suzie has a B in Chem, but whatever.  If they ransomware me I will just upload a new instance and recover the data from backup.  

Link to post
Share on other sites

I spoke with a friend of mine who works for NOAA two days ago and asked her about the issue. She picked up her phone and called someone at their HQ and came back with " yer not gonna believe this, but no shit, a pipe burst in the building where the servers are"

Her speculation is that they shut all the systems down while they are drying out the building. 

Link to post
Share on other sites

I noticed the problem on Wed or Thur when I was trying to see what the wind was doing out at Thomas Pt.

 

Link to post
Share on other sites
27 minutes ago, Rain Man said:

No kidding, my little 3ktsb website for teachers is being hit constantly with SQL injection attempts and probes for vulnerabilities.   It used to be I could ignore the logs for long periods of time.  Not anymore.  

I don't know what use it will be to them to find out little Suzie has a B in Chem, but whatever.  If they ransomware me I will just upload a new instance and recover the data from backup.  

Almost 20 years ago our I.T. engineer (50 person co.) ran an experiment to show us mechanical engineer types what really goes on. He set up a linux box with a host of logging scripts and control files etc and left a port open. Then his logs reported everything. Within minutes there was a knock at the door. Within a half hour the whole machine was compromised--every control file overwritten with backdoors inserted.

The point was to point out that there are no "quiet corners" and that insecure setup is an immediate hazard. Again, almost 20 years ago.

Link to post
Share on other sites
1 hour ago, Bump-n-Grind said:

I spoke with a friend of mine who works for NOAA two days ago and asked her about the issue. She picked up her phone and called someone at their HQ and came back with " yer not gonna believe this, but no shit, a pipe burst in the building where the servers are"

Her speculation is that they shut all the systems down while they are drying out the building. 

Seems like there should be/have been some redundant date center/servers/failover...pretty sure that exists for the forecast modeling servers...but maybe the past few years they focused on making America great again...

  • Like 2
Link to post
Share on other sites
2 hours ago, fastyacht said:

Almost 20 years ago our I.T. engineer (50 person co.) ran an experiment to show us mechanical engineer types what really goes on. He set up a linux box with a host of logging scripts and control files etc and left a port open. Then his logs reported everything. Within minutes there was a knock at the door. Within a half hour the whole machine was compromised--every control file overwritten with backdoors inserted.

The point was to point out that there are no "quiet corners" and that insecure setup is an immediate hazard. Again, almost 20 years ago.

Yep.   20 years ago I had a direct facing ip address and it was regularly hit - especially since I had an ftp server running on it.  If I didn't have a 3 strike you're out policy, multiple ip's from overseas would almost immediately start trying brute force attacks.  They've gotten a lot smarter since then.

Link to post
Share on other sites
9 hours ago, Bump-n-Grind said:

I spoke with a friend of mine who works for NOAA two days ago and asked her about the issue. She picked up her phone and called someone at their HQ and came back with " yer not gonna believe this, but no shit, a pipe burst in the building where the servers are"

Her speculation is that they shut all the systems down while they are drying out the building. 

This is correct.  Pipe burst on the 6th floor of the building. 

Link to post
Share on other sites
7 hours ago, ExOmo said:

Seems like there should be/have been some redundant date center/servers/failover...pretty sure that exists for the forecast modeling servers...but maybe the past few years they focused on making America great again...

That can be really complex and expensive to do, especially hot failover. It is not just a matter of setting up hardware, applications also have to be designed for it as well. Remote facilities, remote backups and such are far easier, but you can also be looking at 24 hours or more to get back, and no system backup is ever completely up to date. Throw in having to maintain the backup site at the same patch levels and configuration, and issues that always surface when you try that exercise.

Link to post
Share on other sites
9 minutes ago, scoopm24 said:

NOAA funds a network of regional coastal observing systems who collect and share data independent of NDBC.

https://ioos.noaa.gov/about/regional-associations/

 Each of the 11 regional systems has a data portal with excellent data.  Some have tailored products to mariners.  All of it is free.  

All of the data streams ended March 9th at 11. This is the same time the servers were shut down at NOAA central. No new data available from any of the buoys since then :-(

Link to post
Share on other sites

Okay, now what are the chances of THIS at the same time as the NBDC issues?

From the REAL TIME LIGHTNING SERVER: This has NOTHING to do with NBDC. 

Technical difficulties due to fire damage. Service will be restored within the next days.

 

Internal Server Error :: Code 500

Sorry, there was an error. Please try again later.

Link to post
Share on other sites
Just now, Rasputin22 said:

Jewish Space Lasers?

you may be onto sumpin

 

Link to post
Share on other sites

I noticed that as of this afternoon sailflow is reporting recent data once again from the west point lighthouse station in seattle, even though the NDBC webpage for it is still dead. Maybe they're starting to get things back up and running?

Link to post
Share on other sites

Some of the same data seems to be getting out at tides&currents. Not all the normal NDBC stations I use on the Chesapeake are available, but some are.

I use this mostly for local water level data at SLIM2 since the Island's drainage system backfills when the tide is too high and my driveway floods at 4.2FT above MLLW...but all the other station data is available too.

On other thing I have noticed is this data doesn't seem to be delayed the way I normally see the NDBC data coming in, and as mentioned, some stations do not report complete data at T&C. It would be nice if they would mirror all the data for redundancy..for example, at Cove Point LNG, no weather data, just current.

https://tidesandcurrents.noaa.gov/

Link to post
Share on other sites
On 3/14/2021 at 8:50 AM, DryArmour said:

This is correct.  Pipe burst on the 6th floor of the building. 

Years ago we ran Payroll for Air Force personnel in the DC area on a Burroughs mainframe at Bolling AFB which is adjacent to Washington Naval Yard.

During a staff meeting the Commanding Officer said there was a leak in the very old building we were located in and the water was accumulating in the the subfloor of the datacenter.  Some of those machines used 440v so it was a serious safety issue due to all the power cables in that subfloor.  They though the water was coming from gutters/roof so it was not like they could just shut off faucet.

I told the colonel that it is easy to resolve.

How? he inquired.

"Lets drill  a hole in the subfloor.  Then it is Finance department's problem since they worked below the datacenter.."   ;<)

Not the solution he was looking for, but it did get him to crack a smile for once in his life.

 

  • Like 1
Link to post
Share on other sites

https://gcaptain.com/flood-knocks-u-s-buoy-data-offline/?subscriber=true&goal=0_f50174ef03-71f6f116d4-170031681&mc_cid=71f6f116d4&mc_eid=90b1ab074a

Flood Knocks U.S. Buoy Data Offline

Mike Schuler
Total Views: 2424 
March 15, 2021

A flood at the U.S. National Weather Service’s headquarters in Silver Spring, Maryland has knocked out power to servers processing NOAA’s marine buoy data, rendering the data inaccessible to the public. 

A timeline for when the data will be restored has not yet been determined.

The NOAA National Data Buoy Center’s mission is to provide quality observations in the marine environment in a safe and sustainable manner to support the understanding of and predictions to changes in weather, climate, oceans and coast, according to the NDBC website. 

The NDBC operates a network of offshore automated weather buoys and Coastal-Marine Automated Network stations that provide hourly reports of marine weather to the National Weather Service and other agencies, providing critical information including wind speed and direction, wave height, pressure changes, and other key data about marine conditions and developing storms along the U.S. coast, Great Lakes, and out at sea. The information is also critical to commercial and recreational mariners.

A statement posted to the NDBC website said the agency’s primary processing servers were shut off due to a facilities issue on March 9. A spokesperson for the National Weather Service offered some additional information about what happened in an emailed statement: 

“On March 9, a burst pipe caused extensive flooding at NWS headquarters in Silver Spring, Maryland, which caused an outage at the data center housed in the building. NOAA’s ocean/marine buoy data are processed on servers located in the affected data center, causing the current buoy data outage. The building is currently being dried out by the building owner/manager, Foulger Pratt. Extensive coordination is taking place between NOAA and Foulger Pratt to develop a comprehensive plan for a full damage assessment, to inspect and restore systems in the building, and to ensure the building is fully repaired and safe for employees to return to their offices. This process will take time to complete. and we do not have an estimation at this time for when the marine buoy data will be available. Alternate solutions to restore buoy data flow are being worked.”

 

 

Link to post
Share on other sites
1 minute ago, robalex117 said:

https://gcaptain.com/flood-knocks-u-s-buoy-data-offline/?subscriber=true&goal=0_f50174ef03-71f6f116d4-170031681&mc_cid=71f6f116d4&mc_eid=90b1ab074a

Flood Knocks U.S. Buoy Data Offline

Mike Schuler
Total Views: 2424 
March 15, 2021

A flood at the U.S. National Weather Service’s headquarters in Silver Spring, Maryland has knocked out power to servers processing NOAA’s marine buoy data, rendering the data inaccessible to the public. 

A timeline for when the data will be restored has not yet been determined.

The NOAA National Data Buoy Center’s mission is to provide quality observations in the marine environment in a safe and sustainable manner to support the understanding of and predictions to changes in weather, climate, oceans and coast, according to the NDBC website. 

The NDBC operates a network of offshore automated weather buoys and Coastal-Marine Automated Network stations that provide hourly reports of marine weather to the National Weather Service and other agencies, providing critical information including wind speed and direction, wave height, pressure changes, and other key data about marine conditions and developing storms along the U.S. coast, Great Lakes, and out at sea. The information is also critical to commercial and recreational mariners.

A statement posted to the NDBC website said the agency’s primary processing servers were shut off due to a facilities issue on March 9. A spokesperson for the National Weather Service offered some additional information about what happened in an emailed statement: 

“On March 9, a burst pipe caused extensive flooding at NWS headquarters in Silver Spring, Maryland, which caused an outage at the data center housed in the building. NOAA’s ocean/marine buoy data are processed on servers located in the affected data center, causing the current buoy data outage. The building is currently being dried out by the building owner/manager, Foulger Pratt. Extensive coordination is taking place between NOAA and Foulger Pratt to develop a comprehensive plan for a full damage assessment, to inspect and restore systems in the building, and to ensure the building is fully repaired and safe for employees to return to their offices. This process will take time to complete. and we do not have an estimation at this time for when the marine buoy data will be available. Alternate solutions to restore buoy data flow are being worked.”

 

 

Not the answer I was hoping for. It is what it is though but be aware that the various weather models depend on that data to create accurate forecasts and now all of the upstream data is unavailable which leaves the West Coast and Hawaii effectively "blind" for forecast accuracy that we have become accustomed to. 

 

Link to post
Share on other sites

Thomas Pt light is back online...  maybe there's hope for the Random Pic thread!!!

 

https://www.ndbc.noaa.gov/station_page.php?station=tplm2

 

Link to post
Share on other sites
  • 2 weeks later...

Kind of related. But lists a bunch of issues facing NOAA.


https://www.newstimes.com/news/article/Weather-Service-internet-systems-crumbling-as-key-16064001.php

Tuesday morning's outage meant the Weather Service's flagship website, weather.gov, was down, cutting off access to its forecasts and warnings.

There is a major, national outage impacting the distribution of NWS products," tweeted the Weather Service's Weather Prediction in College Park, Md.

The Weather Service's central operations center issued a bulletin at 5:11 a.m. highlighting failures nationwide, which included its forecast offices losing contact with the agency's networks "impacting product dissemination and data reception," inoperable websites and no access to NWS Chat.

Link to post
Share on other sites
2 minutes ago, BayRacer said:

Kind of related. But lists a bunch of issues facing NOAA.


https://www.newstimes.com/news/article/Weather-Service-internet-systems-crumbling-as-key-16064001.php

Tuesday morning's outage meant the Weather Service's flagship website, weather.gov, was down, cutting off access to its forecasts and warnings.

There is a major, national outage impacting the distribution of NWS products," tweeted the Weather Service's Weather Prediction in College Park, Md.

The Weather Service's central operations center issued a bulletin at 5:11 a.m. highlighting failures nationwide, which included its forecast offices losing contact with the agency's networks "impacting product dissemination and data reception," inoperable websites and no access to NWS Chat.

I was wondering what the error message I kept getting was indicating. That's the answer. It was up and down on Monday as well. Hope they get this sorted out soon. Seems like upgrading the weather computer infrastructure might be a better way to spend $1.9 trillion dollars. Just my $.02. Your mileage may vary.

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...