News and Events: Archive:

Service Announcement: Digital Repository

The storage system hosting digital repository services (RUcore) is experiencing hardware problems. As a result, the following resources became unavailable at 7:15 p.m., Tuesday, April 12, 2011:

The Libraries' server administrators are working with the hardware manufacturer to re-enable the storage system. We regret any inconvenience and will keep the Rutgers and partner communities advised of the status of the above services.

Integrated Information Services

This is an update at the end of Day 2 of the repository outage (April 14, 6:14 p.m.):

1. We now have a priority 1 service ticket open with the manufacturer of the repository hardware. We have sent them system data for their diagnostics and await further action on their behalf.

2. The repository interface pages have been moved to another server so that users trying to connect will see a service message instead of a blank screen.

3. We have made plans to restore data from repository backup to another server so that information can be made available for searching (but not for ingesting new objects) if the repair takes more than another day.

We will update the Libraries Community again tomorrow. We wouldd like to assure everyone that both SCC and IIS staff members in charge of the repository and the servers are doing their best to bring these services back to life as soon as possible.

Thank you for your patience as we work through these difficulties.

Integrated Information Services

Update, April 15, 5:32 p.m.:

The manufacturers' service engineers are still diagnosing our hardware but at this point we know they will have to make a site visit scheduled for Monday morning.

In the meantime, we have started to build a read-only version of the repository and plan to restore content to it. The latter is a very time-consuming process as over 20,000 objects and almost 100,000 data streams, randomly located on automated tape backup, must be managed. Work will proceed over the weekend but we do not expect to complete it before late Monday.

The good news is that we received five proposals for the new storage system and we are reviewing the competitive bids to make a selection. New technologies allow us to design the next repository with high-level redundancy and faster recovery in mind.

Once again, thank you for your patience.

Integrated Information Services

Update, April 20, 11:30 a.m.:

Dear Colleagues,

Good news, finally!

A read-only version of the repository has been brought online. This is a temporary solution while we rebuild the main system. It will allow users and librarians to search and retrieve more than 95% of the objects and videos in RUcore, RUetd, RUresearch, NJEDL, VMC, and other collections – but not to modify them or create new ones.

SCC and IIS have spent hours testing this read-only replacement. But we understand there might be some unexpected discrepancies so we will keep regenerating indexes in the background. Please let us know if you find any glitch or discover missing content.

Work continues on re-creating our production system but it will take several days. We hope that the temporary replacement will see us through until then. Our long-term plans include keeping a cold standby system like this one in case of any unexpected event in the future.

Thank you for your support and congratulations to team members who made this possible and tested with us: Chad, Isaiah, Jeffrey, Jei, Rhonda, Linda, Dave, Sho, Ashwin, Kalaivani, and Marty.

Integrated Information Services

Posted April 14, 2011; April 15, 2011; April 18, 2011; April 20, 2011