| Description | UPDATE, 25.11.22, 3:25 p.m. Due to very large amounts of data, the file system checks could unfortunately not yet be completed. Access to the data in the LSDF Online Storage is therefore still not possible. We regret this long outage. _ _ _ UPDATE, 24.11.22, 11.15 p.m. Since Monday, 21.11.22, we have performed several consistency checks. The results of these checks are being verified by the manufacturer support. Unfortunately, during the checks, which take up to 24 hours per file system, crashes occurred, so that they have to be repeated. Under normal circumstances such work is done in parallel, but the current problems in the cluster mean that we cannot work in parallel. Unfortunately, this also means that it will take longer until we can make concrete statements about the availability of the data. We apologize for the ongoing outage. _ _ _ UPDATE, 21.11.22, 3.30 p.m. On 18.11.2022 the LSDF had to be taken completely offline and isolated in order to stabilize internal operations for troubleshooting. This was successful, so that important maintenance processes could be carried out for all file systems over the weekend. Whether this work was successful must be confirmed today with load tests and renewed file system checks. Together with the manufacturer support, the resulting data will be evaluated in order to be able to ensure the consistency of the user data. Together with the manufacturer support, a possible candidate for the cause of the file system failure was identified. This points to problems in the Infiniband network that only occur in conjunction with certain firmware/software/configuration states and which may have been triggered by the load of the original file system check. We are currently investigating possible paths to update above components. We apologize for the ongoing outage. _ _ _ UPDATE, 18.11.22, 2 p.m. Due to given reasons and to avoid further problems in the file system, the LSDF Online Storage has to be switched off urgently. This also means that read-only access to the data is NOT possible anymore! Thank you for your understanding. _ _ _ LSDF Online Storage (bulk storage for scientific data) - https://www.scc.kit.edu/en/services/11228.php - is not available. AFFECTED/IMPACTS Access via all normally available protocols (ssh, scp, sftp, https, webdav, nfs, cifs, ..), as well as access via connected HPC systems to the LSDF data are affected. On the HPC login nodes, LSDF login cluster, LSDF WebDAV cluster, and NFS/SMB cluster, all LSDF file systems except lsdf02 read-only are available. We are working with the support team to resolve the issues as quickly as possible. |
|---|---|
| Affected users | LSDF user |
| Exists since | 2022-11-16 12:00 |
Announcements
Subscribe to / unsubscribe from announcements
|
News about the SCC offer: Service News
|
Report malfunction: via ITB or the SCC ticketing system.
|
In case of failure of this website: scc.fail
|
Incidents