NSS4000 connection lost during backup & no space left backup msg

Answered Question
Nov 3rd, 2009
User Badges:

Hi, we have installed an NSS4000 (with 4 x 1Tb WD disks) for a client.

Bay 1 & 2 setup as RAID 1, Bay3 and Bay4 setup as simple normal disks.

I have upgraded the firmware to 1.14-20, and the system is connected to a network though an AD Domain.


We have been having two serious problems with our setup since day one:

1) The system seems to lose connection when backing up through the Windows Server 2003 SBS backup program. It starts backing up but unexpectedly stops due to loss of the M drive (which is the NSS destination drive - bay 3). We have tried the same backup setup with a network drive (\\Dataserver\Meletes) instead of a mapped drive, but with the same results. The backup log is as follows:


Backup Status
Operation: Backup
Active backup destination: File

Media name: "Backup of Data normal 01-10-09.bkf created 10/1/2009 at 1:50 PM"

Volume shadow copy creation: Attempt 1.
Backup of "M: \\dataserver\meletes"
Backup set #1 on media #1
Backup description: "Set created 10/1/2009 at 1:50 PM"
Media name: "Backup of Data normal 01-10-09.bkf created 10/1/2009 at 1:50 PM"
Backup Type: Normal
Backup started on 10/1/2009 at 1:54 PM.
Warning: Unable to open "M:\Μελέτες\P 205_06 Αναδιαμόρφωση Vodafone Shops\e-mails\Εισερχόμενα\Από Vodafone\9464 Γ Σεπτεμβρίου\Έπιπλα για καταστήματα 9180, 9464 & 9469.eml" - skipped.
Reason: The specified network name is no longer available.
Error: The device reported an error on a request to write data to media.
Error reported: Unknown error.
There may be a hardware or media problem.
Please check the system event log for relevant failures.
The operation was ended.
Backup completed on 10/6/2009 at 1:14 PM.
Directories: 433
Files: 3454
Bytes: 1,981,392,881
Time:  11 minutes and  45 seconds
Error: M: is not a valid drive, or you do not have access.


2) NSS reports an "No space left on destination" error when we use the NSS internal backup.

I have a share on my Raid 1 disks (bay 1 & 2) called Meletes which I try to backup to Disk 3 (Bay3).

The share used space is approx 50Gb and the Disk 3 free space more than 900Gb. Result is the same no matter if I use compression or not.

The message I get is from the NSS is:


2009-10-02 13:00:49 Backup meletes to normal_bcp/backup/DATASERVER.meletes_Backup_2009.10.02_00.15.05.tar.gz; No space left on destination
2009-10-02 00:15:05 Backup meletes to normal_bcp starting


I would really appriciate any help, as client has started requesting return of the product.

Thanks

Correct Answer by Apexrp2009 about 7 years 7 months ago

Hallo,


I had the same issue almost half a year ago, and CISCO is still working on the solution. No workaround has been provided.

If you want more info about my problem please see the thread at the other CISCO forum


http://forums.linksysbycisco.com/linksys/board/message?board.id=Network_Storage&thread.id=6346


In general the problem is caused by too many files in one share. The size of the files should not matter. I have tested it on NSS4000 and NSS6000 with 750GB HDD with RAID1, RAID10, and NO RAID. The problem was always the same. I have also tried to backup from one NSS to another (from 4000, to 6000 and from 6000 to 4000), on the same NSS but to another share (on both 4000 and 6000), to a shared folder on a Windows based workstation. The result was always the same "no space left on destination", although, there was plenty of free space.


Oh, I have forgotten to add what files have I tested. I have just taken the "inf" folder from the Windows directory and copied it many times (20-30). It resulted in thousands of files in one directory with a quite small size (about 1GB).


The engineers in CISCO had some problems with replicating my issue, but as far as I know they have finally managed to do so. Moreover, they have checked that it is highly propbably that the previous firmware also had the same issue, but they have not confirmed it, yet.


--

Kind regards,

Marcin

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Alejandro Gallego Wed, 11/04/2009 - 16:54
User Badges:
  • Cisco Employee,

Currently looking into this issue and I am trying to replicate the problem.

Please browse to the destination via windows explorer and make sure that you are able to read and write to the backup destination. Also if possible create a new NTFS back up job using the NSS Admin account not an user account to back up the data and post log file.

organodomi Fri, 11/06/2009 - 00:29
User Badges:

Thanks for your quick response Alegalle,


My NSS is joined to an AD domain, so all users are imported from the server.

I double flashed the NSS4000 firmware yesterday in case that would correct my issues but I got exactly the same results.


- The backup discussed in case (1) above is run from the server within the Administrator account.

Although everything starts to backup as it should the NSS connection is lost later, with no apparent reason. This happens every time!


- The backup in case (2) above - NSS native - does not ask for an account.

I have created 5 shares on my RAID 1 disk (Bay 1 & bay 2 physical disks). Four of them have little data, the fifth has around 40Gb of data. When setting up the backup during my last attempt yesterday, the first four were backed up correctly (log displayed the 'share xxxx completed' message), but during the fifth share backup I got the same unpleasent message of 'no space left on destination'. This is the share that always gives me the problems, and I believe it has to do with the share size. Just for completeness let me note that the first 4 shares contain files which are no more than 8Gb in total (about 2Gb in each share).


Any suggestions?

I heard that a new firmware should be out soon. Could that be the remedy to all this?

Alejandro Gallego Mon, 11/09/2009 - 15:12
User Badges:
  • Cisco Employee,

I am working very hard to find a resolution on this issue in the lab. I have not been able to replicate the error but I am currently trying to find as much random files to put on the NSS to see if there is a problem with file size or are we just seeing some type of configuration error. Granted the config error does not seem likely but you never know. There is another post here that is very similar to this so I will be working hard to find a workaround at best.

In the mean time if you could copy and paste your admin log file to the post there maybe something in there of use. Especially if you see any error concerning user mappings or connection time outs.

Correct Answer
Apexrp2009 Thu, 11/12/2009 - 05:22
User Badges:

Hallo,


I had the same issue almost half a year ago, and CISCO is still working on the solution. No workaround has been provided.

If you want more info about my problem please see the thread at the other CISCO forum


http://forums.linksysbycisco.com/linksys/board/message?board.id=Network_Storage&thread.id=6346


In general the problem is caused by too many files in one share. The size of the files should not matter. I have tested it on NSS4000 and NSS6000 with 750GB HDD with RAID1, RAID10, and NO RAID. The problem was always the same. I have also tried to backup from one NSS to another (from 4000, to 6000 and from 6000 to 4000), on the same NSS but to another share (on both 4000 and 6000), to a shared folder on a Windows based workstation. The result was always the same "no space left on destination", although, there was plenty of free space.


Oh, I have forgotten to add what files have I tested. I have just taken the "inf" folder from the Windows directory and copied it many times (20-30). It resulted in thousands of files in one directory with a quite small size (about 1GB).


The engineers in CISCO had some problems with replicating my issue, but as far as I know they have finally managed to do so. Moreover, they have checked that it is highly propbably that the previous firmware also had the same issue, but they have not confirmed it, yet.


--

Kind regards,

Marcin

organodomi Fri, 11/13/2009 - 00:54
User Badges:

Hi Alex, thanks for your reply.

I read through the above link.

It is really a shame for cisco to offer products with such problems!

Have you tried the last post remedy - downgrading to the previous firmware? Did it solve your issues?

I believe it's time for a return of the NSS, I don't think that my client can wait NO MORE! It's months that I have been trying to resolve this!

Alejandro Gallego Fri, 11/13/2009 - 19:20
User Badges:
  • Cisco Employee,

I have been able to find the following; keep in mind that this does not mean that this is the cause just means that I was able to reproduce the problem twice with the following steps:


Clean NSS (Factory)

Join AD >> make sure to check the CIFS logs for proper binding. You should not "Did we join?" if you do most likely you will also see unable to map local user xxx to sid xxx

Begin to add data once your volumes and shares are created

Begin built in back ups

Once you verify you are not getting errors like below, disconnect the DC long enough to get these errors should be about 5minutes


Screen shot 2009-11-13 at 10.00.22 PM.png

now that we have these errors, user permissions may still work are they are cached, but that behavior will not last.

rejoining the Domain is a good idea but will not work

Screen shot 2009-11-13 at 10.06.30 PM.png

you have to delete the computer account first from AD and then rejoin

if we do and rejoin all is good but i feel that now we still end up with permissions problems.

this does not explain random reboots, as i have not been able to replicate. the long short of it is, call SBSC and open a case so this behavior can be documented as further assist code development.


SBSC: 1.866.606.1866


-Alejandro

tplevrakis Tue, 02/16/2010 - 00:30
User Badges:

Hello to everyone.


I (my client) am a new owner of a Cisco NSS4000.


I have installed four 1TB disks, configured two raid1 arrays and have configured 14 shares on them.


I have the same problem with my backup jobs. I have scheduled a backup job containing my shares and all backups run fine except one of them which does not backup and returns the message.: "No Space left on destination"


Of course there is plenty of space at the destination, no quotas are configured and the read/write permissions are checked and found to be Ok.


I too believe that it is a matter of the number of files and not a matter of the size of the files. My largest back up has 953 files in it and an uncompressed size of aprox. 9 GB and runs fine. But the share that cannot be backed up has an uncompressed size of aprox. 5 GB and 21679 files.


I am using the latest firmware 1.16-3


I also have encountered some extra issues with the NSS4000 like not being able to access the storage share although i supply the correct admin password and also in one occassion it stopped providing the correct permissions configured for each share so i had to rebbot the system, but the backup problem is the most important one and i really need a solution for that.



Thank you

Actions

This Discussion