We are attempting an 80Gb preposition. 5 hours and 18Gb into the session it terminated with the status message of "Internal Error". The CifsAO log of the edge device has nothing in it. The CifsAO monitoring Cache shows current usage to be 20Gb out of 96Gb. There is nothing in the system logs of the edge device and so far have found nothing in the Data Center device. Any recommendations as to where else I can look to see what the cause was? Our 2-3Gb prepositioning tests all ran fine.
Solved! Go to Solution.
I've looked through a couple of scenarios relating to that error and prepositioning. The only thing I can find on 4.1.5a is the following...
In your preposition job, did you limit the "Total Size as % of Cache Volume"? I'm looking at DDTS CSCta55041-Preposition task may terminate with reason as "Internal Error". It say that it may occur when edge side object cache is completely full and preposition is in progress reading into it. Check your Max cache volume for the job and increase it and retry the job to see if it resolves. This DDTS is fixed in upcoming releases 4.1.7 and 4.2.
The other advice I can give is to try to delete your prepositioning job and then reschedule it fresh to see if there was a problem in the scheduler.
I have a TAC case open and our SE is talking with the developers. According to the CifsAO on the remote I have 95G reserved for the prepositioning. My percentage is currently set at 25% which should be the 95G.
I did get a script from the SE that lists the directories and files in the preposition cache and I discovered something odd. The smaller directories for our previous preposition tests are there as well as the directory for this failed preposition. The odd thing is that with the failed preposition it looks like the directories are there but no files, at least as deep as I have been able to look so far.
The way the original data is laid out is that the files, each about 2G in size are about three directories deep. The way the master files are laid out is that there is the root, then backup, then one of two servers, then the individual databases, then the actual files.
Prepositiong shows server name, which would be the root, then backup, then nothing. Not sure if it is related to the issue or not but have not heard back from the SE or TAC or the developers since we discoverd that yesterday. I think they are still going through the Show Tech and system reports we uploaded yesterday.
One thought we had was to simply take 8 or 10 of these 2G files and let them be the only items in the directory and see if the transfer breaks space related. We also had an internal conversation as to if the issue could be related to the directory structure layout. We just do not understand the numbers reported in the preposition status and what we are seeing when we try to look at what is actually in the cache after 5 hours. I am just waiting to see if there is anything else Cisco may need before I restart a preposition and possibly erase / overwrite information they may want.
Will keep updated as best I can.
You are in good hands. Having the TAC and escalation looking into it is the best thing for issues like this. Good luck, I'll watch for updates.
Here is what I received today ...
our Developers confirmed you are hitting the following defect:
CSCta55041 - Preposition task may terminate with reason as "Internal Error"
This defect addresses the log errors and task status, it doesn't actually allow more data to be prepositioned. The real reason why only 18GB is prepositioned is the configuration of preposition parameter "max-cache 20", which allows only 20% of the 90GB object cache to be used for preposition. Change this to "max-cache 100" and the full object cache will be available for prepositioned data.
I have asked if this is the same as the Total Size as % of Cache Volume as that is ths only field I have in the definition that has a 20 in it. Going to see if I can find anything through the CLI that might be different.
80G test completed fine after setting % cache to 100. It looks like there is about a 95G maximum that the cache can be. I know when we first started working with preposition that I was told the cache was like 400G. That is why we set the % to 25 to start.
Thans for the assistance...