We run a comprehensive CVP model. This includes AS5400 for ingress gateway, AS5350 for VXML, Cisco Unified Presence (stateless sip proxy), and CVP/ICM.
Currently the IIS server that is serving the vxml pages/media to the 5350 is currently set to pass the http header cache-control with a duration of 30 mins. When I do a show http client cache I can see that the freshtime on some files is set to 1800 which is the 30 minutes cache-control setting that is configured on IIS, which is a global setting.
However I see files that have the default freshtime such as 864,000 (24 hours) that is somehow getting set by the IOS. I've been told the IIS server is globally configured in such a way that each cache-control header in the http response will be set to 1800.
Is there anything else I can look at here? We believe this is causing issues when new media files are loaded on the IIS server and they never show up in cache on the vxml browser.
Please let me know what info would be helpful.
However I see files that have the default freshtime such as 864,000 (24 hours) that is somehow getting set by the IOS. I
Don't you mean 86,400 (24 x 60 x 60 = 86400) ?
Your gateway probably has:
http client cache refresh 86400
This will be used only when the message does not specify the Fresh Time. It can do this through http message headers (Cache-Control), having Expires, Date in the message header, and sometimes as 10% of diff between Date and Last Modified.
If you are seeing this in your gateway, then it means that despite what the IIS guys say, some files are coming down without the "Content Expiration" being set.
I've been told the IIS server is globally configured in such a way that each cache-control header in the http response will be set to 1800.
What does "globally configured" mean? Right at the top of the tree? Ask them to check.
That said, why are you using such a short content expiration time? 30 minutes is extreme.
I typically use 30 days for high performance, but have some directories in IIS with shorter times - 1 day for some messages, 1 hour for emergency messages.
Why did you chose 30 minutes?
Thanks much for the fast response. You will have to bear with me since our environment is very complex and so many people own different parts of our CVP topology.
I checked in our vxml gateways and the default set is:
http client cache refresh 864000
The IIS guys have confirmed that the cache-control header will get populated at the top level which I'm assuming that means all folders below it. What's strange is I can see some files in the same folder have different values. For example I can see .wav files that have the default of 864000 set and I also see files set with the standard 1800.
I'm assuming since there are probably 2500 - 3000 call scripts in place the vxml dev's want the gateways to refresh quick since they tend to make numerous changes during the day.
Since I posted this I found out the IIS servers in our design are front ended by F5 local content manager switch. I'm now wondering if those devices are to blame for stripping out the content-cache header. I will run some packet caps tomorrow and see what I get.
Minor point - so the setting on the gateway is 10 days.
I checked the CVP7 Config Guide and 864000 is right through it. The CLI syntax is
#http client cache refresh ?
<1-864000> Time value in seconds
so 864000 is the maximum.
According to the Cisco White Paper on HTTP Client Cache, 86400 is the default. I don't know how to set the default - perhaps if you simply don't have this line, the default acts. Funny how you can see these numbers for ages and misplace a few zeros.
You will have to bear with me since our environment is very complex and so many people own different parts of our CVP topology.
No worries - I'm used to complex environments.
If your IIS guys say it's set at a top level directory (or on a web site, or even the default for the web serevr) it certainly will get propagated to all files underneath.
How many Media Stores do you have?
What's strange is I can see some files in the same folder have different values. For example I can see .wav files that have the default of 864000 set and I also see files set with the standard 1800.
Ok, that is very weird.
I have heard that the behaviour of the HTTP client cache on the 5350XMs was not as "reliable" as the ISRs. For example, you can stale the cache on the ISR but you can have problems doing this on the 5350. Maybe there is a bug there.
When you say "same folder", is this the same exact URI?
Are the VXML developers still developing scripts? That's fine, but I would have the top level of the IIS tree set to 30 days and override it somewhere down the line to 30 mins on a folder for the VXML devs. They can use the "default audio path" variable during development, and then fix it for the go live (assuming Audium).
But it sounds like this is not a branch office (split ingress and VXML gwys) and your media store is on the LAN. Correct?
In that case, as short lifetime is not a real problem, but Cisco do claim that router performance under load can suffer when caching is suboptimal. Cached files can be shared, but if it's not cached, multiple calls could be fetching the file at the same time. I'm just a bit obsessive on this.
What about queue music. There is no need to have a 30 minute lifetime on queue music.
We also use a Virtual IP Address - and use the Cisco CSS. These run a heartbeat to the media stores and fetch a small file every second or so. I bet the F5 works the same way. I know that the CSS does not manipulate the header.
Packet captures are always the final arbiter.
Thanks for an interesting post.
Hehe, if you stick around long enough I can make all kinds of interesting posts. I'm convinced that we have the most complex topology not only for CVP but for our other platforms.
So yes, looks like I can't add nor divide, the default timer for these is 10 days not 24 hours. So that explains why the dev's were complaining to me for almost the past two weeks that .wav files are not updating.
If I do a show http client cache on the vxml gateway I can see .wav file that belong to the same folders even down to the same sub-folder that have different freshtime values. The only difference being the actual file name.
And the IIS guys swear by the good book that the folders are set for 1800 for the cache header. I can do a set http client cache stale command on one of the vxml gateways version 12.4(15)T7 and I can watch the cache start to fill up (we do about 3.5 million calls a month and these are split out across 40 some vxml gateways). Almost immediately I see files stored in cache with the default timer. You are correct - are media store is on the LAN and bandwidth is not a concern for us.
In addition, our CVP model is only in place for the IVR treatment. If the customer wants an agent we do a transfer connect (take back and transfer with the carrier). These transfers end up in some pretty beefy Avaya systems. So therefore, we don't have any music on hold that is local to the CVP system.
That's good news to hear that your CSS does not play havoc on the headers.
I've always assumed that was the case but a packet capture tomorrow will prove who is to blame here. I could do some debugs at the vxml gateway but without a syslog to log them to it would be worthless since there is so much volume.
Let me know if you come up with any other thoughts. I'm also thinking about opening up a TAC case just to make sure there are no bugs in the train of code we run on the vxml side.
Can you collect a packet capture while doing "audio-prompt load
It will be interesting to see what's in it.
I was actually able to run a quick debug this morning and I think I've confirmed that the AS5350 is NOT to blame.
Here is what I did:
1. Run "debug http client msg"
2. Run "debug http client cache"
3. Run "set http client cache stale"
I then started watching the cache fill up. My debug started to show that for whatever reason when the 5350 reaches out to the media server to see if a .wav file has been modified that the media server does not always add the cache-control tag one the response. I get the proper response the file was not modified but on other .wav files I get the "not modified" response and I also get the cache-control header.
I then went and did a "show http client cache" and for the ones that I did not see a cache-control header on they were listed at the default timer.
Once again, I checked with the IIS team and they assured me that the cache-control header is set at the top level which includes all folders underneath it. I think at this point I will do the real packet capture and just hand it to the IIS team and have them explain why I don't see the cache-control header on every request.
Is it a requirment of IIS to always send the cache-control header on the requests for last modification?
Packet cap attached.
Take a look at the first GET message sent out and the response is a stripped down 304 http response. This message does not contain any expire headers, cache-control header and not even a last modified date which IMO breaks rules defined in RFC 2616. After this response I can go in and do a "show http client cache" and I can see that this .wav file has been tagged with the default freshtime I have set in IOS.
Look further down and you will see a proper 304 http response come back with cache-control and a last modified date. A show http client cache will show this file set with 1800 seconds as the freshtime since the cache-control flag was set.
So now, I think I've got enough info to go back to the IIS team and tell them the news. I do not think the vxml browser is to blame for any of this. It's got to be something wacky in IIS or the content switch.
Good stuff. I agree with your analysis, but it sure is strange behaviour. I'd guess you'd want the wireshark on the upstream side of the F5 to check.
This topic is interesting enough to make some test in my lab.
With IIS, when I reload my prompt to get GW to send refresh request, when IIS sends 304, it does have cache-control header. So I guess in your case, it could the content switch that is causing the problem.
I've also modified the wave file in IIS not to have expiry. Now, when I tried to reload the prompt, the 304 didn't have any cache-control. So the fresh time in GW defaults back to it's value 3min compared to it was set earlier 1 min.
I guess you want capture what IIS sending before the content switch. Your IIS people may be telling the truth that they have setup right
Thanks for sharing your issue and actions.
It would be pretty easy to verify for the microapps.
Assuming you use a key word in your ICM scripts like "media" and resolve that on the gateway through the ip host table to point to the VIP with "media-backup" also pointing at the VIP, pick one gateway and change the ip host table to go directly to the IIS boxes, then stale the cache. Watch what happens.
If it works, then the F5 is the problem.