Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1663
Views
0
Helpful
20
Replies

911 call went out secondary MGCP gateway

Go to solution
dbraggs107
Level 1
Level 1

‎09-22-2014 06:28 AM - last edited on ‎03-25-2019 08:31 PM by Community Manager

We had someone dial 911 and it went out a secondary MGCP gateway which caused the call to get routed to a incorrect PSAP. 

I've checked the logs on this gateway and the PRI never went down during the time of the call. I also tested the next day and the call went out the correct gateway multiple times without any changes on my end. 

The only thing I can think of is that this particular gateway lost it's MGCP registration during the time of the call and the 911 call got routed to the next available MGCP gateway. However, there is nothing in the logs that shows MGCP was down during the 911 call. 

I have CDR logs showing the 911 being routed to the other gateway, but it doesn't give a reason why it did so. 

 

Any thoughts on how I can track this call more in depth to figure out why it went out the secondary route group member / gateway?

 

 

 

 

Labels:
0 Helpful
2 Accepted Solutions

Accepted Solutions

Go to solution
Brad Martin
Level 3
Level 3

‎09-22-2014 07:20 AM

Did you check distribution algorithm on the route group? Top Down vs. Circular.

View solution in original post

0 Helpful

Go to solution
Brian Meade
Level 7
Level 7
In response to dbraggs107

‎09-23-2014 11:42 AM

It hit your 9.@ Route Pattern in the BI_IA_Unrestricted partition:

05745660.001 |03:23:27.031 |AppInfo  |Digit Analysis: star_DaReq: daReq.partitionSearchSpace(:0820ca35-41ef-4b13-8d5e-bd76bb146425), filteredPartitionSearchSpaceString(BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways), partitionSearchSpaceString(BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways)
05745660.002 |03:23:27.031 |AppInfo  |Digit Analysis: star_DaReq: Matching Legacy Numeric, digits=9911
05745660.003 |03:23:27.032 |AppInfo  |Digit Analysis: getDaRes data: daRes.ssType=[0] Intercept DAMR.sstype=[0], TPcount=[0], DAMR.NotifyCount=[0], DaRes.NotifyCount=[0]
05745660.004 |03:23:27.032 |AppInfo  |DbMobility: getMatchedRemDest starts: cnumber = 911
05745660.005 |03:23:27.032 |AppInfo  |DbMobility: getMatchedRemDest: full match case
05745660.006 |03:23:27.032 |AppInfo  |DbMobility SelectByDestination: no matching RemDestDynamic record exists for remdest [911]
05745660.007 |03:23:27.032 |AppInfo  |DbMobility: can't find remdest 911 in map
05745660.008 |03:23:27.032 |AppInfo  |Digit analysis: match(pi="2", fqcn="5159564404", cn="4404",plv="5", pss="BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways", TodFilteredPss="BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways", dd="9911",dac="0")
05745660.009 |03:23:27.032 |AppInfo  |Digit analysis: analysis results
05745660.010 |03:23:27.032 |AppInfo  ||PretransformCallingPartyNumber=4404
|CallingPartyNumber=5159564404
|DialingPartition=BA_IA_Unrestricted
|DialingPattern=9.@
|FullyQualifiedCalledPartyNumber=9911
|DialingPatternRegularExpression=(9)(911)
|DialingWhere=(AREA-CODE == 800) OR (AREA-CODE == 888) OR (AREA-CODE == 866) OR (AREA-CODE == 877) OR (SERVICE == 911) OR (AREA-CODE == 855)
|PatternType=National
|PotentialMatches=ForegoPotentialMatches
|DialingSdlProcessId=(0,0,0)
|PretransformDigitString=9911
|PretransformTagsList=ACCESS-CODE:SERVICE
|PretransformPositionalMatchList=9:911
|CollectedDigits=911
|UnconsumedDigits=
|TagsList=SERVICE
|PositionalMatchList=911
|VoiceMailbox=9911
|VoiceMailCallingSearchSpace=BA_Internal
|VoiceMailPilotNumber=1900
|RouteBlockFlag=RouteThisPattern
|RouteBlockCause=0
|AlertingName=
|UnicodeDisplayName=
|DisplayNameLocale=1
|OverlapSendingFlagEnabled=0
|WithTags=
|WithValues=
|CallingPartyNumberPi=NotSelected
|ConnectedPartyNumberPi=NotSelected
|CallingPartyNamePi=NotSelected
|ConnectedPartyNamePi=NotSelected
|CallManagerDeviceType=NoDeviceType
|PatternPrecedenceLevel=Routine
|CallableEndPointName=[b3a0f041-f16b-45e9-90de-19bdf999dd59]
|PatternNodeId=[2fb5b40c-c243-485a-98cf-490507dd713d]
|AARNeighborhood=[]
|AARDestinationMask=[]
|AARKeepCallHistory=true
|AARVoiceMailEnabled=false
|NetworkLocation=OffNet
|Calling Party Number Type=Cisco Unified CallManager
|Calling Party Numbering Plan=Cisco Unified CallManager
|Called Party Number Type=Cisco Unified CallManager
|Called Party Numbering Plan=Cisco Unified CallManager
|ProvideOutsideDialtone=true
|AllowDeviceOverride=false
|AlternateMatches= Information Not Available
|TranslationPatternDetails= Information Not Available
|ResourcePriorityNamespace=
|PatternRouteClass=RouteClassDefault

 

Which goes to this Route List:

05745672.001 |03:23:27.032 |AppInfo  |RouteListControl::idle_CcSetupReq - RouteList(RL_IA_IL_Local), numberSetup=0 numberMember=13 vmEnabled=0

 

 

View solution in original post

0 Helpful
20 Replies 20

Go to solution
Brad Martin
Level 3
Level 3

‎09-22-2014 07:20 AM

Did you check distribution algorithm on the route group? Top Down vs. Circular.

0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Brad Martin

‎09-22-2014 07:32 AM

Yes, route group is set to top down. 

0 Helpful

Go to solution
Jaime Valencia
Cisco Employee
Cisco Employee
In response to dbraggs107

‎09-22-2014 07:45 AM

You would need to pull out the CUCM traces for that kind of analysis.

HTH

java

if this helps, please rate
0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Jaime Valencia

‎09-22-2014 07:51 AM

I'm assuming I'll probably have to enable some additional trace options in CUCM. Are the traces you are referring to done in RTMT? Also any chance I'll be able to find out what happened a few days back now? Thanks all. 

0 Helpful

Go to solution
Jaime Valencia
Cisco Employee
Cisco Employee
In response to dbraggs107

‎09-22-2014 07:55 AM

For troubleshooting we usually set them to detailed.

How much detail, and whether they'll contain something related to the root cause, will depend on the level to which they were set when the problem happened.

How many days you can trace back in them, depends on your call volume.

HTH

java

if this helps, please rate
0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Jaime Valencia

‎09-22-2014 07:59 PM

Ok looks like detailed debugging was already enabled and the records go back to last Friday. Here is what I pulled out of RTMT so far: 

 

The 911 call happened at approximately 3:22am on 9/19. You can see from the trace that it's going to the secondary MGCP gateway in the route group. Strange thing is, it doesn't even try to go to the primary gateway VG0A and just sends the 911 call to VG0B. 

 

2014/09/19 02:38:08.109|CC|RELEASE|22263806|22263807|16
2014/09/19 03:03:55.007|CC|SETUP|22263808|22263809|4453|4440|4440
2014/09/19 03:03:55.012|CC|OFFERED|22263808|22263809|4453|4440|4440|SEP00131ADC4823|SEP00152B8F3FAE
2014/09/19 03:04:25.399|CC|RELEASE|22263808|22263809|16
2014/09/19 03:22:40.987|CC|SETUP|22263811|22263812|4440|4404|4404
2014/09/19 03:22:40.993|CC|OFFERED|22263811|22263812|4440|4404|4404|SEP00152B8F3FAE|SEP000AF4BB77D3
2014/09/19 03:23:21.579|CC|RELEASE|22263811|22263812|16
2014/09/19 03:23:27.032|CC|SETUP|22263814|22263815|5159564404|911|911
2014/09/19 03:23:27.036|CC|OFFERED|22263814|22263815|5159564404|911|911|SEP000AF4BB77D3|S0/SU0/DS1-0@VG0B.xxxx.net
2014/09/19 03:24:30.803|CC|RELEASE|22263814|22263815|16
2014/09/19 03:40:29.172|CC|RELEASE|22263816|22263817|16
2014/09/19 03:40:42.267|CC|RELEASE|22263818|22263819|16
2014/09/19 03:40:59.753|CC|REJECT|22263820|22263821|4432|||1
2014/09/19 03:41:13.940|CC|REJECT|22263823|22263824|4432|9515|9515|1
2014/09/19 03:41:20.507|CC|SETUP|22263825|22263826|5159564440|5159889|5159889
2014/09/19 03:41:20.515|CC|OFFERED|22263825|22263826|5159564440|5159889|5159889|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:41:26.805|CC|RELEASE|22263825|22263826|16
2014/09/19 03:41:33.767|CC|SETUP|22263828|22263829|5159564440|5159889|5159889
2014/09/19 03:41:33.788|CC|OFFERED|22263828|22263829|5159564440|5159889|5159889|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:41:43.749|CC|RELEASE|22263828|22263829|16
2014/09/19 03:43:53.995|CC|RELEASE|22263830|22263831|16
2014/09/19 03:44:07.358|CC|SETUP|22263832|22263833|5159564440|6019515|6019515
2014/09/19 03:44:07.364|CC|OFFERED|22263832|22263833|5159564440|6019515|6019515|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:44:14.550|CC|RELEASE|22263832|22263833|16
2014/09/19 03:44:23.736|CC|RELEASE|22263834|22263835|16
2014/09/19 03:44:29.218|CC|SETUP|22263838|22263839|5159564440|9887020|9887020
2014/09/19 03:44:29.270|CC|OFFERED|22263838|22263839|5159564440|9887020|9887020|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:44:32.158|CC|RELEASE|22263838|22263839|16
2014/09/19 03:44:38.368|CC|RELEASE|22263836|22263837|16
2014/09/19 03:44:39.476|CC|REJECT|22263840|22263841|4440|9|9|1
2014/09/19 03:44:43.146|CC|RELEASE|22263842|22263843|16
2014/09/19 03:45:02.286|CC|SETUP|22263845|22263846|5159564440|6019988|6019988
2014/09/19 03:45:02.303|CC|OFFERED|22263845|22263846|5159564440|6019988|6019988|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:45:06.126|CC|RELEASE|22263847|22263848|16
2014/09/19 03:45:12.108|CC|RELEASE|22263845|22263846|16
2014/09/19 03:45:18.042|CC|SETUP|22263849|22263850|5159564432|01139083526015159889020|01139083526015159889020
2014/09/19 03:45:18.049|CC|OFFERED|22263849|22263850|5159564432|01139083526015159889020|01139083526015159889020|SEP00131A8C37F3|S0/SU0/DS1-0@VG0B.xxxx.net
2014/09/19 03:45:20.163|CC|RELEASE|22263851|22263852|16
2014/09/19 03:45:22.299|CC|RELEASE|22263849|22263850|16
2014/09/19 03:45:28.913|CC|REJECT|22263853|22263854|4440|9|9|1
2014/09/19 03:45:34.672|CC|RELEASE|22263858|22263859|16
2014/09/19 03:45:38.893|CC|RELEASE|22263855|22263856|3
2014/09/19 03:45:56.258|CC|SETUP|22263860|22263861|5159564432|01139083526015159889020|01139083526015159889020
2014/09/19 03:45:56.262|CC|OFFERED|22263860|22263861|5159564432|01139083526015159889020|01139083526015159889020|SEP00131A8C37F3|S0/SU0/DS1-0@VG0B.xxxx.net
2014/09/19 03:45:58.204|CC|RELEASE|22263860|22263861|16
2014/09/19 03:46:13.692|CC|RELEASE|22263862|22263863|16
2014/09/19 03:46:28.364|CC|SETUP|22263864|22263865|4432|9889020|9889020
2014/09/19 03:46:28.372|CC|OFFERED|22263864|22263865|4432|9889020|9889020|SEP00131A8C37F3|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:46:34.861|CC|RELEASE|22263864|22263865|16
2014/09/19 03:46:48.460|CC|RELEASE|22263866|22263867|16
2014/09/19 03:46:53.857|CC|SETUP|22263868|22263869|5159564440|5159889|5159889
2014/09/19 03:46:53.864|CC|OFFERED|22263868|22263869|5159564440|5159889|5159889|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:47:01.798|CC|RELEASE|22263868|22263869|16
2014/09/19 03:47:02.979|CC|RELEASE|22263870|22263871|3
2014/09/19 03:47:03.417|CC|RELEASE|22263872|22263873|16
2014/09/19 03:47:05.188|CC|RELEASE|22263874|22263875|16
2014/09/19 03:47:17.918|CC|SETUP|22263876|22263877|4406|18775766676|18775766676
2014/09/19 03:47:17.925|CC|OFFERED|22263876|22263877|4406|18775766676|18775766676|SEP000AB7C6B04E|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:47:33.761|CC|RELEASE|22263876|22263877|16
2014/09/19 03:48:10.213|CC|RELEASE|22263878|22263879|16
2014/09/19 03:50:40.131|CC|SETUP|22263880|22263881|4439|4404|4404
2014/09/19 03:50:40.135|CC|OFFERED|22263880|22263881|4439|4404|4404|SEP0012D9EDDAB0|SEP000AF4BB77D3
2014/09/19 03:51:27.848|CC|RELEASE|22263880|22263881|16
2014/09/19 03:54:35.794|CC|SETUP|22263883|22263884|4404|4494|4494
2014/09/19 03:54:35.798|CC|OFFERED|22263883|22263884|4404|4494|4494|SEP000AF4BB77D3|VGC127f83095a17
2014/09/19 03:55:12.905|CC|RELEASE|22263883|22263884|16
2014/09/19 03:55:44.958|CC|RELEASE|22263885|22263886|16
2014/09/19 03:57:08.368|CC|SETUP|22263887|22263888|5641|918775766676|918775766676
2014/09/19 03:57:08.375|CC|OFFERED|22263887|22263888|5641|918775766676|918775766676|SEP001A6D3FC473|S0/SU0/DS1-0@VG0C.xxxx.net
2014/09/19 03:57:36.528|CC|RELEASE|22263887|22263888|16
2014/09/19 03:59:56.396|CC|SETUP|22263889|22263890|5637|5639|5639
2014/09/19 03:59:56.401|CC|OFFERED|22263889|22263890|5637|5639|5639|SEP001A6D3FC40B|SEP001A6D3FC38A
2014/09/19 04:03:55.084|CC|RELEASE|22263889|22263890|16
2014/09/19 04:12:00.909|CC|SETUP|22263892|22263893|5155209852|4404|4404
2014/09/19 04:12:00.912|CC|OFFERED|22263892|22263893|5155209852|4404|4404|S0/SU0/DS1-0@VG0A.xxxx.net|SEP000AF4BB77D3
2014/09/19 04:12:37.302|CC|RELEASE|22263892|22263893|16
2014/09/19 04:12:54.447|CC|SETUP|22263895|22263896|4404|4494|4494
2014/09/19 04:12:54.453|CC|OFFERED|22263895|22263896|4404|4494|4494|SEP000AF4BB77D3|VGC127f83095a17
2014/09/19 04:12:56.397|CC|RELEASE|22263895|22263896|16
2014/09/19 04:13:04.427|CC|SETUP|22263897|22263898|5159564404|15155209852|15155209852
2014/09/19 04:13:04.434|CC|OFFERED|22263897|22263898|5159564404|15155209852|15155209852|SEP000AF4BB77D3|S0/SU1/DS1-0@VG0A.xxxx.net

0 Helpful

Go to solution
Brian Meade
Level 7
Level 7
In response to dbraggs107

‎09-23-2014 08:29 AM

That's just the call logs.  We need the SDL files.

0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Brian Meade

‎09-23-2014 10:04 AM

Attached 911 call at 3:22am on 9/19

 

 

sdl001_100_000426-mod.zip
0 Helpful

Go to solution
Brian Meade
Level 7
Level 7
In response to dbraggs107

‎09-23-2014 10:16 AM

So it went out this PRI:

05745680.003 |03:23:27.033 |AppInfo  |SMDMSharedData::findLocalDevice - Name=S0/SU0/DS1-0@VG036015.xxxx.net Key=91f798b2-c835-a160-97b7-d6d56d295a13 isActvie=1 Pid=(1,143,38) found

 

 

I'm guessing that's the incorrect one?  Which GW/PRI should it have gone out?  Can you share a screenshot of your route group?

0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Brian Meade

‎09-23-2014 10:28 AM

Yes, it went out S0/SU0/DS1-0@VG036015.xxxx.net and it should have gone out S0/SU0/DS1-0@VG074010.xxxx.net

 

attached 

142 KB
0 Helpful

Go to solution
Brian Meade
Level 7
Level 7
In response to dbraggs107

‎09-23-2014 10:39 AM

The traces you provided are from the publisher?  Is that the node the PRI shows registered to?  If not, we'll need the SDL traces from that node.

0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Brian Meade

‎09-23-2014 10:49 AM

Yes, and both nodes in question are registered to the pub. Thanks for taking a look. 

0 Helpful

Go to solution
Brian Meade
Level 7
Level 7
In response to dbraggs107

‎09-23-2014 11:04 AM

I don't see anything about that PRI being down when it tries to route the call.

 

I see a successful call out the S0/SU0/DS1-0@VG074010.xxxx.net PRI later at 3:41:20 but that call used the "RL_IA_Local" route list.

 

The 911 call used the "RL_IA_IL_Local" route list which seems to have S0/SU0/DS1-0@VG036015.xxxx.net listed as the first endpoint.  Can you share a screenshot of the route groups configured on the "RL_IA_IL_Local" route list?

0 Helpful

Go to solution
dbraggs107
Level 1
Level 1
In response to Brian Meade

‎09-23-2014 11:33 AM

Looks like you found the problem RG_IA_IL_Local was set to circular which was the first one on that RL's list to go out. Strange thing about this is all my test calls that I made went out RG_IA_Local which was already set to top down and had the correct gateway at the top. 

 

The reason why I thought this was for this reason: 

 

  • Results Summary
  • Calling Party Information
  • Dialed Digits = 911
  • Match Result = RouteThisPattern
  • Matched Pattern Information
  • Called Party Number = 911
  • Time Zone = Etc/GMT
  • End Device = RL_IA_Local
  • Call Classification = OnNet
  • InterDigit Timeout = NO
  • Device Override = Disabled
  • Outside Dial Tone = NO
  • Call Flow
  • TranslationPattern :Pattern=
  • Route Pattern :Pattern= 911
  • Route List :Route List Name= RL_IA_Local
    • RouteGroup :RouteGroup Name= RG_IA_Local
    • RouteGroup :RouteGroup Name= RG_IA_LD
    • RouteGroup :RouteGroup Name= RG_Local
    • RouteGroup :RouteGroup Name= RG_LongDistanceCalls
    • RouteGroup :RouteGroup Name= RG_NY_LOCAL
    • RouteGroup :RouteGroup Name= RG_NY_LD
0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents