Troubleshooting Tools

Asterisk's confession booth: pjsip show *

Before any packet capture, before any tcpdump, ask Asterisk what it thinks. pjsip show endpoints is the first thing to type when a customer says "calls are failing". If a trunk's Endpoint state is anything other than In use or Not in use, you have a config error and you can stop debugging the network.

The four pjsip show commands you will reach for ninety percent of the time are endpoints, contacts, channels, and identifies. Below is the cheat sheet — click any row to see a longer example output on the right. Filter by typing.

$ ssh freya@192.168.35.197 \
    "docker exec freya-asterisk asterisk -rx 'pjsip show endpoints'"

The reflex: pjsip set logger on + docker logs -f

When pjsip's snapshots aren't enough, watch the dialog happen. pjsip set logger on tells Asterisk to print every inbound and outbound SIP message to the same log stream as the rest of the agent — which means docker logs -f becomes your live SIP scope. Place a test call, capture the INVITE / 100 / 180 / 200 / ACK arc, then turn the logger off so the logs stop drowning in SIP.

Pair the logger with grep -E 'INVITE|200 OK|ACK|BYE|SIP/2.0' to keep only the headers that matter. If you need the raw bytes (header parsing bugs, suspect transport mismatch), drop to tcpdump in step 89.

# 1. flip logger on
ssh freya@192.168.35.197 "docker exec freya-asterisk asterisk -rx 'pjsip set logger on'"

# 2. tail and filter
ssh freya@192.168.35.197 "docker logs -f freya-asterisk" \
  | grep -E 'INVITE|200 OK|ACK|BYE|SIP/2.0|488|503'

# 3. place the test call from the customer side

# 4. flip logger off (otherwise prod logs drown)
ssh freya@192.168.35.197 "docker exec freya-asterisk asterisk -rx 'pjsip set logger off'"

What you are looking for in the stream:

• No INVITE at all → packet never reached Asterisk. Move to tcpdump; suspect H1 firewall or H5 identify.
• INVITE with no 100 Trying within 200ms → Asterisk silently dropped it. This was the Garanti M08 root cause. Check the identify match block (H5).
• 488 Not Acceptable Here → codec or transport mismatch (H4). Read the SDP carefully.
• 200 OK then call drops at ~32s or ~60s → ACK lost on the return path. Classic stateless proxy symptom (H3).

sngrep — terminal Wireshark for SIP

When you can't get a customer to share a Wireshark capture (compliance, no shell, bureaucracy), sngrep on our side gives you 80% of the same view. It captures live, groups packets by SIP dialog, and renders a ladder diagram per call right inside the terminal. Arrow-keys to select a dialog, Enter for the ladder, F2 to save the dialog as a pcap.

The killer feature is per-dialog grouping: ten concurrent calls become ten clean rows, not 600 interleaved messages. When a dialog row goes red ("FAIL"), it's already pre-grouped — open it and the ladder tells you which leg failed.

tcpdump on the host, Wireshark on your laptop

For media plane debugging — jitter, packet loss, codec negotiation, listening to actual audio — sngrep is too narrow. Capture full UDP with tcpdump, scp the pcap to your laptop, and open it in Wireshark. The trick is filtering only what matters: SIP on 5060 plus the RTP port range Freya uses (10000–10499).

ssh freya@192.168.35.197 \
  "sudo tcpdump -i any -w /tmp/call.pcap udp portrange 10000-10499 or port 5060"
# place a call, then Ctrl-C the tcpdump
scp freya@192.168.35.197:/tmp/call.pcap .
open call.pcap   # macOS opens it in Wireshark

Inside Wireshark, three menu paths are all you need:

• Telephony → VoIP Calls → pick the call → Flow Sequence for the ladder.
• Telephony → RTP → Stream Analysis for jitter, packet loss, sequence gaps.
• Telephony → RTP → RTP Player to listen. If the audio plays cleanly here, your problem is upstream of RTP. If it doesn't, the problem is RTP itself (NAT, port range, codec).

What the Flow Sequence view looks like (mock screenshot):

Read this: KASP forwarded the INVITE at 12:37:16.302 and waited. Asterisk never replied — no 100 Trying, no 180, no 200. After 7 seconds KASP's own timer fired and it returned 502 No More Routes. Absence of a packet, not presence of an error. That is exactly the signature of H5 (identify match miss) — the canonical Garanti M08 root cause.

The 6-hypothesis decision tree

Every recurring SIP integration failure we have ever debugged at Freya falls into one of six bins. freya-onprem/playbooks/sip-asterisk-debug.md codifies them — Ricky's writeup of the Garanti M08 / M09 arc. Memorize this tree. Walking it in order means you do the cheap network checks before the expensive config tuning, which means you fix incidents in 30 minutes instead of three days.

Walk the tree. Each yes/no narrows you to a concrete next command:

For at-a-glance scanning when you don't have time for the tree:

Discipline: every time you face a SIP issue, write down which hypotheses you have ruled out and the evidence that rules them out. Save the artefact (pcap, log snippet, screenshot). The customer will ask the same question two weeks later, and you will be glad you have receipts.