sngrep saving invalid pcap when SIP recieved over EEP listener

[tomponline]

We have recently started using sngrep's eep listener feature combined with Freeswitch's HEP dump feature.
This is working well and we can see SIP traces from encrypted and unencrypted sessions coming from Freeswitch's HEP send feature in sngrep.

However if we dump selected dialogues from inside sngrep to a pcap file then these files are not openable with either sngrep -I or wireshark.

If you choose the txt dump option in sngrep then this works OK, but isn't as useful as the PCAP format.

See attached 2 files for the same SIP dialogue dumped in pcap and txt format.
user.zip

[tomponline]

sngrep config:

set capture.device lo
set eep.listen on
set eep.listen.version 3
set eep.listen.address 127.0.0.1
set eep.listen.port 9060
set eep.listen.pass
set eep.listen.uuid off

Freeswitch config:

<global_settings>
  <param name="capture-server" value="udp:127.0.0.1:9060"/>
</global_settings>

[tomponline]

sngrep version 1.4.6

[Kaian]

Hi @tomponline

Yep, I was aware of this flaw. The problem is that the save process is expecting frames captured from the wire with all packet binary contents (including link layer headers). When capturing from EEP (seems only me is using EEP instead of HEP), there are no link layer information so, in order to save to pcap, we must generate each packet protocol header and pack to binary to create a frame.

I think that's what Homer does, creating a ethernet header with hardcoded MAC addresses, and packing each header.

This is not a quick fix, but I'll try to give it a shot.

Thanks for reporting!

[tomponline]

thanks @Kaian is there a difference between EEP and HEP? Would using one or the other fix this issue?

Thanks for quick response.

[Kaian]

Nope, it's the same protocol.

According to sipcapture hep project it was originally HEP and changed to EEP, although nobody uses EEP actually. Next sngrep major version will have all those setting renamed to HEP, to make it more consistent with the capturing tools environment.

[towe75]

Hi,

we have a related problem when sending hep V3 from kamailio to sngrep.
The payload looks ok but src/dst ip numbers and timestamps are broken.

I checked capture_eep.c and compared it with the HEP3 spec and also the Kamailio sipcapture parser.
IMHO the protocol allows a arbitrary Chunk ordering but sngrep assumes a fixed sequence of Chunks and tries to read src and dst ip directly after the header.

You can see the concrete sequence of Chunks in the Kamailio code: https://github.com/kamailio/kamailio/blob/42129d35e4510092266322168f68f3019d28bdb8/src/modules/siptrace/siptrace_hep.c#L97 and it sends two hep_chunk_uint8_t chunks before src and dst addresses, hence the broken ip numbers in sngrep.

Maybe the Kamailio parser code from https://github.com/kamailio/kamailio/blob/42129d35e4510092266322168f68f3019d28bdb8/src/modules/sipcapture/hep.c#L294 can be used as a draft for a better implementation.

[Kaian]

Hi @towe75

I have updated the parser for HEPv3 based on the kamailio module you linked. I have only tested it sending HEP from another sngrep, so I'm not sure if it will work for kamailio (but it should). If you have time to test it and give some feedback it would be helpful. Thanks for the information.

The parser loop for sngrep is here in case you want to dig in the code: https://github.com/irontec/sngrep/blob/master/src/capture_eep.c#L650

Anyway this was not related to the problem of the issue. As mentioned in notes before, fixing this issue won't be easy, as we have to create a packet builder from HEP to PCAP (like homer does when you download as pcap).

Regards!

[towe75]

Hi @Kaian ,

wow, that was really quick. I will have a look at the commit and i will try it out in our enviroment in a few days (after Kamilioworld 2019).

Cheers

[towe75]

@Kaian : i compiled the latest master. IP's and timestamps are now fine in HEP3 packets.
Thank you!

[Kaian]

Cool, It will be in a future 1.4.7 as soon as I collect some more fixes.

Thanks for the testing, and sorry to the original poster for all these emails you'll be receiving.

[tomponline]

@Kaian no problem :)

[marrold]

Hi, using 1.4.9 from the Irontec repositor I am still missing the IP headers?

Thanks

[Kaian]

There has been no changes in HEP parser in the past years, and I'm not aware of any problem parsing HEP in 1.4.9, so you shouldn't be missing any header. Although saving HEP to PCAP is still not possible in 1.4.9

[marrold]

Although saving HEP to PCAP is still not possible in 1.4.9
Oh, that's what I was doing. Capturing via HEP and then saving to PCAP. The signalling was saved in Ethernet Frames but without the IP headers

[thewilli]

Any news on this issue (i.e. saving HEP to PCAP results in invalid PCAP file)?

[g-v-egidy]

I would also like to write data received over HEP/EEP to a PCAP-file.

I use Asterisk (with pjsip) and all my SIP sessions are TLS-encrypted. While Asterisk has the ability to directly write pcap-files, this option isn't very flexible and will also dump the sip sessions into a logfile. So you can only use it for a short time. I would prefer to let Asterisk always output the data via HEP/EEP and then capture it on demand and with comfortable filtering with sngrep.

[Kaian]

Hi!

I will try to implement something to address this.

Packets received from HEP are being saved as just the payload, so I will try to generate a fake Ethernet + IPv4 + UDP header partially filled with the HEP data, partially invented (like eth mac addresses). This is not a solution but I think homer does a similar thing when you download a pcap from the web, and allows opening the pcap in other libpcap tools (sngrep between them).

This still won't allow saving mixed HEP + network in the same file, but that's another story.

Regards

[Kaian]

Hi!,

Saving received packets in HEP/EEP Listen mode should be possible in the next release. Also, this listen mode can now be used without capturing from network at the same time, so you can run sngrep as just a HEP receiver.

There is still no way to mix multiple capture sources into a single pcap (mixing packets from network and received from hep socket).

Regards!

[g-v-egidy]

Thank you very much for working on this.

I just went ahead and tried it out by taking 1.4.10 and patching in your commit.

Unfortunately it didn't work as expected for me: I could see the packages received from my Asterisk HEP export fine in the sngrep window, but the exported pcap can't be properly parsed by wireshark. Wireshark shows the individual packets, but the decoding is just garbage.

Did I maybe use it wrong? I called sngrep -E -L 127.0.0.1:9061 -O test.pcap to create the pcap.

I have attached a short sample pcap, maybe you can figure out the problem.

[g-v-egidy]

When I downloaded the patch yesterday 1.5.0 wasn't released yet, so I didn't test it with that.

I now took the final 1.5.0 and tried it again - unfortunately it didn't work too. Same result as I got yesterday.

Or do you prefer if I open a new issue for this?

[Kaian]

No, I'll handle it in this issue.

Above commit only changes saving from the UI that is what we mostly use.

I'll check the -O flag and report you back.

Thanks for the testing and feedback!

[g-v-egidy]

I just tried to save with F2 in the gui and it didn't make a difference, the pcap was still not properly parseable by wireshark.

[Kaian]

Hi!

If you're capturing only with HEP (you can check this in the first line of the UI, it must display Capture Mode: Online [L:9061]), then it shouldn't be problems while saving using the ui. If you're mixing capture modes ([multi] usually appears in the capture mode), it may cause unexpected behaviours.

If you check latest master commits, it should be also possible to save from the command line using -O argument. It required a couple of commits because sngrep was not designed to work without a libpcap capturing thread, so be sure to pull them all.

Thanks for the testing!

[g-v-egidy]

Yey! With your latest patches it now works for me.

Thank you for implementing this.

[Kaian]

Nice!

Thanks to you for all the testing!!

Let's hope 1.5.1 won't take so much time to release 0:)