Crashed

Phil - NY4Q · May 10, 2025, 12:07 AM

And the error was :

May 09 14:04:40 websdr start-websdr.sh[487124]: [2025-05-09 14:04:40] [info] asio async_write error: system:32 (Broken pipe)
May 09 14:04:40 websdr start-websdr.sh[487124]: [2025-05-09 14:04:40] [fatal] handle_write_frame error: websocketpp.transport:2 (Underlying Transport Error)
May 09 14:05:47 websdr start-websdr.sh[487124]: [2025-05-09 14:05:47] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:05:47 websdr start-websdr.sh[487124]: [2025-05-09 14:05:47] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:05:47 websdr start-websdr.sh[487124]: [2025-05-09 14:05:47] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:07:41 websdr start-websdr.sh[487124]: [2025-05-09 14:07:41] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:07:41 websdr start-websdr.sh[487124]: [2025-05-09 14:07:41] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:07:41 websdr start-websdr.sh[487124]: [2025-05-09 14:07:41] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:07:41 websdr start-websdr.sh[487124]: [2025-05-09 14:07:41] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
May 09 14:10:20 websdr start-websdr.sh[487124]: [2025-05-09 14:10:20] [error] handle_read_handshake error: websocketpp.transport:7 (End of File)

Bas ON5HB · May 12, 2025, 11:42 PM

Same problems here, I'm testing this type for compiling, also changed in install.sh

Code Select

meson build --optimization=3 --unity='on'
and

Code Select

meson compile -C build
See here what it does:

https://mesonbuild.com/Unity-builds.html

They claim it produces faster and better code.

Fingers crossed

Also the compiled binary is a lot smaller.

Bas ON5HB · May 13, 2025, 04:05 PM

Next step, I noticed 'too slow' messages appear every few seconds. However, I do not believe the message is right.
As the websocket can handle very large ammounts of data in 1 package.

So I looked into backend / websocket.cpp and found the 'slow' part and changed the buffer-count from 50000 to 100000.
Now I see diffent messages appear but no slow anymore.

Code Select

// If the client is slow, avoid unnecessary buffering and drop the
        // audio value was 50000 -> Changed to 100000
        auto con = m_server.get_con_from_hdl(data->hdl);
        if (con->get_buffered_amount() > 100000) {
            printf("Dropping Audio due to buffering slow client\n");
            continue;

And changed this part to match it:

Code Select

// Iterate over each waterfall client and send each slice
        std::scoped_lock lg(waterfall_slice_mtx[i]);
        for (auto &[slice, data] : waterfall_slices[i]) {
            auto &[l_idx, r_idx] = slice;
            // If the client is slow, avoid unnecessary buffering and
            // drop the packet - changed from 50000 to 100000
            auto con = m_server.get_con_from_hdl(data->hdl);
            if (con->get_buffered_amount() > 100000) {
                printf("Dropping Audio due to buffering slow client\n");
                continue;

I did use the unity compiling...as I beleive it's far better.

I do see messages like this now....but no crash (so far!

)

Code Select

mei 13 15:54:12 Websdr start-websdr.sh[977381]: Thank you for using PhantomSDR+, you are supporting the Development of an Open-Source WebSDR Project ♥
mei 13 15:54:12 Websdr start-websdr.sh[977381]: Using OpenCL
mei 13 15:54:12 Websdr start-websdr.sh[977381]: Using OpenCL platform: Intel(R) OpenCL Graphics
mei 13 15:54:12 Websdr start-websdr.sh[977381]: Using OpenCL device: Intel(R) HD Graphics 630
mei 13 15:54:12 Websdr start-websdr.sh[977381]: Markers updated.
mei 13 15:54:14 Websdr start-websdr.sh[977379]: Attenuation: 0
mei 13 15:54:14 Websdr start-websdr.sh[977379]: Gain: 40
mei 13 15:54:14 Websdr start-websdr.sh[977381]: Waterfall is sent every 1 FFTs
mei 13 15:58:34 Websdr start-websdr.sh[977381]: [2025-05-13 15:58:34] [fatal] Fatal error reading request:  error: websocketpp.http:2 (The specified data contains illegal
 characters for the context)
mei 13 15:58:34 Websdr start-websdr.sh[977381]: [2025-05-13 15:58:34] [error] Handshake ended with HTTP error: 400

There is no reason to set the buffer-count so low, I saw tests that sends far bigger amounts.

Fingers crossed.

If you run 'watch service websdr status' you see the status every 2 seconds, very handy to see it change and read the errors constantly.

Phil - NY4Q · May 13, 2025, 05:06 PM

I'm glad you posted this Bas. I had changed my setting to 100000 a while back, but missed the 2nd setting.

Thanks again!

Bas ON5HB · May 13, 2025, 05:29 PM

Well errors started....but no crash....same as yours....

Code Select

mei 13 16:05:30 Websdr start-websdr.sh[989097]: Using OpenCL platform: Intel(R) OpenCL Graphics
mei 13 16:05:30 Websdr start-websdr.sh[989097]: Using OpenCL device: Intel(R) HD Graphics 630
mei 13 16:05:30 Websdr start-websdr.sh[989097]: Markers updated.
mei 13 16:05:32 Websdr start-websdr.sh[989095]: Attenuation: 0
mei 13 16:05:32 Websdr start-websdr.sh[989095]: Gain: 40
mei 13 16:05:33 Websdr start-websdr.sh[989097]: Waterfall is sent every 1 FFTs
mei 13 17:23:52 Websdr start-websdr.sh[989097]: [2025-05-13 17:23:52] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 13 17:23:52 Websdr start-websdr.sh[989097]: [2025-05-13 17:23:52] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 13 17:23:52 Websdr start-websdr.sh[989097]: [2025-05-13 17:23:52] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 13 17:23:52 Websdr start-websdr.sh[989097]: [2025-05-13 17:23:52] [error] handle_read_frame error: websocketpp.transport:7 (End of File)

Pretty sure the buffer-count is the problem....maybe it should be even higher, upto 500K.

We should also change the second audio slow message to waterfall, so we know the difference....

Phil - NY4Q · May 14, 2025, 02:45 PM

So I noticed this error this morning.

Code Select

May 14 10:39:37 vhfsdr start-websdr.sh[1211824]: [2025-05-14 10:39:37] [error] handle_read_handshake error: websocketpp.transport:7 (End of File)
May 14 10:39:37 vhfsdr start-websdr.sh[1211824]: [2025-05-14 10:39:37] [fail] WebSocket Connection [::ffff:202.112.47.208]:51992 - "" - 0 websocketpp.transport:7 End of File

So then I looked in ufw.log for more IPs in that subnet.

Code Select

May 11 14:33:55 vhfsdr kernel: [349721.027333] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.47.34 DST=168.8.96.79 LEN=52 TOS=0x10 PREC=0x20 TTL=36 ID=65436 PROTO=TCP SPT=52517 DPT=11434 WINDOW=65535 RES=0x00 SYN URGP=0
May 11 14:40:02 vhfsdr kernel: [350087.837507] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=1.202.112.142 DST=168.8.96.79 LEN=44 TOS=0x10 PREC=0x20 TTL=236 ID=55390 PROTO=TCP SPT=12062 DPT=55553 WINDOW=1024 RES=0x00 SYN URGP=0
May 11 17:45:14 vhfsdr kernel: [361199.791888] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.47.34 DST=168.8.96.79 LEN=52 TOS=0x10 PREC=0x20 TTL=36 ID=38305 PROTO=TCP SPT=53293 DPT=8080 WINDOW=65535 RES=0x00 SYN URGP=0
May 13 06:40:49 vhfsdr kernel: [494133.708235] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.47.54 DST=168.8.96.79 LEN=40 TOS=0x10 PREC=0x20 TTL=227 ID=54321 PROTO=TCP SPT=58983 DPT=11434 WINDOW=65535 RES=0x00 SYN URGP=0
May 13 13:53:22 vhfsdr kernel: [520086.554215] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.237.201 DST=168.8.96.79 LEN=70 TOS=0x18 PREC=0x20 TTL=2 ID=43016 DF PROTO=UDP SPT=9119 DPT=53 LEN=30
May 13 13:53:34 vhfsdr kernel: [520098.539701] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.237.201 DST=168.8.96.79 LEN=70 TOS=0x18 PREC=0x20 TTL=1 ID=43016 DF PROTO=UDP SPT=9119 DPT=53 LEN=29
May 14 06:07:32 vhfsdr kernel: [578536.362645] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.47.54 DST=168.8.96.79 LEN=40 TOS=0x10 PREC=0x20 TTL=227 ID=54321 PROTO=TCP SPT=58055 DPT=11434 WINDOW=65535 RES=0x00 SYN URGP=0
May 14 09:41:32 vhfsdr kernel: [591376.194423] [UFW BLOCK] IN=enp2s0 OUT= MAC=f4:39:09:0e:cc:3b:f8:6b:d9:3c:78:02:08:00 SRC=202.112.237.201 DST=168.8.96.79 LEN=44 TOS=0x10 PREC=0x20 TTL=100 ID=0 PROTO=TCP SPT=23456 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0

whois on that IP

Code Select

WhatIsMyIP.com Whois Lookup Report for: 202.112.47.208 
% [whois.apnic.net]
% Whois data copyright terms    http://www.apnic.net/db/dbcopyright.html

% Information related to '202.112.0.0 - 202.113.255.255'

% Abuse contact for '202.112.0.0 - 202.113.255.255' is 'abuse@cernet.edu.cn'

inetnum:        202.112.0.0 - 202.113.255.255
netname:        BJR-CERNET
descr:          China Education and Research Network
descr:          Beijing Regional Network
country:        CN
org:            ORG-CEAR1-AP
admin-c:        CER-AP
tech-c:         CER-AP
abuse-c:        AC1685-AP
status:         ALLOCATED PORTABLE
remarks:        origin AS4538
mnt-by:         APNIC-HM
mnt-lower:      MAINT-CERNET-AP
mnt-routes:     MAINT-CERNET-AP
mnt-irt:        IRT-CERNET-AP
last-modified:  2020-05-28T02:35:51Z
source:         APNIC

irt:            IRT-CERNET-AP
address:        Network Research Center,
address:        Main Bldg, Tsinghua Univ
address:        Beijing 100084, China
phone:          +86-10-62784301
fax-no:         +86-10-62785933
e-mail:         abuse@cernet.edu.cn
abuse-mailbox:  abuse@cernet.edu.cn
admin-c:        CER-AP
tech-c:         CER-AP
auth:           # Filtered
remarks:        timezone GMT+8
remarks:        http://www.ccert.edu.cn
remarks:        abuse@cernet.edu.cn was validated on 2025-01-22
mnt-by:         MAINT-CERNET-AP
last-modified:  2025-01-22T13:47:43Z
source:         APNIC

organisation:   ORG-CEAR1-AP
org-name:       China Education and Research Network (CERNET)
country:        CN
address:        FIT 3-220,
address:        Tsinghua University
phone:          +86-10-62785983
fax-no:         +86-10-62785933
e-mail:         hostmaster@cernic.net
mnt-ref:        APNIC-HM
mnt-by:         APNIC-HM
last-modified:  2018-09-13T12:56:34Z
source:         APNIC

role:           ABUSE CERNETAP
country:        ZZ
address:        Network Research Center,
address:        Main Bldg, Tsinghua Univ
address:        Beijing 100084, China
phone:          +86-10-62784301
e-mail:         abuse@cernet.edu.cn
admin-c:        CER-AP
tech-c:         CER-AP
nic-hdl:        AC1685-AP
remarks:        Generated from irt object IRT-CERNET-AP
remarks:        abuse@cernet.edu.cn was validated on 2025-01-22
abuse-mailbox:  abuse@cernet.edu.cn
mnt-by:         APNIC-ABUSE
last-modified:  2025-01-22T16:25:45Z
source:         APNIC

role:           CERNET Helpdesk
address:        CERNET Center
address:        Beijing 100084, China
country:        CN
phone:          +86-10-6278-4049
fax-no:         +86-10-6278-5933
e-mail:         helpdesk@cernet.edu.cn
remarks:        abuse@cernet.edu.cn
admin-c:        XL1-CN
tech-c:         SZ2-AP
nic-hdl:        CER-AP
mnt-by:         MAINT-CERNET-AP
last-modified:  2020-09-03T09:14:12Z
source:         APNIC

% This query was served by the APNIC Whois Service version 1.88.34 (WHOIS-US2)



Tool URL: https://www.whatismyip.com/ip-whois-lookup/

So it looks like nasty folk are trying to run more injections and when they hit port 80, then sometimes it causes that error and once in a while it may cause the system to crash.

Opinions?

Bas ON5HB · May 15, 2025, 03:00 PM

Have a look:

https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands

Nothing special, it's auto-blocking.

I would not worry about it. else use IPtables to block the entire range if they come often.

However, it could be legit but blocked for no reason.

I do not like UFW and it's not active on my machines

Bas ON5HB · May 15, 2025, 04:01 PM

Ok....I found the problem of the slow-errors!!!

Code Select

mei 15 15:57:07 Websdr start-websdr.sh[3484581]: Dropping Waterfall due to buffering slow client
mei 15 15:57:07 Websdr start-websdr.sh[3484581]: Dropping Waterfall due to buffering slow client
mei 15 15:57:07 Websdr start-websdr.sh[3484581]: Dropping Waterfall due to buffering slow client

It's the waterfall and not audio. I upped the waterfallsize (toml) from 2048 to 8192 and for testing 16384.
When you do this, the error will come very quickly when more users enter the websdr.

Ergo, I believe the waterfall buffer should be 1 Mega and the waterfallsize not much bigger then 1024 or 2048 as it costs a lot of bandwidth.
2048 is very high already....anything above it just a waste of traffic.

Going to test same size (to force it fast) and set only waterfall-buffer to 1M and reduce audio to 100K...

Update: the differentce between 16384 and 1024 is at full-range 2Mbit versus 180Kbit per user. Even 2048 is far too much.
My buffer-sizes are now audio 100K / waterfall 1M .... waterfall in toml is now 1024.....fingers crossed.

Update2: Waterfall 1M was still to little....set to 50M now and 100K audio....testing again. As the waterfall is 30MHz wide, hopefully 50M is enough.

Bas ON5HB · May 16, 2025, 02:55 PM

YES YES!!!!

No errors of slow stuff....

Code Select

root@Websdr:~# service websdr status
● websdr.service - PhantomSDRPlus WebSDR
     Loaded: loaded (/lib/systemd/system/websdr.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2025-05-15 16:18:58 CEST; 22h ago
    Process: 3509452 ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh (code=exited, status=0/SUCCESS)
   Main PID: 3509452 (code=exited, status=0/SUCCESS)
      Tasks: 11 (limit: 18787)
     Memory: 668.6M
        CPU: 16h 56min 13.024s
     CGroup: /system.slice/websdr.service
             ├─3509453 rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o -
             └─3509455 build/spectrumserver --config config-rx888mk2.toml

mei 16 14:00:11 Websdr start-websdr.sh[3509455]: [2025-05-16 14:00:11] [info] asio async_read_at_least error: system:104 (Connection reset by peer)
mei 16 14:00:11 Websdr start-websdr.sh[3509455]: [2025-05-16 14:00:11] [error] handle_read_frame error: websocketpp.transport:2 (Underlying Transport Error)
mei 16 14:00:11 Websdr start-websdr.sh[3509455]: [2025-05-16 14:00:11] [info] asio async_read_at_least error: system:104 (Connection reset by peer)
mei 16 14:00:11 Websdr start-websdr.sh[3509455]: [2025-05-16 14:00:11] [error] handle_read_frame error: websocketpp.transport:2 (Underlying Transport Error)
mei 16 14:01:22 Websdr start-websdr.sh[3509455]: [2025-05-16 14:01:22] [error] handle_read_handshake error: websocketpp.transport:7 (End of File)
mei 16 14:01:33 Websdr start-websdr.sh[3509455]: [2025-05-16 14:01:33] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 16 14:15:12 Websdr start-websdr.sh[3509455]: [2025-05-16 14:15:12] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 16 14:15:12 Websdr start-websdr.sh[3509455]: [2025-05-16 14:15:12] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 16 14:16:05 Websdr start-websdr.sh[3509455]: [2025-05-16 14:16:05] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
mei 16 14:35:59 Websdr start-websdr.sh[3509455]: [2025-05-16 14:35:59] [error] handle_read_handshake error: websocketpp.transport:7 (End of File)
root@Websdr:~#

Looks like it's fixed with those numbers

Bas ON5HB · May 18, 2025, 03:44 PM

Too soon, it's there again....

Code Select

mei 18 15:07:13 Websdr start-websdr.sh[3509455]: Dropping Audio due to buffering slow client
mei 18 15:07:13 Websdr start-websdr.sh[3509455]: Dropping Waterfall due to buffering slow client
mei 18 15:07:13 Websdr start-websdr.sh[3509455]: Dropping Audio due to buffering slow client
mei 18 15:07:13 Websdr start-websdr.sh[3509455]: Dropping Waterfall due to buffering slow client
mei 18 15:07:13 Websdr start-websdr.sh[3509455]: Dropping Audio due to buffering slow client

As I do not see crashes. I do believe the counter of the buffer is posting a useless message.

I'm starting to believe we do not need to count the buffer ammount at all.

https://websockets.readthedocs.io/en/stable/topics/memory.html

The document states that memory can be used, but the numbers won't be very high.

So I removed these lines for now, both cases:

Code Select

//        if (con->get_buffered_amount() > 100000) {
//            printf("Dropping Audio due to buffering slow client\n");
//            continue;
//        }

Let's see what happens

Phil - NY4Q · May 18, 2025, 07:17 PM

Well with your previous changes, I no longer see any 'slow client' messages.

I'm looking forward to your results. I have not had any clients mention buffering or such in a while, but they might have just stopped complaining.

Phil

Emmanuel SV1BTL · May 19, 2025, 07:28 AM

Wll, I am not sure what I have to do, after all these tests. Can you please Bas describe all the steps to make the necessary changes?

Bas ON5HB · May 19, 2025, 03:33 PM

Simple, goto the backend, websocket.cpp and change this:

Code Select

//        if (con->get_buffered_amount() > 100000) {
//            printf("Dropping Audio due to buffering slow client\n");
//            continue;
//        }

And a second time:

Code Select

//            if (con->get_buffered_amount() > 50000000) {
//                printf("Dropping Waterfall due to buffering slow client\n");
//                continue;
//            }

Recompile the backend and you are done. If you din't change anything, both time can be about audio, there are just 2 parts with these lines.