Coredumps and crash.

[Bas ON5HB]

Strange behavior during the last month. RX-888 USB disconnections, but the server is still running!
The only new condition is the heat - 38 and even 42 °C
To be honest, I am tired....

No idea what you mean.

But we finally worked out the problem with the long-jmps, now testing.

The problems seem to be in buffering too much causing TCP-stack-problems.

In websocket.cpp you can change 2 lines, as we did for testing:

Code Select Expand

if (con->get_buffered_amount() > 100) {
            printf("Dropping Audio due to buffering slow client\n");
            continue;
        }

Code Select Expand

if (con->get_buffered_amount() > 100) {
              printf("Dropping Waterfall due to buffering slow client\n");
              continue;
           }
Could be your lines are without the message-printing, doesn't matter....change the value to 100.

We hope this ends the crashes for good, it seems the values where way too high. The code is pretty hard to read and messages unclear.

TCP in real-time should not buffer a lot as it fills the backlog for nothing. This is TCP-buffering, not the same as at the client side buffering.

Maybe 100 is still to high. Restarts by systemd will let us know....I hope.

[Bas ON5HB]

Next attempt....maybe the FIFO isn't working properly....

So I made 1 script and 1 systemd

start-websdr.sh

Code Select Expand

#!/bin/bash
cd /opt/PhantomSDR-Plus/

killall -s9 spectrumserver
killall -s9 rx888_stream

## Files to load
#FIFO=fifo.fifo
TOML=config-rx888mk2.toml

#[ ! -e "$FIFO" ] && mkfifo $FIFO

rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 --pga -d -r -g 50 -a 0 -m low -o - | build/spectrumserver --config $TOML

#exit
websdr.service

Code Select Expand

[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target

[Service]
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=exec
Restart=on-failure
RestartSec=5

[Install]
Alias=websdr.service
WantedBy=multi-user.target
Let's see how this works.

[Phil - NY4Q]

Strange behavior during the last month. RX-888 USB disconnections, but the server is still running!
The only new condition is the heat - 38 and even 42 °C
To be honest, I am tired....

My receiver never (rarely) is the culprit. It is always spectrumserver. BUT, after Bas' latest suggestions, last check its at 23 hours with no systemd restart.

[Bas ON5HB]

I updated the buffers from 100 to 500, seems 100 is too low.

And in spectrumserver.cpp I changed the backlog size:

Code Select Expand

m_server.set_listen_backlog(16384); //Backlog upped from 8192 to 16384 for testing - Bas.
Maybe it's the backlog that is too small.

Test is running 4 hours now....so far no issues.

[Phil - NY4Q]

After Bas' latest suggestions, I am at 27 hours with no crashes.

Phil

[9a7aof]

After Bas' latest suggestions, I am at two days and 11 hours with no crashes.

I noticed that spectrumserver was slowly taking up more and more memory,
from 231 to 253 in two days and a few hours (i have 8Gb RAM on this computer).

And then, I change receiver band, only one user, crash. After 5 sec server works.
I edit spectrumserver.cpp, compile, and I back to one startup script (as Bas sugested).

Now server normal work.
Wait for next crash...

Darko, 9a7aof

[Bas ON5HB]

19 hours and no crash, fingers crossed

Update: Active: active (running) since Fri 2025-07-11 16:39:00 CEST; 24h ago

Fingers crossed....

[Phil - NY4Q]

root@websdr:/# systemctl status websdr.service
● websdr.service - PhantomSDRPlus WebSDR
     Loaded: loaded (/lib/systemd/system/websdr.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2025-07-10 18:45:13 UTC; 2 days ago
   Main PID: 780 (start-websdr.sh)
      Tasks: 15 (limit: 18956)
     Memory: 581.8M
        CPU: 22h 35min 48.507s
     CGroup: /system.slice/websdr.service
             ├─780 /bin/bash /opt/PhantomSDR-Plus/start-websdr.sh
             └─911 build/spectrumserver --config config-rx888mk2.toml

[Bas ON5HB]

And we crashed again with longjump...

Pfff....back 2 systemd files as only 1 doesn't restart the spectrum.

1 day and 14 hours....

And again....installed Curl 8.15-DEV and purged Ubuntu Curl, then recompiled spectrum, let's see what happens now.

Also changed compiling to -O1 for compiler options. Let's see what happens. Uses 2% more CPU now.

[Phil - NY4Q]

I have had another crash as of an hour ago.

Code Select Expand

Jul 13 13:38:59 websdr start-websdr.sh[911]: Dropping Waterfall due tterminate called after throwing an instance of 'std::bad_alloc'
Jul 13 13:38:59 websdr start-websdr.sh[911]:   what():  std::bad_alloc
Jul 13 13:39:00 websdr start-websdr.sh[780]: /opt/PhantomSDR-Plus/start-websdr.sh: line 23:   911 Aborted                 (core dumped) build/spectrumserver --config $TOML < $FIFO

I also noticed this.

Code Select Expand

Jul 13 14:06:43 websdr start-websdr.sh[3478320]: build/spectrumserver: /usr/local/lib/libcurl.so.4: no version information available (required by build/spectrumserver)

I think this was caused by the removal of Ubuntu's curl and the install of the latest. I fixed this by deleting the old pointer to library and created a new pointer to the updated library.

Code Select Expand

sudo rm /usr/local/lib/libcurl.so.4

Then make a new symbolic link pointing to the new library.

Code Select Expand

sudo ln -s /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0 /usr/local/lib/libcurl.so.4

Then I recompiled spectrumserver and away we go.

[Bas ON5HB]

The curl from the github has 4.8.0 libs, not 4.7.0

So I removed 4.7.0 and replaced it with 4.8.0 from the github.

And I get the same message too, exactly the same.

[Phil - NY4Q]

The curl from the github has 4.8.0 libs, not 4.7.0

So I removed 4.7.0 and replaced it with 4.8.0 from the github.

And I get the same message too, exactly the same.

Well I tried to point it to 4.8.0, but it gave the same message, so I moved it to 4.7.0 and it was ok then. SO, there is still some of the Ubuntu stuff on there or something. Maybe I need to do apt purge curl...?

[Bas ON5HB]

Purge will remove curl, else it leaves stuff behind.
However, after my purge the 4.7.0 lib was still there.

Mine also says: build/spectrumserver: /usr/local/lib/libcurl.so.4: no version information available (required by build/spectrumserver)

It doesn't matter as it runs

Doesn't matter, it could be because it's not a release but dev version.

So I ran meson --wipe build and compiled again....now it showed: Run-time dependency libcurl found: YES 8.15.0-DEV

And the message about the version during start is gone.

This one is left behind:

/usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0

I deleted those and copied them from /usr/local/lib, after wipe they are found as new version.

[Bas ON5HB]

It restarted again, but I see no reason why it did....so I think systemd is the cause.

So I changed:

Code Select Expand

Requires=network-online.target
Into:

Code Select Expand

After=network-online.target
I did notice before that NIC's tend to reload from time to time. 
So by not requiring it, it doesn't have stop and start.

And we are off for hours again

[Bas ON5HB]

Well removed Curl again and installed the original Ubuntu one. Makes no difference.

Then I noticed that the orignal Phantom has a different Glaze! We seem to use 2.4.0, but the original uses 2.4.4
So I installed it by hand:

Code Select Expand

git clone https://github.com/stephenberry/glaze.git -b v2.4.4
mkdir build / cd build / cmake .. / make / make install  

Then wiped the Phantom build dir and compiled again, also frontend.

Then I noticed something, the TOML doesn't accept http://websdr.heppen.be anymore and comlains about resolving, so I removed http:// in front and the error is gone.

Weird. Of well, let's see how it holds. Hopefully Steven is ready soon with 2.0 and resolved these bugs.