Well I ran out of memory and it crashed with a core-dump.
It seemed to have happened after this message:
jun 22 00:11:56 Websdr start-websdr.sh[1824032]: terminate called after throwing an instance of 'std::bad_alloc'
jun 22 00:11:56 Websdr start-websdr.sh[1824032]: what(): std::bad_alloc
But at the same time systemd-coredump service started acting up and took a very long time to finish.
I'm not certain if it was caused by the core-dumping itself, but it looks like it.
So I decided to remove core-dumping....and here is shown how to do it.
https://linux-audit.com/software/understand-and-configure-core-dumps-work-on-linux/
and here:
https://www.cyberciti.biz/faq/linux-disable-core-dumps/
It seems not usefull other then for Linux developers.
Also added to /etc/sysctl.conf
kernel.perf_cpu_time_max_percent=0
net.core.netdev_max_backlog=8192
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_keepalive_time=300
net.ipv4.tcp_keepalive_probes=4
To make TCP-connection not last too long, as they typical last 7200 seconds, kind of much for an websdr.
And stopped the kernel from throttling.
And another one....this time plocate was running.
I do not need this tool.
So I removed it: apt purge plocate
It makes an index of all the files in the filesystem, and it was running when the errors started.
Maybe it has problems with the fifo-files?
So much useless junk in Ubuntu :'(
Had another again, this time the RX888STREAM, with longjumps and I couldn't start it again.
So I ran install.sh again, after it would start again.
Maybe I messed with the RX888 driver too much 8)
More progress, it seems the program is running out of memory....so I changed the systemd start-script:
[Unit]
Description=PhantomSDRPlus WebSDR
Wants=network-online.target
After=network-online.target
[Service]
PIDFile=/run/websdr.pid
ExecStart=/bin/bash /opt/PhantomSDR-Plus/start-websdr.sh
ExecStop=/bin/bash /opt/PhantomSDR-Plus/stop-websdr.sh
Restart=always
RestartSec=5
Type=Exec
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
Alias=websdr.service
start-websdr.sh (stop is about the same, just killing stream and spectrum)
#!/bin/bash
cd /opt/PhantomSDR-Plus/
### STOP CRASH Injections!!!!
##If no other rules apply, flushing is easier
iptables -D INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -D INPUT -m string --algo kmp --string "device.rsp" -j DROP
iptables -A INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -A INPUT -m string --algo kmp --string "device.rsp" -j DROP
## Files to load
FIFO=fifo.fifo
TOML=config-rx888mk2.toml
[ ! -e "$FIFO" ] && mkfifo $FIFO
#Without PGA
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o - > >
sleep 2
build/spectrumserver --config $TOML < $FIFO &
#exit
Maybe it keeps running now ;)
Ok, that all didn't work.
So I took out some compiler options and let it run with -O0, no optimize....that costed a lot of CPU 68% normal to 97% ;D
However it didn't longjump anymore.
So I removed in install.sh the --optimisze=3, just took it out.
And changed meson.build:
project(
'spectrumdistributor',
'cpp',
default_options : [
'optimization=0',
#'b_sanitize=address'
]
)
add_project_arguments('-march=native', language: 'cpp')
add_project_arguments('-ggdb3', language: 'cpp')
add_project_arguments(['-std=c++23','-Ofast','-Wall','-Wextra','-Wpedantic'], language: 'cpp')
I removed Mason optimize....many say it does nothing, or something....whatever.
And I changed the optimize to be done by the compiler directly.
At the same time checking the memory-usage over time via: 'watch service websdr status' and run 'top' to see if RES number keeps going up for spectrumserver. RX888 doesn't seem to change once started.
I found that longjumps are memory leaks or memory-allocation that is in trouble because of some string or so going out of bounds.
Whatever it means ;D
Fingers crossed, I know '-O0' seems to be fine.....but code is SLOW...real slow :o
I found several articles that Meson has memory leaks, and I did notice while running memory-consumption keeps going up.
Current memory consumption is between 625~640MB.....it changes with number of users.
Hi Bas,
please fix the line:
Quoterx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o - > >
It has to be like this and then it works (for me):
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o - > $FIFO &Best regards,
Darko, 9a7aof
Doesn't matter still crashes with longjumps.
So I changed it again..... :o
systemd websdr.service :
[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target
[Service]
PIDFile=/run/websdr.pid
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=simple
Restart=on-failure
[Install]
WantedBy=multi-user.target
Alias=websdr.service
start-websdr.sh
#!/bin/bash
cd /opt/PhantomSDR-Plus/
### STOP CRASH Injections!!!!
##If no other rules apply, flushing is easier
iptables -D INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -D INPUT -m string --algo kmp --string "device.rsp" -j DROP
iptables -A INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -A INPUT -m string --algo kmp --string "device.rsp" -j DROP
./rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -g 30 -m low -o - | ./build/spectrumserver --config config-rx888mk2.toml
exit
Let's hope this works, maybe it's systemd causing to restart spectrum on the same port, causing it....
Fingers crossed.
Nope, not working either and again bad alloc stuff....
So I took receiver and spectrum apart as service....hopefully this works.
websdr.service
[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target receiver.service
After=receiver.service
[Service]
PIDFile=/run/websdr.pid
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=exec
Restart=on-failure
[Install]
WantedBy=multi-user.target
Alias=websdr.service
receiver.service
[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target
Before=websdr.service
[Service]
PIDFile=/run/receiver.pid
ExecStart=/opt/PhantomSDR-Plus/start-receiver.sh
Type=exec
Restart=on-failure
[Install]
WantedBy=multi-user.target
Alias=receiver.service
start-websdr.sh
#!/bin/bash
cd /opt/PhantomSDR-Plus/
## Files to load
FIFO=fifo.fifo
TOML=config-rx888mk2.toml
[ ! -e "$FIFO" ] && mkfifo $FIFO
build/spectrumserver --config $TOML < $FIFO
#exit
start-receiver.sh
#!/bin/bash
cd /opt/PhantomSDR-Plus/
## Files to load
FIFO=fifo.fifo
TOML=config-rx888mk2.toml
[ ! -e "$FIFO" ] && mkfifo $FIFO
#Without PGA
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -g 40 -a 0 -m low -o - > $FIFO
#exit
And we are testing again....pfffff....
Hi Bas,
I testing new new version script for start phantom websdr :)
Quoteroot@Siemens:/home/websdr# service websdr status
● websdr.service - PhantomSDRPlus WebSDR
Loaded: loaded (/etc/systemd/system/websdr.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-07-05 17:07:08 CEST; 11min ago
Main PID: 694 (start-websdr.sh)
Tasks: 11 (limit: 9253)
Memory: 231.0M
CPU: 25min 28.976s
CGroup: /system.slice/websdr.service
├─694 /bin/bash /home/websdr/PhantomSDR-Plus/start-websdr.sh
└─697 build/spectrumserver --config config-rx888mk2.toml
Quoteroot@Siemens:/home/websdr# service receiver status
● receiver.service - PhantomSDRPlus WebSDR
Loaded: loaded (/etc/systemd/system/receiver.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-07-05 17:07:08 CEST; 31min ago
Main PID: 630 (start-receiver.)
Tasks: 4 (limit: 9253)
Memory: 7.8M
CPU: 3min 51.355s
CGroup: /system.slice/receiver.service
├─630 /bin/bash /home/websdr/PhantomSDR-Plus/start-receiver.sh
└─698 ./rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 44000000 -g 30 -m high -a 13 -d --pga -o -
srp 05 17:07:06 Siemens systemd[1]: Starting receiver.service - PhantomSDRPlus WebSDR...
srp 05 17:07:08 Siemens systemd[1]: Started receiver.service - PhantomSDRPlus WebSDR.
srp 05 17:07:13 Siemens start-receiver.sh[698]: Attenuation: 13
srp 05 17:07:13 Siemens start-receiver.sh[698]: Gain: 158
I run iptables from cron.
Best regards,
Darko,9a7aof
It tried to restart, but failed.
Changed some more parameters, test some more, if they work I post them.
As for putting IP-tables in cron, that is not a good idea.
They only need to be inserted once after a reboot, else they keep growing to no purpose.
See here to make them stick: https://linuxconfig.org/how-to-make-iptables-rules-persistent-after-reboot-on-linux
I just did a quick blocking as I have no other rules. ;)
Hi,
Quote@reboot /bin/bash -lc 'cd /home/websdr/PhantomSDR-Plus/ && ./ipt.sh'
Quoteroot@Siemens:/home/websdr# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- anywhere anywhere STRING match "%3C%" ALGO name kmp
DROP all -- anywhere anywhere STRING match "device.rsp" ALGO name kmp
I run iptables from cron only once, when restarting the linux computer.
Thanks!
Darko, 9a7aof
Ok, I think I found the crashing, it has been running the entire day without longjumps....it's buffering of slow clients that causes this.
Edit the websocket.cpp
//if (con->get_buffered_amount() > 1000) {
// printf("Dropping Audio due to buffering slow client\n");
// continue;
//}
//Fixed version, no need to output - Bas ON5HB
if (con->get_buffered_amount() > 500) { continue; }
And it's there a second time for the waterfall
// Fixed version,no need to output - Bas ON5HB
if (con->get_buffered_amount() > 500) { continue; }
I removed the printing and adjusted the numbers.
We remmed those lines, but turns out websocket keeps buffering, some sort of bug and over time it crashes by keeping connections open.
start-websdr.sh script:
#!/bin/bash
cd /opt/PhantomSDR-Plus/
### STOP CRASH Injections!!!!
##If no other rules apply, flushing is easier
iptables -D INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -D INPUT -m string --algo kmp --string "device.rsp" -j DROP
iptables -A INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -A INPUT -m string --algo kmp --string "device.rsp" -j DROP
## Files to load
FIFO=fifo.fifo
TOML=config-rx888mk2.toml
[ ! -e "$FIFO" ] && mkfifo $FIFO
build/spectrumserver --config $TOML < $FIFO
#exit
start-receiver.sh script:
#!/bin/bash
cd /opt/PhantomSDR-Plus/
## Files to load
FIFO=fifo.fifo
TOML=config-rx888mk2.toml
[ ! -e "$FIFO" ] && mkfifo $FIFO
#Without PGA
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -g 45 -a 0 -m low -o - > $FIFO
#exit
systemd receiver.service
[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target receiver.service
Before=websdr.service
[Service]
PIDFile=/run/receiver.pid
ExecStart=/opt/PhantomSDR-Plus/start-receiver.sh
Type=exec
Restart=on-failure
RestartSec=5
[Install]
Alias=receiver.service
WantedBy=multi-user.target
Systemd websdr.service
[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target
After=receiver.service
[Service]
PIDFile=/run/websdr.pid
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=exec
Restart=on-failure
RestartSec=5
[Install]
Alias=websdr.service
WantedBy=multi-user.target
That seems to do the trick. We got mixedup because of these messages it was printing in the log, but at the same time people injected the system with nasty code and crashed the chatbox with weird characters.
It keeps running stable so far, not seen if it restarted, but I really don't care if it does. As long as it keeps online ;D
Fingers crossed that this keeps it from crashing.
Strange behavior during the last month. RX-888 USB disconnections, but the server is still running!
The only new condition is the heat - 38 and even 42 °C
To be honest, I am tired....
Just to let you know, I've fixed my cron script bug:
And you can see the RX-888 on the same antenna on port 9074
www.unixservice.com.au:9074 and port 9073 for the RSP1 and PhantomSDR
#!/bin/bash
PATH=/usr/local/bin:$PATH
cd /home/alanb/src/ny4q/PhantomSDR-Plus
[ `pgrep -c -u $USER miri_sdr` = 1 ] && \
[ `pgrep -c -u $USER spectrumserver` = 1 ] && exit 0
pkill -u $USER spectrumserver
pkill -u $USER miri_sdr
echo "" >> MiriErrors
data >> MiriErrors
sleep 2
miri_sdr -m 252 -f 6900000 -s 2000000 - 2>MiriErrors | ./build/spectrumserver --config config-miri.toml >> specMiri 2>&1 &
exit
Thanks Alan,
Sdr-list is now working stably, I'm including my server on the list again.
Best regards,
Darko, 9a7aof
Quote from: Emmanuel SV1BTL on Jul 07, 2025, 10:29 PMStrange behavior during the last month. RX-888 USB disconnections, but the server is still running!
The only new condition is the heat - 38 and even 42 °C
To be honest, I am tired....
No idea what you mean.
But we finally worked out the problem with the long-jmps, now testing.
The problems seem to be in buffering too much causing TCP-stack-problems.
In websocket.cpp you can change 2 lines, as we did for testing:
if (con->get_buffered_amount() > 100) {
printf("Dropping Audio due to buffering slow client\n");
continue;
}if (con->get_buffered_amount() > 100) {
printf("Dropping Waterfall due to buffering slow client\n");
continue;
}Could be your lines are without the message-printing, doesn't matter....change the value to 100.
We hope this ends the crashes for good, it seems the values where way too high. The code is pretty hard to read and messages unclear.
TCP in real-time should not buffer a lot as it fills the backlog for nothing. This is TCP-buffering, not the same as at the client side buffering.
Maybe 100 is still to high. Restarts by systemd will let us know....I hope.
Next attempt....maybe the FIFO isn't working properly....
So I made 1 script and 1 systemd
start-websdr.sh
#!/bin/bash
cd /opt/PhantomSDR-Plus/
killall -s9 spectrumserver
killall -s9 rx888_stream
## Files to load
#FIFO=fifo.fifo
TOML=config-rx888mk2.toml
#[ ! -e "$FIFO" ] && mkfifo $FIFO
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 --pga -d -r -g 50 -a 0 -m low -o - | build/spectrumserver --config $TOML
#exit
websdr.service
[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target
[Service]
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=exec
Restart=on-failure
RestartSec=5
[Install]
Alias=websdr.service
WantedBy=multi-user.target
Let's see how this works.
Quote from: Emmanuel SV1BTL on Jul 07, 2025, 10:29 PMStrange behavior during the last month. RX-888 USB disconnections, but the server is still running!
The only new condition is the heat - 38 and even 42 °C
To be honest, I am tired....
My receiver never (rarely) is the culprit. It is always spectrumserver. BUT, after Bas' latest suggestions, last check its at 23 hours with no systemd restart.
I updated the buffers from 100 to 500, seems 100 is too low.
And in spectrumserver.cpp I changed the backlog size:
m_server.set_listen_backlog(16384); //Backlog upped from 8192 to 16384 for testing - Bas.
Maybe it's the backlog that is too small.
Test is running 4 hours now....so far no issues.
After Bas' latest suggestions, I am at 27 hours with no crashes.
Phil
After Bas' latest suggestions, I am at two days and 11 hours with no crashes.
I noticed that spectrumserver was slowly taking up more and more memory,
from 231 to 253 in two days and a few hours (i have 8Gb RAM on this computer).
And then, I change receiver band, only one user, crash. After 5 sec server works.
I edit spectrumserver.cpp, compile, and I back to one startup script (as Bas sugested).
Now server normal work.
Wait for next crash... :)
Darko, 9a7aof
19 hours and no crash, fingers crossed ;)
Update: Active: active (running) since Fri 2025-07-11 16:39:00 CEST; 24h ago
Fingers crossed....
root@websdr:/# systemctl status websdr.service
● websdr.service - PhantomSDRPlus WebSDR
Loaded: loaded (/lib/systemd/system/websdr.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2025-07-10 18:45:13 UTC; 2 days ago
Main PID: 780 (start-websdr.sh)
Tasks: 15 (limit: 18956)
Memory: 581.8M
CPU: 22h 35min 48.507s
CGroup: /system.slice/websdr.service
├─780 /bin/bash /opt/PhantomSDR-Plus/start-websdr.sh
└─911 build/spectrumserver --config config-rx888mk2.toml
And we crashed again with longjump...
Pfff....back 2 systemd files as only 1 doesn't restart the spectrum. :'(
1 day and 14 hours....
And again....installed Curl 8.15-DEV and purged Ubuntu Curl, then recompiled spectrum, let's see what happens now.
Also changed compiling to -O1 for compiler options. Let's see what happens. Uses 2% more CPU now.
I have had another crash as of an hour ago.
Jul 13 13:38:59 websdr start-websdr.sh[911]: Dropping Waterfall due tterminate called after throwing an instance of 'std::bad_alloc'
Jul 13 13:38:59 websdr start-websdr.sh[911]: what(): std::bad_alloc
Jul 13 13:39:00 websdr start-websdr.sh[780]: /opt/PhantomSDR-Plus/start-websdr.sh: line 23: 911 Aborted (core dumped) build/spectrumserver --config $TOML < $FIFO
I also noticed this.
Jul 13 14:06:43 websdr start-websdr.sh[3478320]: build/spectrumserver: /usr/local/lib/libcurl.so.4: no version information available (required by build/spectrumserver)
I think this was caused by the removal of Ubuntu's curl and the install of the latest. I fixed this by deleting the old pointer to library and created a new pointer to the updated library.
sudo rm /usr/local/lib/libcurl.so.4
Then make a new symbolic link pointing to the new library.
sudo ln -s /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0 /usr/local/lib/libcurl.so.4
Then I recompiled spectrumserver and away we go.
The curl from the github has 4.8.0 libs, not 4.7.0
So I removed 4.7.0 and replaced it with 4.8.0 from the github.
And I get the same message too, exactly the same.
Quote from: Bas ON5HB on Jul 13, 2025, 06:59 PMThe curl from the github has 4.8.0 libs, not 4.7.0
So I removed 4.7.0 and replaced it with 4.8.0 from the github.
And I get the same message too, exactly the same.
Well I tried to point it to 4.8.0, but it gave the same message, so I moved it to 4.7.0 and it was ok then. SO, there is still some of the Ubuntu stuff on there or something. Maybe I need to do apt purge curl...?
Purge will remove curl, else it leaves stuff behind.
However, after my purge the 4.7.0 lib was still there.
Mine also says: build/spectrumserver: /usr/local/lib/libcurl.so.4: no version information available (required by build/spectrumserver)
It doesn't matter as it runs ;D
Doesn't matter, it could be because it's not a release but dev version.
So I ran meson --wipe build and compiled again....now it showed: Run-time dependency libcurl found: YES 8.15.0-DEV
And the message about the version during start is gone.
This one is left behind:
/usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0
I deleted those and copied them from /usr/local/lib, after wipe they are found as new version.
It restarted again, but I see no reason why it did....so I think systemd is the cause.
So I changed:
Requires=network-online.target
Into:
After=network-online.target
I did notice before that NIC's tend to reload from time to time.
So by not requiring it, it doesn't have stop and start.
And we are off for hours again :o
Well removed Curl again and installed the original Ubuntu one. Makes no difference.
Then I noticed that the orignal Phantom has a different Glaze! We seem to use 2.4.0, but the original uses 2.4.4
So I installed it by hand:
git clone https://github.com/stephenberry/glaze.git -b v2.4.4
mkdir build / cd build / cmake .. / make / make install ;D
Then wiped the Phantom build dir and compiled again, also frontend.
Then I noticed something, the TOML doesn't accept http://websdr.heppen.be anymore and comlains about resolving, so I removed http:// in front and the error is gone.
Weird. Of well, let's see how it holds. Hopefully Steven is ready soon with 2.0 and resolved these bugs.
Hi!
PhantomSDR+ uptime 4 days:
Quoteroot@Siemens:/home/websdr# service websdr status
● websdr.service - PhantomSDRPlus WebSDR
Loaded: loaded (/etc/systemd/system/websdr.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-07-12 07:46:18 CEST; 4 days ago
Main PID: 636 (start-websdr.sh)
Tasks: 15 (limit: 9253)
Memory: 263.0M
CPU: 1w 3h 39min 1.151s
CGroup: /system.slice/websdr.service
├─636 /bin/bash /home/websdr/PhantomSDR-Plus/start-websdr.sh
├─694 rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 44000000 -g 30 -m high -a 13 -d --pga -o -
└─695 build/spectrumserver --config config-rx888mk2.toml
Quoteroot@Siemens
.:++++ooooosssoo:. ------------
.+o++::. `.:oos+. OS: LMDE 6 (faye) x86_64
:oo:.` -+oo: Host: LIFEBOOK S752 10601581625
`+o/` .::::::-. .++-` Kernel: 6.1.0-37-amd64
`/s/ .yyyyyyyyyyo: +o-` Uptime: 4 days, 2 mins
`so .ss ohyo` :s-: Packages: 3404 (dpkg)
`s/ .ss h m myy/ /s`` Shell: bash 5.2.15
`s: `oo s m Myy+-o:` Resolution: 1366x768
`oo :+sdoohyoydyso/. CPU: Intel i5-3320M (4) @ 3.300GHz
:o. .:////////++: GPU: Intel 3rd Gen Core processor Graphics Controller
`/++ -:::::- Memory: 489MiB / 7800MiB
Best Regards,
Darko, 9a7aof