Coredumps and crash.

Bas ON5HB · Jun 22, 2025, 11:28 AM

Well I ran out of memory and it crashed with a core-dump.

It seemed to have happened after this message:

jun 22 00:11:56 Websdr start-websdr.sh[1824032]: terminate called after throwing an instance of 'std::bad_alloc'
jun 22 00:11:56 Websdr start-websdr.sh[1824032]: what(): std::bad_alloc

But at the same time systemd-coredump service started acting up and took a very long time to finish.

I'm not certain if it was caused by the core-dumping itself, but it looks like it.

So I decided to remove core-dumping....and here is shown how to do it.

https://linux-audit.com/software/understand-and-configure-core-dumps-work-on-linux/

and here:

https://www.cyberciti.biz/faq/linux-disable-core-dumps/

It seems not usefull other then for Linux developers.

Also added to /etc/sysctl.conf

Code Select

kernel.perf_cpu_time_max_percent=0
net.core.netdev_max_backlog=8192
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_keepalive_time=300
net.ipv4.tcp_keepalive_probes=4

To make TCP-connection not last too long, as they typical last 7200 seconds, kind of much for an websdr.
And stopped the kernel from throttling.

Bas ON5HB · Jul 01, 2025, 04:24 PM

And another one....this time plocate was running.
I do not need this tool.

So I removed it: apt purge plocate

It makes an index of all the files in the filesystem, and it was running when the errors started.

Maybe it has problems with the fifo-files?

So much useless junk in Ubuntu

Bas ON5HB · Jul 03, 2025, 05:49 PM

Had another again, this time the RX888STREAM, with longjumps and I couldn't start it again.

So I ran install.sh again, after it would start again.

Maybe I messed with the RX888 driver too much

Bas ON5HB · Jul 04, 2025, 02:32 PM

More progress, it seems the program is running out of memory....so I changed the systemd start-script:

Code Select

[Unit]
Description=PhantomSDRPlus WebSDR
Wants=network-online.target
After=network-online.target

[Service]
PIDFile=/run/websdr.pid
ExecStart=/bin/bash /opt/PhantomSDR-Plus/start-websdr.sh
ExecStop=/bin/bash /opt/PhantomSDR-Plus/stop-websdr.sh
Restart=always
RestartSec=5
Type=Exec
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
Alias=websdr.service

start-websdr.sh (stop is about the same, just killing stream and spectrum)

Code Select

#!/bin/bash
cd /opt/PhantomSDR-Plus/

### STOP CRASH Injections!!!!
##If no other rules apply, flushing is easier

iptables -D INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -D INPUT -m string --algo kmp --string "device.rsp" -j DROP

iptables -A INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -A INPUT -m string --algo kmp --string "device.rsp" -j DROP

## Files to load 
FIFO=fifo.fifo
TOML=config-rx888mk2.toml

[ ! -e "$FIFO" ] && mkfifo $FIFO

#Without PGA
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o - > >

sleep 2

build/spectrumserver --config $TOML < $FIFO &

#exit

Maybe it keeps running now

Bas ON5HB · Jul 04, 2025, 05:29 PM

Ok, that all didn't work.

So I took out some compiler options and let it run with -O0, no optimize....that costed a lot of CPU 68% normal to 97%

However it didn't longjump anymore.

So I removed in install.sh the --optimisze=3, just took it out.

And changed meson.build:

Code Select

project(
    'spectrumdistributor',
    'cpp',
    default_options : [
        'optimization=0',
        #'b_sanitize=address'
    ]
)

add_project_arguments('-march=native', language: 'cpp')
add_project_arguments('-ggdb3', language: 'cpp')
add_project_arguments(['-std=c++23','-Ofast','-Wall','-Wextra','-Wpedantic'], language: 'cpp')

I removed Mason optimize....many say it does nothing, or something....whatever.
And I changed the optimize to be done by the compiler directly.

At the same time checking the memory-usage over time via: 'watch service websdr status' and run 'top' to see if RES number keeps going up for spectrumserver. RX888 doesn't seem to change once started.

I found that longjumps are memory leaks or memory-allocation that is in trouble because of some string or so going out of bounds.
Whatever it means

Fingers crossed, I know '-O0' seems to be fine.....but code is SLOW...real slow

I found several articles that Meson has memory leaks, and I did notice while running memory-consumption keeps going up.

Current memory consumption is between 625~640MB.....it changes with number of users.

9a7aof · Jul 04, 2025, 07:11 PM

Hi Bas,

please fix the line:

Quoterx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o - > >

It has to be like this and then it works (for me):

Code Select

rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -a 0 -m low -g 40 -o - > $FIFO &
Best regards,
Darko, 9a7aof

Bas ON5HB · Jul 05, 2025, 09:03 AM

Doesn't matter still crashes with longjumps.
So I changed it again.....

systemd websdr.service :

Code Select

[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target

[Service]
PIDFile=/run/websdr.pid
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=simple
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=websdr.service

start-websdr.sh

Code Select

#!/bin/bash
cd /opt/PhantomSDR-Plus/

### STOP CRASH Injections!!!!
##If no other rules apply, flushing is easier

iptables -D INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -D INPUT -m string --algo kmp --string "device.rsp" -j DROP

iptables -A INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -A INPUT -m string --algo kmp --string "device.rsp" -j DROP

./rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -g 30 -m low -o - | ./build/spectrumserver --config config-rx888mk2.toml

exit

Let's hope this works, maybe it's systemd causing to restart spectrum on the same port, causing it....

Fingers crossed.

Bas ON5HB · Jul 05, 2025, 01:53 PM

Nope, not working either and again bad alloc stuff....

So I took receiver and spectrum apart as service....hopefully this works.

websdr.service

Code Select

[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target receiver.service
After=receiver.service

[Service]
PIDFile=/run/websdr.pid
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=exec
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=websdr.service

receiver.service

Code Select

[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target
Before=websdr.service

[Service]
PIDFile=/run/receiver.pid
ExecStart=/opt/PhantomSDR-Plus/start-receiver.sh
Type=exec
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=receiver.service

start-websdr.sh

Code Select

#!/bin/bash
cd /opt/PhantomSDR-Plus/

## Files to load 
FIFO=fifo.fifo
TOML=config-rx888mk2.toml

[ ! -e "$FIFO" ] && mkfifo $FIFO

build/spectrumserver --config $TOML < $FIFO

#exit

start-receiver.sh

Code Select

#!/bin/bash
cd /opt/PhantomSDR-Plus/

## Files to load 
FIFO=fifo.fifo
TOML=config-rx888mk2.toml

[ ! -e "$FIFO" ] && mkfifo $FIFO

#Without PGA
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -g 40 -a 0 -m low -o - > $FIFO 

#exit

And we are testing again....pfffff....

9a7aof · Jul 05, 2025, 05:39 PM

Hi Bas,

I testing new new version script for start phantom websdr

Quoteroot@Siemens:/home/websdr# service websdr status
● websdr.service - PhantomSDRPlus WebSDR
Loaded: loaded (/etc/systemd/system/websdr.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-07-05 17:07:08 CEST; 11min ago
Main PID: 694 (start-websdr.sh)
Tasks: 11 (limit: 9253)
Memory: 231.0M
CPU: 25min 28.976s
CGroup: /system.slice/websdr.service
├─694 /bin/bash /home/websdr/PhantomSDR-Plus/start-websdr.sh
└─697 build/spectrumserver --config config-rx888mk2.toml

Quoteroot@Siemens:/home/websdr# service receiver status
● receiver.service - PhantomSDRPlus WebSDR
Loaded: loaded (/etc/systemd/system/receiver.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-07-05 17:07:08 CEST; 31min ago
Main PID: 630 (start-receiver.)
Tasks: 4 (limit: 9253)
Memory: 7.8M
CPU: 3min 51.355s
CGroup: /system.slice/receiver.service
├─630 /bin/bash /home/websdr/PhantomSDR-Plus/start-receiver.sh
└─698 ./rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 44000000 -g 30 -m high -a 13 -d --pga -o -

srp 05 17:07:06 Siemens systemd[1]: Starting receiver.service - PhantomSDRPlus WebSDR...
srp 05 17:07:08 Siemens systemd[1]: Started receiver.service - PhantomSDRPlus WebSDR.
srp 05 17:07:13 Siemens start-receiver.sh[698]: Attenuation: 13
srp 05 17:07:13 Siemens start-receiver.sh[698]: Gain: 158

I run iptables from cron.

Best regards,
Darko,9a7aof

Bas ON5HB · Jul 06, 2025, 06:21 AM

It tried to restart, but failed.
Changed some more parameters, test some more, if they work I post them.

As for putting IP-tables in cron, that is not a good idea.
They only need to be inserted once after a reboot, else they keep growing to no purpose.

See here to make them stick: https://linuxconfig.org/how-to-make-iptables-rules-persistent-after-reboot-on-linux

I just did a quick blocking as I have no other rules.

9a7aof · Jul 06, 2025, 06:38 AM

Hi,

Quote@reboot /bin/bash -lc 'cd /home/websdr/PhantomSDR-Plus/ && ./ipt.sh'

Quoteroot@Siemens:/home/websdr# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- anywhere anywhere STRING match "%3C%" ALGO name kmp
DROP all -- anywhere anywhere STRING match "device.rsp" ALGO name kmp

I run iptables from cron only once, when restarting the linux computer.
Thanks!
Darko, 9a7aof

Bas ON5HB · Jul 06, 2025, 05:38 PM

Ok, I think I found the crashing, it has been running the entire day without longjumps....it's buffering of slow clients that causes this.

Edit the websocket.cpp

Code Select

        //if (con->get_buffered_amount() > 1000) {
        //    printf("Dropping Audio due to buffering slow client\n");
        //    continue;
        //}

        //Fixed version, no need to output - Bas ON5HB
        if (con->get_buffered_amount() > 500) { continue; }

And it's there a second time for the waterfall

Code Select

          // Fixed version,no need to output - Bas ON5HB
          if (con->get_buffered_amount() > 500) { continue; }

I removed the printing and adjusted the numbers.

We remmed those lines, but turns out websocket keeps buffering, some sort of bug and over time it crashes by keeping connections open.

start-websdr.sh script:

Code Select

#!/bin/bash
cd /opt/PhantomSDR-Plus/

### STOP CRASH Injections!!!!
##If no other rules apply, flushing is easier

iptables -D INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -D INPUT -m string --algo kmp --string "device.rsp" -j DROP

iptables -A INPUT -m string --algo kmp --string "%3C%" -j DROP
iptables -A INPUT -m string --algo kmp --string "device.rsp" -j DROP

## Files to load 
FIFO=fifo.fifo
TOML=config-rx888mk2.toml

[ ! -e "$FIFO" ] && mkfifo $FIFO

build/spectrumserver --config $TOML < $FIFO

#exit

start-receiver.sh script:

Code Select

#!/bin/bash
cd /opt/PhantomSDR-Plus/

## Files to load 
FIFO=fifo.fifo
TOML=config-rx888mk2.toml

[ ! -e "$FIFO" ] && mkfifo $FIFO

#Without PGA
rx888_stream/target/release/rx888_stream -f ./rx888_stream/SDDC_FX3.img -s 60000000 -g 45 -a 0 -m low -o - > $FIFO 

#exit

systemd receiver.service

Code Select

[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target receiver.service
Before=websdr.service
[Service]
PIDFile=/run/receiver.pid
ExecStart=/opt/PhantomSDR-Plus/start-receiver.sh
Type=exec
Restart=on-failure
RestartSec=5
[Install]
Alias=receiver.service
WantedBy=multi-user.target

Systemd websdr.service

Code Select

[Unit]
Description=PhantomSDRPlus WebSDR
Requires=network-online.target
After=receiver.service
[Service]
PIDFile=/run/websdr.pid
ExecStart=/opt/PhantomSDR-Plus/start-websdr.sh
Type=exec
Restart=on-failure
RestartSec=5
[Install]
Alias=websdr.service
WantedBy=multi-user.target

That seems to do the trick. We got mixedup because of these messages it was printing in the log, but at the same time people injected the system with nasty code and crashed the chatbox with weird characters.

It keeps running stable so far, not seen if it restarted, but I really don't care if it does. As long as it keeps online

Fingers crossed that this keeps it from crashing.

Emmanuel SV1BTL · Jul 07, 2025, 10:29 PM

Strange behavior during the last month. RX-888 USB disconnections, but the server is still running!
The only new condition is the heat - 38 and even 42 °C
To be honest, I am tired....

alanVK2ZIW · Jul 08, 2025, 02:08 PM

Just to let you know, I've fixed my cron script bug:
And you can see the RX-888 on the same antenna on port 9074
www.unixservice.com.au:9074 and port 9073 for the RSP1 and PhantomSDR

#!/bin/bash
PATH=/usr/local/bin:$PATH
cd /home/alanb/src/ny4q/PhantomSDR-Plus
[ `pgrep -c -u $USER miri_sdr` = 1 ] && \
[ `pgrep -c -u $USER spectrumserver` = 1 ] && exit 0

pkill -u $USER spectrumserver
pkill -u $USER miri_sdr

echo "" >> MiriErrors
data >> MiriErrors

sleep 2
miri_sdr -m 252 -f 6900000 -s 2000000 - 2>MiriErrors | ./build/spectrumserver --config config-miri.toml >> specMiri 2>&1 &

exit

9a7aof · Jul 08, 2025, 03:17 PM

Thanks Alan,

Sdr-list is now working stably, I'm including my server on the list again.

Best regards,
Darko, 9a7aof