In case you are wondering what happened to my channel last night...
You were probably bombarded with 2 weeks worth of my messages and I also received a lot of messages I didn't realize I was missing.
For the past 2 weeks, my worker queue has been clogged up and most of my messages weren't going anywhere. A few people got them. There were some replies to some of my test messages and real messages, but when I looked my channel's clone, there was nothing new after about 2 weeks ago. When I looked at another channel on my server, there were also no new messages from this channel.
If you follow the Hubzilla Support channel, you've seen me flailing around the past couple of days and eventually what fixed it.
A couple days after Hurricane Helene passed, we had a long power outage. As usual, I had my server running on UPS at the start of it. It claimed to still have a couple hours left, so I went to walk the dog for an hour and it ran out just before I got back. My best guess is that the Hubzilla cron job had just started when the server lost power. I have no proof that this is how this started, but it fits the timing and the end result is the same regardless of the cause.
I barely understand the processes I'm about to describe. If someone who understands them better would correct anything i get wrong, I would appreciate it.
A Cron task was added to the workerq table, but was never processed. Once power was restored and things began running again, the cron job would run every 10 minutes, attempt to add a Cron task to the queue only to have it rejected because there was already one there. I'm assuming that this would normally mean that the queue worker was already working on the Cron task and we don't want to start another. Except, in this case, there wasn't one running. So, for two weeks, the periodic cron job was ineffective, leaving tasks undone. My workerq table had just over 1000 records when it started working again.
By adding additional log messages, I was able to determine that the code which takes items from the queue and processes them was never getting run. I worked back from there to find that when the cron job ran, it was bailing out when trying to add a Cron task because the task already existed. At this point, I figured I had nothing to lose by deleting the Cron record in the workerq table, so I did. The floodgates opened! From the 1000 entries I had at that point, I saw my queue climb past 4500 before it started coming back down. tasks about new posts were being turned into delivery tasks to all of you. Other tasks delivered things to me. It took hours to work through the queue and, when I checked this morning, my queue was empty and I had over 100 notifications for new posts and replies.
