Release: CBC News App (RSS Reader)

Basic and Machine Language

Moderator: Moderators

craftsman1234
Vic 20 Dabbler
Posts: 87
Joined: Thu Aug 29, 2013 3:14 pm

Re: Release: CBC News App (RSS Reader)

Post by craftsman1234 »

Bobbi wrote:I would love to get to the bottom of the issue with the random lockups. It is interesting to hear that it occurs with a device other than the WiModem. It seems that when we request a URL with a longer body (such as New York Times RSS feeds) then sometimes the body is truncated in the middle. I have a simple test program (without the XML parsing) which shows the same behaviour.

I do plan to make it so that users can add their own feeds eventually.

Glad you enjoy the program. I like to leave it running all day on my TV in the living room :)
I like having it running in the background as well.

I have not been able to get it to run at 2400 baud with my set up errors with stock settings and nothing with the RS232 patch.

I've also tested the 64 version but was not able to get it to work. This was with a Strikelink setup at 1200 or 2400 baud. I could see the original HTML come across but the feed would never start up. Like I said though not a WiModem set up.
Commodore Pet 4032, Vic-20, 2 C64, 2 SX64, PAL 64c, 128DCR. Commodore Since 1980
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

1200 is as fast as the kernal routines will go on either VIC-20 or C64.

On the C64, if the commodoreserver.com drivers are loaded, it is possible to do 2400 baud also. In principal, 38.4Kbaud is also possible with the commodoreserver.com drivers (but I think the RSS Reader code would need to be modified to use hardware flow control to pause the modem when it is unable to keep up!)

The 2400 baud option never works on VIC-20 and only works on C64 if a third-party RS-232 driver is already loaded.

I have made a small change for the next version I release - on VIC-20 2400 baud is not offered. On C64 it is offered but I added some text to make it clear that the commodoreserver.com driver is needed.
craftsman1234
Vic 20 Dabbler
Posts: 87
Joined: Thu Aug 29, 2013 3:14 pm

Re: Release: CBC News App (RSS Reader)

Post by craftsman1234 »

Bobbi wrote:1200 is as fast as the kernal routines will go on either VIC-20 or C64.

On the C64, if the commodoreserver.com drivers are loaded, it is possible to do 2400 baud also. In principal, 38.4Kbaud is also possible with the commodoreserver.com drivers (but I think the RSS Reader code would need to be modified to use hardware flow control to pause the modem when it is unable to keep up!)

The 2400 baud option never works on VIC-20 and only works on C64 if a third-party RS-232 driver is already loaded.

I have made a small change for the next version I release - on VIC-20 2400 baud is not offered. On C64 it is offered but I added some text to make it clear that the commodoreserver.com driver is needed.

I'm not sure how important speed really is for the amount of text being sent. 1200 is working fine here. A suggestion for future updates to have a selection for the amount of time between each news post or space bar to go to the next if you read it faster then the timer.
Commodore Pet 4032, Vic-20, 2 C64, 2 SX64, PAL 64c, 128DCR. Commodore Since 1980
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

In v0.8 you can hit space or return to pause the output. A page-by-page mode would be quite nice too, I agree!

Right now I am working on adding support for a configuration file so that you guys can add your own feeds. I should have a beta release for testing pretty soon.
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

RSS Reader v0.9

Post by Bobbi »

Here is v0.9 of the RSS Reader.

The big improvement with this release is that the news feeds are no longer hard-coded. Now there is a config file (rss.conf) which allows you to configure your own feeds. See rss.readme for full details. The file is in ASCII, so you can edit it on a PC and just copy it to your SD card in the same folder as RSS20.PRG.

A .D64 image is also included in the ZIP file.

If anyone sets up their own custom feeds, please share them and I will include them in the next version of the RSS Reader (at least until we run out of function keys to assign them to!)

I feel a v1.0 release is in sight!!

EDIT: I am just testing this version with the New York Times feed and it seems the problem we had with lock-ups / incomplete downloads seems to be fixed. Not sure what I did to fix it frankly, but I am happy if it is indeed working now!
Attachments
rss-0.9.zip
(29.15 KiB) Downloaded 55 times
craftsman1234
Vic 20 Dabbler
Posts: 87
Joined: Thu Aug 29, 2013 3:14 pm

Re: Release: CBC News App (RSS Reader)

Post by craftsman1234 »

I've added a few feeds.

Main ones being NASA and CSDb

url=csdb.dk,CSDb
feed=/rss/latestforumposts.php/rss.xml,Latest Posts
feed=/rss/latestreleases.php/rss.xml,Latest Release
feed=/rss/latestadditions.php?type=release/rss.xml,Latest Addition
feed=/rss/scenenews.php/rss.xml,Scene News
feed=/rss/latestcomments.php/rss.xml,User Comments
feed=/rss/upcomingevents.php/rss.xml,Upcoming Event

url=www.nasa.gov,NASA
feed=/rss/dyn/breaking_news.rss/rss.xml,Breaking News
feed=/rss/dyn/educationnews.rss/rss.xml,Education News
feed=/rss/dyn/shuttle_station.rss/rss.xml,ISS News
feed=/rss/dyn/onthestation_rss.rss/rss.xml,On the Station
feed=/rss/dyn/solar_system.rss/rss.xml,Solar System
feed=/rss/dyn/earth.rss/rss.xml,Earth News
feed=/rss/dyn/mission_pages/kepler/news/kepler-newsandfeatures-RSS.rss/rss.xml,Kepler
feed=/rss/dyn/aeronautics.rss/rss.xml,Aeronautics

Tried to figure out Reuters with no luck. Well maybe just too much HTML in it

url=feeds.reuters.com,Reuters
feed=/Reuters/businessNews,Business
feed=/Reuters/MostRead,Most Read
feed=/Reuters/scienceNews,Science
feed=/Reuters/technologyNews,Technology
feed=/Reuters/topNews,Top News
feed=/Reuters/domesticNews,US News
feed=/Reuters/worldNews,World
feed=/Reuters/oddlyEnoughNews,Oddly

I still have the lock ups as before :( but the program is coming along. I wonder if there would be anyway to have the program remove html links from the feeds. Also if you can set the screen/border and text color at start-up it would help. I am running it from an sd2iec and the file program changes the screen black.
Commodore Pet 4032, Vic-20, 2 C64, 2 SX64, PAL 64c, 128DCR. Commodore Since 1980
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

Awesome, maybe I will add the CSDb and NASA feeds to the included config file! I also discovered how to add weather feeds for a location using The Weather Network.

Interesting that you are still experiencing lock-ups. Is this on VIC-20 or C64? If it is on VIC-20, are you using srowe's RS-232 wedge (SYS 40960)?

I have been running the program in the background on both systems over the last couple of days. On both VIC-20 and C64, reliability seems much improved compared with v0.8, but there are still occasional freeze-ups. Running on VIC-20, with srowe's wedge, I was able to continuously show feeds (CBC, NYT, BBC) in kiosk mode for several hours without any freeze ups. (This is using the yet-unreleased v0.91 which has an improved kiosk mode which works both per-provider as before or globally showing feeds from all providers.)

The code does not seem to be quite as reliable on C64 as VIC-20. Lock ups on C64 occur maybe every 20 minutes or so rather than every couple of hours. The fact that it the two platforms behave differently seems to suggest that the issue is with RS-232 kernal code. I don't really understand what I changed from v0.8 -> v0.9 (I did a big clean up and refactoring but functionality is much the same) that would affect the lock-ups. Up until that point I had thought it was happening within the WiModem, but that seems not to be that case.

There is one interesting thing that I have observed, and which I hope is a clue. If I turn on the flashing border by pressing 'B' I can see when bytes are being transferred from the modem. Occasionally it stops flashing for five or six seconds meaning no bytes are transferred, then spontaneously starts again. When it continues there does not appear to be a backlog of bytes in the RS-232 RX buffer to process (you can see that pattern in the border flashes when it occurs) so it seems that the RS-232 kernal NMI code stalled somehow, rather than the RSS Reader (which is the one emptying that buffer.) This makes me think there are still bugs in the kernal RS-232 routines, even with the wedge. Those bugs seem to be worse on C64 than VIC-20 with wedge.

I will do some more testing with C64 at 2400 baud using commodoreserver.com drivers. If that works reliably then we can conclude it is an issue with the kernal RS-232 routines. The search goes on!

I'll fix the screen and border colour at start up!

Right now the code filters out a lot of HTML, unicode and other stuff it can't deal with. Let me take a look at the Reuters feed and see what the issue is with that one.
craftsman1234
Vic 20 Dabbler
Posts: 87
Joined: Thu Aug 29, 2013 3:14 pm

Re: Release: CBC News App (RSS Reader)

Post by craftsman1234 »

Bobbi wrote:Awesome, maybe I will add the CSDb and NASA feeds to the included config file! I also discovered how to add weather feeds for a location using The Weather Network.

Interesting that you are still experiencing lock-ups. Is this on VIC-20 or C64? If it is on VIC-20, are you using srowe's RS-232 wedge (SYS 40960)?

I have been running the program in the background on both systems over the last couple of days. On both VIC-20 and C64, reliability seems much improved compared with v0.8, but there are still occasional freeze-ups. Running on VIC-20, with srowe's wedge, I was able to continuously show feeds (CBC, NYT, BBC) in kiosk mode for several hours without any freeze ups. (This is using the yet-unreleased v0.91 which has an improved kiosk mode which works both per-provider as before or globally showing feeds from all providers.)

The code does not seem to be quite as reliable on C64 as VIC-20. Lock ups on C64 occur maybe every 20 minutes or so rather than every couple of hours. The fact that it the two platforms behave differently seems to suggest that the issue is with RS-232 kernal code. I don't really understand what I changed from v0.8 -> v0.9 (I did a big clean up and refactoring but functionality is much the same) that would affect the lock-ups. Up until that point I had thought it was happening within the WiModem, but that seems not to be that case.

There is one interesting thing that I have observed, and which I hope is a clue. If I turn on the flashing border by pressing 'B' I can see when bytes are being transferred from the modem. Occasionally it stops flashing for five or six seconds meaning no bytes are transferred, then spontaneously starts again. When it continues there does not appear to be a backlog of bytes in the RS-232 RX buffer to process (you can see that pattern in the border flashes when it occurs) so it seems that the RS-232 kernal NMI code stalled somehow, rather than the RSS Reader (which is the one emptying that buffer.) This makes me think there are still bugs in the kernal RS-232 routines, even with the wedge. Those bugs seem to be worse on C64 than VIC-20 with wedge.

I will do some more testing with C64 at 2400 baud using commodoreserver.com drivers. If that works reliably then we can conclude it is an issue with the kernal RS-232 routines. The search goes on!

I'll fix the screen and border colour at start up!

Right now the code filters out a lot of HTML, unicode and other stuff it can't deal with. Let me take a look at the Reuters feed and see what the issue is with that one.
Thanks for the work on this. My set up is different instead of the wimodem (Vic-20/gglabs to MAX3232/PI3). The RS232 wedge does not seem to work for me on the Vic but I'll test it out again with version .9.

I have not been able to get it to work on the 64 at all but again it is not a wimodem. I'm trying it with a Striklink setup.

I've noticed that with my set-up (64) even connecting to BBS the first time I run the program (Strikterm included) it will not connect the first time. Like the modem needs reset or buffer cleared. After the first time I can then access a BBS. With the RSS reader there is no way to retry short of a reset and reloading the program on the 64 as the program will not allow you to go back to the main menu so you can not try a second time.
Commodore Pet 4032, Vic-20, 2 C64, 2 SX64, PAL 64c, 128DCR. Commodore Since 1980
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

I am playing with the new feeds. I see with some of the longer articles I am getting RS-232 RX overruns (lost characters). The VIC-20 display shows '[ST rx overrun]' when this occurs.

I am going to play around with the code to address this issue. I can use flow control (RTS) to stop the modem when displaying longer articles. I should probably also introduce some deliberate delay to scroll the article at human-readable speed when it is more than one screenful!

Maybe it was these Rx overruns that are causing you to experience freeze-ups? Do you still see frequent freeze-ups with CBC or New York Times feeds for example?

Thank you for your testing :)
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

It is good to know that the RSS Reader works with other user port Internet solutions :)
craftsman1234
Vic 20 Dabbler
Posts: 87
Joined: Thu Aug 29, 2013 3:14 pm

Re: Release: CBC News App (RSS Reader)

Post by craftsman1234 »

Bobbi wrote:It is good to know that the RSS Reader works with other user port Internet solutions :)
Yes it is great to see it works with my hack jobs...

I have seen the same overflow (ST rx) issue but is has never seemed to cause the freeze up. This is mainly in the Reuters feed and other newsgroups that I added. Not with NASA etc. Very large news listings with a lot of html seem to cause it.

I still see the freeze ups with CBC and NYT I check it further but it does not seem to go more than 2 times in Kiosk mode.
Commodore Pet 4032, Vic-20, 2 C64, 2 SX64, PAL 64c, 128DCR. Commodore Since 1980
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

I have been running the NASA feeds in kiosk mode repeatedly for the last hour or so without any issue. (While I was writing this is locked up after maybe an hour.)

In v0.9 kiosk mode only goes through the feeds once. I have fixed that for upcoming releases and it will iterate continuously.

Looking at the XML for the Reuters feed, there is masses of HTML included in the <description> tag. The RSS Reader is never really going to be able to deal with this in any useful way. I have some heuristic code to discard unicode characters (which is why some feeds do not have apostrophes and accented characters are omitted.) I also have some code to skip over (some) HTML tags. However it is never going to be able to deal with the Reuters feed without including a proper HTML parser.
craftsman1234
Vic 20 Dabbler
Posts: 87
Joined: Thu Aug 29, 2013 3:14 pm

Re: Release: CBC News App (RSS Reader)

Post by craftsman1234 »

Bobbi wrote:Looking at the XML for the Reuters feed, there is masses of HTML included in the <description> tag. The RSS Reader is never really going to be able to deal with this in any useful way. I have some heuristic code to discard unicode characters (which is why some feeds do not have apostrophes and accented characters are omitted.) I also have some code to skip over (some) HTML tags. However it is never going to be able to deal with the Reuters feed without including a proper HTML parser.
They don't seem to follow a limited amount of data. I not sure if this really is a true RSS feed as I tried it numerous different ways.
Commodore Pet 4032, Vic-20, 2 C64, 2 SX64, PAL 64c, 128DCR. Commodore Since 1980
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

Here is v0.91, a minor update to v0.90.

- Kiosk mode now loops forever once again. (I broke that in v0.9).
- Added 'K' option to provider menu which does kiosk mode through all providers and all feeds. There is still a per-provided 'K' option to do kiosk mode for that provider only.
- I use hardware flow control (RTS) to pause the modem while drawing the screen and unpauseit afterwards. This should prevent the RX buffer overruns we were seeing with longer articles.
- Bug fixes, including setting the screen and border colours properly on VIC-20.

I have just done a test run on VIC-20 for two hours or so at 1200 baud. It ran continuously showing CBC, NYT, BBC, pages in sequence until I unplugged it in order to test on C64. It is now running on C64 without a hitch so far. Maybe clearing RTS to pause the modem while drawing the screen makes things more reliable.

I added NASA, CSDb and a sample Weather Network entry in the rss.conf file.

Give it a try and see how it goes.
Attachments
rss-0.91.zip
(30.61 KiB) Downloaded 64 times
Bobbi
Vic 20 Afficionado
Posts: 355
Joined: Thu Oct 13, 2016 11:35 am
Location: Toronto
Occupation: Programmer

Re: Release: CBC News App (RSS Reader)

Post by Bobbi »

Right after I posted this it locked up on C64 (on a CBC feed).

Still, it seems pretty reliable on (NTSC) VIC-20 with WiModem now.
Post Reply