revision3 videocast downloader perl script

[ieatpunk]

For the console junkies:

I attached a script I made to automate the downloading of my favorite rev3 feeds. I included instructions at the top of the script, but I will give a quick getting started tutorial.

This script uses wget to retrieve the video files from the web. Most distros have this installed by default. If you don't have it, get it from your distro's package repo.

You may also need a few perl modules, depending on whether or not you have previously downloaded them. To install the ones for this script, issue the following commands as root:

$ perl -MCPAN -e "install XML::RSS"
$ perl -MCPAN -e "install XML::Simple"
$ perl -MCPAN -e "install LWP::Simple"

1) Download this script to any directory, preferably one in your path so you can type rev3 in any directory to start the script.

2) Make the script executable.

$ chmod +x rev3

3) If you have already manually downloaded most of your favorite feeds up to this point, you will want to start the rev3 script with the --nodownload parameter, so that you do not download all of the videos you have already seen in the show's RSS feed.

$ rev3 --nodownload

4) If this is the first time running the script, you will be greeted with a configuration wizard. It will ask you what directory you would like to save videos in, how long between retries for files that could not be retrieved, and then a loop for adding feed URLs. Make sure to copy and paste the feed URL from the Subscription tab under your favorite shows. When finished adding feeds, leave the input blank and hit enter to end. The script will then begin downloading each RSS feed.

5) To add new feeds to the script, use the --add switch.

$ rev3 --add

You will be prompted for the feed's URL, type it and hit enter.

6) To remove feeds, use the --remove switch.

$ rev3 --remove

A list of current feeds will be displayed with a number next to each. Type that number at the prompt to delete the feed.

7) Once fully configured, just run this script daily to download any new shows from your feed list. Even cooler, make a crontab for this script to regularly check for new videos automatically. You can simply run the script with no parameters:

$ rev3

[basskozz]

It's like you read my mind: Help me automate Rev3 show downloads, please (http://revision3.com/forum/showthread.php?p=484186)
Thanks ieatpunk ;-)

Couple of questions:

How can I reconfigure?
- I messed up on the initial config (specifically on the download location for the videos) and I need to reconfigure, but I can't seem to figure out how.
- The script makes mention of a rev3.xml file, but I can't seem to locate it?
- When I issue ./rev3 --nodownload I get:Reading configuration... [success]

[http://revision3.com/diggnation/feed/xvid-large]
Skipping download of http://revision3.com/diggnation/clipshow2008...
Skipping download of http://revision3.com/diggnation/happy2k9...
Skipping download of http://revision3.com/diggnation/inLAagain...
Skipping download of http://revision3.com/diggnation/alexhouse...
Skipping download of http://revision3.com/diggnation/strangerscouch...
Skipping download of http://revision3.com/diggnation/hungover...
Skipping download of http://revision3.com/diggnation/bradsavedus...
Skipping download of http://revision3.com/diggnation/drkevin...
Skipping download of http://revision3.com/diggnation/halogram...
Skipping download of http://revision3.com/diggnation/2008-11-7awards...
Skipping download of http://revision3.com/diggnation/2008-10-31cfree...
Skipping download of http://revision3.com/diggnation/2008-10-24virginhq...
Skipping download of http://revision3.com/diggnation/2008-10-17fowa...
Skipping download of http://revision3.com/diggnation/2008-10-10hummer...
Skipping download of http://revision3.com/diggnation/2008-10-03jalbaminus...
Skipping download of http://revision3.com/diggnation/2008-09-25nojackblack2...
Skipping download of http://revision3.com/diggnation/2008-09-19michelob...
Skipping download of http://revision3.com/diggnation/2008-09-13hotels...
Skipping download of http://revision3.com/diggnation/2008-09-05pdoodles...
Skipping download of http://revision3.com/diggnation/2008-08-29nvisionsj...
Skipping download of http://revision3.com/diggnation/2008-08-21polepoll...
Skipping download of http://revision3.com/diggnation/2008-08-14dolby...
Skipping download of http://revision3.com/diggnation/2008-08-06beefu...
Skipping download of http://revision3.com/diggnation/2008-07-31cwalk...

Done.
but no way to reconfigure?
How many episodes of each feed does it download?
- i.e. the # most recent of each feed?

Is the number of most recent episodes to download configurable?
- For example, I would like the script to only download the 3 most recent episodes of each show/feed.
- It would also be nice if the script could delete old episodes, but I think I can figure out how to create a cronjob to do this manually if it's too much to add to this script.


Thanks again for the great work on the script,
-BassKozz

[ieatpunk]

It's like you read my mind: Help me automate Rev3 show downloads, please (http://revision3.com/forum/showthread.php?p=484186)
Thanks ieatpunk ;-)

Couple of questions:

How can I reconfigure?
- I messed up on the initial config (specifically on the download location for the videos) and I need to reconfigure, but I can't seem to figure out how.
- The script makes mention of a rev3.xml file, but I can't seem to locate it?
- When I issue ./rev3 --nodownload I get:Reading configuration... [success]

[http://revision3.com/diggnation/feed/xvid-large]
Skipping download of http://revision3.com/diggnation/clipshow2008...
Skipping download of http://revision3.com/diggnation/happy2k9...
Skipping download of http://revision3.com/diggnation/inLAagain...
Skipping download of http://revision3.com/diggnation/alexhouse...
Skipping download of http://revision3.com/diggnation/strangerscouch...
Skipping download of http://revision3.com/diggnation/hungover...
Skipping download of http://revision3.com/diggnation/bradsavedus...
Skipping download of http://revision3.com/diggnation/drkevin...
Skipping download of http://revision3.com/diggnation/halogram...
Skipping download of http://revision3.com/diggnation/2008-11-7awards...
Skipping download of http://revision3.com/diggnation/2008-10-31cfree...
Skipping download of http://revision3.com/diggnation/2008-10-24virginhq...
Skipping download of http://revision3.com/diggnation/2008-10-17fowa...
Skipping download of http://revision3.com/diggnation/2008-10-10hummer...
Skipping download of http://revision3.com/diggnation/2008-10-03jalbaminus...
Skipping download of http://revision3.com/diggnation/2008-09-25nojackblack2...
Skipping download of http://revision3.com/diggnation/2008-09-19michelob...
Skipping download of http://revision3.com/diggnation/2008-09-13hotels...
Skipping download of http://revision3.com/diggnation/2008-09-05pdoodles...
Skipping download of http://revision3.com/diggnation/2008-08-29nvisionsj...
Skipping download of http://revision3.com/diggnation/2008-08-21polepoll...
Skipping download of http://revision3.com/diggnation/2008-08-14dolby...
Skipping download of http://revision3.com/diggnation/2008-08-06beefu...
Skipping download of http://revision3.com/diggnation/2008-07-31cwalk...

Done.
but no way to reconfigure?
How many episodes of each feed does it download?
- i.e. the # most recent of each feed?

Is the number of most recent episodes to download configurable?
- For example, I would like the script to only download the 3 most recent episodes of each show/feed.
- It would also be nice if the script could delete old episodes, but I think I can figure out how to create a cronjob to do this manually if it's too much to add to this script.


Thanks again for the great work on the script,
-BassKozz

Thanks for trying the script!

For reconfiguring, I do not currently have a command line option for that, but, if you go to ~/.rev3/, and list the contents, you will find a rev3.xml and rev3_downloaded.xml. You can simply delete rev3.xml and you will be greeted by the configuration wizard on the first run again. rev3_downloaded.xml contains a list of previously downloaded episodes, so that it won't redownload them.

You asked how many new episodes will it download. Unfortunately right now it downloads as many episodes as are given by the feed. I've noticed that around 20 of the latest episodes are usually delivered by the feed. Your idea of setting a number is good, if you check back in a few days I think I will update my script to include a setting like this.

So, check back, I will try to add a --configure option to the script and a configurable new episode download limit as well.

[ieatpunk]

So, I updated the script to include a --reconfig command line option, which I recommend you run so that you can take advantage of the new Maximum download option. This will create a limit like you were speaking of above. So, tell me how this works for you.

[anti-cmos]

Hey this thing is great!!! Thanx man... I do however seem to be having a problem with it. It seems to be stuck downloading the same file over and over. It thinks the file was incomplete for some reason, a little odd if it was a coding issue as all the other files it has d/l'd have been fine, so this makes me ask do you do a MD5 check (comparing against a md5sum file on rev3's website) or something that may not be correct for this file??

B.T.W Here's the stdout. (below)

2009-01-29 23:44:12 (488 KB/s) - `/mnt/hd/Media/iTV/rev3/Hak5_(Large_Xvid)/hak5--0401--pineapples--large.xvid.avi' saved [364497974/364497974]

File size not correct! Deleting incomplete download...
File was not received! Trying again in 30 seconds...

[ieatpunk]

Hey this thing is great!!! Thanx man... I do however seem to be having a problem with it. It seems to be stuck downloading the same file over and over. It thinks the file was incomplete for some reason, a little odd if it was a coding issue as all the other files it has d/l'd have been fine, so this makes me ask do you do a MD5 check (comparing against a md5sum file on rev3's website) or something that may not be correct for this file??

B.T.W Here's the stdout. (below)

2009-01-29 23:44:12 (488 KB/s) - `/mnt/hd/Media/iTV/rev3/Hak5_(Large_Xvid)/hak5--0401--pineapples--large.xvid.avi' saved [364497974/364497974]

File size not correct! Deleting incomplete download...
File was not received! Trying again in 30 seconds...

Yea, this actually is a problem with some of revision3's RSS feeds. They post the wrong filesize sometimes. I attached a copy that ignores checking the file size. I'll try to come up with a better idea later.

[anti-cmos]

Cool thanks again, you don't know how much trouble this saved me. (the script in general) :D

BTW, you might just do a md5 check on the file b4 d/l and after, that would take care of the whole file size thing.