Remember image URLs that have already been downloaded

[metaprime]

Remember image URLs that have already been downloaded and don't re-rip if they have been deleted or renamed.

I think this makes sense as an addition to the history feature. I often download threads from 4chan or users on Reddit and then start to organize the collection by adding tags to file names to help me find pictures by category later on. Often I delete pictures I don't like and I'd like them to stay gone.

Right now I think it made sense to simply check if a URL had already been downloaded by checking the disk for the existence of the expected target filename, but that basically means I can never organize my ripme folders :(

Also, I think it would make sense to include a checkbox for re-ripping all images, even if they have been deleted, which would fall back on the current behavior.

I agree this would be a nice upgrade in functionality. I've started adding tags as names for empty folders inside the ripme saved folders. Helps to organize but I can't delete pics for good.

[clb92]

It should work like youtube-dl's --download-archive "archive.txt" property, in my opinion. A simple list of the URLs that have already been downloaded, so that it can quickly skip those in the future. Just like you described.

[4pr0n]

Just to verify from you guys, the request is:

• Save every URL somewhere (like history.json)
• Re-ripping an album will ignore already-ripped image URLs
• Ability to remove history for a specific rip (the next rip would re-rip everything).

I'm worried about scaling issues; history.json would get extremely bloated and could lag the program significantly... Maybe a history.json file inside the rip folder? Or just a urls.txt that just has a list of URLs?

There's already an option to save image URLs to urls.txt, but this option won't download the images. It seems trivial to do both (rip & save image URLs) ... as for ignoring duplicate image URLs, that might be harder.

[clb92]

Maybe a history.json file inside the rip folder? Or just a urls.txt that just has a list of URLs?

Yeah, that could work. I'm thinking it should just be a simple text file with URLs, like

http://url1.com/path/here
https://www.url2.com/stuff
http://yet.another.url/

...and so on.

So when I run ripme.jar -u "http://reddit.com/r/pics" -history "urls.txt" again it'll check in the rips/reddit_sub_pics (default) folder for a the history.txt file that I specified, right? But I would also like to be able to specify an absolute or relative path to where I put the history file, like -history "/volume1/home/admin/reddit_pics_history.txt", if that wouldn't be too hard. And if the specified history file does not exist, then create it.

Ripme is a really great program so far. Thank you.

[metaprime]

The above two posts are spot on. I noticed another problem that comes up on i.e. chan pages where people sometimes delete images and that affects the sequence numbers for the rest of the album... so many duplicates.

I like the idea of storing a history.txt or urls.txt inside of every ripped folder. It won't bother me that it is there, and when I'm done ripping I could always delete that file. On a folder-to-folder basis it wouldn't be that bloated.

As for detecting duplicate URLs I don't believe it would be that hard. If you sort the list of URLs the ripper should always end up finding the same URLs and detecting duplicates should be fairly easy.

[icannotfly]

Seconding this; storing ripped URLs per-folder to avoid redownloading files would be very nice.

I third this.