Tumblr NSFW Upload thread

@redweasel  
Thanks! Just making sure there isn’t some thread I’m missing.

So, I have good news and bad news. I’ll start with good ones: stage-2 list is finally complete. But the bad news is… Tumblr API no longer allows to dump blogs that redirect to https://www.tumblr.com/dashboard/blog/{blogname}. For me it doesn’t even let me do that when OAuth header is present (using authorized account and not a random one). Would be glad to see if there exists some kind of a workaround…

@Micro-Cyberbrony  
That cmd line script will rename files about as fast as the operating system can access them (very fast). However, I should note that this does give the same prefix to ALL files in the directory. This would incorrectly name pictures that have been made by other artists and rebloged. It seems that TumblOne does not write any meta data so there is no way of renaming pictures from many different artists automatically. You can manually sort them by making a new folder for an artist and viewing all the pictures as large icons so you can sort them visually.
 
I’m using TumblThree and have if write a .json file for each blog with all the information that indicates if something was a reblog with a reference to the psudo-random file names. I’m not good enough with programing to deal with this data but I’m planning on making the archive available to the other people here that are.
 
TumblThree settings you should use in order to get everything.

@twkr  
I’m currently re-crawling everything with TumblThree and it at least didn’t complain at any hidden blogs so I guess it still works there. If you have an example URL I could test it, I don’t want to remove and add an existing one right now in case that breaks it.
 
 
@ElectricGears  
I haven’t looked into what they do, but should I switch from text to json for metadata and turn dump crawler data on?  
Also is there a reason image size is set to 1280 (vs “raw”)?

@rannius  
The example link: http://quiet-ala.tumblr.com

@twkr  
I think it worked, 516 files.

@LostPone
 
It’s still here….. I knew he had buried the actual blog’s content under a lot of unrelated stuff ever since 2014, but I went back a lot of pages and didn’t find anything, so I thought it’s just a mirror of his modblog. I didn’t know how much unrelated stuff he reblogged on there and with things related to Jan, I think I have come to expect the worst….. The blog is still in danger, though, so it needs to be saved.
 
This isn’t just what Tumblr does right now, there’s also the fact that a blog gets apparently deleted when the creator doesn’t show any activity on it for six months or so. Which might be the reason why Jan has reblogged so much stuff on it over the years….. But now, Jan has officially left Tumblr and it’s not sure if he will return to it once everything calms down, so we have six months to save the best CMC askblog now.  
I will go and download it in its entirety and probably also create new WaybackMachine archives where you can enlarge the pictures.
 
 
Maybe I can sleep better now, suddenly, a thousand rocks have fallen off of my heart…..

@rannius  
Looked through TumblrThree code. MY FUCKING GOD! It grabs web-like user session key for dashboard, hijacks CSRF tokens, fakes HTTP_REFERER and makes calls to Tumblr’s service API for dashboard AJAX. I’m not sure if it’s possible to grab those using console-only app because it pretty much requires virtual browser inside grabber tool that you just can’t get in terminal applications. Headless Chrome may work but FOR FUCK’S SAKE that would take ages to code properly =~=

@twkr  
How many blogs are we talking about?

@rannius  
That’s what I’m trying to calculate right now. At worst 8664 but I seriously doubt that.

@rannius  
Json file are more structured and I’m guessing would be easier to parse for automation. It looks like you don’t need the crawler data per post, they all are concatenated in text.json, images.json, answers.json. A while ago tumblr disabled public access to the _raw files so I don’t think there’s a need for the extra requests. (I don’t know if the crawler actually attempts to grab the _raw file, gets a 404, then falls back to _1280 or lower resolutions; or if the original request get a list of available files and of course it doesn’t include the _raw version.)

@twkr  
Oh, that’s quite a few. I’ll have to change my setup for that.
 
 
@ElectricGears  
Thanks, I hope the text files and missing ones won’t be an issue then.

@The Frowning Pony
 
I’m gonna use that program. Just queue as many sites as I can. I’ll do my part and help as much as I can.

@CMC Scootaloo  

 
I’ve seen Tumblrs that have been inactive for longer, like this one, that are still up. Do keep an eye on that one particular Tumblr though, if it suddenly gets deleted on January 6 then we suddenly know No One ran the blog and something’s happened to them…

Anyway, i archived all the ones i claimed on archive.is or web.archive. And also a few one that weren’t listed in the list:
 
Anthroquestria  
Ask Milky Way  
Ask Pun (incomplete)

Does anyone actually have an archive of mylittleanthros.tumblr.com? I don’t know how to back up the whole blog and all pics, kinda new to the whole archiving thing. Also, would that blog be considered NSFW?
 
My problem is naming each individual pic from the blog by the artist’s name from 2012 to 2018, and it’s a royal pain in the neck to name the pic by artist and download each pic one by one. What can I do?

@Micro-Cyberbrony  
Started archiving  
script seems to started processing it alright.  
And amount of posts is over 20k  
Still didn’t managed to make it downloading audio and video though, so it will only backup posts with pics.

@TotallyNotANoob  
Thank you! PM me if finished, OK? Pics on that blog can’t go to waste.

@TotallyNotANoob  
@Micro-Cyberbrony  
I got mylittleanthros backed up with the script and you can use Jdownloader 2 to grab the videos if you want.

I never had an account, and I couldn’t watch nsfw art/blogs there anyway since months. Meaning, I can’t help with the effort. It would be pointless to register just for 7 days.

@Xaxu-Slyph  
Can you send a link then? And do you have pics with videos?

@Micro-Cyberbrony  
The entire thing is 10gb and it doesn’t include any videos. As I said. Jdownloader 2 would be the best way to go in that regard. I’m currently concentrating on collecting as many ask blogs as possible. The script only grabs pictures and html files so I won’t have the videos for EVERY blog(I will have some as I was using Jdownloader 2 before to grab as much as possible. Which also parsed video files.) I am not familiar with the intricacies of torrenting and I don’t currently have a mega account. Someone else may offer what you need at an earlier time than I. Sorry that I cannot provide a link as you wish, but this whole endeavor is taking hours of my time as it is. The only free time as well, but I will trudge onward and help as I can. I will most likely be the last one to fill in the gaps if possible with how my life situation is. I am doing my part in saving as much as I possibly can. A time will come after the 12/17 deadline to get everything organized. If you wish to have JUST mylittleanthro yourself then I suggest grabbing Jdownloader 2, load it up, copy the link, and drop it into the parser. There are various tutorials on it if needed. Celestia speed to you.
 
@CMC Scootaloo  
I have confirmed that I have these:  
ask-berry-punch  
ohdatcheerilee  
naughtynaughtyluna
 
I get nothing from rainbownspeedash  
And magnalunansfw seems to lead to nothing really.(Must have been grabbed by bots.)

@rannius  
Rewrote stage-1 tumblr checker (now with proper locked and password-protected blog detection (sic!)) and fed it tumblrs that didn’t make it into the alive list. We’re already down to 8537 tumblrs at worst. Now just need to wait for it to check the rest. Shouldn’t take more than a couple of hours.

@Xaxu-Slyph  
Add mrmeowart.tumblr.com to that list, I grabbed it before it went down though (same with 3rd, 4th and 5th).
 
 
@twkr  
Good, ideally by friday I can switch to a 100mbit connection and have my new 8TB drive.
 
BTW does anyone know if TumblThree runs on WINE.