Prevent AI crawlers

The Smiling Pony

( ͠° ͟ʖ ͡° )

Due to ongoing architectural rework, there haven’t been any published changes to the site code in over half a year, and that was specifically to deal with child porn being spammed. This wouldn’t be simply adding a meta tag everywhere, but also user-facing UI for enabling and disabling it per image, per tag, etc.

I’m also not sure that meta tag would actually be used by any crawler outside of a couple commercial AIs (like NovelAI or Dall-e 2); even Google and Bing ignore all the no-bots/crawler tags when they’re feeling curious enough. Most AIs will just ignore it, same as they do for DA and other sites; it’s effectively just an empty gesture.

To note that DA’s shitstorm wasn’t that AI was crawling the site (as mentioned, they can’t do anything about that, meta tag or no), but that they were setting up their own commercial AI using their site’s own database, and automatically opting everyone in.

Posted about a year ago Report

Fyrestar

Lexi Fyrestar

It was both but in the end all about opting in all artists to let AI steal their art.

Do you have any sources that confirm they ignore these tags? Adding it wouldn’t do any harm still. It would also help giving people the option to spoiler their images from crawlers entirely.

Posted about a year ago Report

The Smiling Pony

( ͠° ͟ʖ ͡° )

@Fyrestar
I don’t have, like, a list of which publicly accessible AIs ignore it, but some definitely don’t, and all this is open source stuff so anyone can just turn it off / delete the line when they set theirs up. For example, https://twitter.com/isyourguy/status/1591149452643172353

There was also some post going around from the internal communications in DA about the engineers freaking out that the feature had been released before it was ready, and that it was pretty much useless for non-DA AIs anyway.

Posted about a year ago Report

Ciaran

Senior Moderator

友情は魔法だ

@Fyrestar
Even DeviantArt notes that opting out doesn’t mean that your images won’t appear in datasets.

And they flat out state that it does not affect whether your art appears in their own DreamUp dataset.

Want to learn more about all of this? Do this.

Fill out DeviantArt’s DreamUp Username Opt-Out Request
It will be reviewed by a human within 10 days.

• Please note this is only available to artists who DeviantArt believes are ‘most likely to have been sourced by Stable Diffusion’.

Assuming you pass, and are opted-out, then your name can not be used in DreamUp as a text prompt.

Even if you do this and are approved, your images are still harvested and used in DreamUp.

All that changes is that your name can’t be used as a part of a text prompt.

Now, let’s opt out our images.

Go to each of your images on DeviantArt and click this box:

As noted above, this only works if crawlers respect the meta directive, and DeviantArt does not. Regardless of your setting for this option, DreamUp still gets your images.

And, remember, there’s no guarantee that bots are crawling DeviantArt. So even if DeviantArt makes the meta directive available for the crawler, the crawler might not be getting the image via DeviantArt.

They can just crawl Google, or YIS, or some other image search tool’s cache instead.

So the meta directive at that point doesn’t even matter - the data’s not being crawled ‘from DeviantArt’, and Google and YIS aren’t going to respect a meta directive for something they are not.

FIND OUT IF YOU ARE IN THE DATA SET

If you want to find out if you are in the dataset of 5.8 billion images being used by many of these tools, you may do so here.

It works best if you give it an image to search for in the data. Names sometimes work, but if you give it an image you almost always get a 100% positive or 100% negative.

I’m in there so many times, and from so many places, I’m not even sure what the pathway was that resulted in the images being snagged. Like - I don’t believe DeviantArt was in any way involved in most of the grabs, because even images from my vanity websites are in there. Images from friends web comics are in there too, so to keep from being snagged you’d need to implement this tag every place your images appear ever.

That’s why I think this doesn’t have anything to do with the images being grabbed from DeviantArt or even this site - to me it looks like they’re just coming out of image searches on something like Google.

Because the images aren’t ‘from this site’ or ‘from that site’ - in some cases the images are SO OLD the site they were on doesn’t even exist any more.

So … for me … it looks like they’re eating search engine caches. Because even if you went to that site today there’s no site to grab the image off of.

At which point, site directives stop having any meaning.

To make this really work, I think you’d almost have to do it as meta data in the image itself.

But that too would have to be supported and respected end-to-end.

And we never, ever got that working for Copyright, so I don’t know how you could successfully get it working for something like this.

My guess is that someone will start selling an invisible watermark service, like the old copyright keys that supposedly tracked artist copyrights. The ones that seemed to work were incredibly expensive, and the ones that were free never seemed to work.

Maybe there’s an opportunity here for someone to turn a fast buck by scamming people with promises that can’t be delivered. Like DeviantArt has done.

So, anyway, yes, the meta directive is nice. But if even a single crawler ignores it, you’re in the dataset. And, today at least, there is nothing you can do about that.

So to me at least it seems like a lot of work to make people feel like something is being done. But it doesn’t actually do anything.

Just look at DeviantArt’s implementation - they right out admit that they themselves are ignoring their own implementation, and the only thing they’ll consider allowing people to opt-out from is the artist name.

Almost assuredly because they want to avoid violating an artist’s trademark.

So, names CAN be protected, on DeviantArt at least. Most likely because of trademark liabilities.

But not images.

BE THE CHANGE YOU WANT TO SEE

Philomena is public source, and anyone who wants to is welcome to extend its functionality to support this meta directive. That would make the new features available to any site that is based Philomena.

You don’t have to wait until this site’s dev team has the resources and time for it - you can do it yourself, and then sites that use Philomena may opt-in to the new features if they are functional and efficacious and integrate with their own processes and features.

PS: For me, my favorite thing about all of this is that DeviantArt’s DreamUp opt-out is careful to say it is ‘currently free’, and only available to artists who they judge to be ‘most likely’ to have been harvested.

So … not everyone gets to play. And if even hardly ever did anything artists like me are in the dataset, there’s no way of knowing what ‘most likely’ means. I’m absolutely not ‘most likely’, but I’m already in it.

If you do get to play and fill out DeviantArt’s forms, god bless. But it doesn’t do anything except protect any potential trademark you might have on your artist’s name - not even for DeviantArt’s own DreamUp. And even for people whom DeviantArt considers ‘most likely’ there’s no guarantee that it will remain free.

DeviantArt has been very clever how they played this. They gave people the impression that they’ve done something, and people keep asking why others aren’t doing the same.

But all they’ve done is give the appearance of this being optional somehow.

For me, that’s kind of DeviantArt’s whole thing. Respond to drama, like selling prints of people’s DA submissions, by giving people more boxes to click. As soon as people get lost in the box clicking and demanding that other sites ‘do the say’, DeviantArt goes ahead and does what they had been doing all along.

Posted about a year ago Report
Edited about a year ago

Fyrestar

Lexi Fyrestar

@Fyrestar

It was both but in the end all about opting in all artists to let AI steal their art.

Do you have any sources that confirm they ignore these tags? Adding it wouldn’t do any harm still. It would also help giving people the option to spoiler their images from crawlers entirely.

Do you have a link or image of it?

@Cirian i was aware it’s just a directive, just like webcrawlers, but that it’s completely ignored and those ass*** ignore it even themselves is massive. I will flush my art from all public crawlable platforms till they implement something fighting them, i will setup a own site gallery now that blocks off the crawlers.

Might even consider using shit like onlyfans and paid access for just everything anymore, this is a stab into every artists back. AI is cancer.

Posted about a year ago Report

Background Pony #AA9F

@Fyrestar

I will flush my art from all public crawlable platforms

Hey, I just saw your dA. Kudos to you for really following through on your principles. There are lots of people out there who talk big but never take the plunge (or don’t stick with it in the long term.)

till they implement something fighting them, i will setup a own site gallery now that blocks off the crawlers.

Both are impossible.

Might even consider using shit like onlyfans and paid access for just everything anymore

This will not work. People will pay and repost it purely for the sake of doing it, and then wherever it appears is out of control. There’s at least one moderately well-known website dedicated to this purpose. The only options I can think of are to never show anyone art, to watermark it so severely no-one would want to look at it anyway, or to not worry about it.

Posted about a year ago Report

Ciaran

Senior Moderator

友情は魔法だ

@Fyrestar

i will setup a own site gallery now that blocks off the crawlers.

Paywalling your art probably would work, for awhile. Eventually you will probably find your art being shared someplace like kemono.

If people can see it on the web, it can be grabbed by a crawler. And even if you paywall it, some asshole will steal it and share it for you anyway.

I’m not saying that is good - it sucks. Really really sucks. But … I’ve been dealing with the whole “make it so no one else can get at this thing we want to put on the internet” for a long time, and there really is no simple answer.

Good luck though. If you try, maybe you’ll figure out something that no one else has.

@Background Pony #AA9F

This will not work. People will pay and repost it purely for the sake of doing it

Basically, yes. This.

For me, the only solution I’ve found is:

Make More Art

And once something is made, let it go. If you paywall it, collect what you can while its still secure - but make sure you’re already making your next thing.

Posted about a year ago Report

Fyrestar

Lexi Fyrestar

I think i already know a way to make images invisibly identifiable to every of the registered viewers so if it’s leaked the responsible one can be hunted legally.

It’s obvious they will still be able to share it, it’s rather about making the responsible identifiable, so someone might leak it, you can sue them collect some cash the person could go to jail for since they signed a agreement. But these details need to be done by a lawyer and the agreement/contract needs to use anything the law offers.

For me, the only solution I’ve found is:

Make More Art

And once something is made, let it go. If you paywall it, collect what you can while its still secure - but make sure you’re already making your next thing.

I don’t think this is a good idea at all, you basically just keep feeding those monsters, and you should not forget, no one will commission you anymore if they can give the commission prompt to a cheaper or free AI instead of you, letting it create something that will look like you did it. Art becomes completely worthless if we don’t fight it. And i don’t just mean passively.

Posted about a year ago Report
Edited about a year ago

LemonDrop

C++ Crazed

@Fyrestar
I think you have an overly optimistic view about the legality of ignoring something like a robots meta tag, that is to say, it’s not illegal. If a human can view the art legally an AI can and there’s really nothing you can do to change that (lest some laws actually change). It (right now at least) is fair use, you do not get absolute authority over what people are allowed to do with your work, copyright has exceptions and this is one.

Posted about a year ago Report
Edited about a year ago

Exedrus

@LemonDrop
The act of ignoring a robots.txt file on its own may not be illegal, but it’s possible to release an image under a license that explicitly disallows use of the image to train an AI. Whether that will hold up in a court case is debatable. Training an AI might very well be considered fair use, especially since the output of generative AI usually doesn’t have a copyright. It also might not be fair use since it’s clearly got potential to run artist out of business. I don’t know of any court cases that settle this for AI training in particular, though I’m not a lawyer. So I might be missing something.

Posted about a year ago Report

The Smiling Pony

( ͠° ͟ʖ ͡° )

It also might not be fair use since it’s clearly got potential to run artist out of business.

That’s a bit like arguing the Tolkein estate can sue Hasbro because their D&D novels and premade campaigns can run them out of business.

Posted about a year ago Report

Exedrus

@The Smiling Pony
Can the Tolkien estate prove that the people who wrote those D&D novels explicitly based their ideas on Tolkien’s work? I’m sure Tolkien had contemporaries. And his ideas were based on existing folklore/mythology. D&D novelists could argue they haven’t even read his books, they just read medieval European history.

If an AI is trained on an image, integrates it into it’s model, and then is explicitly used to produce images that would replace the need for that image’s creator, there is a very clear line from the use of the artist’s works to them being flooded out of the market. This seems like the sort of thing that the “market impacts” part of the fair use doctrine is intended to prevent.

Posted about a year ago Report

Ciaran

Senior Moderator

友情は魔法だ

It’s also a little like the Tolkein estate suing Terry Pratchett’s estate because he once read The Hobbit.

As I’ve said several times now, my hook into all of this is protecting the copyrights of artists.

If the resulting image is derivative, then if the copyright holder wants the derivative image deleted, it gets deleted. The original artist’s rights are reserved.

It does not matter if the derivative work was made by a human, or by a machine. The original artist’s rights are reserved.

But … if the resulting work is not derivative, then it is not derivative.

The only real difference is that only humans can have original copyrightable works. Anything made by a machine is, by definition, not protected.

At least, not unless the ‘anything’ is music, and the machine was built by Sony.

Posted about a year ago Report

Ciaran

Senior Moderator

友情は魔法だ

If an AI is trained on an image, integrates it into it’s model, and then is explicitly used to produce images that would replace the need for that image’s creator …

If the resulting works are derivative of the original artist’s works, then they are a copyright violation.

But are AIs actually being used to replace individual artists?

At best, I see them sometimes able to emulate style or palette, and when given a source image to work with they are really good at filling in detail.

Admittedly, in music this is an entirely different world. Emulators fit in pedals now, and you can have something remarkably close to an actual, specific, artist playing drums for you in a package that fits in your hand. But there’s usually quite a bit of licensing tied into all of that, so the artist (or their representatives) are still getting a cut.

If you have an example of someone using AI or machine generated art to replace an artist here on this site, please report it or send me a link via PM. I would very much appreciate it if you would share it. If someone is literally doing what you are describing, then that would probably be a Rule #1.

Posted about a year ago Report

Exedrus

If you have an example of …

I have not seen that happen. It’s just a theoretical that I imagine could be thrown around if the question of AI training as fair use went to a court.

Though it is good to hear that that’s not allowed on this site.

Posted about a year ago Report
Edited about a year ago because: Trying to be clearer on what I responded to.

Ciaran

Senior Moderator

友情は魔法だ

@Exedrus
Well, still speculating, but someone doing that intentionally would probably either be trying to replicate a Patreon alternative image, which is very much not ok here (Rule #1). Or, would probably be doing an edit to embarrass or harass the original artist or something like that.

That kind of stuff happens now, but it’s always humans doing it.

If people start leveraging AI to accomplish the same thing, it will still be rule breaking. What tools they are using to break the rule doesn’t matter.

Posted about a year ago Report

The Smiling Pony

( ͠° ͟ʖ ͡° )

Can the Tolkien estate prove that the people who wrote those D&D novels explicitly based their ideas on Tolkien’s work? I’m sure Tolkien had contemporaries. And his ideas were based on existing folklore/mythology. D&D novelists could argue they haven’t even read his books, they just read medieval European history.

lmao

90% of fantasy authors proudly proclaim how early they read Tolkein’s work, and what they learned from it. And in terms of orcs and dwarves, and elves and halflings, no, Tolkien had no contemporaries; he invented the modern conception for those fantasy races. Effectively anyone that writes a novel or script or game that has point-eared long-living elves, humanoid brutish orcs, stouty mountain-minin dwarves, etc. is as much of a “thief” as any random AI is for having been trained on someone’s image.

AND FURTHERMORE

An AI is as trained on an artist’s image as an artist is trained on what they themselves are copying from. It’s not like artists suddenly dreamt up Princess Celestia’s form, eye colour, mane style, etc. out of nowhere; they watched the show, they studied screencaps, they looked at other artist’s images (without their permission!). AI does the same; it looks at a whole lot of images and figures out the commonalities when people prompt “princess celestia”.

I really understand a lot of the concerns about AI art replacing artists, commission artists losing revenue, the “worth” of art being diminished. But this angle of “AI steals!” ain’t it. Everyone steals, everyone. AI just does it faster and is more honest about it.

Eventually people will train AIs starting up from basic geometric shapes and millions of iterations to recreate all sorts of objects and styles, with zero human art training. What then? AI will be creating new art based only on the subjective approval of viewers agreeing that it looks like what was asked and/or it’s aesthetically pleasing.

Posted about a year ago Report

Exedrus

@The Smiling Pony
This is going way beyond what I was trying to say. My point is that it is not currently clear whether the act of downloading an image for AI training is fair use. If it turns out to not be fair use, then artists could likely license their works to explicitly disallow that.

The similarities between AI training and human learning may play some part in the debate. I’m not sure. (Though the feeling I get when reading through legal documents is that they’re more focused on precedent and the exact letter of the law than the philosophical implications.) Minimally, AIs aren’t given the exact same rights as humans since (in the US at least) AI-generated art doesn’t generally have copyright protection, while human-created art gets it by default. And it’s not unreasonable to have special rules for AI in particular, since AIs can do things that humans can’t: I can copy a single AI onto a thousand computers and have them churn out enormous numbers of images per second. That’s the sort of competitive advantage that could break the underlying balance that US copyright law exists to protect; a balance between encouraging creativity and protecting the results of those creative pursuits.

Posted about a year ago Report

Background Pony #AA9F

If the resulting image is derivative, then if the copyright holder wants the derivative image deleted, it gets deleted. The original artist’s rights are reserved.

It does not matter if the derivative work was made by a human, or by a machine. The original artist’s rights are reserved.

But … if the resulting work is not derivative, then it is not derivative.

If the resulting works are derivative of the original artist’s works, then they are a copyright violation.

Do you have a definition of a derivative work (which is not as simple as “was used during its creation” because that would incorrectly include learning/reference material and transformative works) and a way to ascertain whether something is or isn’t derivative? These are necessary for your statements to have meaning.

If you have an example of someone using AI or machine generated art to replace an artist here on this site, please report it or send me a link via PM. I would very much appreciate it if you would share it. If someone is literally doing what you are describing, then that would probably be a Rule #1.

I have one example of the reverse, where they used another artist’s work as the foundation and reinterpreted it via AI: (explicit) >>2955564 and see the comments which refer to (explicit) >>1437713. I have not reported it because I do not see a “no edits” DNP for the artist whose drawing it is unquestionably derivative from.

Posted about a year ago Report

Ciaran

Senior Moderator

友情は魔法だ

@Background Pony #AA9F
This is the policy that we use to work these kids of things here. Substitute “derivative work” in place of “traced” and this document is a good introduction to the concept of derivative works and the protections they may have or may not have. Of course, this begs the question of “transformative” works but we approach those on a case by case basis here, usually working to find consensus between the artists involved.

And, keep in mind we’re not counting lawyers on pinheads.

I looked at what you linked, and I don’t see how >>2955564 is derivative, or any kind of a violation of copyright or violation of Rule #1, based on >>1437713.

They appear to be two different works.

If >>2955564 were drawn by a human who said they were inspired by >>1437713, would you still see it as a violation of artist:howl echoes’s copyright?

Posted about a year ago Report

Background Pony #AA9F

@Ciaran
The pose lines up exactly and the uploader commented that they “used” Howl’s picture, which almost certainly means it was operated upon by the “image to image” function. If a human made it freehand by referencing, it is close enough in my eyes that I would expect them to credit the original picture when they share the new version - which they did, in the comments. Whether it’s a “violation of copyright” is above my paygrade to figure out.

Posted about a year ago Report

The Smiling Pony

( ͠° ͟ʖ ͡° )

@Background Pony #AA9F
That case is at its base no more different than any of the hundreds (or thousands, idk) of cum edits or similar that people have done with GIMP or Photoshop and unrelated to the issue of AI training datasets.

Posted about a year ago Report

Bigcheese

@The Smiling Pony
This is going way beyond what I was trying to say. My point is that it is not currently clear whether the act of downloading an image for AI training is fair use. If it turns out to not be fair use, then artists could likely license their works to explicitly disallow that.

It’s incredibly difficult to add a license on top of an image publicly posted on the internet. You have to somehow get people to agree to the license before downloading the image. Source code solves this by putting the copyright at the top of each file, and using the DMCA to make it illegal to remove the copyright notice.

I looked at what you linked, and I don’t see how >>2955564 is derivative, or any kind of a violation of copyright or violation of Rule #1, based on >>1437713.

They appear to be two different works.

This is img2img meaning the AI was given >>1437713 as input, and it modified it to be >>2955564. Depending on the settings, this can be equivalent to a human artist redrawing an image while basically using the original image as a base layer. There’s no current case law here for AI, but had a human drawn >>2955564, I expect it would be classified as not a copy, thus the artist tag would be incorrect under my understanding of the derpi rules.

Posted about a year ago Report

Ciaran

Senior Moderator

友情は魔法だ

@Bigcheese
Yeah, the artist tag definitely doesn’t apply there.

Regardless of the inspiration, the resulting image is not derivative of the original artist’s copyright.

Posted about a year ago Report