ccunning@lemmy.world to News@lemmy.world · 1 day agoPay-per-output? AI firms blindsided by beefed up robots.txt instructions.arstechnica.comexternal-linkmessage-square8fedilinkarrow-up126arrow-down11cross-posted to: technology@lemmy.world
arrow-up125arrow-down1external-linkPay-per-output? AI firms blindsided by beefed up robots.txt instructions.arstechnica.comccunning@lemmy.world to News@lemmy.world · 1 day agomessage-square8fedilinkcross-posted to: technology@lemmy.world
minus-squareTheMightyCat@ani.sociallinkfedilinkarrow-up2·1 day agoIf there is a way to combine this with something like anubis this could be very interesting, then you are also not dependent on the honesty of reading the robots.txt Have a RSL license? here you go scrape the content. Trying to freeload? anubis time.
If there is a way to combine this with something like anubis this could be very interesting, then you are also not dependent on the honesty of reading the robots.txt
Have a RSL license? here you go scrape the content.
Trying to freeload? anubis time.