Nangabe ungafuni kutsi tinkampani te-AI ticeceshe emamodeli ato ngalokucuketfwe kwewebhusayithi yakho, engeta imigca lemibili kulifayela le-robots.txt lewebhusayithi yakho: User-Agent: __usage_ai-training bese ngaphansi kwawo Disallow: /. Mhla ti-30 Imphala 2026, i-OpenAI, i-Google kanye ne-Microsoft tamemetela kutsi titawuhlonipha loluphawu loluhlanganyelwe: i-Microsoft icala mhla ti-1 Indlovana 2027, i-Google ihlose 2027, kantsi i-OpenAI itsi itawulufaka etinyangeni letitako. Loluphawu lutsintsa kucecesha kuphela, ngako aluyisusi iwebhusayithi yakho emiphumeleni yekusesha ye-Google nobe ye-Bing. Kuze kube ngulapho inkampani ngayinye iluvula, gcina tilungiselelo tayo letikhona. Nangabe ufundze sihloko setfu mayelana nemibandzela lemisha ye-Meta levumela ema-bot ayo kutsi ahlole iwebhusayithi yakho, loku kusinyatselo lesilandzelako.

Kwamenyetelwa ini ngco?

Kuze kube nyalo, umnikati wewebhusayithi lofuna kugcina lokucuketfwe kwakhe kungasetjentiswa ekuceceshweni kwemamodeli bekufanele aliti ligama le-crawler yenkampani ngayinye futsi abhale umtsetfo lohlukile kuyo: i-GPTBot ye-OpenAI, i-ClaudeBot ye-Anthropic, i-Google-Extended ye-Google, kanye neligama lelisha ngaso sonkhe sikhatsi lapho kuvela inkampani lensha. Luphawu lolusha lushintja loluhlu ngeligama linye lelihlanganyelwe. Luvela eluhlakeni (Internet-Draft) lolwetfulwe ku-Internet Engineering Task Force (IETF), sigungu lesisemuva kwemazinga lamanyenti etheknikhi e-inthanethi, futsi lwabhalwa nguKevin Kelley we-Anthropic ngekhatsi kwelicembu lekusebenta le-IETF letincumo te-AI. Luhlaka lunelusuku lwamhla ti-30 Imphala 2026, luphelelwa sikhatsi mhla ti-3 Inkhwekhweti 2027, futsi luhloselwe indlela yemazinga, loku kusho kutsi lusesesiphakamiso, hhayi lizinga leliphelele.

Lombono ulula ngenhloso. Luhlaka alwengeti ticondziso letinsha ku-robots.txt. Lusebentisa licembu lelivamile lapho umugca walo we-User-Agent uphetse ligama lelikhetsekile, lelibitwa ngekutsi lithokheni lenchubomgomo (policy token), bese imitsetfo levamile ye-Allow ne-Disallow ikhombisa kutsi ngutiphi tingcenye tewebhusayithi letingasetjentiswa nalokungafanele tisetjentiswe. Sibonelo, ungala kucecesha kuyo yonkhe iwebhusayithi kepha ukuvumele engcenyeni yakho yetindzaba temaphephandzaba:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Luhlaka luchaza kucecesha i-AI ngekutsi kusebentisa lokucuketfwe kute kugucule emapharamitha lafundziwe yimodeli ye-AI lekhicitako, lokusho imodeli lekhicita umbhalo, titfombe nobe lokunye lokucuketfwe. Emamodeli lasetjentiswa kuphela ekuhlukaniseni, ekuhleleni nobe ekunikeni emaphuzu akafakwa.

Ngubani lotawuluhlonipha, futsi nini?

Inkampani Leyakumemetela Nini Lokuhlala kuhlukene
OpenAI Itawuhlonipha luphawu loluhlanganyelwe; tilawuli tayo letikhona tisachubeka «Etinyangeni letitako» I-GPTBot yekucecesha, i-OAI-SearchBot yetimphendvulo tekusesha ku-ChatGPT
Google Itawutsatsa kwala ngelithokheni njengekuvimbela i-Google-Extended, ngaphandle nangabe i-Google-Extended ibekwe ngalokucacile Ihlose 2027 I-Googlebot ye-Google Search; i-Google-Extended yekucecesha i-Gemini kanye ne-grounding
Microsoft Itawuhlonipha lithokheni futsi yengete sincumo lesehlukile ekhasini ngalinye lesibitwa ngekutsi yi-AI Grounding Kusukela mhla ti-1 Indlovana 2027; tilungiselelo ngaphambi kwamhla ti-1 Indlovulenkhulu 2027 Kufaneleka kwekuvela emiphumeleni ye-Bing akuguculeki
Anthropic Umbhali weluhlaka usebenta lapho Akukho lusuku ekhasini layo lelusito I-ClaudeBot isesilawuli sayo lesibhaliwe sekucecesha

Imininingwane lemibili yalezimemetelo ibalulekile kulo lonkhe libhizinisi lelinewebhusayithi. Kwekucala, i-Microsoft itsi ngemuva kwamhla ti-1 Indlovulenkhulu 2027 ngeke isayifundza imaki lendzala i-NoArchive kanye ne-NoCache njengetincumo te-AI; nangabe iwebhusayithi yakho itsembele kuto, shicilela tincumo letinsha ngaphambi kwalelilanga. Tilungiselelo tekushintja te-grounding tivuleka ku-Bing Webmaster Tools mhla ti-15 Ingongoni 2026, emawebhusayithi lasavele asebentisa lemaki kuphela, futsi tiphelelwa sikhatsi mhla ti-30 Inyoni 2027. Kwesibili, luhlaka lutsi umtsetfo lojwayelekile lobhalelwe onkhe ema-crawler, User-Agent: *, awubalwa njengesincumo sekucecesha. Kuvimbela konkhe akusho kutsi «akukho kucecesha»; lithokheni leliphetse ligama kuphela lelisho loko.

Kucecesha akufanani nekuvela etimphendvulweni

Lona kumehluko lobanyenti banikati bemawebhusayithi labangawuboni. Kucecesha kusho kutsi lokucuketfwe kwakho kusita kwakha imodeli yesikhatsi lesitako. Timphendvulo, i-Microsoft letibita ngekutsi yi-grounding, tisho kutsi umsiti we-AI ufundza likhasi lakho ngesikhatsi likhasimende libuta umbuto, bese uyalicaphuna kanye nesixhumanisi. Lithokheni lelisha litsintsa kucecesha kuphela. I-OpenAI ikugcina kuhlukene loku kokubili: i-GPTBot ingeyekucecesha, kantsi i-OAI-SearchBot incuma kutsi iwebhusayithi yakho ingavela yini etimphendvulweni tekusesha ku-ChatGPT. I-Microsoft iya embili futsi yengeta sincumo se-grounding lesibekwa likhasi ngelikhasi, sinetinketfo letintsatfu: Allow; DisallowExtended, lekhawulela kusetjentiswa kutimphahla te-Microsoft uqobo lwayo njenge-Copilot; kanye ne-Disallow. I-Microsoft iyecwayisa kutsi kukhawulela i-grounding kunciphisa kakhulu kubonakala kwakho etimphendvulweni te-AI, nekutsi kute nalesinye setinketfo lesicinisekisa kufakwa, kuvakashelwa nobe kukhokhelwa.

I-Google iyinhlukano lekufanele uyibuke. Silawuli sayo i-Google-Extended sihlanganisa kucecesha emamodeli e-Gemini kanye ne-grounding kuma-app e-Gemini, futsi i-Google itsi emawebhusayithi lavumela i-Google-Extended ngalokucacile angavela etimphendvulweni tema-app e-Gemini futsi atfole kuchofotwa. Ngako nangabe wala ngelithokheni lelihlanganyelwe, i-Google itawukutsatsa njengekuvimbela i-Google-Extended. Uhlala ku-Google Search, ngoba i-Google-Extended ayitsintsi kufakwa nobe kuhlelwa ngelizinga lapho, kepha uyayiyekela i-grounding kuma-app e-Gemini. Nangabe ufuna lokubonakala, beka i-Google-Extended ngalokucacile. Kuze wati kutsi timphendvulo te-AI titikhetsa njani tinsita tato, fundza kutsi ungayenta njani inkampani yakho ivele ku-ChatGPT.

Inkampani yakho kufanele yale yini?

Kute imphendvulo yinye leliciniso; kuya ngekutsi iwebhusayithi yakho iphetseni. Kwala kunengcondvo nangabe lokucuketfwe ngekwako kungumkhicito: tifundvo lokukhokhelwa tona, lucwaningo loluyiwo lwangempela, luhlu lwemanani lolunemininingwane, nobe tinkhulungwane tetinchazelo temikhicito letibhalwe licembu lakho. Kuvumela akubiti lokunyenti nangabe iwebhusayithi yetfula tinsita takho futsi ufuna kutsi ligama lenkampani yakho latiwe kakhulu. Kuko kokubili, chubeka uvumela ema-crawler ekusesha netimphendvulo, ngoba ngiwo laletsa emakhasimende.

Tinyatselo letisihlanu talelisonto

  1. Fundza lifayela lakho lelikhona. Vula idomeyini.com/robots.txt ku-browser. Funa User-agent: * lokhohliwe kanye ne-Disallow: /, nobe umtsetfo lovimbela i-OAI-SearchBot, i-Googlebot nobe i-Bingbot.
  2. Ncuma tintfo letimbili ngalokwehlukana. Kucecesha: yebo nobe cha. Timphendvulo: yebo nobe cha. Bhala phansi sincumo kute umakhi wesofthiwe lolandzelako angasihlehlisi.
  3. Engeta lithokheni lelihlanganyelwe bese ugcina imitsetfo lekhona. Kute nayinye kuletinkampani letintsatfu leseyihlonipha lithokheni, ngako gcina imitsetfo yakho ye-GPTBot, ye-ClaudeBot kanye ne-Google-Extended eceleni kwalo kuze kube ngulapho ngayinye iyalihlonipha. Likhasi lelusito le-Anthropic lisabhala i-ClaudeBot njengendlela yekwala.
  4. Hlola imaki yakho ye-Bing. Nangabe emakhasi akho asebentisa i-NoArchive nobe i-NoCache, hlela kushintjela etincumweni letinsha ngaphambi kwamhla ti-1 Indlovulenkhulu 2027.
  5. Beka konkhe lokuyimfihlo ngemuva kwekungena ngemvume. I-robots.txt luphawu lolusidlangalaleni futsi lwekutitsandzela: luhlaka luvumela tivumelwano letiyimfihlo kutsi tiludlule, futsi sincumo sisebenta kuloko lokutsatfwa ngemuva kwekutsi usishicilele, hhayi kuloko lokwacoceleleka ngaphambili.

Umbono we-Origami

Iminyaka leminyenti, i-robots.txt bekulifayela lokute namunye enkampanini lobelikhumbula, lelabhalwa kanye ngulowakha iwebhusayithi. Loko kuyashintja: lifayela nyalo lincuma kutsi lokucuketfwe kwakho kuyawacecesha yini emamodeli e-AI, kutsi emakhasi akho ayavela yini etimphendvulweni te-AI, futsi masinyane nekutsi i-Microsoft iliphatsa njani likhasi ngalinye. Ligama lelihlanganyelwe lingintfutfuko yangempela uma ulicatsanisa neluhlu lwema-crawler lebelukhula njalo ngenyanga, kepha alisitsatsi sikhundla sesincumo. Lesivame kusibona yiwebhusayithi lapho kute lowatiko kutsi ngubani lowagcina kulungisa i-robots.txt, nobe linethiwekhi lekusabalalisa lokucuketfwe (CDN) levimbela ema-crawler umnikati lebekafuna kuwamukela. Ku-Origami sakha emawebhusayithi, emaphulathifomu kanye nemasistimu lentiwe ngekwetidzingo takho, futsi letilungiselelo sitsatsa njengencenye yekwakha, tibhaliwe futsi tihloliwe. Nangabe ufuna kutsi i-robots.txt yakho, imitsetfo ye-CDN kanye nekubonakala kwakho ku-AI kuhlolwe ndzawonye, sitsintse.

Imitfombo