Wb crwlr, smtms clld spdr r spdrbt nd ftn shrtnd t crwlr, s n ntrnt bt tht systmtclly brwss th Wrld Wd Wb nd tht s typclly prtd by srch ngns fr th prps f Wb ndxng (wb spdrng).
Wb srch ngns nd sm thr wbsts s Wb crwlng r spdrng sftwr t pdt thr wb cntnt r ndcs f thr sts' wb cntnt. Wb crwlrs cpy pgs fr prcssng by srch ngn, whch ndxs th dwnldd pgs s tht srs cn srch mr ffcntly.
Crwlrs cnsm rsrcs n vstd systms nd ftn vst sts nprmptd. sss f schdl, ld, nd "pltnss" cm nt ply whn lrg cllctns f pgs r ccssd. Mchnsms xst fr pblc sts nt wshng t b crwld t mk ths knwn t th crwlng gnt. Fr xmpl, ncldng rbts.txt fl cn rqst bts t ndx nly prts f wbst, r nthng t ll.
Th nmbr f ntrnt pgs s xtrmly lrg; vn th lrgst crwlrs fll shrt f mkng cmplt ndx. Fr ths rsn, srch ngns strggld t gv rlvnt srch rslts n th rly yrs f th Wrld Wd Wb, bfr 2000. Tdy, rlvnt rslts r gvn lmst nstntly.
Crwlrs cn vldt hyprlnks nd HTML cd. Thy cn ls b sd fr wb scrpng nd dt-drvn prgrmmng.