Caching Youtube
Puas rasanya, akhirnya bisa caching youtube (dan yang similar) :D.
Setelah sekian bulan “mangkrak” akhirnya jalan juga walau masih di mesin virtualbox.
Howto ini bukan untuk para pembenci youtube dan googlemap.
Tapi untuk youtube dan googlemap lovers.
bahan referensi yang jadi bacaan.
http://www.mail-archive.com/squid-users@squid-cache.org/msg54605.html
http://www.mail-archive.com/squid-users@squid-cache.org/msg51076.html
http://wiki.squid-cache.org/Features/StoreUrlRewrite
http://wiki.squid-cache.org/Features/StoreUrlRewrite/RewriteScript
Versi yang saya pakai adalah squid-2.7.STABLE3, tidak tahu dukungan untuk versi yang lain.
- buat script untuk manipulasi youtube.
#!/usr/bin/perl $|=1; while (<>) { @X = split; $url = $X[0]; $url =~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)&.*@squid://videos.youtube.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)$@squid://videos.youtube.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/videodownload\?(.*)docid=(.*?)$@squid://videos.google.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/videodownload\?(.*)docid=(.*?)&.*@squid://videos.google.INTERNAL/ID=$3@; print "$url\n"; } - Lalu di squid.conf-nya edit seperti yang dibawah ini:
acl store_rewrite_list url_regex ^http://(.*?)/get_video\? acl store_rewrite_list url_regex ^http://(.*?)/videodownload\? cache allow store_rewrite_list # Had to uncomment this again, because I couln'd login to google mail using IE6 (firefox had no trouble): acl QUERY urlpath_regex cgi-bin \? cache deny QUERY refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999 override-expire ignore-no-cache ignore-private storeurl_access allow store_rewrite_list storeurl_access deny all storeurl_rewrite_program /usr/local/bin/store_url_rewrite
Hasilnya bisa dilihat di access-log, pada saat mengakses video yang sama, akan langsung hit.
# grep youtube access.log | grep TCP_HIT 1214834411.379 735 192.168.1.89 TCP_HIT/200 1604459 GET http://youtube.com/get_video?video_id=2d55B-SiJdM&t=OEgsToPDskKrwAAE_vVIhOqMhPqmPDUQ - NONE/- video/flv 1214834487.090 818 192.168.1.94 TCP_HIT/200 1604459 GET http://youtube.com/get_video?video_id=2d55B-SiJdM&t=OEgsToPDskLGVqEnxKjLEN4DGA3HYGse - NONE/- video/flv 1214836269.353 4383 192.168.1.91 TCP_HIT/200 9533167 GET http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskKeQxYVvYZ7fgEIW4UNC_U- - NONE/- video/flv 1214836514.802 3757 192.168.1.91 TCP_HIT/200 9533167 GET http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskIEwsTb26LiGFc96hBUUa9Z - NONE/- video/flv
Satu pesan dari Horacio Herrera Gonzalez, karena basic scriptnya tidak spesifik ke url tertentu, maka :
Warning! This code may match other sites not related to YT or GV.
He he he he, watching your bandwidth.
Karena beberapa user merasa kesulitan untuk mengaplied caching youtube.
Langkah dibawah adalah urutan di server saya.
- Saya pakai distro TSL 3.05, dengan squid squid-2.7.STABLE3
-
./configure \ --sysconfdir=/etc/squid \ --prefix=/usr \ --enable-async-io \ --enable-removal-policies=lru,heap \ --disable-delay-pools \ --disable-wccp \ --disable-wccp2 \ --enable-kill-parent-hack \ --enable-snmp \ --enable-default-err-languages=English --enable-err-languages=English \ --enable-linux-netfilter \ --disable-auth
- config hasil parsing ^# dari squid.conf
acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access deny all icp_access allow localnet icp_access deny all http_port 3128 transparent hierarchy_stoplist cgi-bin ? cache_mem 6 MB maximum_object_size_in_memory 32 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /nfs/cache 20000 16 256 maximum_object_size 64 MB cache_swap_low 98 cache_swap_high 99 access_log /var/log/squid/access.log squid cache_log /var/log/squid/cache.log cache_store_log none log_fqdn off storeurl_rewrite_program /etc/squid/store_url_rewrite acl store_rewrite_list url_regex ^http://(.*?)/get_video\? acl store_rewrite_list url_regex ^http://(.*?)/videodownload\? storeurl_access allow store_rewrite_list storeurl_access deny all cache allow store_rewrite_list acl QUERY urlpath_regex cgi-bin \? cache deny QUERY refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 quick_abort_min 0 quick_abort_max 0 quick_abort_pct 98 acl apache rep_header Server ^Apache broken_vary_encoding allow apache vary_ignore_expire on cache_effective_user squid cache_effective_group squid log_icp_queries off ipcache_size 2048 ipcache_low 98 ipcache_high 99 memory_pools off reload_into_ims on coredump_dir /usr/var/cache pipeline_prefetch on
Caching photobucket
Kontribusi apit (Ym-id relative_04), caching untuk photobucket yang banyak di pakai di friendster.
di store_url_rewrite
$url =~s@^http://(.*?)/albums\?&.*@squid://images.photobucket.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/albums\?$@squid://images.photobucket.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/albums\?&.*@squid://videos.photobucket.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/albums\?$@squid://videos.photobucket.INTERNAL/ID=$3@;
di squid.conf
acl store_rewrite_list url_regex ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? acl store_rewrite_list url_regex ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\? refresh_pattern ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? 43200 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\? 43200 90% 999999 override-expire ignore-no-cache ignore-private
Hasilnya
TCP_HIT/200 5474813 GET http://vid264.photobucket.com/albums/ii163/shannonwiseman12/DSCN0212.flv - NONE/- text/plain
Update script
Diperkirakan youtube merubah sistem mereka, sekitar quartal pertama tahun 2009.
Akibatnya script diatas sudah tidak berfungsi, untuk mengatasinya perlu diubah script dan beberapa bagian di konfigurasi.
Untung saja sudah ada panduannya di http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
konfigurasi di bawah saya coba dimesin vmware dengan os centos 5.2, juli 2009
Untuk mempermudah saya sertakan squid.conf yang sudah dimodifikasi dan script url rewriternya.
acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl localnet src 10.0.0.0/8 acl localnet src 172.16.0.0/12 acl localnet src 192.168.0.0/16 acl SSL_ports port 443 acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777 acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access deny all icp_access allow localnet icp_access deny all http_port 3128 transparent hierarchy_stoplist cgi-bin ? cache_mem 6 MB maximum_object_size_in_memory 32 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /cache 20000 16 256 maximum_object_size 64 MB cache_swap_low 98 cache_swap_high 99 access_log /var/log/squid/access.log squid cache_log /var/log/squid/cache.log cache_store_log none log_fqdn off #storeurl_rewrite_program /etc/squid/store_url_rewrite #acl store_rewrite_list url_regex ^http://(.*?)/get_video\? #acl store_rewrite_list url_regex ^http://(.*?)/videoplayback\? acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback.*id) \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)\? \/ads\? acl store_rewrite_list_web url_regex ^http:\/\/([A-Za-z-]+[0-9]+)*\.[A-Za-z]*\.[A-Za-z]* acl store_rewrite_list_path urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)$ acl store_rewrite_list_web_CDN url_regex ^http:\/\/[a-z]+[0-9]\.google\.com doubleclick\.net acl QUERY2 urlpath_regex get_video\? videoplayback\? \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)\? cache allow QUERY2 cache allow store_rewrite_list_web_CDN acl QUERY urlpath_regex cgi-bin \? cache deny QUERY storeurl_access allow store_rewrite_list #this is not related to youtube video its only for CDN pictures storeurl_access allow store_rewrite_list_web_CDN storeurl_access allow store_rewrite_list_web store_rewrite_list_path storeurl_access deny all #rewrite_program path is base on windows so use use your own path storeurl_rewrite_program /etc/squid/cacheyoutube2.pl storeurl_rewrite_children 1 storeurl_rewrite_concurrency 10 refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^http://(.*?)/videoplayback\? 10080 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern -i (get_video\?|videoplayback\?id|videoplayback.*id) 161280 50000% 525948 override-expire ignore-reload #and for pictures refresh_pattern -i \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv)(\?|$) 161280 3000% 525948 override-expire reload-into-ims refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 quick_abort_min 0 quick_abort_max 0 quick_abort_pct 98 acl apache rep_header Server ^Apache broken_vary_encoding allow apache vary_ignore_expire on cache_effective_user squid cache_effective_group squid log_icp_queries off ipcache_size 2048 ipcache_low 98 ipcache_high 99 memory_pools off reload_into_ims on coredump_dir /usr/var/cache pipeline_prefetch on
sedangkan untuk storeurl programnya sebagai berikut
isi file cacheyoutube2.pl
#!/usr/bin/perl
$|=1;
while (<>) {
@X = split;
$x = $X[0];
$_ = $X[1];
$u = $X[1];
if (m/^http:\/\/([0-9.]{4}|www\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/; # compatible to old cached get_video?video_id
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";
# new youtube
} elsif (m/^http:\/\/([0-9.]{4}|www\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(id=[a-zA-Z0-9]*)/) {
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $2 . "\n";
} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";
#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2 . "\n";
# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";
#rapidshare
} elsif ( ($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";
} elsif ( ($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
# $z = $1; $z =~ s/[A-Za-z]+[0-9-.]+/cdn/;
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";
#like porn hub variables url and center part of the path, filename etention 3 or 4 with or withour ? at the end
} elsif ( ($u =~ /tube8|pornhub/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $3 . $5 . "\n";
#...spicific servers end here.
#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2 . "." . $3 . "\n";
#for yimg.com
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*?)/) {
print $x . "http://cdn.yimg.com/" . $3 . "\n";
#generic http://variable.domain.com/path/filename."ext" or "exte" with or withour "?"
} elsif (m/^http:\/\/( ([A-Za-z]+[0-9-.]+)*?)\.(.*?)\.(.*?)\/(.*?)\.([^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 . "." . $6 . "\n";
# generic http://variable.domain.com/...
} elsif (m/^http:\/\/( ([A-Za-z]+[0-9-.]+)*?)\.(.*?)\.(.*?)\/(.*)$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 . "\n";
# spicific extention that ends with ?
} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)\?(.*)/) {
print $x . "http://" . $1 . "/" . $2 . "." . $3 . "\n";
# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2 . "\n";
} else {
print $x . $_ . "\n";
}
}
Jangan lupa di chmod +x agar file perl-nya bisa di exekusi.
Sebelum diconfigure squidnya dipacth dengan file diff (http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion) di bawah:
— src/client_side.c 2008-10-30 07:37:56 +0800
+++ src/client_side.c 2008-11-05 23:44:55 +0800
@@ -2399,6 +2399,18 @@
is_modified = 0;
}
}
+ /* bug fix for 302 moved_temporarily loop bug when using storeurl*/
+ if (mem->reply->sline.status == HTTP_MOVED_TEMPORARILY) {
+ const char *cloc = httpHeaderGetStr(&e->mem_obj->reply->header, HDR_LOCATION);
+ if (!strcmp(http->uri,cloc)) {
+ debug(33, 1) (“Loop Detected: %s Redirect to: %s\n”,
+ http->uri,cloc);
+ http->log_type = LOG_TCP_MISS;
+ clientProcessMiss(http);
+ return;
+ }
+ }
+ /* bug fix end here*/
stale = refreshCheckHTTPStale(e, r);
debug(33, 2) (“clientCacheHit: refreshCheckHTTPStale returned %d\n”, stale);
if (stale == 0) {
Dua kali posting di sini, squid.conf, 2x pula tidak muncul. Ini dibuka via proxy spidi. Saya coba posting lagi squid.conf dalam beberapa bagian:
squid-2.7.STABLE6
squid.conf hampir sama, hanya saya tulis prosedural sesuai dengan squid.conf default dan dikasih keterangan. Nanti saya posting lagi saya bagi dalam beberapa bagian.
squid.conf SQUID-2.7.STABLE6 bagian 1/2:
#
# SQUID 2.7.STABLE6
#
# Sebelum diconfigure squidnya dipacth dengan file diff yg dicomot dari
# http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
# Squid Cache: Version 2.7.STABLE6
# configure options: ‘–prefix=/usr/local/squid’ ‘–libdir=/usr/local/lib’ ‘–includedir=/usr/include’ \
# ‘–mandir=/usr/local/man’ ‘–libexecdir=/usr/local/lib/squid’ ‘–with-maxfd=2048′ ‘–with-aufs-threads=24′ \
# ‘–with-pthreads’ ‘–with-aio’ ‘–with-dl’ ‘–enable-gnuregex’ ‘–enable-async-io=24′ ‘–enable-storeio=aufs’ \
# ‘–enable-removal-policies=lru,heap’ ‘–enable-icmp’ ‘–enable-kill-parent-hack’ ‘–disable-delay-pools’ \
# ‘–disable-wccp’ ‘–disable-wccpv2′ ‘–enable-snmp’ ‘–enable-ssl’ ‘–enable-cache-digests’ \
# ‘–enable-default-err-language=English’ ‘–enable-err-languages=English’ ‘–enable-linux-netfilter’ \
# ‘–disable-ident-lookups’ ‘–enable-cachemgr-hostname=localhost’ ‘–disable-auth’
# ACCESS CONTROLS
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl localnet src 192.168.100.0/255.255.255.0
acl localnet src 192.168.0.0/255.255.255.0
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 88 # to my local webserver
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl PURGE method PURGE
acl snmppublic snmp_community public
acl QUERY urlpath_regex cgi-bin \?
acl store_rewrite_list url_regex ^http://(.*?)/get_video\?
acl store_rewrite_list url_regex ^http://(.*?)/videoplayback\?
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow localnet
http_access allow PURGE localhost
http_access deny PURGE
http_access deny all
http_reply_access allow manager
http_reply_access allow all
icp_access deny all
# NETWORK OPTIONS
http_port 8080 transparent
# zph
zph_mode tos
zph_local 0×30
zph_parent 0
zph_option 136
# MEMORY CACHE OPTIONS
cache_mem 16 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
# DISK CACHE OPTIONS
cache_replacement_policy heap LFUDA
cache_dir aufs /cache 50000 16 256
maximum_object_size 128l MB
cache_swap_low 98
cache_swap_high 99
# LOGFILE OPTIONS
access_log none
logfile_daemon /usr/local/lib/squid/logfile-daemon
cache_log /var/log/squid/cache.log
cache_store_log none
mime_table /usr/local/squid/etc/mime.conf
pid_filename /var/run/squid.pid
netdb_filename /usr/local/squid/var/logs/netdb.state
Hehehe …
Soalnya squid cache video ini mulai sdh jadi obsesi, saat gagal percobaan saya posting di sini, begitu berhasil saya posting pula di sini :)
squid.conf SQUID-2.7.STABLE6 bagian 2/2:
# OPTIONS FOR FTP GATEWAYING
ftp_user user@ableh.org
ftp_passive on
# OPTIONS FOR EXTERNAL SUPPORT PROGRAMS
unlinkd_program /usr/local/lib/squid/unlinkd
pinger_program /usr/local/lib/squid/pinger
# OPTIONS FOR URL REWRITING
storeurl_rewrite_program /etc/squid/store_url_rewrite_1.pl
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 10
storeurl_access allow store_rewrite_list
storeurl_access deny all
# OPTIONS FOR TUNING THE CACHE
## For squid-2.6
## We recommend you to use the following two lines.
# acl QUERY urlpath_regex cgi-bin \?
# cache deny QUERY
cache allow store_rewrite_list
cache deny QUERY
refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private
refresh_pattern ^http://(.*?)/videoplayback\? 10080 90% 999999 override-expire ignore-no-cache ignore-private
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
# HTTP OPTIONS
request_header_max_size 20 KB
reply_header_max_size 20 KB
request_body_max_size 0 KB
via on
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
ie_refresh on
vary_ignore_expire on
# TIMEOUT
connect_timeout 3 minute
request_timeout 7 minute
persistent_request_timeout 5 minute
# ADMINISTRATIVE PARAMETERS
cache_mgr admin-proxy@ableh.org
cache_effective_user nobody
cache_effective_group nogroup
visible_hostname proxy.ableh.org
# PERSISTENT CONNECTION HANDLING
client_persistent_connections off
server_persistent_connections off
# SNMP OPTIONS
snmp_access allow snmppublic localhost
snmp_access deny all
# ICP OPTIONS
icp_port 3130
log_icp_queries off
# DNS OPTIONS
check_hostnames on
allow_underscore on
ignore_unknown_nameservers off
ipcache_size 2048
ipcache_low 98
ipcache_high 99
fqdncache_size 8192
# MISCELLANEOUS
memory_pools off
cachemgr_passwd blablabla info config *
client_db off
reload_into_ims on
coredump_dir /cache
pipeline_prefetch on
Beberapa hari yl saya download video youtube, barusan download lagi url yang sama, lah kok ngulang dr awal ya? Kalau download lagi pada hari yg sama tetap cache hit. Keliatannya rule refresh_pattern ada yg perlu dicermati nich. PR buat pembaca … :) Soalnya cobanya nun jauh di sana, dari ujung sumatera remote ke server di ujung timur pulau jawa :D …
klo dari pengalaman saya, mungkin dikarenakan squid -z untuk mengaktifkan cache yang saya edit pada squid.conf, dikarenakan saya mencari angka yang pas buat cache agar larinya kenceng. Hanya caching youtube aja yang hilang ketika ketik squid -z, obyek yang lain tetap ada.
Coba aja mas arif buka movie di youtube, metacafe, atau streaming lainnya, sebelum di squid -z, coba mas buka lagi streaming yang udah di cache. Setelah itu squid nya di stop, kemudian ketik squid -z, kemudian aktifkan lagi squid nya, streaming yang udah di cache udah gak ada. Jadi klo pendapat saya, walaupun rule refresh_pattern di gedein sampe bertahun2, kalau udah di squid -z tetap aja streaming dimulai dari awal lagi.
Kalau gak di squid -z, Streaming yang udah dicache tetap ada terus sampai kapanpun. Ini mah pendapat saya lho ya… mohon dikoreksi kalau salah…. Untuk bro human gimana solusinya nih kalau udah di squid -z nih.. pusing euy T_T
mas adi, mas arief, mas materpop3, serta teman2…
udah baca http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion paling bawah ? maksudnya patch apa yah ? selama saya menggunakan squid versi 2.7 Stable3 belum dipatch, caching streaming gak ada masalah
Tolong jelasin yah maksudnya apa patch tersebut… thx
Saya persilakan mas arief yang jawab dah :D
Di url ini: http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
Simpan file diff yang ada di bagian bawah halaman sebagai squid.diff:
— src/client_side.c 2008-10-30 07:37:56 +0800
+++ src/client_side.c 2008-11-05 23:44:55 +0800
@@ -2399,6 +2399,18 @@
is_modified = 0;
}
}
+ /* bug fix for 302 moved_temporarily loop bug when using storeurl*/
+ if (mem->reply->sline.status == HTTP_MOVED_TEMPORARILY) {
+ const char *cloc = httpHeaderGetStr(&e->mem_obj->reply->header, HDR_LOCATION);
+ if (!strcmp(http->uri,cloc)) {
+ debug(33, 1) (“Loop Detected: %s Redirect to: %s\n”,
+ http->uri,cloc);
+ http->log_type = LOG_TCP_MISS;
+ clientProcessMiss(http);
+ return;
+ }
+ }
+ /* bug fix end here*/
stale = refreshCheckHTTPStale(e, r);
debug(33, 2) (“clientCacheHit: refreshCheckHTTPStale returned %d\n”, stale);
if (stale == 0) {
Setelah extract source squid, lakukan patch dengan option dry-run untuk tes file diff:
# tar jxf squid-2.7.STABLE6.tar.bz2
# cd squid-2.7.STABLE6
# patch -p0 –dry-run < ../squid.diff
patching file src/client_side.c
Hunk #1 succeeded at 2407 with fuzz 2 (offset 8 lines).
#
Kalau hasil patch dg option –dry-run seperti di atas berarti patch bisa dipakai, patch kedua kalinya dengan menghilangkan option –dry-run:
# patch -p0 < ../squid.diff
patching file src/client_side.c
Hunk #1 succeeded at 2407 with fuzz 2 (offset 8 lines).
#
Setelah itu baru configure lalu make squid.
ini cara ngepatch…
mohon dijelaskan maksud dan tujuannya patch yang mas arif terangkan diatas? sorry banget, gw nubie abezzzz
wah… ternyata squid versi 2.7 memang harus di patch euy, klo versi lain gw lom coba
Makasih banget mas arief….
n buat mas adi, mas arief, serta teman2, jangan bosan2 posting kesini yah klo ada update squid lagi, biar teman2 yang sering mampir kesini gampang klo ada update squid, daripada cari di google, susah n lama nyarinya :D
Bang itu setingan conf nya di psang di squidnya ipcop bisa ga yaa ?
soalnya aku liat setingannya conf nya tuh standart cache alaman site aja nah pengennya sih ada delay pool ma cache video streaming …
IPcop 1.4.21 squidnya 2.7stable6
Secara teori sudah bisa, tinggal di lihat perl-nya sudah support atau belum.
Dear,
Mo tanya mass.. dolo pakai squid waktu masih pakai fedora Sekarang server di ganti sama ipcop kira2 itu configrasi squidnya beda apa sama ? kalau sama kan tinggal copas conf yang lama ke dlm squidnya ipcop.
Mohon penjelasannnya …
Oh ya.. saya pakai ipcop 1.4.21 squid 2.7 stable6
Setingan squid confnya di pasang di squid ipcop bisa tak tuh ?
Hadooohhhh…
udah di patch masih juga loading lagi euy
gw kira bisa seterusnya…
apa lagi nih bro human T_T
Waduw
bro human…
diyoutube ada tulisan kayak gini :
“An error occurred, please try again later.”
klo gak pake perl youtube atau di disable di squid.conf file perl nya, baru jalan, cuma judul tertentu error muncul tulisan itu, kenapa yah ? ada solusi ?
Belum pernah nemuin kasus seperti ini.
Coba di cek di log squid, access dan error log, mungkin ada clue disitu.
udah solved bro
sepertinya ip youtube ada yang persis dengan ip facebook, gw dapet info dari forummikrotik, di squid.conf diisi skrip server_http11 on n http_port ip_proxy:80 act-as-origin http11, dan satu lagi gw pake ubuntu server, cache_dir nya pake type diskd euy, klo pake type aufs masih tetep aja ada error, aneh bener bro, kok beda diskd dengan aufs dalam segi meng cache streaming, kenapa bisa beda bro ? Padahal diskd bagusnya untuk keluarga BSD kan, bisa jelasin diskd dengan aufs bro
mas human… saya udah coba sesuaikan config dan cacheyoutube2.pl (kebetulan saya make squid-2.7.STABLE3). Kok gak ada yang masuk hit, miss terus walaupun udah saya coba buka situs youtube tersebut dari beberapa pc yang berbeda. apa ada perubahan yang terbaru mas… tolong donk update cara mencachenya. terimakasih sebelum nya.
Nb. Apakah squid-2.7.STABLE3 harus di patch agar bisa mencache (squid-2.7.STABLE, karena squid tersebut udah terinstall langsung dari ubuntunya nih)
terimakasih.
ganti repo nya sama repo kermic, disitu sdh squid 2.7 stable6. pake yg lokal biar cpt donlodnya…
saya pake ubuntu 8.10, tp reponya ambil dr ubuntu 9.10 dr ugm
Artikel ini bagus, mungkin yang perlu ditambahkan adalah alat atau tool untuk menganalisa squid cache video youtube. Alat bantu yang saya pakai waktu mengaplikasikan isi artikel ini:
1. youtube-dl, script untuk download video di link youtube. Nanti akan menghasilkan url yg bisa di-query oleh HEAD, wget dan juga squidclient.
2. HEAD ini package lwp-request (perl) untuk mengetahui apakah url yg diperoleh dari langkah 1 di atas ada informasi HIT atau tidak.
3. wget, untuk download video flv dari link yg dihasilkan oleh no 1.
4. squidclient, gunanya untuk menghapus content / video youtube yang telah dicache di lokal.
5. Dari link url yg dihasilkan di atas, bisa difeed-kan ke script cacheyoutube2.pl untuk melihat apakah scriptnya bekerja atau tidak.
Nantilah saya bedah, tapi di blog saya saja ya :).
Sip sip, ditunggu artikel mas …
Soalnya yang bocor sudah ditambal jadi ndak bisa nonton youtube lagi kayak nontok tv. Padahal ada rencana untuk download video youtube master konduktor herbert von karajan, simphony beethoven lengkap.
Betul bung kutukupret dengan cara juga berhasil untuk caching youtube.
Kak… saya mendapat error message sama kayak Bang andik:
2010/01/31 03:27:49| squid.conf line 4972: acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-$
2010/01/31 03:27:49| aclParseRegexList: Invalid regular expression '\.google\.[a-z][a-$': Invalid range end
2010/01/31 03:27:49| squid.conf line 4973: acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-$
2010/01/31 03:27:49| aclParseRegexList: Invalid regular expression '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-$': Invalid range end
2010/01/31 03:27:49| squid.conf line 4974: acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-$
2010/01/31 03:27:49| aclParseRegexList: Invalid regular expression '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-$': Invalid range end
2010/01/31 03:27:49| squid.conf line 4975: acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-$
2010/01/31 03:27:49| aclParseRegexList: Invalid regular expression '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-$’: Invalid range end
Sy install squidnya pake apt-get install squid, ga tau gmn caranya ngompile ulang pake –enable-gnuregex. Tolong dijelaskan step by stepnya. makasih…
Bos human… bisa tolong jelaskan, bagaimana kita bisa tau kalo link youtube selalu di berubah? maap nubi kang :D
di bandingkan saja dengan cache log squid yang lama.
Atau paling gampang terasa pada waktu squid sudah tidak caching lagi.
mas gmana ya kalau kita pakai Clark Connect ? atau sama aja settng ya ?
terima kasih atas infonya
Secara umum untuk semua linux sama.
Pastikan saja clark connect memakai squid 2.7X dan sudah tersedia perl.
sip dah
Bos human… emang itu harus di kompile sendiri ya?
Kalo misal saya pake squid yang dari apt-get install masih bisa ato ngga? Terimakasih..
Takutnya kalau pakai apt-get, fungsi regexnya bermasalah, karena tidak jadi default kompile dari squidnya
Terimakasih kang human, Saya memakai squid 2.7 STABLE6 ga perlu di compile ulang dan ternyata berhasil dengan mulus, sebelumnya saya memakai software videocache yg bayar itu (meski sy dapetnya ga bayar) dan mengalami bug sehingga harus me-request 2 kali, baru bisa ter-cache.
Tapi dengan jampi2 diatas, wuuuuzzz… jadi ngacir kang!
Matur sembah nuwun.. ^^
Sip dah kalau sudah bisa jalan, jangan biasakan memakai barang haram ya … he he he
Mohon Bantuan
1281035522.448 258366 192.168.99.1 TCP_MISS/200 21647496 GET http://v19.lscache4.c.youtube.com/videoplayback? – DIRECT/208.117.252.152 video/x-flv
1281035572.956 1133 192.168.99.1 TCP_MISS/200 19055 GET http://www.youtube.com/watch? – DIRECT/64.233.181.93 text/html
1281035573.688 122292 192.168.99.1 TCP_MISS/200 8171268 GET http://v5.cache8.c.youtube.com/videoplayback? – DIRECT/74.125.209.244 video/x-flv
1281035574.013 298 192.168.99.1 TCP_MISS/204 462 GET http://www.youtube.com/get_video? – DIRECT/64.233.181.93 text/html
1281035574.169 932 192.168.99.1 TCP_MISS/204 260 GET http://v22.lscache3.c.youtube.com/generate_204? – DIRECT/208.117.241.33 text/html
1281035589.145 1021 192.168.99.1 TCP_MISS/200 20486 GET http://www.youtube.com/watch? – DIRECT/64.233.181.93 text/html
1281035590.070 315 192.168.99.1 TCP_MISS/204 462 GET http://www.youtube.com/get_video? – DIRECT/64.233.181.93 text/html
1281035590.371 962 192.168.99.1 TCP_MISS/204 260 GET http://v14.lscache1.c.youtube.com/generate_204? – DIRECT/204.246.232.32 text/html
^C
kok ga ada yg HIT yach
bisa dibantu ?
ada cara tuk cache biar lama umurnya.. sama settingnya yang mana ya mas? tks
[...] apit1, caching untuk photobucket yang banyak di pakai di [...]
gimana mahu update pada squid 2.7 stable3..saya masih pakai clearos squid 2.6..saya mahu buat cache youtube mass bantuan dongg…:(
bro mohon bantuannya…gmna cara membuat delaypool di server clearos 5.1 yg benar,gagal terus bro,buat membatasi downlad trus menghendel streming you tube…kasian yg maen PB yg ngelak…oiya bro aku pake speedy 2mb….thanks bro…
acl video_youtube dstdomain .youtube.com
delay_pools 1
delay_class 1 2
delay_parameters -1/-1 8000/1000000
delay_access allow video_youtube
delay_access deny all
squid w ga bisa jga padahal dah ikutin caranya tetep aja banyak yg error,.apaannya yg salah ya,.w pake squid Version 2.7.STABLE6,.tolong dong di posting ulang aja biar ga ribet,.
saya mau tanya apakah squid contoh diatas masih bisa cache youtube, dahulu memang bisa, baru sadar sekarang youtubenya ga cache lagi di tempat saya. tq
#############################################
acl youtube dstdomain .youtube.com
cache allow youtube
## atau as simple just ##
cache allow all
#############################################
## kalau squid belum di-patch tambahkan ##
minimum_object_size 512 bytes
## pastikan maximum_object_size yg sesuai ##
maximum_object_size 2 GB
####################################################################
## Note: script yang ditampilkan human mungkin hanya untuk video youtube lama !!
####################################################################
terimakasih infonya mr x, jadi sekarang hanya se-simple itu ya, pakai squid versi berapa ?
memang betul saya tidak mengupdate script di blog saya.
Sudah tidak ada lagi mesin proxy untuk di oprek.
Sekarang sedang dipaksa maen poip2-an.
kalo udah kadarluasa info nya di hapus aja bro…