1 / 55

INN

INN. InterNet News. Introduction to INN (1). Originally written by Rich Salz ISC took over development in 1996 http://www.isc.org/software/inn INN is a complete Usenet system innd – NNTP server, main program, inbound receiver Exchange news articles with peers

celine
Download Presentation

INN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. INN InterNet News

  2. Introduction to INN (1) • Originally written by Rich Salz • ISC took over development in 1996 • http://www.isc.org/software/inn • INN is a complete Usenet system • innd – NNTP server, main program, inbound receiver • Exchange news articles with peers • innfeed: outbound, feed articles to peers • nnrpd – NNTP reader server • Separate readers and peers • For readers: readers.conf • For peers: incoming.conf、innfeed.conf、newsfeeds • A lot of components…

  3. Introduction to INN (2)

  4. Introduction to INN (3) • Main configuration • inn.conf • Components • innd – main daemon for peers (and users) • incoming.conf • innfeed – NNTP feeder • innfeed.conf、newsfeeds • nnrpd – for users • readers.conf • Newsgroups database • active、active.times、newsgroups、history • Overview database • buffindexed.conf、ovdb.conf、overview.fmt • Spool • storage.conf、cycbuff.conf

  5. Introduction to INN (4) • Specific functions • Article and history expiration • expire.ctl • Handling of control messages • control.ctl • Submission addresses for moderated groups • moderators • Cron jobs • news.daily • Summery of INN log files • innreports.conf • Others • …

  6. INN Installation

  7. Install INN (1) • In FreeBSD ports • news/inn: INN 2.4.6 • 2009-06-08 - INN 2.4.6 was made EOL by the release of 2.5.0. • 2010-Mar-25 - INN v2.5.2 , ISC is pleased to announce a new bug fixed version of INN 2.5 • How to install INN 2.5.2? • Tarball: ftp://ftp.isc.org/isc/inn/inn-2.5.2.tar.gz • Dependencies: “Requirements” in http://www.eyrie.org/~eagle/software/inn/ • C compiler gcc: built-in • Perl and MIME::Parser: lang/perl5.8+、mail/p5-MIME-Tools、graphics/p5-GD • GNU make: devel/gmake • GnuPG: security/gnupg • Berkeley DB: databases/db47+ • Yacc implementation: devel/bison

  8. Install INN (2) • http://www.eyrie.org/~eagle/software/inn/docs-2.5/checklist.html • http://www.eyrie.org/~eagle/software/inn/docs-2.5/install.html • Set up account ‘news’ • Set up the home directory for ‘news’ • # mkdir –p /home/news && chown news:news /home/news • You *CANNOT* chown -R news:newsafter installing. Some bins are SUID root. • Login as ‘news’ • su - news / sudo -u news … • rlogin –l news localhost • Enable rlogind via inetd, and set ~news/.rhosts • localhost username news:*:8:8::0:0:News Subsystem:/home/news:/bin/tcsh

  9. Install INN (3) • The source • Fetch the tarball and put it in /tmp by fetch or wget(ftp/wget) • Extract it: tar xzf inn-2.5.2.tar.gz • Configure • ./configure --help • --prefix=$HOME • --with-http-dir=$HOME/public_html • --enable-largefiles cannot be used with --enable-tagged-hash • --enable-keywords • --with-perl • --with-berkeleydb=/usr/local • You can copy some environment variables from ports/news/inn • make -n CONFIG_DONE_INN=1 /usr/ports/news/inn/work/.configure_done.inn._usr_local • CC, CFLAGS, CXXFLAGS, LDFLAGS

  10. Install INN (4) • After configure • Modify Makefile.global Please check the following files before running make, to ensure that everything was set correctly. Makefile.global include/config.h include/inn/options.h include/inn/paths.h innfeed/innfeed.h ## Berkeley DB support. If this support is configured, anything linking ## against libstorage also needs to link against DB_LDFLAGS and DB_LIBS. DB_CPPFLAGS = -I/usr/local/include/db47 DB_LDFLAGS = -L/usr/local/lib DB_LIBS = -ldb-4.7 –lz DBM_CPPFLAGS = -I/usr/local/include/db47 DBM_LIBS = -L/usr/local/lib -ldb-4.7

  11. Install INN (5) • Compile INN • $ gmake (make) • Install INN • Login as ‘root’ • # gmake (make) install • To upgrade INN • # gmake (make) update • upgrade binaries, scripts, and manual pages, but not config files • Be sure to configure INN with the same options that was used previously. • Finally • Create mail alias for usenet@ (which is defined in --with-news-master) • Set .tcshrc setenv PATH $HOME/bin:$PATH setenv MANPATH $HOME/share/man:`manpath`

  12. Install INN (6) • Set up syslog • In /etc/syslog.conf • $ cd ~/log && touch news.crit news.err news.notice • # cd /var/log && ln –s ~news/log news • # /etc/rc.d/syslogd reload • Log rotate • Done by news.daily, which is triggered by cron # uncomment these if you're running inn news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice /var/log/news/news.notice

  13. Install INN (7) • Set up cron jobs • crontab for ‘news’ • $ crontab –e • news.daily • If you're using any non-CNFS storage methods, add delayrm to the above option list for news.daily. • ~news/log/expire.rm • expirerm file • rnews • Processing spooled messages to the server again 0 3 * * * ~news/bin/news.dailyexpireoverlowmark 0 3 * * * ~news/bin/rnews -U

  14. Install INN (8) • Copy start-up script from news/inn/files/innd.sh.in • Put it in /usr/local/etc/rc.d/innd • Modify ‘%%VAR%%’s • %%PREFIX%%  ~news (/home/news、/usr/local/news、…) • %%DBDIR$$  ~news/db • %%NEWSBASE%%  ~news • # chmod 555 /usr/local/etc/rc.d/innd • Modify /etc/rc.conf • A lot of configurations remain… innd_enable=“YES”

  15. INN Databases ~news/db/

  16. active (1) • List the newsgroups carried by INN. • Maintained using ctlinnd or control messages • Edit it directly: active.times, overview database • Format • Each newsgroup is listed only once • Each line specifies one newsgroup • Four fields separated by a space: • <name> <high> <low> <status> • cs.test 0000003681 0000000285 y • name: junk、control* • high is the highest article number ever seen • low is not guaranteed to be accurate, and should only be taken to be a hint. • Normally updated nightly as part of the expire process • If the lowest article number is greater than the highest article number, then there are no articles in the newsgroup.

  17. active (2) • Format • status: • y: allow posting and feeding • m: moderated, all posting must be approved • n: allow feeding only • j: fed articles are filed in junk group • x: disallow posting and feeding • =foo.bar: articles are filed in the newsgroup foo.bar • Get a close to complete newsgroups database file • ftp://ftp.isc.org/pub/usenet/CONFIG/ • active、newsgroups • tw.bbs.* • Big5 newsgroups for tw.bbs.*

  18. active.times • Provide a chronological record of when newsgroups were created on the local server • Format • Three fields • <name> <time> <creator> • cs.test 1128713744 news@news.cs.nctu.edu.tw • time: the time it was created, expressed as Unix timestamp • date -r <time> • creator: plain text intended to describe the entity that created the newsgroup • This field should be encoded in UTF-8 • Usually the e-mail address of the creator

  19. newsgroups • Contain a list of newsgroups and its short description • It is not necessary that all the groups carried by the news server be listed in the newsgroups file • It is also not necessary that all the groups listed in the newsgroups file be carried by the news server • If you use ctlinnd newgroup to manually create a group, only the active file is updated • edit newsgroups file to add a short description for the created group • Format • Each line consists of two fields separated by at least one tab • <name>\t<description> • cs.computer-center [系計中公告] (Moderated) • Preferred format: The total line length should be at most 79 columns. The description should start with a capital and not be more than 55 characters long • using UTF-8 for non-ASCII characters in description is recommended

  20. history (1) • Keep a record of current and recently expired articles • Format • [<Hash>]\t<date>\t<token> • [E7516D3A69051BC9712D093D52A4C67B] 1274105567~-~1274105565 @0305414C543100000000000E92A20000025E@ • Each line corresponds to one article • Hash: ASCII representation of the hash of the Message-ID header • date: three sub-fields separated by a ‘~’, each contains Unix timestamp • 1st sub-field: arrival time of the article • 2nd sub-field: ‘Expires:’ header or ‘-’ • 3rd sub-field: ‘Date:’ header, record of the posting time • token: a token of the article. • This field is empty if the article has been expired • Use sm utility to retrieve the article

  21. history (2) • grephistory • Query the INN history database via Message-ID • Output the token if the article is still on the news server • By use of sm to retrieve the article • makehistory • Initialize or rebuild INN history database • Also can be used to rebuild the overview database • makehistory –FOx • Be sure to stop innd and delete or zero out the existing database

  22. INN Configurations ~news/etc/

  23. Article Storage Format • The supported storage formats • tradspool • Articles are stored as individual text files, named by the article number, and divided up into directories based on the newsgroups name • article 12345 in cs.test would be stored as cs/test/12345 • timehash • Articles are stored as individual text files, and divided into directories based on the arrival time. • timecaf • Similar to timehash, but multiple articles are put in the same file • cnfs • Articles are stored in pre-configured buffer files • Advantages and Disadvantages • http://www.eyrie.org/~eagle/software/inn/docs-2.5/install.html#S6 • INN can support all four at the same time

  24. Overview Storage Mechanism • Overview is for readers • Three overview mechanisms • tradindexed • Uses two files per newsgroup, one containing the overview data and one containing the index • Fast for readers, but slow to write to • buffindexed • Stores overview data and index information into buffers, which are preconfigured files defined in buffindexed.conf, similar to CNFS buffers • Slower for readers • ovdb • Stores overview data in a Berkeley DB database • Fast and robust, but may require more disk space

  25. Configration outline • Access controls • Peers: incoming.conf、newsfeeds、innfeed.conf • Readers: readers.conf • Storage • cycbuff.conf、storage.conf • Overview • ovdb.conf、buffindexed.conf、overview.fmt • Other controls • Expiration: expire.ctl • Moderators list: moderators • INN report: innreport.conf • Default subscriptions for a new client: subscriptions • …

  26. inn.conf (1) • The primary general configuration file for all INN programs • Most likely to need setting • organization: “College of Computer Science, NCTU” • It will be filled in the ‘Organization:’ header for posts • ovmethod: ovdb • The overview mechanism • General Settings • server: csnews2.cs.nctu.edu.tw • Default NNTP server for nnrpd, actsync, nntpget, getlist • Feed Configuration • pathalias: news.cs.nctu.edu.tw • News servers within a particular organization to add a common identity • Article Storage • extraoverviewadvertised: [ ] • Extraoverviewhidden: [ ] • See inn.conf(5)

  27. inn.conf (2) • Reading • noreader: true • For separately running innd and nnrpd • Posting • nntpauthsender: true • Add ‘Sender:’ header to local posts containing the identity assigned by readers.conf • Monitoring • doinnwatch: false • Disable the innwatch • Logging • docnfsstat: true • Start cnfsstat when innd is started • logcycles: 3 • How many old logs scanlogs keeps

  28. incoming.conf • Specify who are permitted to connect to your host and feed it articles • Connections from hosts not listed in this file will be rejected or be handed off to nnrpd and checked against the access restrictions in readers.conf • Three types of entries: key/value, peer, group streaming: true # streaming allowed by default max-connections: 8 # per feed groupcs { max-connections: 16 patterns: "*,@local*" peer CSBBS { hostname: "bbs.cs.nctu.edu.tw, 140.113.168.8" } peer FREEBSD { hostname: "freebsd.cs.nctu.edu.tw, 140.113.17.209" patterns: "!*,freebsd.*,mailing.*" } } group peering { max-connections: 16 patterns: "*,@cis.*,@cs.*,@csie.*,@local*" peer NCTUPEER { hostname: "news-peer.nctu.edu.tw, 140.113.250.119" max-connections: 8 } }

  29. newsfeeds (1) • Determine how incoming articles are redistributed to your peers • Organized as a series of feed entries • Each entry is composed of four fields separated by ‘:’ • 1st field: <name>/<exclude>,<exclude>,… • If you don't want a feed to receive articles from a certain source, then … • 2nd field: <newsgroups-pattern>/<distrbution-list> • wildmat pattern matching syntax • These are simple wildcard matches using the ‘*’ as the wildcard character • wildmat patterns can be specified in a comma-separated list • The last pattern in the line that matches the group name is used • Patterns beginning with ! mean to exclude groups matching that pattern • *,!comp.*,comp.os.* • wildmat patterns also support "poison" patterns (patterns starting with @) • “misc.*,@misc.bar” vs. “misc.*,!misc.bar” • For a article crossposted between misc.foo and misc.bar • For a article posted only to misc.bar

  30. newsfeeds (2) • 3rd field: comma-separated list of flags • http://www.eyrie.org/~eagle/software/inn/docs-2.5/newsfeeds.html#S2 • 4th field: multi-purpose parameter, meaning depends on the flags in 3rd field • Feeding peers via innfeed • Site settings ## Uncomment if you're using innfeed. ## Add "-y" as an option to innfeed to use the name of each feed as the ## name of the host to feed articles to; without "-y" an innfeed.conf ## file is needed. # innfeed funnel master. innfeed!\ :!*\ :Tc,Wnm*:/home/news/bin/innfeed-y news.uu.net/uunet\ :*,!junk,!control,!control.*/!foo\ :Tm:innfeed!

  31. newsfeeds (3) • Other examples • News overview used when “useoverchan: true” in inn.conf • CS Archive-SQLite • Mirror nctu.talk and tw.bbs.campus.nctu # News overview. overview!:*:Tc,WnteO:/news/bin/overchan archive_sqlite_cs!\ :!*,cs.*,@cs.cc.*\ :Tc,Wn:/news/bin/archive-sqlite.pl cs # mirror mirrorpost!/mirrorpost,netnews.cs.nctu.edu.tw\   :!*,nctu.talk,tw.bbs.campus.nctu\  :Tc,Wf:/news/bin/mirrorpost.pl nctu.talktw.bbs.campus.nctu

  32. innfeed.conf • Used to control the innfeed program • Three types of entries: key/value, peer, group (as incoming.conf) • peer <name> { # body} • <name> is the same as the site name in ‘newsfeeds’ • If you do not specify -y in ‘innfeed!’ entry in newsfeeds, then … • In newsfeeds • In innfeed.conf NCTUCSBBS/bbs.cs.nctu.edu.tw,BS2\ :!*,cs.*,tw.bbs.*,twbbs.*,taiwan.*,nctu.*,mailing.*, :Tm:innfeed! group BBS { max-connections: 2 streaming: false port-number: 7777 peer NCTUCSBBS { ip-name: bbs.cs.nctu.edu.tw } }

  33. readers.conf (1) • Specify access control for nnrpd • Two types of entries: parameter/value, configuration groups • <parameter>: <value> • No way to continue a line on the next line, and no way to have a value longer than about 8,180 characters • Configuration groups • auth: provides mechanisms to establish the identity of the user • access: given the user's identity, what that user is permitted to do • readers.conf is a two-step process • Put all of the auth groups first, and all of the access groups below, last match • A user identity, as established by an auth group, looks like an e-mail address "<username>@<domain>“ • When matching access groups the last matching one in the file is used to determine the user's permissions • http://www.eyrie.org/~eagle/software/inn/docs-2.5/readers.conf.html#S4

  34. readers.conf (2) • auth group • Generate the user identity like • <UNKNOWN>@EVERYWHER • liuyh@NCTUCSCC • nnrpdauthsender • http://www.eyrie.org/~eagle/software/inn/docs-2.5/readers.conf.html#S2 auth "EVERYWHERE" { hosts: "*" default: "<UNKNOWN>" default-domain: "EVERYWHERE" } auth "NCTUCSCC" { hosts: "bsd*.cs.nctu.edu.tw, linux*.cs.nctu.edu.tw, sun*.cs.nctu.edu.tw" default-domain: "NCTUCSCC" res: "ident -t" }

  35. readers.conf (3) • access group • newsgroupsvs. read + post • access: R(read)、P(Post)、I(IHAVE)、A(Approver)、N(NEWNEWS)、L(post to non-local posting newsgroups, [jnx] in active) • http://www.eyrie.org/~eagle/software/inn/docs-2.5/readers.conf.html#S3 access "READING_LIMIT" { users: "*@*“ access: "R" read: "!*,cs.help,freebsd.*,gmane.*,mailing.*" } access "CS" { users: "*@NCTUCS, *@NCTUCSCC, *@NCTUCISCC, *@NCTUCSIECC“ access: "RP" newsgroups: "*,!control*,!cs.prsystem,!cs.cc.*,!csie.cc.*,!cis.sysadmin.*,!csie.sysadmin.*,!cis.computer-center.staff,!cis.syslog.*,!cis.bbs.admin.*,!junk" }

  36. cycbuff.conf • Define the cyclical buffers that make up the storage pools for CNFS • To use any sysbuff larger than 2GB: --enable-largefiles • Formats • cycbuffupdate:<interval> • refreshinterval:<interval> • cycbuff:<name>:<file>:<size> • No longer than 7 characters in <name>, and than 63 in <file> • <size> is the length in KB (2GB=2097152) • cycbuff:BIG80:/home/news/cycbuffs/BIG80:524288 cycbuff:BIG81:/home/news/cycbuffs/BIG81:524288 • dd if=/dev/zero of=/home/news/cycbuffs/BIG80 bs=1k count=524288 • metacycbuff:<name>:<buffer>[,<buffer>,...][:<mode>] • Mode: INTERLEAVE(default) or SEQUENTIAL • metacycbuff:BIG8:BIG80,BIG81

  37. storage.conf (1) • Contain the rules used to assign articles to different storage methods • Format • <methodname>: tradspool、timehash、timecaf、cnfs、trash • <storage_class>: a unique number in 0-255 to identify the storage • Used in expire.ctl, timehash, and timecaf • <options>: only for CNFS to specify the metacycbuff name method <methodname> { newsgroups: <wildmat> class: <storage_class> size: <minsize>[,<maxsize>] expires: <mintime>[,<maxtime>] options: <options> exactmatch: <bool> }

  38. storage.conf (2) • An example method tradspool { newsgroups: cs.cc.* class: 0 } method cnfs { newsgroups: cs.*, eecsep.*,nctu.* class: 1 size: 0,8191 options: NCTUCS } method cnfs { newsgroups: comp.*,humanities.*,misc.*,news.*,rec.*,sci.*,soc.*,talk.* class: 2 options: BIG8 }

  39. overview • ovdb.conf • Parameters for tuning ovdb, no need to modify • buffinedxed.conf • <index>:<filename>:<size> entries (similar to cycbuff) • <index> is unique in 0-65535, <size> is in KB • overview.fmt • No longer used by INN since v2.5 • Use extraoverviewadvertised and extraoverviewhidden in inn.conf

  40. expire.ctl (1) • Default configuration file for expire and expireover, define how long • History entries for expired or rejected articles are remembered • /remember/:<days> • Articles stored on the server are retained • <wildmat>:<flag>:<min>:<default>:<max> • if ‘groupbaseexpiry: true‘ in inn.conf (default) • Last match • <wildmat>: ! and @ not permitted, and only a single expression • <flag>: M(moderated)、U(unmoderated)、A(all)、X(delete from all groups) • <storage_class>:<min>:<default>:<max> • if ‘groupbaseexpiry: false‘ in inn.conf • <storage_class>: defined in storage.conf, ‘*’ to specify a default for all classes • <min>, <default>, <max>: decimal number of days, fraction is allowed or ‘never’ • <min>, <max>: come into play with an ‘Expire:’ header • <default>: used as the expiration period for most articles

  41. expire.ctl (2) • An example # Keep expired article history for 11 days, matching artcutoff plus one. /remember/:11 # Most articles stay for two weeks, ignoring Expires: headers. *:A:14:14:14 # Accept Expires: headers in moderated groups for up to a year and # retain moderated groups for a bit longer. *:M:1:30:365 # Keep local groups for a long time and local project groups forever. example.*:A:1:90:90 example.project.*:A:never:never:never

  42. moderators • Determine submission addresses for moderated newsgroups • Format • <pattern>:<address> • First match for <pattern> • More specific patterns should be listed before general patterns • <address>: a simple e-mail address • At most one ‘%s’ may occur and be replaced by the name of the newsgroup, with all ‘.’ in the name changed to ‘-’ ## CS moderated newsgroup cs.computer-center:newsTA@cs.nctu.edu.tw ## Public hierarchies with exceptions. ffm.*:%s@moderators.arcornews.de fj.*:%s@moderators.fj-news.org medlux.*:%s@news.medlux.ru ## Direct all other public hierarchies to the master moderator database. *:%s@moderators.isc.org

  43. innreport.conf • Configuration file for innreport • Default parameters • section default: • html true; • title "Daily Usenet report for mynews"; • footer "Local contact: usenet@mynews"; • html_css_url "innreport.css"; • A lot of options for report content and appearance

  44. Control Messages

  45. Control Messages (1) • Cancels are handled internally by INN • Others are processed by controlchan (newsfeeds) • The actions of controlchan are determined by control.ctl • ftp://ftp.isc.org/pub/usenet/CONFIG/control.ctl • newgroup、rmgroup、checkgroups • control* • control • control.cancel • control.checkgroups • control.newgroup • control.rmgroup

  46. Control Messages (2) • Authenticate control messages based on the ‘From:’ header • Obviously perilous and control messages are widely forged • Many hierarchies sign all of their control messages with PGP • controlchan knows how to do this (using pgpverify) without additional configuration • PGP key import • ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS • $ gpg --import PGPKEYS • $ ln -s ~/.gnupg ~/etc/pgp

  47. Control Messages (3) Path: csnews2!csnews.cs.nctu.edu.tw!news.cs.nctu.edu.tw!ctu-peer!ctu-gate! news.nctu.edu.tw!newsfeed.berkeley.edu!ucberkeley!solaris.cc.vt.edu!news.vt.edu!guardian.oit.duke.edu!news.glorb.com!news2.glorb.com!usenet.stanford.edu!usenet-its.stanford.edu!bounce-back From: group-admin@isc.org Newsgroups: sci.physics.acoustics Subject: cmsg newgroup sci.physics.acoustics Control: newgroup sci.physics.acoustics Approved: group-admin@isc.org Date: Sat, 15 May 2010 13:00:01 -0700 Message-ID: <cmsg-20100515200001$2fac@isc.org> X-PGP-Sig: 2.6.3a Subject,Control,Message-ID,Date,From,Sender \011iQCVAwUBS+79QsJdOtO4janBAQGNsAP7BAj2Vl4LS2RoIZmYTfHc9GFg/nFve8Hj \011iOYqpW+WiF7pI6JaNzPaeS/Y3Dh7G9HRjjhuYbQ/+bTUHWeDItRFBbGVe3t+yXEv \011TbQ/NqWqMug/OKujGsOs9wy1HQKZBErDIfxCf+XD6JenRccrBPAZoCMdSFieztZe \011vFU9B4Az+ew= \011=d9AE Xref: csnews.cs.nctu.edu.tw control.newgroup:168581 For your newsgroups file: sci.physics.acoustics\011Topics in acoustics and vibrations.

  48. Control Messages (4) • control.ctl • Last match • Format • <type>:<from>:<newsgroups>:<action> • <type>: all, newgroup, rmgroup, checkgroups, … • <from>: match the ‘From:’ header • <action>: • doit、drop、log、mail、verify-pgp_userid • Encoding of newgroup and checkgroups control messages • /encoding/:*:nctu.*:big5 • /encoding/:*:tw.*:big5 • /encoding/:*:fido7.*:koi8-r • /encoding/:*:fido.*:utf-8 • /localencoding/:utf-8 • Checkgroups • /maxdocheckgroups/:*:*:10

  49. More on INN control and operations

  50. Start and Stop • Before starting, check the configurations • $ inncheck -a -v -f -pedantic -perm • /usr/local/etc/rc.d/innd • su -fm news -c "/home/news/bin/rc.news start" • su -fm news -c "/home/news/bin/rc.news stop" • If you run nnrpd separately • su -fm news -c "/home/news/bin/nnrpd –D" • su -fm news -c "killall nnrpd"

More Related