Using Usenet - teabagger of torrents

[retlix]

A few weeks ago a couple people suggested that I write a tutorial on Usenet. Most people these days use torrents. There are however certain advantages and disadvantages of Usenet as compared to torrents. For example, recent legal developments where ISP's have ratted out torrent users has made Usenet more attractive.

I don't know how many parts this will be since I'm in the process of writing it, but over the next couple weeks I should be able to give you all the information you will need.

So, let's start at the beginning.

History

Centuries ago (circa 1985), before there was an internet, people still liked to chat. So some of the proto-geeks who could afford it (back then, geeks got even less respect than now) would get a couple extra phone lines and a spare computer and set it up as a bulletin board service (BBS). Other geeks would dial in and post messages much like we do today on internet forums.

As the internet began to grow, a protocol was established for creating BBS's in cyberspace. In form and function they were much like a normal BBS except that they didn't exist in any particular place. Rather, these “newsgroups” were mirrored on whatever servers wished to mirror them.

The collection of internet newsgroups eventually came to be known as Usenet. This is not to be confused with the newsgroup provider Usenet.com.

Although newsgroups were originally meant to be a means of exchanging text messages, with a little encoding magic one could also upload binary files. Data is normally encoded in bytes of 8 bits but Unix-type data communication required that one bit be used for parity checking. That left only 7 bits for data. Converting from one form to another wasn't a real issue though. There was even a program that could be transmitted and run without any decoding. It was written in assembler using only the first 128 commands recognized by a CPU and therefore only 7 of the 8 bits.

Legend has it that the first binaries to be uploaded were pornographic in nature. That may or may not be true, but porn did become a popular theme for many newsgroups. And the desire to share one's porn collection with others became an important factor in the growth of the internet – even before the appearance of the Worldwide Web.

Today, Usenet has largely been replaced by web sites in regard to the distribution of pornography. However it is still a very popular medium for the exchange of warez (cracked software) and multimedia such as television shows and movies.

The main limitation of Usenet however was the fact that your access to a particular newsgroup depended mainly on your ISP. Each mirror site can determine which newsgroups to mirror.

Something called the “retention period” was and sometimes still is another problem. In addition to deciding which newsgroups to host, the mirror can determine the length of time for which it will keep posts made to the group. Ten years or so ago, my ISP only had a retention period of several days.

To be completely accurate though, ISP's often were not the actual host. Instead they would contract this function out.

Because of these limitations, as well as other issues such as privacy, companies arose that specialized in mirroring Usenet newsgroups. They are always subscription based and today have extensive retention periods. One of the largest, Giganews, now retains posts, including attached binary files, for approximately one year.

Another advantage of such a service is that for a small fee, you will have the option of accessing newsgroups via an encrypted link.

But more on that anon.

Next time: How Usenet works.

[retlix]

Usenet Basics

OK, nobody really cares about networking protocols and other geek yak (i.e., superfluous technical jargon spewed by geeks and similar to the ink a cuttlefish squirts when in danger). So we're going to try to stay close to the bone. By the end, you should be ready to start leaching.

Historically, there have been many newsreader programs – i.e., programs you would use to download binaries from Usenet. However, sometimes one program will provide advantages that are unmatched and will become the default standard.

Today, that standard is Newsbin. I am not hawking this program and if you don't want to use it, that's cool. But if you decide to use another program (like News Boy or News file grabber), make sure that it supports NZB files. You'll see why in a bit.

If you subscribe to Giganews, they have a version of the program that they sell for $25 but it only works with their service. But since GN has a one year retention period and doesn't block any newsgroups (AFAIK), I never bother looking for binaries anywhere else.

The advantage of Newsbin (and NZB compatible readers) is that you can aggregate all of the posts making up the file (or files) you want to download into a single NZB file which can be read by your news reader.

This may not sound like a big deal, so let me give you an example.

For many years there was no easy way to search all of Usenet. If you were interested in a particular TV serial for example, you would have to scan through all of the Usenet groups that you thought someone might upload that serial to. Maybe it would be alt.binaries.tv or maybe some other group. There was also the possibility that it would be cross-posted. In other words, the poster would tell his posting program to upload the files to multiple newsgroups.

Cross-posting was a good thing up to a point since it would make your files easier to find. But in extreme cases started to feel a lot like spam. It didn't really matter to the Usenet mirror site since they would only store one version of the file and just have pointers to it in the additional newsgroups.

Eventually someone came up with the idea of creating a Usenet index. There are probably several such web sites today but the one I use exclusively is binsearch.info. This site specializes in newsgroups used to upload binary files such as programs, multimedia, pix, etc. It does not provide an exhaustive search of all of Usenet, but it's pretty rare that I haven't found what I was looking for by using it.

So, you go to binsearch and enter the name of the tv serial and hit enter. It will return a list of all posts in the newsgroups that it indexes that have that string in the subject line. You can then check off the posts you want to download and there is a button labeled “create nzb.” If you click this, it will create the NZB file that will tell your news reader where all of the components of this binary are found.

You should also know that binsearch has an advanced search which can give you much better search results since it allows you to specify a number of parameters that the basic search doesn't.

Depending on how you have set up the file association parameters for NZB files, it will either ask you where it should save the file or it will automatically invoke your news reader and start the download.

THAT is why you want a news reader that supports NZB files. Otherwise you would have to go to the news reader, have it download the headers for that group and then check off each part of the archive you want to download.

That may not sound like a big deal, but when you consider that some groups have hundreds of millions of posts (you read that correctly), just downloading the headers can take a very, very long time. With an NZB file, you don't have to worry. Just search and leach (note: “leach” means to dl from a group or FTP site without contributing anything in return. While it used to be a very derogatory term, I would guess that most Usenet users are in fact leaches.)

So, at this point, you know everything you need to start dl'ing.

1.get an NZB compatible news reader
2.check with your ISP for the necessary parameters which will include, at a minimum, the name of the news server – usually something like news.my.isp or nntp.my.isp
3.use binsearch to generate an NZB for what you want to dl.
4.Invoke the NZB from your newsreader and chill.

If you have to specify a port setting, the default is 519, but I think most programs will pre-fill this for your. For SSL connections it's a different port.

One thing we haven't covered is parity files. You will see these in your dl's as PAR or PAR2 files. If your dl included all of the parts for the archive and no parts are corrupted, then you won't need the PAR files. I'll go over this next time.

WARNING: There is a lot of nasty malware on Usenet. It's gotten so bad, that probably 50% or more of warez that I've dl'ed recently has been infected. Usually, it will be the key generator or crack program that is infected, but not always. Video files (aside from WMV) files are harder to infect so the route used here will be some proprietary codec included with the download or the files will be in a self-extracting archive where the EXE portion is infected. There are other tricks that are also used so this list is not exclusive.

Therefore, I would recommend the following

1.Do not dl files that were posted less than 1 week ago. Actually, depending on how good your AV software is and how frequently you download updates, you might want to extend that out to a couple weeks or a month.
2.Be sure to scan files that have been dl'ed after they have been extracted. Many files will use RAR compression. Some newsreaders like Newsbin will un-rar automatically if you tell it to in the program settings. Also, some AV's like Avira will inspect archives up to several levels deep automatically. Some don't do either so you have to be sure to do this yourself
3.Use a non-standard multimedia program like VLC or Gomplayer to view multimedia files. There are too many security holes in Windows Media Player and it's too easy to get sucked in to going to some bogus, malware web site.

In other words, if you are not prepared for possible malware attacks, don't dl from Usenet.

Next time: Not sure yet. PAR files and whatever else I can think of.

[retlix]

Parity files

I'll start by saying that I have no fucking clue how these things works. As far as I'm concerned, it's magic.

Parity files are a way for you to recover parts of an archive that have been lost or corrupted. So let's say that I upload the latest episode of Opra (ew, just puked a little). It will probably be an AVI file that I've split into sections using Winrar. Let's say that I split it into 10 100meg parts. As long as there are no problems with the upload AND none of the parts get corrupted as they're being mirrored to other sites AND the people dl'ing don't have any issues, then it's all good.

Most of the time that will be the case. But it wasn't always so and even with broadband connections and premium Usenet providers, you can still run into problems.

That's where PAR files come in. The original poster will tell the PAR program that he wants, let's say, 10% redundancy with the RAR files. In other word, one of my 100 meg files could be lost and with the remaining files plus the PAR files, the PAR program can rebuild the missing file.

Also, the dl'er might get all 10 of the files, but some might be corrupt. That's also cool. The program can isolate the bad blocks in those files and as long as they are less than 10% of the total, it can rebuild each corrupt file.

How fucking cool is that? Like I said – magic!

The program I use for decoding parity files is Quickpar. The latest version seems to be 0.9, indicating that it is a beta version. However I think it's been in beta for at least 10 years, so I wouldn't get hung up on that.

When you install it, you will have the option to incorporate it into the file manager shell. I think that is the default, but if not, be sure to select that option.

Normally, the PAR files will have the same root file name as the RAR files, but not always.

If you try to un-rar an archive and it bombs, then just double click the associated PAR or PAR2 file. If you have enough PAR file blocks to cover those that are damaged or missing, you'll be good to go.

Even on a fast machine, the checking and rebuilding process can take many minutes. I've never timed it but I would estimate that to rebuild an entire 100meg file would take as long 20-40 minutes depending on your CPU and memory. It is an exceptionally CPU intensive process so it will slow you down. I've tried to run Quickpar on 5 or 6 archives at a time on a quad core and the machine became unusable until the process was finished.

If you are using Newsbin, you will not see any PAR files in your dl directory since by default, it puts those into the 'scratch' queue. If you don't have a broadband connection, this is the way to go since there is no point dl'ing them if you don't need them. But for people with broadband, fish around in the program options and change this setting so that they are automatically dl'ed.

[retlix]

reserved 3

[retlix]

reserved 4

[retlix]

Part 2 posted above.

If anyone has questions, please feel free to post them. I'm sure there is a lot of stuff that isn't completely clear so let me know what needs more explanation.

Happy leaching.

[retlix]

New post - #3 above on parity files.

It's been a rough couple days so that's all I've got for now. I'll try to have another installment over the weekend maybe.

This isn't usenet related, but to help prevent your browser from getting jacked by malware you may dl, check into these firefox addons

1. noscript
2. flashblock

Since most web based malware will attack via either scripts or flash, these are worth having. They only work with firefox though so if you're using IE, well, that's just wrong.

Another program that will work with IE and can help improve computer security is keyscrambler. It will prevent a key logger from reading anything you type into a web page. They have other versions that go beyond just web pages but they're not free.