August 2009 - State of the Site

Once a month now I'm going to start doing a short post that I've dubbed "State of the Site". I intend to cover what our eventual goals are, how we're working towards those goals, how we're doing financially, and any other relevant information.

Where We've Excelled...

This month has been an amazing one for the website. We broke some new records for # of visitors, we overtook some of our most popular competitors (Plube.com and Goleech.org), and we released some very cool new features. In addition, we've had a surge of additional content, caused by a new tool for our top administrators, which helped make specific tasks much easier for them.

New Features

I'd like to highlight a post that was made just a few hours ago:

http://www.sinlesslinks.com/Movies/worlds-greatest-dad-2009-ppvrip-xvid-devise/

This is an excellent example of what the new "Multiple File Hosts" feature allows. Originally only the top Rapidshare linkset was posted, and the "AutoBot" transferred those links over to MegaUpload and MegaShares, while an excellent submitter Kan submitted linksets for several other file hosts, to make it a post that likely will never be seen without at least one set of working links.

The AutoBot's ability to transfer links between sites isn't perfect, but it does the best it can. Storage.to has a hard time with files sometimes, and it takes a lot longer, but we feel it's a great addition. Eventually we'd love to support auto transfers to Hotfile.com, but there are so many restrictions on uploading and downloading there, that it will be a miracle if we're ever able to pull that off.

Beating the Competition

Here's a great graph demonstrating how over the past 3 months we've overtaken a few of our top competitors, and how this month we've steadily been climbing above them. I personally have nothing against either of these sites, we're all just out to accomplish different things. However I also don't recommend them, due to excessive ads and pressure to subscribe on one and content inappropriate for children on the other (not that we're perfect in either area either though).

Note that we're the blue line.

Where We've Had Problems...

Most notably this past month were bugs in the AutoBot system. For about a week and a half it was consistently adding MegaShares linksets that didn't have any links in them. This was due to some improper linkchecking by the AutoBot system, and it has since been resolved.

In addition, we ran into some communication problems this past month or so. I learned I need to make a more concerted effort to regulate the quality of the titles, etc. we use, and to make sure I treat everyone fairly. Too often good intentions can get lost in translation, and its important that I do my best to prevent this.

Where Your Money Went...

The money for the second server was a huge help this past month. The AutoBot is a direct result of that, as well as the new administrative tool that I mentioned earlier. In addition, I recently launched a program I've titled "SinlessLinks OS", which was made possible by our new second server.

SinlessLinks OS?

Using the new server, I've set up a "sandbox" (aka a Development Area), and I've given out access to this development server to a few programmers that expressed an interest in helping out in the past. Together we're going to be reprogramming the entire site, and adding a whole slew of features. I hope to finish up this "overhaul" in December or January, and when we do, I'll be making all the code for the website Open Source (hence the OS in SinlessLinks OS).

That means that early next year, anyone that wants to can have a copy of the website, along with the entire database. This is being done for several reasons...

First, I want to make sure the website "lives forever". One way to help make sure this happens is to make sure that everyone has a copy, and that if anything happens to the website, someone else can put up an exact copy, replicating what the site looked like the day it went down.

Second, I want the website to always be improving. I don't have time to always work on it though, and so this will allow any well-meaning, skilled developer to make additions to the site.

Third, I can't do this forever. The career path I'm pursuing would be a conflict of interest with what SinlessLinks is, and this is the right way for me to be able to "hand over" the site to the public. No worries though, I'm not going anywhere anytime soon, I'm just making all the preparations way in advance so that nobody will miss me when I'm gone :-).

The Vision...

You've caught a glimpse of the vision in what I've written above, but I figured I'd give you what my current "official" vision looks like:

Welcome to SinlessLinks OS, where every possible online file host is supported, as well as a multitude of torrent indexing websites, and possibly direct links to torrent files themselves. Every quality version of every post is easily accessible, and links are automatically transferred between dozens of file hosts. You can submit a review, rate a particular quality version, or an entire television series. Anyone can post items, but only those by approved posters or given enough thumbs up in the free-for-all area will actually get seen by the majority of the visitors. Any set of posts or new “non-posts” can be formed into a “collection” that’s easily brows-able. The search feature is updated with the new information constantly, and delivers consistent and reliable results, along with matching users, reviews, requests, and more. The best part – the entire operation is open source, copies of the latest code and database are available on the website every day, and the site is handed from caretaker to caretaker to ensure that it will never die.

~SinlessLinks

Support Added for Storage.to and Hotfile.com

Hey Everyone,

I just wanted to write a quick post to let you guys and gals know that we've added support for Storage.to and Hotfile.com links.

This means that all our linkcheckers now properly check these kinds of links, and you will be able to properly filter these sites just like Rapidshare and MegaUpload, etc.

You probably have also noticed that we have a lot more MegaShare links now, and we hope to have just as many Storage.to and Hotfile.com links within a few weeks. Within another month or so, odds are every file hosting site that you might want will be available on almost every post we have.

Thanks again to our donators for making this possible. We feel it's a major step forward. I realize that not everyone will like having all types of links like this, but all they have to do is turn it off so they don't see them. Getting links of all types will help everyone, because the site will grow which gives us opportunities to do more. In addition, we'll eventually be transferring every type of link to every other type of site, so it won't matter what links are posted, everyone will benefit.

~SinlessLinks

Well Shoot! What's been going on with the Shoutboxes?

I wanted to take an opportunity to post this little notice letting everyone know we apologize about the shoutboxes being down. After all, we just purchased another shoutbox for our members with the full idea of letting them finally get to use the box without the ever present "Shoutbox exceeded Bandwidth" issues. But more to the point, we are aware of the problem and wanted to let you know that it isn't anything we caused. We just have to wait it out and let the company get their servers back online.

In the meantime, feel free to comment crazy excitement here as I personally know I've missed all the exciting bantor. Hope all is well though with you, and Sinless, Lufrebent, and I look forward to continually offering additional exciting features.

New Server Put To Use: RS to MU

Hey Everyone!

You all were kind enough to donate your money so that we could rent a second server. Well, beginning today you'll get to see that money put to good use.

Starting today, every single Rapidshare link that is posted on the site will AUTOMATICALLY be transfered to MegaUpload and added to the post. These links will be exact mirrors of the Rapidshare links, and therefore interchangeable. The whole process of transfering the files to MegaUpload takes about 5 minutes (depending on how many links there are, it may take a bit longer).

During Rapidshare "Happy Hour" times (2am to 10am CET), the same system will work to convert all our past posts over to MegaUpload as well.

Q: Can I have it transfer it to my MegaUpload Account?

A: No, unfortunately that would be far too complicated, and we don't want to handle other people's MU passwords.

Q: So you're stealing points that are rightfully mine? How's that Sinless?!

A: We're not stealing any points. MegaUpload does not award points for any files over 100mb, and about 90% of all the links posted these days are over that limit. And any of the sparse points we might get will go only towards keeping that account alive, so that your donation money doesn't have to pay for it.

Q: Can I still post MegaUpload mirrors?

A: Yes! Of course! Just because this automatic system will always post them, doesn't mean you can't. The more linksets the better!

Q: Can I setup a system like this?

A: Not one exactly like ours, however if you use Google, you'll find plenty of other sites to help you do this kind of thing.

Q: I did Google it, and it says you have to have a Rapidshare Premium account to do it.

A: Yes you do. Our system uses about 8 different Rapidshare accounts, and it is predicted that it will use almost all their available Download Traffic every day. Yes that means we'll be using between 80-800GB of download traffic a day. Be glad it's not your Rapidshare account.

Q: How can I help?

A: Submit Rapidshare Links, Post Rapidshare Links, spread the word about how we have MegaUpload stuff now, and a little monetary donation will go a long way to helping make sure we can keep the server.

What's Next?

The next thing(s) the new server will be used for...

1) Transfer all MegaUpload Links posted by users over to Rapidshare.

2) Transfer all RS and MU links over to MegaShares, MediaFire, FileFactory, Storage.to, hotfile.com, etc.

~SinlessLinks

We Have Liftoff!

We were able to get a server for the price we originally intended. The processor is 200Mhz slower than I would have liked, but we're still very excited about our new acquisition. We paid for the server today and should be able to start using it in the next 24 hours.

I'd also like to take a moment to simply honor the achievement that humankind accomplished 40 years ago. True, this is the day we *left* the moon, but that's a commonly overlooked achievement as well. Space Exploration would likely not be what it is today if something had gone wrong to prevent the astronaughts from returning to earth. Thank you to the brave Men and Women acrossed the world that work every day to make science fiction a reality. I look forward to sending you all a Google Wave from my Firefly class vessel someday.

~SinlessLinks

Best Laid Plans of Mice and Men

Quick update on our second server...

Today we unfortunately found out that we had to pay three months in advance for our second server. This was dissapointing, but nevertheless it didn't stop us.

However, by the time we were able to raise the funds, the specific server we were attempting to rent was no longer being offered by our hosting company. So to help rectify this, I've emailed our hosting company asking if we can still get the rate that was advertised this morning. I believe we'll succeed in getting the rate, but if not, we will unfortunately either not be able to get the server at this time, or have to look for a different company to host it with, as the next cheapest server we could get is around $20 more a month, which makes it more expensive than our current server.

Just wanted to update anyone interested. I'll write more tomorrow, when (hopefully) we get confirmation that we can have the server for the price we were expecting.

~SinlessLinks

A Second Server on the Way...

Just a few hours ago, I wrote about how I had dreams of getting a second server to be able to do some cool extra things for the website. Well thanks to 7 very generous donators, we have raised more than enough money to make that happen.

I'm going to do my best to make sure that I am very transparent about what we use the server for, and keep everyone up to date on how it's helping out the site. I want everyone to know exactly what the money went towards, without any questions or doubts about our motives.

I'll be ordering the server asap tomorrow (I have to wait for our hosting company to be open during business hours), and it'll take a day or two before we're setup and ready to use it, and then I'll start letting you guys know exactly what we're going to be doing with it.

Once again, thank you very much for your generosity, and stay tuned for more updates.

~SinlessLinks

A Second Server for Related Projects...

Want to help the site improve? We have a need! Read on to find out how...

Something I've wanted to do for a long time is get a second server that can be used to do things like...

1) Keep Rapidshare links from dying because of inactivity.

2) Run our link checker from that server to keep this one fast.

3) Have a remote server to store backups on.

4) Give us the ability to try new projects.

5) Start developing our own ShoutBox system at some point.

6) Potentially develop a system that would automatically (without downloading anything) mirror all Rapidshare links on MegaUpload/MegaShares and vice-versa (and other sites as well).

7) Possibly setup an SVN test server to allow others to collaborate in improving the site with me.

SO, HOW CAN YOU HELP??

Adding this second server would cost roughly $50 USD a month. Our current cost ranges from $70 a month to $85 dollars a month (depending on if we have to pay for the shout boxes or other quarterly server costs).

This would mean that our monthly site costs would run between $120-$130 a month. I feel like that's asking a lot.

However I'd rent the server on a monthly basis. So aside from shutdown fees, we'd be able to drop the second server whenever we need to if we were unable to get the funds.

In addition, the site ads are helping out some now, which would mitigate any increase in donations that would be required.

I also believe that the addition of this second server will help bring in more visitors, because of some of the things we'll be able to do with it.

So, that said. I thought this would be a great month to mention it and ask for your help.

Why this month? Because we already have $45 ready for this month's server fees, which would mean that the second server is pretty much already paid for, and we only need to raise our normal amount.

So it's up to you guys. If we can get the extra money for this server, I think you guys will be happy with what we're able to do with it over time. And if we get more visitors, there will be more people to share the burden of the cost.

And I want to reiterate that we don't make any money off the site. Everything donated goes right into paying for server costs. All of our admins and site partners, etc. are strictly volunteers.

I'd love to be able to do this, but I'll leave it up to you guys, since you finance everything.

~SinlessLinks

Site Architecture of a Rapidshare Site

When the site started up, I knew very little about MySQL or proper database design. Thankfully I got pretty lucky and what I designed worked, though it's far from ideal.

You may have noticed that at SinlessLinks we have a few features that other websites don't have. The most noticeable is the new way to select just what websites you want to view. This is made possible because of something that I've called a "linkset". I can't remember for the life of me if I heard that word somewhere else or coined it as I created the site, but it's that part of the database design that makes it all possible.

A LinkSet is intended to be a collection of links that contains all the links necessary to download and unrar a particular item. Or in the case of YouTube, it would contain all the links necessary to watch a full tv-episode or movie.

Often on other sites you'll see them post a "primary" set of links, and then some alternate sets. On a few "news only" blogs, you'll see a lot of different sets of links in the comments, like you have started to see here more often. This is the "problem" I was trying to solve with LinkSets. Just because the main links that the original poster provided go dead, doesn't mean that the post should be deleted. And anyone that's trusted should be able to add links of any type to a post, and have them show up, but not clutter up the page. It needed to be easy to look at and quickly grab the links you want.

And in TV-Series, I had seen many posts where there were 3 sets of alternates for each episode. Now this is great for making sure there's always working links, but when you're looking at something like Seinfeld or The Daily Show, it's just not practical to have that many different sets of links all visible at the same time. So it was decided that while many linksets can be added, only one will show up per episode. Then as soon as one set goes dead, the other would come into place. (Now mind you this has a long way to go before it's perfect -- we need an easy way to check and flag the links in a single episode or season so that the dead links will go away and working ones will show up -- and we also need a way to view all the links available for a particular episode, a page devoted to that episode mind you.)

Okay, so now that what we were trying to achieve has been mentioned, a few words about how I achieved it, what I learned, and how I would do it differently if I could start over.

The main post is stored in a table of its own. However, this table only stores information like the url, the title, the year, the creator of the post, when it was created, etc. It doesn't actually store the body of the post. Instead, it is linked to a "linksets" table, which stores cohesive sets of links. These linksets have a "Type" which a number standing for Rapidshare, MegaUpload, etc., they have an author, and they have a "status" (which is normally 1 for good-working links, but the linkchecker can switch it to 2 for dead, and we also have 3 for "submitted" but not yet approved).

In addition to linksets, their is a table for series, seasons, and episodes, which help store all the information to make up a proper TV-Series post.

All posts are created when you view them. This has it's up sides and down sides. On the up side, it means that as soon as a change occurs (such as adding a linkset, or adding/deleting an episode) everyone sees that change immediately. The downside is that even one little portion of bad code can slow down or even crash the whole site, because it is run hundreds if not thousands of times a minute. Eventually I hope to create a "cache" system that builds out the post anytime a change is made, and then that is used to display the post to the user.

Some things that I would change about this architecture...

There's no need for the "series" table. It contains redundant information to the posts table, and needs to be eliminated eventually. Not sure if I will ever do this though, as it would be tons of work, with very little speed improvement.

There's no need for the "links" table that I use. Every line in a linkset is stored in a separate row in a links table. The original thought was to be able to store the status of a single link as being dead and show that every time someone views a post. But after trying that, it was just overkill. Besides, if one link in a linkset is dead, it's normally safe to assume that people can't properly use the content at all.

I originally connected a linkset to a post (if it was a regular movie, etc.), or an episode (if it belonged to a tv-series). This was a nightmare, because in order to get all the linksets for the main page, I had to do queries that involved the posts, series, seasons, episodes, and linksets table. While that may be a good normalized setup, it makes a lot more sense for optimization to just have all the linksets directly connected to a post, and then just get the rest of the tv-series information when you have to. This is one of the biggest changes I made in the past week.

Those are really the only things I see wrong with the current database design. Though I see TONS wrong with the code that builds out the posts... the php is very redundant and extremely inefficient. To the point where it always amazes me how fast pages can still load most of the time. I hope to fix this eventually, and make everything object orientated in the process (it's currently procedural), but it's such a big project that I want to make sure all other important features are finished first.

Well those were just some thoughts I wanted to share... it's probably not well written at all, but I hope those of you that were interested actually learned something about how the site works.

~SinlessLinks

Spam Control

I've received a positive response to my recent posts about developing the website, so I'm going to continue that trend.

Today I wanted to quickly talk about something every web developer despises... SPAM. While most of us appreciate the tribute to Monty Python, we'd prefer not to have to deal with it, and we definitely don't want it cluttering up our website.

Spam comes in all forms: email, comments, posts, contact submission forms, even our requests page gets spam sometimes.

At SinlessLinks I have built in a lot of different ways to help cut down on Spam....

One of the most obvious is that anywhere there's a submission page, there's a captcha for the person to fill out. This has helped cut down on the spam tremendously. If a spammer has to do extra work to place their spam on your website, odds are they'll move on to someplace else. One thing you'll notice about our Captchas though, is that they're fairly easy to read, and they're extremely simple. I hold projects such as ReCaptcha in high regard for what they attempt to do, but I absolutely despise how hard it is to read and get the right answer to the captcha. I believe that a captcha shouldn't inconvenience your real users more than it has to. This was one of the big reasons for creating the user registration: our regular users can sign up and not have to deal with the evil captcha's any longer.

The downside to having such a simple captcha system, is that some spam still gets through, even from automated bots. So more needed to be done.

To take the spam prevention one step further, I created what is currently still a hard coded spam filter. It checks every comment against an array of various words that we've found only exist in spam comments. (Such as the names of various adult orientated medications.)

Another deterrent I put in place is "nofollow". Using rel="nofollow" in any links in the comments section helps to deter any spammers, as they won't get any page rank boost, nor will their pages get indexed faster by spamming us. You'll also notice that I don't use BB code. I personally believe that comments are secondary to the actual post, and therefore don't need to use special formatting such as italics and bold to get their point across. Spammers often use these techniques to draw attention to their post, and when their standard posts don't look good at all, they get frustrated and move on.

But alas, even with all these spam prevention techniques in place, we still get up to ten spam comments a day. Now that's not too bad, but Ribbitz and I are picky... we don't want any spam. So our foolproof method is to simply delete the spam. We have a "Recent Comments" page that about 20 admins have very easy access to. Between these 20 people, this page is checked in upwards of 100 times a day, and every spam comment found can easily be deleted by clicking a red "x" next to the comment. Beyond that, there's a "Ban" link right next to the red x. Clicking on the Ban link prevents that particular IP from posting anymore comments, or if they're logged in, it prevents that user from posting anymore comments. Yes we do run the risk of accidentally preventing legit users from posting comments, but so far we haven't run into any complaints of that sort.

From the whole process, I've learned that while you can work forever to create an automatic spam prevention system, the best method is to simply make it easy for devoted users to take care of the spam quickly and quietly.

~SinlessLinks

Here's a pic of just how easy it is for our Admins to take care of nefarious comments: