Well Shoot! What's been going on with the Shoutboxes?

I wanted to take an opportunity to post this little notice letting everyone know we apologize about the shoutboxes being down. After all, we just purchased another shoutbox for our members with the full idea of letting them finally get to use the box without the ever present "Shoutbox exceeded Bandwidth" issues. But more to the point, we are aware of the problem and wanted to let you know that it isn't anything we caused. We just have to wait it out and let the company get their servers back online.

In the meantime, feel free to comment crazy excitement here as I personally know I've missed all the exciting bantor. Hope all is well though with you, and Sinless, Lufrebent, and I look forward to continually offering additional exciting features.

New Server Put To Use: RS to MU

Hey Everyone!

You all were kind enough to donate your money so that we could rent a second server. Well, beginning today you'll get to see that money put to good use.

Starting today, every single Rapidshare link that is posted on the site will AUTOMATICALLY be transfered to MegaUpload and added to the post. These links will be exact mirrors of the Rapidshare links, and therefore interchangeable. The whole process of transfering the files to MegaUpload takes about 5 minutes (depending on how many links there are, it may take a bit longer).

During Rapidshare "Happy Hour" times (2am to 10am CET), the same system will work to convert all our past posts over to MegaUpload as well.

Q: Can I have it transfer it to my MegaUpload Account?

A: No, unfortunately that would be far too complicated, and we don't want to handle other people's MU passwords.

Q: So you're stealing points that are rightfully mine? How's that Sinless?!

A: We're not stealing any points. MegaUpload does not award points for any files over 100mb, and about 90% of all the links posted these days are over that limit. And any of the sparse points we might get will go only towards keeping that account alive, so that your donation money doesn't have to pay for it.

Q: Can I still post MegaUpload mirrors?

A: Yes! Of course! Just because this automatic system will always post them, doesn't mean you can't. The more linksets the better!

Q: Can I setup a system like this?

A: Not one exactly like ours, however if you use Google, you'll find plenty of other sites to help you do this kind of thing.

Q: I did Google it, and it says you have to have a Rapidshare Premium account to do it.

A: Yes you do. Our system uses about 8 different Rapidshare accounts, and it is predicted that it will use almost all their available Download Traffic every day. Yes that means we'll be using between 80-800GB of download traffic a day. Be glad it's not your Rapidshare account.

Q: How can I help?

A: Submit Rapidshare Links, Post Rapidshare Links, spread the word about how we have MegaUpload stuff now, and a little monetary donation will go a long way to helping make sure we can keep the server.

What's Next?

The next thing(s) the new server will be used for...

1) Transfer all MegaUpload Links posted by users over to Rapidshare.

2) Transfer all RS and MU links over to MegaShares, MediaFire, FileFactory, Storage.to, hotfile.com, etc.

~SinlessLinks

We Have Liftoff!

We were able to get a server for the price we originally intended. The processor is 200Mhz slower than I would have liked, but we're still very excited about our new acquisition. We paid for the server today and should be able to start using it in the next 24 hours.

I'd also like to take a moment to simply honor the achievement that humankind accomplished 40 years ago. True, this is the day we *left* the moon, but that's a commonly overlooked achievement as well. Space Exploration would likely not be what it is today if something had gone wrong to prevent the astronaughts from returning to earth. Thank you to the brave Men and Women acrossed the world that work every day to make science fiction a reality. I look forward to sending you all a Google Wave from my Firefly class vessel someday.

~SinlessLinks

Best Laid Plans of Mice and Men

Quick update on our second server...

Today we unfortunately found out that we had to pay three months in advance for our second server. This was dissapointing, but nevertheless it didn't stop us.

However, by the time we were able to raise the funds, the specific server we were attempting to rent was no longer being offered by our hosting company. So to help rectify this, I've emailed our hosting company asking if we can still get the rate that was advertised this morning. I believe we'll succeed in getting the rate, but if not, we will unfortunately either not be able to get the server at this time, or have to look for a different company to host it with, as the next cheapest server we could get is around $20 more a month, which makes it more expensive than our current server.

Just wanted to update anyone interested. I'll write more tomorrow, when (hopefully) we get confirmation that we can have the server for the price we were expecting.

~SinlessLinks

A Second Server on the Way...

Just a few hours ago, I wrote about how I had dreams of getting a second server to be able to do some cool extra things for the website. Well thanks to 7 very generous donators, we have raised more than enough money to make that happen.

I'm going to do my best to make sure that I am very transparent about what we use the server for, and keep everyone up to date on how it's helping out the site. I want everyone to know exactly what the money went towards, without any questions or doubts about our motives.

I'll be ordering the server asap tomorrow (I have to wait for our hosting company to be open during business hours), and it'll take a day or two before we're setup and ready to use it, and then I'll start letting you guys know exactly what we're going to be doing with it.

Once again, thank you very much for your generosity, and stay tuned for more updates.

~SinlessLinks

A Second Server for Related Projects...

Want to help the site improve? We have a need! Read on to find out how...

Something I've wanted to do for a long time is get a second server that can be used to do things like...

1) Keep Rapidshare links from dying because of inactivity.

2) Run our link checker from that server to keep this one fast.

3) Have a remote server to store backups on.

4) Give us the ability to try new projects.

5) Start developing our own ShoutBox system at some point.

6) Potentially develop a system that would automatically (without downloading anything) mirror all Rapidshare links on MegaUpload/MegaShares and vice-versa (and other sites as well).

7) Possibly setup an SVN test server to allow others to collaborate in improving the site with me.

SO, HOW CAN YOU HELP??

Adding this second server would cost roughly $50 USD a month. Our current cost ranges from $70 a month to $85 dollars a month (depending on if we have to pay for the shout boxes or other quarterly server costs).

This would mean that our monthly site costs would run between $120-$130 a month. I feel like that's asking a lot.

However I'd rent the server on a monthly basis. So aside from shutdown fees, we'd be able to drop the second server whenever we need to if we were unable to get the funds.

In addition, the site ads are helping out some now, which would mitigate any increase in donations that would be required.

I also believe that the addition of this second server will help bring in more visitors, because of some of the things we'll be able to do with it.

So, that said. I thought this would be a great month to mention it and ask for your help.

Why this month? Because we already have $45 ready for this month's server fees, which would mean that the second server is pretty much already paid for, and we only need to raise our normal amount.

So it's up to you guys. If we can get the extra money for this server, I think you guys will be happy with what we're able to do with it over time. And if we get more visitors, there will be more people to share the burden of the cost.

And I want to reiterate that we don't make any money off the site. Everything donated goes right into paying for server costs. All of our admins and site partners, etc. are strictly volunteers.

I'd love to be able to do this, but I'll leave it up to you guys, since you finance everything.

~SinlessLinks

Site Architecture of a Rapidshare Site

When the site started up, I knew very little about MySQL or proper database design. Thankfully I got pretty lucky and what I designed worked, though it's far from ideal.

You may have noticed that at SinlessLinks we have a few features that other websites don't have. The most noticeable is the new way to select just what websites you want to view. This is made possible because of something that I've called a "linkset". I can't remember for the life of me if I heard that word somewhere else or coined it as I created the site, but it's that part of the database design that makes it all possible.

A LinkSet is intended to be a collection of links that contains all the links necessary to download and unrar a particular item. Or in the case of YouTube, it would contain all the links necessary to watch a full tv-episode or movie.

Often on other sites you'll see them post a "primary" set of links, and then some alternate sets. On a few "news only" blogs, you'll see a lot of different sets of links in the comments, like you have started to see here more often. This is the "problem" I was trying to solve with LinkSets. Just because the main links that the original poster provided go dead, doesn't mean that the post should be deleted. And anyone that's trusted should be able to add links of any type to a post, and have them show up, but not clutter up the page. It needed to be easy to look at and quickly grab the links you want.

And in TV-Series, I had seen many posts where there were 3 sets of alternates for each episode. Now this is great for making sure there's always working links, but when you're looking at something like Seinfeld or The Daily Show, it's just not practical to have that many different sets of links all visible at the same time. So it was decided that while many linksets can be added, only one will show up per episode. Then as soon as one set goes dead, the other would come into place. (Now mind you this has a long way to go before it's perfect -- we need an easy way to check and flag the links in a single episode or season so that the dead links will go away and working ones will show up -- and we also need a way to view all the links available for a particular episode, a page devoted to that episode mind you.)

Okay, so now that what we were trying to achieve has been mentioned, a few words about how I achieved it, what I learned, and how I would do it differently if I could start over.

The main post is stored in a table of its own. However, this table only stores information like the url, the title, the year, the creator of the post, when it was created, etc. It doesn't actually store the body of the post. Instead, it is linked to a "linksets" table, which stores cohesive sets of links. These linksets have a "Type" which a number standing for Rapidshare, MegaUpload, etc., they have an author, and they have a "status" (which is normally 1 for good-working links, but the linkchecker can switch it to 2 for dead, and we also have 3 for "submitted" but not yet approved).

In addition to linksets, their is a table for series, seasons, and episodes, which help store all the information to make up a proper TV-Series post.

All posts are created when you view them. This has it's up sides and down sides. On the up side, it means that as soon as a change occurs (such as adding a linkset, or adding/deleting an episode) everyone sees that change immediately. The downside is that even one little portion of bad code can slow down or even crash the whole site, because it is run hundreds if not thousands of times a minute. Eventually I hope to create a "cache" system that builds out the post anytime a change is made, and then that is used to display the post to the user.

Some things that I would change about this architecture...

There's no need for the "series" table. It contains redundant information to the posts table, and needs to be eliminated eventually. Not sure if I will ever do this though, as it would be tons of work, with very little speed improvement.

There's no need for the "links" table that I use. Every line in a linkset is stored in a separate row in a links table. The original thought was to be able to store the status of a single link as being dead and show that every time someone views a post. But after trying that, it was just overkill. Besides, if one link in a linkset is dead, it's normally safe to assume that people can't properly use the content at all.

I originally connected a linkset to a post (if it was a regular movie, etc.), or an episode (if it belonged to a tv-series). This was a nightmare, because in order to get all the linksets for the main page, I had to do queries that involved the posts, series, seasons, episodes, and linksets table. While that may be a good normalized setup, it makes a lot more sense for optimization to just have all the linksets directly connected to a post, and then just get the rest of the tv-series information when you have to. This is one of the biggest changes I made in the past week.

Those are really the only things I see wrong with the current database design. Though I see TONS wrong with the code that builds out the posts... the php is very redundant and extremely inefficient. To the point where it always amazes me how fast pages can still load most of the time. I hope to fix this eventually, and make everything object orientated in the process (it's currently procedural), but it's such a big project that I want to make sure all other important features are finished first.

Well those were just some thoughts I wanted to share... it's probably not well written at all, but I hope those of you that were interested actually learned something about how the site works.

~SinlessLinks

Spam Control

I've received a positive response to my recent posts about developing the website, so I'm going to continue that trend.

Today I wanted to quickly talk about something every web developer despises... SPAM. While most of us appreciate the tribute to Monty Python, we'd prefer not to have to deal with it, and we definitely don't want it cluttering up our website.

Spam comes in all forms: email, comments, posts, contact submission forms, even our requests page gets spam sometimes.

At SinlessLinks I have built in a lot of different ways to help cut down on Spam....

One of the most obvious is that anywhere there's a submission page, there's a captcha for the person to fill out. This has helped cut down on the spam tremendously. If a spammer has to do extra work to place their spam on your website, odds are they'll move on to someplace else. One thing you'll notice about our Captchas though, is that they're fairly easy to read, and they're extremely simple. I hold projects such as ReCaptcha in high regard for what they attempt to do, but I absolutely despise how hard it is to read and get the right answer to the captcha. I believe that a captcha shouldn't inconvenience your real users more than it has to. This was one of the big reasons for creating the user registration: our regular users can sign up and not have to deal with the evil captcha's any longer.

The downside to having such a simple captcha system, is that some spam still gets through, even from automated bots. So more needed to be done.

To take the spam prevention one step further, I created what is currently still a hard coded spam filter. It checks every comment against an array of various words that we've found only exist in spam comments. (Such as the names of various adult orientated medications.)

Another deterrent I put in place is "nofollow". Using rel="nofollow" in any links in the comments section helps to deter any spammers, as they won't get any page rank boost, nor will their pages get indexed faster by spamming us. You'll also notice that I don't use BB code. I personally believe that comments are secondary to the actual post, and therefore don't need to use special formatting such as italics and bold to get their point across. Spammers often use these techniques to draw attention to their post, and when their standard posts don't look good at all, they get frustrated and move on.

But alas, even with all these spam prevention techniques in place, we still get up to ten spam comments a day. Now that's not too bad, but Ribbitz and I are picky... we don't want any spam. So our foolproof method is to simply delete the spam. We have a "Recent Comments" page that about 20 admins have very easy access to. Between these 20 people, this page is checked in upwards of 100 times a day, and every spam comment found can easily be deleted by clicking a red "x" next to the comment. Beyond that, there's a "Ban" link right next to the red x. Clicking on the Ban link prevents that particular IP from posting anymore comments, or if they're logged in, it prevents that user from posting anymore comments. Yes we do run the risk of accidentally preventing legit users from posting comments, but so far we haven't run into any complaints of that sort.

From the whole process, I've learned that while you can work forever to create an automatic spam prevention system, the best method is to simply make it easy for devoted users to take care of the spam quickly and quietly.

~SinlessLinks

Here's a pic of just how easy it is for our Admins to take care of nefarious comments:

New Group Management

One of the most important things I've learned over the years is how helpful it is to make administrative tasks easy for those that have to do them.

We have a pretty elaborate and very customizable User Management system at SinlessLinks.com. Each user belongs to a group, and it is very easy for the super admins to change what group an individual belongs to, and emails are automatically sent out explaining the new features, etc. A lot of work done with very little effort by the admins.

One thing we've been lacking though, is an easy way to edit the groups themselves. Before today, anytime we needed to add, edit, or delete a group, I had to manually go into the database to make the necessary changes. This wasn't too bad, since it only needs done every once in a blue moon. However, I like to make a user interface for everything I can, because it cuts down on the work that I have to do myself in the long run.

Today I created a Group Management feature that allows the super admins to easily edit what permissions any group has, as well as add groups, and delete groups (except the most basic groups -- those can't be deleted). I believe this feature will help us keep the site up to date and very secure, as it will be easy to create new groups and grant only very specific permissions to some individuals.

While I can't allow everyone to visit the page and play around with it, I did take a quick screenshot of what it looks like.

~SinlessLinks

Oh... and for anyone wondering, no that's not a frame. I don't use any frames, and avoid iframes whenever possible (I use the new object tag instead if I have to). That's just a scrollable div.

Thoughts on Site Backups...

Hey Everyone,

So every major website such as SinlessLinks.com should have a regular backup system. Even if that system is just that someone does a site backup once a week or once a month.

For the longest time that was our backup system. Whenever I remembered and could be hassled to do it, I would make a database backup and a backup of the various files required for the site. Obviously this isn't a very good system... and potentially disasterous if we were to get attacked by some malicious individual.

So overtime I've been playing around with various automatic backup solutions. Finally tonight I've implemented a custom one I've created that is by far my favorite.

Every night, our database and all our site files are automatically backed up. The database is dumped into a file and then all the files are either gzipped or rared.

Now that much is great, but it's also pretty normal. But what if our server were to crash or suffer a harddrive failure? All our backups are on the server. That's a problem. So I went one step further.

After backing up, the files are automatically uploaded to Rapidshare, and the download links for those files are then automatically emailed out to certain designated email addresses.

I think that's a pretty cool system :-). Plus the poetry of it is beautiful. What better way to backup a Rapidshare Links website than to upload it to Rapidshare and save the links?

Learned something cool while testing it too that will make all of you uploaders as jealous as I am right now. The server that the website runs on can upload 40mb to Rapidshare in about 10-15 seconds... that's insanely awesome. Don't we all wish we had that speed?

~SinlessLinks

Downtime Apologies

Hey Everyone, SinlessLinks here with a quick apology...


Bad News:
Last night I attempted to make some improvements to our Searching system, but failed to test one particular feature before uploading the changes to the website. Unfortunately, the one particular feature I forgot to test had two misplaced parenthesis that caused the server to crash (literally... it was all because of two parenthesis... like these two). I immediately saw the problem and started working to fix my mistake. However, due to one problem after another, we were unable to get the server restarted until about 20 minutes ago.


In short, we had technical problems, and it was all my fault. I'm deeply sorry for the downtime, and I promise you now that it won't happen again anytime in the foreseeable future. I've learned my lesson and will triple check everything before I make any changes.


GOOD NEWS:

The search feature has been improved! Remember how it used to bold like half the page for no apparent reason? I fixed that! And now if you want to find a particular user's profile... say AlexisRod's... you can search for it in the search box and it'll show up! Check out the following link for an example:

http://www.sinlesslinks.com/search/alexisrod


So again, I'm very very sorry for the problems, and we fixed it as fast as possible. We didn't lose anything, and are better prepared for next time our server crashes (hopefully never again...).


Rest assured, no one is out to get us, I just make mistakes every once in a while ;-).



~SinlessLinks