Download Counter not working

Do you have anything you'd like to bring to the Site Admins attention? Suggestions? Problems etc? Please note: Beginners should go to the 'help for beginners' forum below!

Moderator: Moderators

Re: Download Counter not working

Postby NeutronIC on Sun Mar 19, 2017 4:10 pm

Perhaps I should give a little more background on the download counter issue as well...

The system used to simply add 1 to a download count variable in the database a long time ago - really simple, exactly how you'd expect to do it! However, it became apparent that download numbers were lower than expected and didn't actually match up with web logs if I manually checked. This was down to a good old fashioned programming problem when you have multiple things updating a counter at the same time - let's say counter is 0, two near simultaneous downloads start and both read the value as zero, update it to 1 and put it back in the database - end result, it shows 1, not 2. This ought to have been an almost non existent problem - but it turns out the site got so busy that it became near rampant - and people were justifiably confused about why their files were apparently not getting downloaded very much.

Now, at the same time I set the file library up many many moons ago, I also had the system keep a record of every individual download ever made from the system in another table - and as this simply adds a new record, it doesn't suffer from any of these concurrency issues. For information, you can view the table from your perspective by going to "My Downloads" along the left hand menu on the main page of the site. This allows you to see when you downloaded every file you've downloaded, and if the file has been updated since you downloaded it, and if there have been any new comments left since you downloaded it.

Again, some years ago, it occurred to me that I could use this massive data table to get accurate download counts, so I wrote a script that would sum up this by file and put the values back in to the database. This worked ok, until the file library got bigger and bigger, and the download record table became enormous.

It's run in to various problems over the years and i've helped it along - the main one being that while it was running, it would basically stop the web site from working at all because it was making such a large load on the database server. Tuning the database and refining the queries has improved its performance.

The most recent issue was that it was being process limited - in two ways. I had put a max counter on it, one of those "well, it will never get this big, so stop it if it ever feels it has to process this many files, because that probably means it's gone mental" - turns out, it now has to process that many files (many more in fact)... Also, having fixed that, it was then taking so long to run that it was hitting the max time limit - so i've resolved that too.

Download counts are never lost, the way the system works it always goes back to that download record and counts it fresh - so even if it stops working at all, it just regenerates it, so don't worry about that :)

The whole process is taking around 10 minutes now which isn't too bad, considering the vast number of records it is processing. It should now re-run every day to keep the numbers current.

Hope that helps explain the background.

Matt.
NeutronIC
Atomic Systems Team
Atomic Systems Team
 
Posts: 12625
Joined: Fri Oct 05, 2001 12:00 am
Location: E11, London, England

Re: Download Counter not working

Postby chrisonline on Mon Mar 20, 2017 4:21 am

Matt

Thanks for the update. I have not tried the FTP method, and thought it was the "old" method, being superseded by the more user-friendly browser method. I FTP to my own website, so that holds no fears for me, but the "small configuration file" and subsequent "side-loading" baffle me. I will look it up and give it a go.

However my recent problem raised a question which has not been answered. Can the PREVIOUS file (which disappeared when the new failed upload took place) be recovered? It may sound trivial, but it concerns me that the download count for that is lost, and a fresh upload will now reset to zero.

So is the old file lost forever, or is it retrievable from a backup somewhere, to reinstate it so that it can then be updated - by the FTP method if necessary?

Chris
chrisonline
Very Active Forum Member
 
Posts: 1022
Joined: Sat Apr 27, 2013 1:10 pm
Location: Front Royal, Virginia, USA

Re: Download Counter not working

Postby NeutronIC on Mon Mar 20, 2017 9:56 am

The small configuration file is literally a 3 line text file which says what your username is, what filename is for the main file and what filename is for the image - that's it.

The sideloading is basically automatic - you go in to "view my uploads" and the system detects any configuration files for you and copies them in to the file library so that you can just edit them using the usual edit screen.

If you know the file id of a removed file then upload a new one and I can just swap the file id's over relatively easily - and then the nightly download counter tool should actually recover its download count automatically, since the record of a file id being downloaded doesnt get removed if the file is removed.

Matt.
NeutronIC
Atomic Systems Team
Atomic Systems Team
 
Posts: 12625
Joined: Fri Oct 05, 2001 12:00 am
Location: E11, London, England

Re: Download Counter not working

Postby chrisonline on Mon Mar 20, 2017 7:34 pm

Hi Matt

Tried to PM you but I'm getting a fail error, with an SQL problem fetching the page?

Anyway, my message was as follows:

Thanks for your help regarding my previously "failed" upload. I have now FTP'd it using your guidelines, and it is sitting awaiting approval as file UKTS 36672.

It is v3.0 of a previous file. The "lost" previous file was UKTS 36360. If, as you suggested, you are able to swap the file number and ensure that this v3.0 update picks up and runs with the previous Download Count, it would be appreciated.

Thanks again for your help

Chris Laurence (chrisonline)
Front Royal, Virginia, United States
chrisonline
Very Active Forum Member
 
Posts: 1022
Joined: Sat Apr 27, 2013 1:10 pm
Location: Front Royal, Virginia, USA

Re: Download Counter not working

Postby chrisonline on Mon Mar 20, 2017 9:22 pm

FYI, here is the error message I got when trying to PM the above message to you

General Error
SQL ERROR [ mysqli ]

Out of range value for column 'user_unread_privmsg' at row 1 [1264]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.
chrisonline
Very Active Forum Member
 
Posts: 1022
Joined: Sat Apr 27, 2013 1:10 pm
Location: Front Royal, Virginia, USA

Re: Download Counter not working

Postby Easilyconfused on Mon Mar 20, 2017 10:07 pm

I think Matt's PMs are broken due to the sheer amount of junk in there. I can't PM him either.
Kindest regards

John Lewis

Member of the forum moderation team
User avatar
Easilyconfused
Worried about Silent Chickens
 
Posts: 12855
Joined: Tue Dec 31, 2002 9:06 am
Location: Portsmouth & Bristol

Re: Download Counter not working

Postby chrisonline on Wed Mar 22, 2017 12:26 pm

Matt.

Your PM is still not working, but I want to say . . . as and when you see this, it seems the other file ID I gave you was wrong. It was actually ID 36322 (originally uploaded Dec 18, 2016) - that was version 2.0 of the Class 150/1 physics & sound patch.

But as the new file (ID 36672) has now been up since late March 20, with 118 downloads so far, I'm not sure it is worth changing anything now.

I will know henceforth to use the FTP method

Chris
chrisonline
Very Active Forum Member
 
Posts: 1022
Joined: Sat Apr 27, 2013 1:10 pm
Location: Front Royal, Virginia, USA

Previous

Return to Site Admin Queries

Who is online

Users browsing this forum: No registered users and 10 guests