Pritlog - performance

Hardkap » Pritlog - Blogging made simple

Pritlog - performance

(35 posts) (4 voices)

Started 3 years ago by Prit
Latest reply from Prit

Tags:

suggestions

12 Next »

Prit
Pritlogger

I just wanted to get a rough idea of the number of posts you all estimate to see in your blog as months/years pass by. A rough maximum value.

I have been doing performance tests on Pritlog with varying number of posts. With the current approach of using individual files for each post, when I get to more than couple of hundreds of posts, the performance starts falling and if you reach a 1000, the performance will become much slower.

I am trying mockups to create posts in a single file and some other alternatives. I found the most performance with all posts in a single file. I could easily navigate Pritlog with 5000 posts (reasonable large ones) without a problem. Not much performance impact visible to the end user at all.

I think, as time permits, the next thing I want to work on would be the performance. For now, everyone's Pritlog will work absolutely fine. Nothing to panic. Its when you are above 500 posts that things may start slowing down. But before Pritlog users reach there, I want to be able to have a solution for this.

Posted 3 years ago #
john
Member

Hi Prit,

I've experimented extensively with flatfile scripts during the last years, including solutions with everything on a single db.

The best resource I found ever, is this wonderful library that allows to use SQL like statements on flatfiles. I've used it and it works great !

Maybe you find it useful too. Documentation and examples available at > http://lukeplant.me.uk/resources/flatfile/

Regarding max number of posts, I guess not more than 2000.

Posted 3 years ago #
Prit
Pritlogger

Thanks John.

I will try this one. I found couple of solutions like this.

Posted 3 years ago #
john
Member

I don't know if this idea has any chance to work, but assuming that the performance issue relates to opening and reading too many individual files:

Woudn't it be possible to have some sort of "master index" file, containing at least 2 fields - post nbr and category - and use it to only open and read the max number of posts as defined in pagination setting, at the time a new page is requested ?

Index update could be triggered when some post is not found deleting that line to continue at the next.

What do you think ?

Posted 3 years ago #
Prit
Pritlogger

Yes. John. This is a good idea. I will try this one.

I had tried just opening the required number of files for that particular page, but something was causing a time lapse again. I will need to look further.

Posted 3 years ago #
Prit
Pritlogger

I tried the above flat file db solution. When trying with 5000 records, the access is pretty fast, but the update is very time consuming. When reading about flat file db's, there were in many places advice to go with Sqlite DB as it comes by default with many server installations (php5) by default and it stores data in something like flat files and you can move it around easily.

Do you all have Sqlite installed in your server? The below link has a simple tutorial. You can try the initial example to see if Sqlite database is created and queried.
http://devzone.zend.com/node/view/id/760

I read in several places that Sqlite would be much more efficient compared to any flat file DB system.

Posted 3 years ago #
john
Member

You mean slow when updating a record ? what results did you get ?
Re. Sqlite I'll check and let you know.

Posted 3 years ago #
Prit
Pritlogger

Yes, slow when updating a record when using 5000 records. Reading is pretty fast even without a text db class. Direct read of 5000 records and locating it just using simple php is very fast.

I even tried on a free php hosting site called 110mb.com and Sqlite is present there by default. This looks like a good solution to me if this is as common as they claim.

Ofcourse whatever the decision is, it would require a reasonable rewrite of the current Pritlog. If this happens, the update script should automatically take care of the migration of data.

Posted 3 years ago #
Prit
Pritlogger

I tried Sqlite DB with roughly 5000 records. All operations (read, update, search) were done instantly. Superfast I mean. You wont even know there are 5000 records.

Posted 3 years ago #
john
Member

SQLite doesn't seem to work on my hosting (I have the choice to switch between PHP4 and 5. but currently using PHP4 due issues with a script I run)

Anyway searching my hosting's KB doesn't return any results for SQLite, so I send a mail to support to confirm its availability.

I also noted that the summary section on http://devzone.zend.com/node/view/id/760 states: "Like every tool, SQLite has its strengths and weaknesses. While being an ideal solution for small and/or mostly-read applications, it is not well suited for large-scale applications performing frequent writes. This limitation is due to SQLite’s single file based architecture, which doesn’t allow multiplexing across servers, or the usage of database-wide locks on writes.

Posted 3 years ago #
Prit
Pritlogger

I listened to an interview with the author of Sqlite and by large, they mean that applications using millions of records and transactions per second. Pritlog does not fall into that category. Anyway, it would be much better than doing files.

Posted 3 years ago #
john
Member
Prit,

Its clear that there are other ways to handle the data, offering better performance, but wouldn't it be interesting if we could find a solution to keep using your original approach of separate files per post ?

Building on the idea to have a "master index" of the post directory, I put some code together. I tested it and it works but I don't have a clue if it could help to increase the overall performance of your script (I leave that part to you :)

The code:

part 1 shows a sorted array that is created only once per session, containing all filenames that exist in the post directory.

part 2 shows a foreach loop to read a subset of that array, as needed to populate the current page. This part could be integrated in the pagination function (it will also require to calculate an offset to define the index of the first post in a given page)

On a previous post I mentioned that the index would require at least 2 fields: post nbr, and category. What about making filenames look like 00005_somecategory.prt ?
In that way the 2 values could be extracted directly from the filename itself.

In fact this code doesn't create or read any index file, the index only exists in a session var, which is build once per user session and used to get the post nbrs and category to display next.
```
<?
session_start();

// 1. build post index array if it doesn't exist and store in session var

if (!isset($_SESSION['posts_index'])){

// Define absolute path to your folder
$path =$_SERVER['DOCUMENT_ROOT'].'blog1/posts/';

// Open the folder
$dir_handle = @opendir($path) or die("Unable to open $path");

// Loop through the files and populate session array
$listDir = array();
$i = 0;
while (false !== ($file = readdir($dir_handle))) {
if($file == "." || $file == ".."){continue;}
$_SESSION['posts_index'][$i]=$file;
$i++;
}
closedir($dir_handle);

sort ($_SESSION['posts_index']);

}

/////////////////////////////////////////////

// 2. read only number of post files required by each page request

// loop through the session array with foreach

$config_entriesPerPage = 4; // only for testing !!

foreach($_SESSION['posts_index'] as $key=>$value){
if ($key <= $config_entriesPerPage){
echo $_SESSION['posts_index'][$key].'';
}
}

/////////////////////////////////////////////

?>
```
Maybe the code is useless to increase performance, but perhaps other ideas could grow from it. Let me know.
Posted 3 years ago #
Prit
Pritlogger

Thanks John.

In one of my tests, I tried using our original file approach, with opening only the required posts depending on the page/category (from one of your suggestion earlier). This works much efficient. But, when it comes to searching, we would need the contents of all the posts - which would again make it humongous. I believe we will have the same limitation in the approach you mentioned.

I tried couple of text db's, but performance is not improving because its basically flat files again.

With plain flat files, one approach that got me high performance is having all 5000 posts in the same file and parsing it and loading it into an array. But as this array is humongous, memory used on the server will increase. This didnt cause any problems on my test or actual servers. But again, updating - is still an issue. We lose performance when updating.

Posted 3 years ago #
john
Member

Please correct me if I'm wrong, but I make 2 assumptions when you mention "performance loss when updating":

1. that your metrics are primarily based on php execution time
2. that "loss when updating" is only experienced by users when writing to a file

If the above is true, what is its relative impact in % when considering all factors determining the time to load an updated page ?

With 5000 posts, are we talking about delays of 5, 10 or 15 sec. or less than a second ?

On the other hand, using separate files per post doesn't need any file locking since there are not concurrent authors editting the same file, right ?
How does this statement apply to comments that are stored on single files per post ?

Thanks Prit !

Posted 3 years ago #
Prit
Pritlogger

Hi John,

The delay in updating would be more than 30 seconds or more.

Also, since the actual write of file happens in 1 statement instantly, we usually dont see any file locking issues. I have not seen that yet.

Posted 3 years ago #
Bert
Member

Hi all, I shouldn't be posting here, as this technical discussion is way above my head. So these are just some notes from a beginner's point of view:

1. I've been testing Pritlog on a free host, 000hosted.com, which seems to be owned by Byethost, and although I've chosen PHP 5, it doesn't look like SQLite is supported. It should be embedded in PHP5, but it's an option a host can untick.

2. Barry's PPLOG blog is still as fast (=really fast) and trouble-free as on day one, after about 550 posts. Do you think it will slow down when reaching say 2000 posts?

3. Some flatfile cms's seem to be less flat then others, having implemented an internal database to speed up the execution. SQLite is called "embeddable". Does this mean it, or another non-text database, could be inside Pritlog?

4. Not exactly on-topic, but related(?) :

boost-your-website-load-time-with-3-lines-of-code

5. When googling to learn more about SQLite, I came across endless forum discussions between people having troubles with setting up the database. It made me realize the beauty of Pritlog: just upload, set permissions and you can start blogging...It would sure be nice if this simplicity could be continued somehow.

Posted 3 years ago #
john
Member

I placed 200 posts on my server (before I had only 11)

Loading the main page with 11 reported an execution time around 1.2 sec which increased up to a max of 2sec for 200 posts (not realy proportional)

Editing a post takes around 0,25sec until I see "Updated successfuly"
Adding a new post ~ 0,50sec until "Added successfuly"

Tomorrow I will load 2000 posts and see what happens if I change some code

Just playing :)

as Bert says "It would sure be nice if this simplicity could be continued somehow"
My impression is that with some modifications the original script could be quite acceptable for most users.

Posted 3 years ago #
Prit
Pritlogger

Bert,

Regarding your observation on PPLOG, I tried around 500 posts with simple perl script and php script. All these scripts do is do directory listing and load the posts to an array. Perl script finished in 2 seconds and PHP took 10 seconds to do the same simple thing. Maybe this is why PPLOG did not have any issues.

Thanks for the link for speeding up web pages. We can use that. But, our performance issue is mostly related to server side stuff. Anyway, this will help.

I created a test account on 000hosted and checked a simple sqlite php and here is the result:
http://mattie.000hosted.com/sqlite1.php
Looks like it has Sqlite.

John .. thanks for the tests. Will wait to hear back from you.

Posted 3 years ago #
Bert
Member

Good morning John and Prit,

Thanks for your reply and ongoing experiments.

Prit wrote:

Perl script finished in 2 seconds and PHP took 10 seconds to do the same simple thing.

Hmm, that makes Perl 500% more efficient..a Practical Extraction and Reporting Language indeed. PHP will have to come up with lots of other persuading advantages to balance this weakness.

I created a test account on 000hosted (...)Looks like it has Sqlite.

That's strange. I found an official confirmation that SQLite is not supported:

"I had a client enquiry about sqlite3 for free hosting and support informed me that they do not support sqlite at the moment, however the pure mysqld / mysqli are fully supported."

Found in this May 2008 discussion

Posted 3 years ago #
john
Member

Hi Prit and Bert,

I made my first tests using 3000 posts (2285Kb)

The first time is for IE7 second corresponds to Firefox. Results:

Step 1. Testing Original Script
===============================

Load Script 1st time 8,159 | 8,607 reload 2,5s(IE) / 4,4s(FF) browser cache ?
Show 1 Category 3,300 | 3,621
Display an Entry 3,818 | 2,647
Form for New Entry 3,038 | 3,053
Display a Comment 2,779 | 3,102
List All Comments 4,026 | 2,797
Display Archive 8,630 | 9,799 all 3000 posts get listed
Create New Entry 2,997 | 3,067 see (1) below !!
Edit Existing Entry 3,797 | 2,879 see (2) below !!
Click Tag Cloud link 6,761 | 5,428 click initiates search for that word

(1) overwrites existing post (#2988 same for IE and FF !!) / doesnt create new file !!
(2) overwrites existing post (#2993 for IE and 2990 for FF)/ doesnt create new file !!

Step 2. Testing Indexing Snippets
=================================

Indexing to session var 0,032 | 0,051 indexing Post Directory to Session var
Indexing to regular var 0,039 | 0,032 use normal var and reindex as needed
Indexing + n Posts 0,034 | 0,034 index + loop to get posts for current page

The bigest issue seems that creating new posts or editing existing ones doesnt work as expected, in comparison time performance moves to a second place.

I'm confident that the "indexing" snippet will improve execution times and hopefuly solve the new post / edit issue.

I'll need some time to embed the new code, start testing the modified script, and document the changes. More news in a couple of hours ..

Posted 3 years ago #
john
Member

As I started to modify the script I realized that it requires much more rework than expected, with unsure results. Sorry, but I don't have time to complete this experiment now.

As Bert, I liked the simplicity of the original script, and it works great but within its own limits.

Prit mentioned that "when I get to more than couple of hundreds of posts, the performance starts falling and if you reach a 1000, the performance will become much slower"

I thought the problem was only time, but according to my tests with 3000 posts the script starts to FAIL, overwritting existing files when creating / editing posts.

My conclusion: in its current state, Pritlog is realy a nice script if you just need a small blog not exceeding lets say 500 posts. Only keep in mind to clean oldest entries before reaching that mark.

Prit: you made your point, single db is the way to go.

Posted 3 years ago #
Prit
Pritlogger

John .. thanks a lot for performing the tests. Can you please explain the numbers? I am sorry, I did not understand.

I will do more experiments on trying to make the flat file more efficient so that we can stick to it even for larger number of posts. As you both mentioned, it is definitely the most simple.

Posted 3 years ago #
john
Member
Prit,

Numbers show execution times in secs both for IE7 and FF when performing different actions. As expected, longest times appear when 1st loading (all posts are read into an array, on listing the Archive (all posts are shown).

For me the biggest issue was to discover that with 3000 posts the script actually fails >> a. creating a new entry doesn't create a new file, instead it overwrites an existing one or b. editing a existing entry results in overwriting a different file when updating) I found the overwritten file mostly within the last 100 posts (2900 - 3000). Note: this happened before making any changes !

Following is the part of the code that I expected to add to index the post directory, it works but later I realized the logic of the entire script needs too many changes and I don't have the time.
```
// Define absolute path to your folder
$path =$_SERVER['DOCUMENT_ROOT'].'blog1/posts/';

// Open the folder
$dir_handle = @opendir($path) or die("Unable to open $path");

// Loop through the files to populate array
$posts_index = array();
$i = 0;
while (false !== ($file = readdir($dir_handle))) {
if($file == "." || $file == ".."){continue;}
$posts_index[$i]=$file;
$i++;
}
closedir($dir_handle);

// sort ascending
sort ($posts_index);

// get 5 digit code of last post
$posts_last = substr((sizeof($posts_index)), 0, 4);
```
Posted 3 years ago #
Prit
Pritlogger

@John,
I have no idea why the script might misbehave or overwrite existing posts.

Regarding numbers again, let me try to understand it:
When you say - "Show 1 Category 3,300 | 3,621"

Do you mean it took 3300 seconds to show a category?

@Bert,
Regd Perl Vs PHP, when I tested on my production server, it works differently. In fact PHP is faster than Perl to do the same thing. So I guess it was my test server on my local machine that was acting differently.

@All,
I experimented a little with Sqlite and tried to change the posts to a Sqlite DB and see the performance. The below test blog is using Sqlite for posts (only for posts) and has around 5000 posts. You can see the performance. There is no performance drop at all. And it stores the posts in a single file (less size than if we store all posts in a single text file). This file can be moved around to different directories, servers and will continue to work as long as the file path is right.
http://hardkap.net/labs/pritlog55/index.php/mainPage

Posted 3 years ago #
Bert
Member

Thanks for this test Prit. I've read it all till page 41 :)
Would it be different if these posts were all different, of differnet length or containing images?

While you posted the above, I prepared the following..in defence of flatfile:

I've been trying to digest this fascinating and important discussion from a reader's point of view. In my experience, it is very rare I dig into a blog's archive, even if this blog is written by one of my "heros". Some blogs that I visit daily, have years of archives. But I seldom or, in most cases, never open them. As a blog reader my main interest is about what's happening Now. That's what essentially blogs are for. If I wanted to do a more profound study, I would be reading the author's static pages, downloading PDF's or ordering (e)books. So blogs are what they are, momentary expressions of thought, desire, frustration, marketing or creativity. No one expects to read a complete bible in blog form.
Based on this, I got an idea that I think could greatly help a flatfile blog system like Pritlog: Suppose, Pritlog runs optimally up to 300 posts. Starting from post 301, we delegate all tasks to Archie, Pritlog's much younger brother.
Archie knows nothing about php, dynamic content, rss, etc.. all he knows is archives. From post 301 to the oldest post, he just collects them in a single file, a thread of posts, that no longer have any interactivity. They're just there, presented for the sake of history and completeness.
Archie could be a second, much slimmer instance of Pritlog, automatically started up from, say, post 301 and in this way greatly relieving Pritlog, the active and ever fresh flatfile blog software.
Just a thought from an avid reader.

Posted 3 years ago #
klhrevolution
Member

I gotta agree with Bert's above comment. But there is always the option of having an entry in the config file that allows one to write posts to a set html template upon posting to the blog. And for pritlog to display these entries we can use an iframe.

Posted 3 years ago #
Prit
Pritlogger

@Bert,
Thanks for this idea. This discussion is really exciting to me as I like trying out different ideas. I think, based on your idea, if we assume that there wont be any edits to posts beyond the latest 300 posts, and move these posts to a single flat file - just for searching and viewing, without an option to edit, this would make it much faster. Good one Bert.

For the Sqlite test, no, it should not make any change in performance if the posts had different lengths or images(images are only links to the actual images).

@Ken,
If we use iframes to load posts, will it change the performance? I might test it out when I get a chance.

Posted 3 years ago #
klhrevolution
Member

Should not change the performance at all. But, I'm not sure how to solve the issue of prit having all the posts (flat files). I'm thinking out loud here so follow me:

If prit just keeps the $config_entriesPerPage in text file form and each posts is saved as html then maybe that will solve the issue. So only the $config_entriesPerPage are saved as text file, so the most recent posts are displayed in text form and those that are "archived" are saved as html. This follows along the lines of what Bert was saying: "As a blog reader my main interest is about what's happening Now"

example: $config_entriesPerPage is set to 5 so 5 posts are displayed (flat files) so you go to post number 6 and number 5 is now saved as html, you go to post your 7th entry and number 6 is now saved as html and so forth.. And if a reader wants to view a post that is in archive, they have to visit the archive page (obviously - or they find it via search) and they click on the archive link it displays the posts either with an iframe or the archive function could be expanded to fit this scenario and it simply takes them to said html file.

Posted 3 years ago #
john
Member

@Prit,

Just to clarify: my test times are in seconds with 3 decimal places (in Spanish "," is used as the decimal separator) So the largest time I saw was under 10 sec.

Posted 3 years ago #
Prit
Pritlogger

Thanks John. Less than 10 seconds does not make it that bad for so many posts and to work as-is without any changes. I think the performance will depend on the type of server that is hosting the website also. Mine (Godaddy shared) looked worse with that many posts.

This will definitely be an area of improvement for Pritlog.

Posted 3 years ago #

RSS feed for this topic

12 Next »

Reply »

You must log in to post.