the fancy part of the web

Crawler 3.1 release + podcast

Nearly a month after version 3.0.5, we published 3.1.0 of the well known crawler extension today. The main work for that step was related to several bugfixes, a better scheduler integration, some CLI-API enhancements, better testcases and (finally) some documentation updates. Most kudos go to Michael Klapper who took over the responsibility for this release and who also fixed a decent amount of bugs.
You can download the extension in the TER and hopefully this time OpenOffice didn’t trick us again and you’ll be able to see a shiny new extension manual on typo3.org as well.

Besides that, Kasper just released the podcast with me talking about version 3.0 of the crawler and about further improvements for the staticpub extension. You’ll find that on typo3.org/podcasts/kasper.

TemplaVoila 1.4.2 released

Yesterday the 2nd team release of TemplaVoila was uploaded into the TER. It’s basically a maintenance release which fixes more than 100 bugs. But since we haven’t been that straight distinguishing between bug and usability feature, you’ll see a couple of new things within this release.

The high level release notes are:

• page module is now customizable with CSS and JavaScript
• handling of static data structures are improved and fully working now
• wizards are improved, new page wizard is more explaining
• visual cleanups
• new hooks for eTypes (elements added by mapping interface)
• new classes for preview in page module, easy to override by extensions
• added missing localisations
• enhanced drag-and-drop in page module
• over 100 Bugs are fixed
• updated manual

During the installation your TYPO3 Extension Manager will ask to perform a couple of database upgrades. These upgrades aren’t really critical because they just enlarge some database fields, which will make sure that your data really fit’s in.

Just to point one thing out – especially the page module has been improved to be more flexible in certain parts.

These lines of TSConfig can be used to add CSS or JavaScript into the page module and enable easy customizations:

mod.web_txtemplavoilaM1.stylesheet = ../fileadmin/css/tvpagemodule.css

mod.web_txtemplavoilaM1.javascript {
  file1 = ../fileadmin/templates/js/jquery.js
  file2 = ../fileadmin/templates/css/backend.js
}

Further customizations is provided using the “mod.web_txtemplavoilaM1.blindIcons” configuration or with individual content preview classes (configured though “$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['templavoila']['mod1']['renderPreviewContent']” – see ext_localconf.php).

We haven’t discussed what the direction for the next versions really looks like. From my perspective better TYPO3 integration, some kind of code cleanup and also the integration of some features which arise with TYPO3 4.4 will be somehow on our schedule. But since TemplaVoila isn’t a one man show anymore, this isn’t just my decision and in addition I’d like to encourage everyone to send feedback, bugfixes or new features just to give us an impression what you like or dislike in the current version.

Last but not least, I’d like to thank everyone who was somehow involved in the release, especially Steffen Kamper who shared lot of inspiration and who has spent many hours to debug and fix some really tricky issues.

Re: Farewell, TemplaVoila!

Dmitry decided to leave the TemplaVoilà project today and he handed the extension-leadership to me. Since this is a very abrupt change and since there was lot’s of (mis)communication involved I’d like to use this blogpost to answer his “Farewell, TemplaVoila!” post (he turned off the commenting function).

As you might know Steffen Kamper and I joined the TemplaVoilà team some time ago and since Dmitry wasn’t very motivated to maintain the extension anymore, we took over and tried to make TemplaVoilà ready for TYPO3 4.3 (see Dmitry’s clarification below). We also tried to get rid of the over 250 listed bugs from bugs.typo3.org. This went fine for quite some time and we released version 1.4.0 and 1.4.1 in November parallel to TYPO3 4.3. Unfortunately the release come a little too fast for us and a couple of major bugs couldn’t be fixed by that time. After that release we had a meeting with Dmitry and we all agreed to release 1.4.2 in the beginning of January. We found and fixed tons of bugs and also implemented long awaited features in the meantime. We also had the luck that others found new motivation and started to send feedback and started to test TemplaVoilà with us. (Special thanks to Uschi, Jeff and Ron).
Three weeks ago we decided that the current state is “ready to release” and we told Dmitry that it’s up to him to release 1.4.2. Quite some time passed by and today he used several channels (Twitter, Facebook and Newsgroups) to tell the world what he found:

No TV 1.4.2 release soon. Found a bug in page module with unlinking in 1 minute after starting tests. I am severely disappointed by this

He was right – a very obvious bug showed up in the page-module within TYPO3 4.2. Unfortunately nobody saw it and it also seems that some of us are still unable to reproduce it. Anyways — in my opinion — he choose the wrong way to communicate that. Instead of talking to Steffen and me, he decided to talk to anyone else. I wrote him a mail and told him that I didn’t like the way he brought this up and asked for a Skype meeting to discuss how to proceed. After serveral emails back and forth and after others joined the communication, Dmitry decided (without anyone of us asking for it) to leave the team and hand over the leadership. To make it clear to everyone: the discussion started because of one JavaScript error and a few icons in the backend which he didn’t like very much. The Skype meeting never happend – although it would have saved lot’s of time and confusion for all of us.

I’m not very happy with his decision, because this leaves lot’s of questions and because we loose a very diligent developer. But it seems that there’s currently no way to convince him that he might be wrong.

As team Steffen and I will try to continue the development and improvement of TemplaVoilà, especially because it’s one of the most important benefits TYPO3 can offer. I ask everyone to join and contribute some time for testing or feedback. I dislike Dmitry’s approach to fork TemplaVoila and host it somewhere else (see Dmitry’s clarification below). I’m inviting him to rejoin the team at any time and work in a constructive way with us. I’m still convinced that all this just happend because miscommunication and not because of “real” issues.

—
In addition it also seems that others offended Dmitry and asked him not to stop the process anymore – I understand his distraction. He did a very good job in the past, he was open for our improvements and his biggest concern was quality not power or money or anything. Being offended for something you do in your freetime without any payment and being offended by people, who earn money from their clients with your (free) work, is always distracting. Please think about that in future communications with any open source software developer.

—
If you wonder how to get in touch with us: http://bugs.typo3.org lists all known bugs and their status. Report new bugs there or try to add new information to existing ones. Team discussion, in regards to fixed bugs or new features, happens, compareable to the TYPO3 Core, in the “typo3.team.templavoila” list on “lists.typo3.org”. If you need regular feedback or help with TemplaVoilà please use the “typo3.projects.templavoila” list.

—
* Update 05.03.2010: After I posted that the link to that blogpost on Facebook, Dmitry commented it and just to be fair, that’s what he said:

Thanks for your post. It is fair and explains the situation well. I would only clarify two moments: (1) I am not forking TV but creating a version for myself. I do not plan to release it to TER or anywhere. It is a TV for me as personally I like it (2) “Dmitry wasn’t very motivated to maintain the extension anymore” is not exactly so, rather to my taste TV already did all it had to do. I invited others to move TV further because I did not need anything else from it. Thanks again.

Handling data in TYPO3 with tcemain

TYPO3 is (by definition) a powerful tool when it comes to data. Besides creating, updating and deleting data there are also localizing and versioning, logging and even rollbacks. All this is provided through the GUI of TYPO3 and all the technical stuff in working under the hood of TYPO3 for nearly every piece of data. But what if you’re asked to write a script which imports or updates data, how can you make sure that all this is done in a TYPO3 compatible way?

The lazy programmers approach is to write up SQL, but that’s not what’s recommended if you still want the full TYPO3 featureset to be available for you (without reinventing the wheel). In this case the TYPO3 core class tslib_tcemain (short tcemain) is what you’re looking for. For the mentioned tasks there are two main functions relevant – process_cmdmap() and process_datamap(). The process_cmdmap() performs actions like “move”, “copy”, “localize”, “version” (create, stage, swap, flush), “delete” and “undelete”. The process_datamap() does the rest – creating records, updating datafields. Controlling both of them is done with configuration arrays and that’s how it looks like**:

Creating a record*:

$data = array();
$data['tt_content']['NEW'] = array(
	'pid' => 100,
	'header' => 'A new thing'
);

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start ($data, array());
$tce->process_datamap ();

echo "The new element has the uid ".$tce->substNEWwithIDs['NEW'];

Creates a new tt_content record on page 100 with the header set to “A new thing”.

Updating data*:

$data = array();
$data['tt_content']['110'] = array(
	'header' => 'A really new thing'
);

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start ($data, array());
$tce->process_datamap ();

Updates the header field of the content element with the uid 110 to “A really new thing”.

Move data from one page to another*:

$cmd = array();
$cmd['tt_content']['110']['move'] = 101;

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start (array(), $cmd);
$tce->process_cmdmap ();

Moves the tt_content record with the uid 110 to the page 101.

Copy data from one page to another*:

$cmd = array();
$cmd['tt_content']['110']['copy'] = 101;

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start (array(), $cmd);
$tce->process_cmdmap ();

Copythe tt_content record with the uid 110 to the page 101.

Localize your record*:

$cmd = array();
$cmd['tt_content']['110']['localize'] = 5;

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start (array(), $cmd);
$tce->process_cmdmap ();

This creates a localization for the language 5 of the tt_content record with uid 110 (assuming that the tt_content record 110 is a default language record).

Delete*:

$cmd = array();
$cmd['tt_content']['110']['delete'] = true;

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start (array(), $cmd);
$tce->process_cmdmap ();

Deletes the tt_content record with the uid 110.

Undelete*:

$cmd = array();
$cmd['tt_content']['110']['undelete'] = true;

$tce = t3lib_div::makeInstance ('t3lib_TCEmain');
$tce->start (array(), $cmd);
$tce->process_cmdmap ();

Restores the tt_content record with the uid 110 – if it’s deleted.

—-

* Running the codes requires a TYPO3 backend context with a logged in backend user who has the right to perform all these actions. In addition tcemain has some configuration options to change the behaviour of the actions, e.g. “$enableLogging” or “$bypassWorkspaceRestrictions” – they come with useful defaults but you might need to change them in certain situations ~ so looking into the code documentation might save you some time.

** I left out the “version” part since this requires some more explanation than just a few lines of code.

crawler extension version 3.0.5 released

Quite some time after the 4.3 release of TYPO3, we published the necessary compatibility version of the crawler extension. Besides the compatibility fixes for TYPO3 and also for PHP 5.3 we also included some handy features:

First of all there’s now, besides the CLI interface, also a full integration with the scheduler extension, which is available in TYPO3 4.3. This enables to setup crawler runs and manage all crawler releated tasks through the TYPO3 backend.

The “crawler_flush” interface was added to the CLI (and scheduler). It helps to clean up the crawler queue and enables to remove finished or unfinished entries.

In addition the CLI was cleaned up a little bit and behaves more intuitive in most situations. Also the help pages should now really tell you what options you have.

And last but not least we’ve added the possibility to avoid an additional HTTP request and have the crawler rendering the page directly.

Big kudos for their work and their support goes especially to Mick, Fabrizio and Timo from AOE media.