What happend?

In his post Robert kind of gave up his attempts to establish a TYPO3 product board [1]. The reasons for this are very wide-spread and mainly the various flame-wars in the last year and the very personal attacts brought him to the conclusion to:

* no longer invest my time into setting up or participating in an overall product team
* refrain from trying to establish leadership for the TYPO3 project
* concentrate on Flow and Neos and invite teams to participate in frequent meetings about it
* unsubscribe from the core internal mailing list

Even thought this shortens his entire mail a bit – that’s the bottom line it comes down to****.

Looking at the recording of the “Not-the-product-board” [7] meeting from last Tuesday [2] it seems that these steps aren’t necessary, as everybody seemed to be happy with the setup. So what’s the criticism actually about? Lucky enough I don’t have to describe it myself, but can cite an (again internal) response to an earlier mail from one of my fellow Core Team members:

Suddenly, when you (Robert) left the steering committee of the T3A because of the
new bodies and bylaws, you realized that there is no more power for you
to decide about things. That’s why you brought up the concept of a
product team. It’s an attempt to replace the former steering committee,
which has backed up the decisions of the core team in a nice but
undemocratic way.

In addition to these two standpoints we’ve a mixture of opinions and directions in our community and all of them cause quite some irritation [3,4,5]. With all these repeating fights and discussions Robert and (I guess) most of us ask ourselves how the community could avoid the fighting?

So what’s my opinion:

What we should avoid

• Closed door meetings and private discussions – most of the “emotional outbursts” we saw where caused when the seemingly final results where presented to the community out of the blue. The fact that Core-internal is still a vivid place is a problem and we all should be abandon it.
• Working without vision – the long holding support (since 2006) for Neos and the nice drive gridelements got from it’s fans [9] are two good examples that a vision can move mountains. But we should renew the vision statement and provide roadmaps on how we want to achieve it – mainly to motivate contributors and to enable collaboration between the teams.
• Leaders or leader groups – other OpenSource products have their benevolent dictator – this wouldn’t work for our community. Their focus will never reach the entire community and in the end their decisions will always cause confusion. Imho Kasper made a great job to bring TYPO3 to life – but he/we saw that the “dictatorship” didn’t scale in the end [12] and imho Robert’s response and the controversy around the product board kind of shows that too.**
• Personal attacks – nobody joined the TYPO3 community to fight, so there’s no reason to fight back. Within all controversy and disappointment everyone tries to improve TYPO3 in his way [10]

What we need

• Diverse groups from all parts of the community – there’s no other way to capture all ideas and to gather various people from the entire community. As a nice side effect they make the “surface” of our community much larger and might help to involve new users. Btw. our community manager Ben van’t Ende will be happy to help kickstarting and coordinating groups.
• Open meeting protocols and discussions – to make sure there’s a chance for everyone to catch-up later and to avoid bad surprises as we saw during the rebranding [3] or during the version schema change [6] In both cases small groups of people made major (not necessarily bad) decisions on their own behind closed doors – the following controversies showed that good intentions can turn out very negative.
• Clear communication structures in the groups – to make sure things are “presented” appropriately. As see in [6] or in more recent situations [8] things would improve if groups had clear communication-channels. In [6] and [8]. seemingly official messages turned out to be personal attempts instead without actual “approval” / “consensus” from the related groups.
• Rough product roadmaps – They should exist to make sure people share a common vision but they should not be too straight to make sure nobody feels too bound to it. They should also exist to enable some kind of measurement. We should be able to ask the Neos or CMS team whether they “live” along their vision or whether it would be better to adjust it.
• Constructive honesty – people should discuss openly, in a respectful way [11]. Technical doubts and constructive criticism should be allowed, but it should be fair.

What does that mean for our products?

Honestly: I don’t know. Looking at the current situations with the lists from above: we have tons of groups working in many directions, we have a group that agreed on a common product-vision (two times) [13], but the group communicates behind closed doors. I’m not aware of any Neos or Flow roadmaps, but TYPO3 CMS has at least a (short) roadmap [14]. The Core team doesn’t have clear communication structures yet and as shown this raises ton’s of confusion. In addition our discussions tend to get very personal. It seems that sth. like a “product board” could help but…

Product board and leadership

On one hand I don’t like the way how the “product board” was positioned in the beginning – it should not lead anything or decide anything. On the other hand it’s great to have a group of people taking care to formulate a vision. The access to this group should be open to everyone, not just group leaders. Inner-circles, Top-10 groups, leader groups, Core-internal discussions should be avoided and open group-”setups” should be emphasized.

Hope this made sense?

Read on:*

[1] – Google Doc: Product Board (or “Product Team”)
[2] – TYPO3 Product Hangout (On Air)x
[3] – Die Marke TYPO3 erfindet sich neu
[4] – Rebranding: Get the green back
[5] – Wieviel Kommunikation und Roadmaps braucht ein Open Source Projekt?
[6] – TYPO3 6.0 at the corner? How is it possible?
[7] – @kdambekalns: Now a first #TYPO3 “product …” meeting, not official, no decisions, nothing.
[8] – @WrYBiT: @benvantende .. I expected, a mail by you and a news on T3O about it…
[9] – Startnext: Verbessertes TYPO3-Backend mit neuen Features
[10] – @tom_noice: . @thomas_hempel The discussions are there because so many people care.
[11] – Community Code of Conduct
[12] – King for a day, but not for a lifetime
[13] – The Phoenix team reports on the Developer Days 2012
[14] – Proposal for the upcoming Roadmap and LTS

Setup

The basic Varnish setup is more or less always the same and best described by Farbrizio Branca. On top of that we need some TypoScript parameter tweaking to get the cache-control-headers in TYPO3 straight – Daniel Pötzinger’s article covers them best. Another very handy thing which can be found in Fabrizio’s blog is the simplified flow chart for the various Varnish subroutines.Based on that all pages should be cached properly and your site should run smoothly. But in case you have a page with personalized content, you’ll have to reconsider some parts.

Problem

The event submission page on typo3.org is a good example. In case the user is not logged in (a.k.a public page), we just want to show a message which guides him to the login. If there’s a login active (a.k.a user page), we’ll show the submission form instead. In both cases we could cache the content nicely, but how would we ensure that Varnish delivers the correct content?

Solution

Ajax could be a solution, but for large sites it’s usually smarter to avoid as much JavaScript as possible. EdgeSideIncludes (ESI) are another option, but I agree with Daniel, they’re not really useful in this case and I’d rather go with Ajax than with ESI.

What we want in this scenario, is to cache the public page in Varnish and pass to the user page generated by TYPO3 if we find that the user is logged in. But this should of course only happen on pages where this is really necessary – normal pages should just ignore the login state of the user. Therefore we need sth. to distinguish normal from login specific pages in Varnish. Lucky enough TYPO3 already provides a field in the pages properties which allows this distinction. Using the Login Behaviour (pages.fe_login_mode) field, you can enable and disable the user-login for specific branches and pages*. As we want to whitelist login specific pages, our root page should have the default setting “Disable Login” – this will be inherited to all sub-pages. All the login specific pages should have the setting “Re-Enable login”.

Once this is done, we need a way to carry that out to Varnish. We improved EXT:cacheinfo for that purpose, with that it now carries a “loginAllowedInBranch” or “noLoginAllowedInBranch” value in the “X-T3CacheInfo” header. Using all that, the Varnish VCL can be extended to make use of it like this:

sub vcl_hit {
  if (obj.http.X-T3CacheInfo ~ "loginAllowedInBranch") {
    set obj.http.Cache-Control = "private";
    if (req.http.Cookie ~ "(e_typo_user|PHPSESSID|_pk_.*)") {
      # Do not cache requests which come from a logged in user
      return (pass);
    }
  }
}

This is straight forward. For every page which allows logins, we make sure that the client does not keep them in his cache. In case we’re actually on such a page and find the related login-cookies, we pass the request along to TYPO3, otherwise we deliver the public page right away from the cache. The fact that we pass the request along to TYPO3 in some cases doesn’t mean that we’ll deliver the user page, it just indicates that we’ve to rely on TYPO3 to make the right choice based on the actual login state.

Conclusion

For me the beauty here lies in the simplicity. Once you managed to wrap your head around the flow chart and once you managed to deliver appropriate meta-data to Varnish, many more complex scenarios can be resolved equally.

As most of the typo3.org stuff, this solution came from a great team. In this case Michael Stucki and Daniel Pötzinger helped to craft the final solution – thanks guys

* the naming of the field’s labels is really irritating – especially “0 – Enable login” should be “Inherit setting” as it really does not force any setting.

So that’s what you might see in your TypoScript Object Browser:

 Switching from there to the Template Analyzer:

At the bottom of the Template Analyzer, you’ll find a “Complete TS” section and a link ~which looks like normal text.

Clicking on that link will give you the entire concatenated TypoScript and here you’ll also find that the line numbers finally match to the error message.

A well hidden gem which works most likely in all TYPO3 4.x versions

DISCLAIMER: Don’t take the following too serious – when reading this post please keep in mind that the TYPO3 community itself consists of much more than just the Core team activity – many things take place outside of code repositories and can’t be measured anyhow. Every contribution is important and every single action has value.

Using Gerrit sometimes feels quite lonesome – you don’t really see who’s active and you don’t really get a feeling on how much is done in the TYPO3 Core at a certain point. To visualize on how active the contributors are and to show the most active members I applied a little scoring and summed up the results for the time since 2006.

Early 2006 impact chart with few early contributers - click the image to see the full chart

The scoring is quite easy – every author of a patch gets 10 points, testers get 3 and reviewers 1 point*. Looking at the stats it seems that even the statistics pulled from the old Subversion days seem to meet up with today’s numbers – of course we’ve to keep in mind that everything which was pulled from Subversion doesn’t really point to the author but to the actual committer (except if there was a “Thanks to XXXXX” reference in the commit).

To visualize the numbers I choose a Github like impact chart**. Each contributor has it’s own color and line in it and whenever he got active the width of the line scales up. To maintain the overview the scale of the width isn’t linear and every contributor who’s not within the “Top 20″ had to be scaled down to “1″. The line stops if the contributor didn’t get active anymore. The scores are grouped and compared by month.

Taken the last 3months - see the entire stats on the referenced page.

In addition to the chart I also created a table based overview for the “Top 20″ contributors with their score***.

Few things I got from the numbers. There have been 290 contributors already – simply amazing. Comparing the (huge) scores of the release managers with their fellow contributors shows once more that they can be really proud about the job they did or do. It’s also quite cool to see that many people keep sticking around. And finally it was quite surprising to see that our community manager Ben van’t Ende can also be found in the stats. Besides that it’s up to everyone else to find their conclusions from these numbers. I hope it’s motivating everyone to see that the community is active as always.

The script I created to generate the charts isn’t too nice at the moment, but I promise to publish it once I cleaned it up a bit. If you can’t wait to get your hands on it, feel free to send me a mail or tweet. – but I published it anyways: github.com/tolleiv/Repo-Activity-Monitor

Links in short: Impact chart, Monthly “Top 20″ tables.

* The scoring includes the commits to the Core master and all related submodule commits. Unfortunately the submodule commits don’t hold the reviewer and tester information. Also some of the latest changes made in the submodules may not show up at the moment because the submodule pointers haven’t been updated yet.
** Inspired by the incredible RaphaëlJS vector graphics library which is distributed with an MIT license and can be found on raphaeljs.com
*** Again – don’t take it too serious – the scoring doesn’t take into account that some patches can be written in 15 minutes where others take days. It also only uses the final testers and reviewers mentioned in the commit message, other contributions are not counted atm

First of all TemplaVoila 1.6 was released parallel to TYPO3 4.6. It shipped with 22 bug and compatibility fixes. In general the 1.6.x branch is supposed to be compatible with TYPO3 4.4+ which also made it possible to clean up the extension quite a bit. In addition to that TemplaVoila 1.6.1 will show up in the TER in the next few hours. It fixes 10 additional issues.

Besides that, the main repository for TemplaVoila was moved to git.typo3.org and the old Subversion repository and my Github repository have been removed. The new repository location also enables and enforces a new way to contribute code changes for the project. Comparable to the TYPO3v4 Core every change request can be sent to Gerrit, where I can review the changes before they get merged into the repository. That workflow turns out to be very efficient for me. A detailed summary on how to contribute code to any repository hosted on git.typo3.org can be found on wiki.typo3.org. To sum up some of the steps – here’s how you submit a patch*:

git clone git://git.typo3.org/TYPO3v4/Extensions/templavoila.git
cd templavoila
scp -p -P 29418 <username>@review.typo3.org:hooks/commit-msg .git/hooks/
git checkout -b workingBranch
# ... work on the files ...
git add <changedFile>
git commit
git push origin HEAD:refs/for/master/<topic>

A further general change to the team happend more or less silently. During 2011 nobody from the old team or any new developer showed interest in the project and only few contributors helped with bugfixes or reviews. Due to that I also changed my attitude regarding future releases.
I’m currently planning to release one version parallel with every main TYPO3 4.x release, to make sure that TemplaVoila works in new versions and to make sure people can enjoy TYPO3 with TemplaVoila in the future as well. I’ll also try to keep it compatible with all “stable” releases and aim to keep the issue count within the bugtracker as low as possible.
But I’m not planning to integrate new major features such as the field content sliding (use EXT:kb_tv_cont_slide please) or a context sensitive content wizard (a.k.a “content firewall”). Also major refactorings won’t happen because they’d consume far too much of my time – therefore also the desperately needed update of the mapping module or a new new page module will not be implemented in the near future.

Nevertheless I’m open for any type of contribution and I’d be happy to review and test any patch showing up in Mantis or Gerrit – I’m just not able to spent a major part of my freetime for it.

• The Pragmatic Programmer: From Journeyman to Master – a must read for every programmer and once in a while worth to take a look at. It also opens up tons of other topics which you should be pragmatic at.
• Why Functional Programming Matters – introducing the concepts of lazy evaluation and high-order functions for a couple of mathematical examples. With Steam.js as implementation of lazy evaluation in JavaScript.
• MapReduce: Simplified Data Processing on Large Clusters – every now and then I’m trying to get my hands dirty with CouchDB and re-reading this paper and the related works is always a good start.
• Some texts worth looking at when it comes to user stories Imperative vs Declarative Scenarios in User Stories, The Easy Way to Writing Good User Stories, A User Story Primer, How do you estimate the Business Value of User Stories? You don’t., Story Smells: The Valueless Story
• The training wheels came off – shows how valuable BDD scenarios should look like and why the cucumber-rails (ruby gem) stopped shipping predefined step definitions to enforce better scenarios among newcomers.
• What Your Dog Can Teach You About Building Teams - a blog post which has a good point, it misses some background information though. In depth insights might come from Management 3.0: Leading Agile Developers, Developing Agile Leaders and Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers which I won’t have time to look into
• Coding Conway’s Game of Life in Ruby the TDD Way with RSpec #1 – a great screencast showing TDD in action with Ruby and RSpec. This 50min screencast shows how things evolve during TDD coding sessions and how models can grow within test suits.

What’s on your list?

TCEMAIN.clearCacheCmd=all|<pid>

But of course it isn’t very smart to clear all caches once you have filled them up for hundred or even thousand records. To work around these limits it’s quite handy to use an API which lives in TYPO3 since 2008. It allows to add tags to the page cache and removes the caches by tag.

Adding tags to the current page can be done with this block:

$GLOBALS['TSFE']->addCacheTags(array('tx_example_model:' . $id));

Removing the caches for that specific page could look like this:

$GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('tx_example_model:' . $id);

Correctly integrating this with your extension is quite easy. To set the tags, add the first block into your controller and make sure you provide the proper uid for the rendered domain objects. To remove the caches once a record is saved in the backend just register a t3lib_tcemain hook and flush the tagged caches.

Credits: Fabrizio Branca @fbrnc – thx for pointing me to it

class Tx_Example_Hooks_Tcemain {
	public function processDatamap_afterDatabaseOperations($status, $table, $id, $fieldArray, &$reference) {
		if ($table == 'tx_example_model') {
			$GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('tx_example_model:' . $id);
		}
	}
}

EXT:example/ext_localconf.php

$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['processDatamapClass']['example'] = 'EXT:example/Classes/Hooks/Tcemain.php:Tx_Example_Hooks_Tcemain';

First of all you should understand the concept. This nice little “definition” (from flow3.typo3.org) sums it up pretty well:

A signal, which contains event information as it makes sense in the case at hand, can be emitted (sent) by any part of the code and is received by one or more slots, which can be any function in FLOW3 in extbase.

To get this running in extbase, you’ve to get hold of the Tx_Extbase_SignalSlot_Dispatcher, which is the central instance to manage all of it. Within Extbase that’s done easily with this snippet within your classes:

   ...
	/**
	 * @var Tx_Extbase_SignalSlot_Dispatcher
	 */
	protected $signalSlotDispatcher;
   
	/**
	 * @param Tx_Extbase_SignalSlot_Dispatcher $signalSlotDispatcher
	 */
	public function injectSignalSlotDispatcher(Tx_Extbase_SignalSlot_Dispatcher $signalSlotDispatcher) {
		$this->signalSlotDispatcher = $signalSlotDispatcher;
	}
   ...

Next thing is to make use of it. The Slot (listener) part could look like one of following blocks. In all cases you define the Signal by it’s class (not necessarily a PHP Class) and it’s name. Next to that the Slot can either be defined by a Closure, an object with a method name or a PHP-Class and a method name.

   ...
// Using a closure
$this->signalSlotDispatcher->connect(
      'Crunching', 'emitDataReady', function($data) { crunch($data) }, NULL, FALSE
);
   ...
// Using a method of the current object
$this->signalSlotDispatcher->connect(
     'Crunching', 'emitDataReady', $this, 'crunch', FALSE
);
   ...
// Using a method of the specified class
$this->signalSlotDispatcher->connect(
     'Crunching', 'emitDataReady', 'Cruncher', 'crunch', FALSE
);
   ...

To trigger the Signal which invokes the Slots registered above, you’ve to run the following code.

$this->signalSlotDispatcher->dispatch('Crunching', 'emitDataReady', array($data));

One thing I found was that by default the Tx_Extbase_SignalSlot_Dispatcher it not a Singleton in older extbase versions. Bastian fixed that already in the master and 1-4 branches and lucky enough this change was part within the TYPO3 4.6.1 release. But I think it’s still important to mention that this wasn’t the default from the beginning on.

Even if AOP is a nicer way to implement this feature, the extbase backport still works pretty straigh forward.

Edit: One thing I’ve to add – Felix is not “just” talking about Signal/Slots – he’s the one to thank for the backport. And now that his blog is running again – this post seems like a summary

TemplaVoila 1.5.4
The current release focussed on 4.5 compatibility. It uses the new sys_language flag “format” to support sprites, it hooks into the new backend-form (TCA) “layout” and add’s it’s fields to the right tabs within the backend forms and adjusts everything to work fine with the new CSRF mechanism.

Besides that bug fixes for the section index, performance improvements and a couple more are included.

One thing in conjunction with 4.5 you should be aware of is that copied elements are hidden by default. In older versions hidden elements won’t show up in the page module by default and therefore it might seem that nothing was copied, but that’s not right. With 1.5.4 the default setting was changed so hidden elements will show up in the page module. Unfortunately the old setting (to skip hidden elements) might still be present in your session settings – so please either clear your session settings or use the “Advanced function” tab in the page module to change to setting and avoid confusion.

Imagemap_wizard 0.6.0
The last versions proved to be very stable and with some additional sponsoring I was able to improve the DAM and TYPO3 workspaces support. Besides that a couple of issues which showed up in 4.4 and 4.5 were fixed. One of the next features will hopefully be a useful point of interest implementation – keep your fingers crossed that someone’s clients want’s to sponsor some time for that

Workspaces 4.5.1 / 4.5.2
Even if it’s shipped with the Core and included in the official release notes, here’s my summary of the improvements. The workspace module itself brought us lot’s of good feedback and also the new workspace preview raised some attention. Even though 4.5.0 was quite stable we weren’t able to get it working perfectly. The fixes made for 4.5.1 made sure that especially the preview window works much more stable, it introduced state persistence (so switching preview modes or module settings are memorized properly) and it brought some performance improvements.
Btw. if you didn’t check out the new workspaces features and improvements the new workspaces documentation from Susanne Moog is a good point to start.

As member of the TYPO3 workspace team I’d like to encourage everyone to send as much feedback as possible for all the workspace related changes and since maintaining TemplaVoila is also still on my list, it would be great if you let me know if you find any TemplaVoila related issues with the new TYPO3 version.

If you’d like to stay up-to-date with the latest changes in the TYPO3 Core, without reading all the newsgroup messages, you might be interested to use the TYPO3 trunk Changelog RSS feed. Besides the feed there’s a great service from the Core team, they started to publish the protocol of their weekly meeting as “Minutes from the meeting of the release 4.5 team” to the typo3.project.v4 newsgroup. I hope this will be kept up in the future.