Symphony.

There's an idea in my head I've been walking around with for quite some time now, and I just want to share this with you:

Multilanguage

As most of you know, creating multilingual sites with Symphony is currently still a but of a pain in the bottoks. The only way to currently have a multilingual site in Symphony is by relying on a bunch of extensions which do the magic for you, and some workarounds here and there to get everything working.

But most important of all: There is more than one way to achieve a multilingual site, and it's still not exactly clear which way is the right one.

Since multilinguality (I sure hope that that's a word) comes ever more important as soon as Symphony is getting more and more used for the sites of bigger coorperations, I think there should be an out-of-the-box solution for this.

In the core

The way Symphony stores data is quite straight-forward: each field determines it's own structure for it's entries_data-table, and stores it's value there. This however is also the biggest obstruction for a multilingual site, since each entry can only have one value for it's field:

Entry
  |- Field
      |- Value
  |- Field
      |- Value

please note: 'Value' in this example is abstract could be anything: a value and it's handle for a text-input, the geo-location for a google maps-input, a relation-id for subsection manager, etc...

Since this is a fundamental way how Symphony works, any multilingual approach on this would mean that they would have to 'hack' around this way of working. For example: the well-used multilangual field-extension creates it's entries_data-table according to which languages are determined by the language redirect-extension.

Adding a language identifier when storing your values

In my opinion languages should be managed by Symphony itself. They get stored in the database, and each entries_data-table should have an extra option lang_id. This lang_id stores the ID of the language of the current value.

This way, each field that Symphony has, or is supplied by an extension could potentially be multilingual. A simple checkbox called 'multilanguage' in the section editor would be enough to make it multilingual.

On the publish page you could have a couple of tabs on top of your page, one for each language, switching between the different languages setup by Symphony.

What does this implement?

As you can imagine, this implements a lot on the current state of Symphony. Some things should be taken care of, like:

• All existing fields (also those provided by extensions), should be changed to support the lang_id-attribute.
• The publish page should be changed to support multilanguage.
• The sorting and filtering of datasource should be well-taken care of, because that's also one little puppy that might be barking at the wrong tree.
• The .htaccess (or index.php) should be adjusted to support a language-code as first URL-parameter, or revert to the default language when no URL-parameter is set (and the first parameter is just the actual page).
• ...and there are probably a lot more things I am overseeing here.

Where from here?

Offcourse, this is just an idea I've got, but I'm thinking lately about forking Symphony and putting this into a proof-of-concept. Unfortunately, I don't have that much time at the moment, but I thought to share this idea with you guys, perhaps share some thoughts... So tell me: what do you think of this approach?

Thumbs up from me! Multilingual is a daily part of what we do in the (UK - Wales) as it forms part of a Government initiaitve to provide key public information websites in Welsh and English. Having toyed with Symphony for multilingal sites for a while, I'm reluctant to delve in due to the cumbersome (yet perfectly adequate) approach to implementing Multilingual sites with it.

If this idea was a reality, I'm sure a lot of other adopters would take note and want to develop with Symphony. Symphony is in a unique position to be anything the developer wants it to be.. but having a leg up in terms of thinking about multilingual would propel Symphony to the top of my list if not many others lists of tools to use for modern website development for a Global marketplace.

How about leaving the tables as they are and modify the columns? value_en, value_fr etc, like multilingual field proposed.

Is there any difference in performance when searching 1 column in n-tables versus n-columns in 1 table?

Furthermore ... are the options @kanduvisla proposed viable for Symphony 2? They seem pretty radical and reflect in many parts of the code. Wouldn't it be wiser to insert them in Symphony 3?

Entry based multilingual (which you described) is the approach that has the most favour in the team as well. One of the reason's we are holding off at the moment is because similar functionality can/might be achieved with the Subsection Tabs extension as it adds all those features you have mentioned :) We're in a similar predicament to yourself, lots of ideas, so little time! At the moment it is not being considered for 2.3 purely because our todo is fairly hefty at the moment and there are some pressing things that need to be resolved first so this can follow elegantly. It is definitely on our radar though. We're in a similar predicament to yourself, lots of ideas, so little time! At the moment it is not being considered for 2.3 purely because our todo is fairly hefty at the moment and there are some pressing things that need to be resolved first so this can follow elegantly. It is definitely on our radar though.

I don't support this approach.

• It breaks with the Symphony philosophy which has been the cornerstone since its inception.

If an extension needs tweaks/hooks in the core (i.e. the Members extension required a few changes) to make something impossible, possible, that's one thing. But to add it as a feature to the core would be a mistake because it would only be beneficial to those that make multilingual sites, bloat to the rest of us.

Why can't a better extension be made?

The solution will be added as an extension for that very reason, its the 'impossible to possible' that we're hoping to correct :) Its hoped that the language solution might be able to be leveraged for versioning extensions as well! I suppose its a bigger question though, why can't this be done right now with an extension? How does Symphony make this impossible?

@vladG: I thought of this as well, but the fact is that since fields determine what fields they need in the database. Therefore not all extensions have fields like value or handle. Some examples: file uploads, select box links, subsection manager, and a lot more. The aim here is to make a uniform solution for multilanguage support.

@brendo: It's good to hear this is on the radar. Just want to share my thoughts on this with you. Although I'm not sure if my described approach is Entry based multilingual support, but rather field-based multilingual support. I'm not quite sure about SubSection Manager Tabs, I haven't used/tested that extension yet...

@Lewis: Although I understand your concerns and agree that nothing should break with the Symphony philosophy, I do think that something as fundamental as multilanguage support should be embedded into the core, because there are some fundamental hooks and issues which extensions just can't resolve. These go very deep in the fundament on how Symphony handles it's data, sorting, filtering, etc... And you shouldn't see it as an additional feature, but more than another way to handle information. It would be best if one extension could do all of this all by itself, but the solutions that have come up so far still don't feel like a rock-solid solution to the problem.

@brendo: That's an interesting question. I think part of the problem is (as I described above) true multilanguage support affects multiple aspects:

• It should be able to make each field multilingual (for example: checkboxes, images/uploads, relationships with selectbox link or SMM).
• It should give a solution in the publish page to switch between languages.
• It should cover datasource generation in the correct language.
• It should cover filtering datasources.

I tried to cover most of these aspects in a multilanguage extension I made a year ago or something, but I did most of it with JavaScript. Perhaps there are currently more hooks in the system to do this in a better way.

But I do think the bigger problem is that bacause of it's openess, Symphony allows each field to determine how it wants to store it's value in the database. I believe there should be a hook on this level to influence this process. Perhaps even the whole entries_data-table-generation of each field should be abstracted into a Symphony function. It is already a given that such table should always have an entry_id-field. Yet, each field-type writes it's own SQL-query to create the table (in the public function createTable()). Perhaps abstracting this into a Symphony-function allows other extension developers to add additional fields to these entries_data-tables.

Although I understand your concerns and agree that nothing should break with the Symphony philosophy, I do think that something as fundamental as multilanguage support should be embedded into the core

Well, I don't think you do. Your argument is based on an assumption that multilanguage support is essential. That is your opinion.

Why is it not possible with the current core and how could modifications that support multilanguage be incorporated that are generic enough to provide solutions for other barriers and future ones that we don't know about yet.

And don't get me wrong, just because I don't build multilingual sites doesn't mean I don't support the idea, I do support the idea if that's how you and others are using Symphony. I simply don't agree that it should be part of the core.

@Lewis: I think brendo just hit it spot-on: the question isn't how to implement multilanguage support to the core but:

Why can't this be done right now with an extension? How does Symphony make this impossible?

As I mentioned in my previous comment, I think the whole idea of fields creating their own tables directly with SQL-queries should be abstracted. This could give extension developers more hooks to their extensions. This also alligns with your response to

provide solutions for other barriers and future ones that we don't know about yet.

I could imagine a whole range of extensions who could benefit from something like this. Kinda think of it, other field-driven SQL-queries should also be abstracted, like the ones in the install() and uninstall()-functions.

As I mentioned in my previous comment, I think the whole idea of fields creating their own tables directly with SQL-queries should be abstracted.

I see it now (I was too busy thinking and writing my last post, hate when that happens!). I'm not sure of the implications, but that's the kind of thinking that I believe is true to Symphony.

Why can't this be done right now with an extension? How does Symphony make this impossible?

That's what I meant in my original post, but brendo definitely summed it up more clearly :)

Regarding the question core vs extension I completely agree with Lewis — although I am German and might have to build a multilingual website some day.

The feature is definitely not essential to most users. The majority of registered members on this website come from English-speaking countries. In these countries being "multilingual" is no issue at all. (Please forgive me if Wales is a bit different.)

As Lewis said, this does not mean that the core should not be made as flexible as possible to make such an extension work as smooth as possible.

@michael-e we're Welsh first then English here.. but I use both on a daily basis. Flexibility in the core to allow an extension to provide multilingual fields as and when would be the ideal solution for my workflow as it's sometimes just a content filed.. other times.. the whole shabang is needed to be translated on a 1 to 1 and 1 to many approach.

Although it's not essential to most users, it's the one thing which is a showstopper at the moment for Symphony. A decent CMS should at least have a decent multilanguage support. At the moment (especially for newcomers) this already requires some Symphony-experience to set this up.

I am from The Netherlands, but a lot of companies where we work for are operating in different countries, so we have a fairly amount of multilingual sites we should consider.

I don't think the core should be as flexible as possible, but it should provide with some hooks to allow extensions to do some magic. I think you can also differ extensions then into 2 categories:

1. Extensions
2. Core Extensions

@brendo: What do you think of the suggestion to abstract the database-table-generating functions for field-extensions?

@kanduvisla: Please don't only look at your country. I know that TV/websites/whatever in the Netherlands is multilingual — but in this respect your country is definitely special! So if you say "showstopper", you should better say "showstopper in the Netherlands (and Wales)".

Please think about the fact that more than half of the registered members here are natively speaking English (most of them never learning or using any other language). There are many more countries where multi-language websites are much more uncommon than in your country. I am not speaking against cool features, but I simply think that your point of view is a very subjective one. Try and be more objective! Symphony is a world-wide project!

Such a feature should never be a part of the Symphony core, simply because most developers and users will never need it.

That said, I'd love to see a community effort to create a mighty multi-language extension. In the past I have been reading so many discussions about this topic, and still — if I had to build a multi-lingual site tomorrow, I would have no idea which extensions to choose, how to make it right. So I do hope that your impetus will help to solve this issue now and forever!

@michael-e: Although I disagree with you on that our country is special, I am trying to be as objective as possible on this. Perhaps 'Multilanguage from the core' isn't that good of an idea, but as @brendo mentioned later on, the question should be 'Why does Symphony make it so hard to create a multilanguage extension?'. When we can find a solution for this, we probably pave the road for an all-in-one solution for multilanguage, but also (like @Lewis said) provide solutions for other barriers and future ones that we don't know about yet.

What we should do is inventise the barriers there are now for multilanguage support for example, and think of modular solutions to 'make it all work'. Just like there were some slight modifications to get the Members extension working we should also look at what need to be done for this matter.

Abstraction of the generation of tables for fields are one thing that I think needs to be done. Allthough it would require a lot of extensions to alter their code (however, I wouldn't expect that this is quite easy), it opens new windows for more flexibility for extensions.

For example, the following code which is used now thourougly by a lot of fields (to create a field-table in the database):

Symphony::Database()->query('
    CREATE TABLE IF NOT EXISTS `sym_fields_private_upload` (
      `id` int(11) unsigned NOT NULL auto_increment,
      `field_id` int(11) unsigned NOT NULL,
      `destination` varchar(255) collate utf8_unicode_ci default NULL,
      `validator` varchar(255) collate utf8_unicode_ci default NULL,
      PRIMARY KEY  (`id`),
      KEY `field_id` (`field_id`)
    );
');

Could be abstracted to something like:

Symphony::createFieldTable('private_upload', array(
    'destination' => array('type'=>'varchar', 'default'=>'null'),
    'validator' => array('type'=>'varchar', 'default'=>'null')
    )
);

Symphony could create the id and field_id-columns (since these are required anyway), but other extenions could hook in in such a function by adding extra columns by their own.

The same could be done for when a field creates a entries_data-table, or on uninstallation-scripts and such.

This could even be backward-compatible with older extensions, since they would just execute the SQL-query instead of using the abstract-function, but it would be up to the new extension to detect this.

What are other people thoughts on this? And then I don't mean just my example, but also what I said in the first two paragraphs of this post?

'Why makes Symphony it so hard to create a multilanguage extension?'

OK, we are on the same track. :-)

What we should do is inventise the barriers there are now for multilanguage support for example, and think of modular solutions to 'make it all work'.

Agreed. Any additional flexibility added to the core is definitely good.

Symphony could create the id and field_id-columns (since these are required anyway), but other extenions could hook in in such a function by adding extra columns by their own.

Is this a good idea? Why can't the additional columns be a separate table that has a relation to the field's table?

Giel, to be honest I don't think the database table is the problem here.

As I am typing this on my phone I can't go into details on why I think it's not a solution, but maybe we can talk about this on the phone? To get my number, please send an email to huib at my user name dot com.

Once we discussed this we can put a summary on the forum to keep the rest on par.

I'd like to quickly chime in and +1 this idea although I cannot comment much re: the specific technical implementation.

I've personally experienced this obstacle to the adoption of Symphony adoption a couple of times. People chose Typo3 (in this case) over Symphony mainly because of the fact that multi-language was a core feature of Typo3.

A Typo3 developer actually showed it to me: "Add a few languages… Click a few options… Set!" While I personally strongly favor Symphony over all CMS'es I know today, I can see the appeal of this core feature.

I am still hesitant re: 'selling' Symphony when multi-language is a requirement ("Well, it can be done a couple of ways. Let me think… We could use this-and-this extension and maybe work around this by creating this custom DS and…")

Disclaimer: I am 'special' (Dutch :) ) and I actually encounter multi-language quite often. Disclaimer 2: I am not saying it should be a core feature. I am just +1-ing the fact that lack of a solid Symphony multi-language solution is holding Symphony adoption back in my experience and that it would be a huge plus if it were less of a hassle.