One of spring days 2012 brought us a new project. One of our regular customers recommended our company to a very ambitious and engaged Moscow businessman. After reading the specifications we were at a loss… after 5 years of active web-development! HiConversion project demanded the following:
- advertisement posting and administration in VK social network;
- statistics gathering from all the ads published on outer resources: VK, Google Analytics, Yandex.Metrica;
- ability to monitor statisitcs on all the published ads for any time period;
- depending on results of the above-mentioned statistics - automatic or manual ads administration;
- access to all targeting settings (both one-by-one or bulks);
- interfaces for advertising agencies, hierarchical roles and permissions system;
- control over user account: ability to replenish account, automatic publishing blocking if there is budget is not full.
And that is just tip of the iceberg. But let’s now concentrate at the architectural approach to implementation of such a complicated project.
As you see all the above-mentioned demands are far from trivial web-application, not to mention Drupal default functionality. Necessity to update dozens of thousands queries per minute with outer resources information forced us to look for roundabouts to overcome both Drupal core and other services limitations.
To get the most of these limitations we had to re-think queries saving and processing formats in Drupal. Various data formats, principles of their communication and authentication needed to be taken into consideration, of course. As a final result - our queries had been processed not by cron scheduled tasks but by php-demon. It had been monitoring elements presenсe in queue and implemented tasks at the very maximum speed which could have been achieved. A complicated ‘semaphore’ system (based on Drupal Lock API) was in charge of monitoring resource limits. Also our interface allowed to easily add the new types of queries’ elements.
We also were forced to modify the very principle of fields processing in Drupal due to enormous statistical data storages (~1 million queries). At the same time we wanted to stick to Drupal philosophy and to have the possibility to use standard implementations. That is why we created new types of fields which were more adjusted to our needs and used 100% of Drupal Field Storage API solutions. In our case the majority of operations demanded a vast amount of fields to be updated with one-type kind of information in one go. This condition actually helped us to optimise the system - we added the functionality of fields mass updating and the posterior calling of the needed hooks. Advantages of such approach showed up very soon - we were able to use modules to build views with very complicated and interdependent filters and save them.
Regardless of system complexity we have left enough space for further functionality enlargement, e.g. queue event handler with flexible setting was added easily. On user-interface it looked like addition of Field Collection element with set of field settings. From the code perspective we only needed to add these settings and reaction rules handler expanding its basic behavior class.