Models data union

So I have only 1 model and 10 tables for example, I want to bring all the data from the tables into my 1 model, so I can use the Grid/Crud search/paginator etc. If I use setSource on the grid i lose all that.
Note: Tables have no relations between them (they just have the same columns and diff data) and we are talking about milions of records.
Tried custom unions with dsql but it’s really slow and i have no paginator in the grid.
Any ideas ?

Why wouldn’t you like to create separate models for each table if they have no relation?

I don’t think that would help me very much … I would still have to use setSource on the grid and not setModel, after I get the data from all models. And setSource offers virtually no functionality for the grid. We’re talking about 50-100 mil records and I really need search and paginator for the grid/crud.
For example a big table is broken into 10 tables, with no relations between them, but when presenting the data to the user I still have to union all the data from the 10 tables (there’s a filter too so he can pick from and to dates for the results) and I would still have to create models every year of each table …

Hi @pure_snow99.

I’m still working on making this into a proper add-on, but I wanted to share this with you:

Once I’m done, I will announce it on my blog (http://nearly.guru/blog/).

Thanks, Romans.
I’m asumming this works only if you have models for all the tables … How about if you have just 1 model but you want to bring all the data from 10 tables into 1 model, so you don’t have to make 10 models to achieve this ?
There can be a case where you want to make changes just in the db (like breaking a big table into tables / holding data for each year), and you dont want those changes to reflect into the code (like creating models for each table).
Would that be fast enough if we talk millions of records ?
On how many records have you tested Model Union ?

Didn’t tested but can you do this

class Model_Alltables extends romaninsh\ModelUnion\ModelUnion {
    function init(){
        parent::init();
         
       $this->addNestesModel($this->add('SQL_Model',['table'=>'table_1']),'alias1');
       $this->addNestesModel($this->add('SQL_Model',['table'=>'table_2']),'alias2');
       $this->addField('alias1_field1','actual_field1');
      $this->addField('alias1_field2','actual_field2');

@gowravvishwakarma , this wont work because addField assumes the field is present for all nested models.

@pure_snow99, define intermediate models, it shouldn’t be difficult. The union model produces a single query (you can test it against debug() ). I’ll send you invitation to our slack chat so we can discuss this further.

I’d like to use your use-case as a test for my model union before publishing it.

Sure , do we have something close to an addon ?

Yeah, getting there. Implementing aggregation interface, which, actually, could also be ported to regular models also.

Waiting anxiously :slight_smile: have you tested it on millions of records?
Please post git link when it’s ready.
Thank you.

We are actually testing it on 2M + record database :smiley:

Any updates ? : ) How did the test go ?